ELK搭建指南

很久没有发文了，今天水一篇操作文章分享一下经验吧，后续陆续推出深度好文

springboot+logStash+elasticsearch+kibana

版本

• elasticsearch 7.4.2
• logStash 7.4.2
• springboot 2.1.10

下载地址

选择下载的产品和版本，进行下载即可

https://www.elastic.co/cn/downloads/past-releases 

部署

启动Elasticsearch

• 设置配置文件elasticsearch

  cluster.name: my-application node.name: node-1 path.data: /cxt/software/maces/7.4.2/elasticsearch-7.4.2/data path.logs: /cxt/software/maces/7.4.2/elasticsearch-7.4.2/logs network.host: 0.0.0.0 http.port: 9200 discovery.seed_hosts: ["127.0.0.1"] cluster.initial_master_nodes: ["node-1"] 

• 启动

  bin/elasticsearch 

启动Kibana

• 直接去bin目录下启动即可， 都是本机启动无需修改配置

  bin/kibana 

启动LogStash

• config文件夹下创建springboot-log.conf,该配置文件的作用是启动在本机9600端口，后面springboot应用可以直接往9600发送日志。input为日志输入，output为日志输出到elasticsearch

  input{ 
        # 启动在9600端口，输出在控制台 tcp { 
        mode => "server" host => "0.0.0.0" port => 9600 codec => json_lines } } output{ 
        elasticsearch{ 
        hosts=>["192.168.123.166:9200"] index => "springboot-logstash-%{ 
       +YYYY.MM.dd}" } # stdout{ 
        # codec => rubydebug # } } 

• 启动

  bin/logstash -f config/springboot-log.conf 

启动SpringBoot应用

• pom

   
       
       
         net.logstash.logback 
        
       
         logstash-logback-encoder 
        
       
         7.0 
        
       

• testController 方法

  @RestController public class TestController { public static final Logger log = LoggerFactory.getLogger(TestController.class); @RequestMapping("/test") public String test(){ log.info("this is a log from springboot"); log.trace("this is a trace log "); return "success"; } } 

• 启动类main方法加入自动生成日志代码

   @SpringBootApplication public class ElkApplication { public static final Logger log = LoggerFactory.getLogger(ElkApplication.class); Random random = new Random(10000); public static void main(String[] args) { SpringApplication.run(ElkApplication.class, args); new ElkApplication().initTask(); } private void initTask() { Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(new Runnable() { @Override public void run() { log.info("seed info msg :" + random.nextInt()); } }, 100, 100, TimeUnit.MILLISECONDS); } } 

• resource新建logback-spring.xml

   
        <configuration> <include resource="org/springframework/boot/logging/logback/base.xml" /> <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">  
        <destination>192.168.123.166:9600 
         destination>  
         <encoder charset="UTF-8" class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"> <providers> <timestamp> <timeZone>UTC 
          timeZone>  
           timestamp> <pattern> <pattern> { "logLevel": "%level", "serviceName": "${springAppName:-}", "pid": "${PID:-}", "thread": "%thread", "class": "%logger{40}", "detail": "%message" }  
            pattern>  
             pattern>  
              providers>  
               encoder>  
                appender> <root level="INFO"> <appender-ref ref="LOGSTASH" /> <appender-ref ref="CONSOLE" />  
                 root>  
                  configuration> 

验证

• 按顺序启动好了之后打开es-head插件即可查看索引信息，其中有一条调用接口的信息，还有应用启动的消息

  

• Kibana数据展示
  • 设置索引规则

    

    输入之后设置时间戳匹配

  • 展示数据,选择Discover

    

加入filebeat

• logstash 新建filebeat-logstash-log.conf

  input{ 
        beats { 
        host => "192.168.123.166" port => 9600 } } output{ 
        elasticsearch{ 
        hosts=>["192.168.123.166:9200"] index => "%{ 
       [@metadata][beat]}-%{ 
       [@metadata][version]}-%{ 
       +YYYY.MM.dd}" } } 

• 启动

  bin/logstash -f filebeat-logstash-log.conf 

• Filebeat 修改配置文件,找到下方修改的地方修改即可，主要监听的日志文件和输出的logstash服务器地址

  filebeat.inputs: - type: log enabled: true paths: - /cxt/codework/java/springboot-demo/logs/springboot-elk/2022-06-04/info.2022-06-04.0.log setup.kibana: Host: "192.168.123.166:5601" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["192.168.123.166:9600"] 

• springboot 应用程序配置生成日志文件的位置,resource下新建logback-spring-file.xml

   
        <configuration debug="false" scan="false">  
        <property name="log.path" value="logs/springboot-elk"/>  
        <appender name="console" class="ch.qos.logback.core.ConsoleAppender"> <encoder> <pattern>%d{MM-dd HH:mm:ss.SSS} %-5level [%logger{50}] - %msg%n  
         pattern>  
          encoder>  
           appender>  
           <appender name="fileRolling_info" class="ch.qos.logback.core.rolling.RollingFileAppender"> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>${log.path}/%d{yyyy-MM-dd}/info.%d{yyyy-MM-dd}.%i.log 
            fileNamePattern> <TimeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <maxFileSize>50MB 
             maxFileSize>  
              TimeBasedFileNamingAndTriggeringPolicy>  
               rollingPolicy> <encoder> <pattern>%date [%thread] %-5level [%logger{50}] %file:%line - %msg%n  
                pattern>  
                 encoder>  
                  
                  appender>  
                  <appender name="fileRolling_error" class="ch.qos.logback.core.rolling.RollingFileAppender"> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>${log.path}/%d{yyyy-MM-dd}/error.%d{yyyy-MM-dd}.%i.log 
                   fileNamePattern> <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <maxFileSize>50MB 
                    maxFileSize>  
                     timeBasedFileNamingAndTriggeringPolicy>  
                      rollingPolicy> <encoder> <pattern>%date [%thread] %-5level [%logger{50}] %file:%line - %msg%n  
                       pattern>  
                        encoder> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>ERROR 
                         level>  
                          filter>  
                           appender>  
                           <root level="info">  
                           <appender-ref ref="console"/>  
                            
                            
                           <appender-ref ref="fileRolling_info"/>  
                           <appender-ref ref="fileRolling_error"/>  
                            root>  
                             
                             
                             
                            <logger name="org.springframework" level="INFO"> 
                             logger> <logger name="org.mybatis" level="INFO"> 
                              logger>  
                               configuration> 

• application.yml文件中指定logback-spring-file.xml

  logging: # 默认logback-spring.xml 使用logstash传输到es； # 改为logback-spring-file.xml传输日志到归档日志文件，使用filebeat监听日志 config: classpath:logback-spring-file.xml 

• 到这流程就结束了，现在流程就是

  springboot

  logs/springboot-elk/*.log

  监听logs/springboot-elk/*.log

  filebeat 发送数据到logStash

  logStash 数据保存到ES

  ES

  Kibana 对ES中数据进行渲染展示

好了，以上就是ELK的搭建流程，顺带使用filebeat监听日志文件的也做了，大体就是这个酱紫，好久没有肝文章了，后面会陆续推出更多深度理论好文，欢迎关注微信公众号：《醉鱼JAVA》一起进步学习

github