Spring Boot and Filebeat locally without registration and sms

In this tutorial, we will look at connecting and configuring a logging system in a Spring Boot project and sending logs to ELK using Filebeat . This guide is intended for entry-level developers.

Logging and why is it needed

When I first started working as a programmer, one of my senior colleagues liked to repeat: "If you have no logs, then you have nothing . " Indeed, faced with the first bug on test benches or worse in an industrial environment, the first thing we need is application logs and easy access to them. The developers of the application are responsible for the logs themselves, who must ensure that the behavior of the system is logged in such a way that at any time it is possible to understand what is happening with the system and, most importantly, what is wrong with it.

The next question is the convenience of access to logs. Usually, during local testing, we see the log in the application console, and on the test bench - in special log files on the server. Is it convenient and safe to connect to the stand every time, search for the required directory and read log files from there? Practice shows that no, and this is the second problem that a number of products are designed to solve that provide convenient access to logs and search for important information in them. Today we will talk very briefly about one of the groups of such products, the so-called ELK stack (Elasticsearch - Logstash - Kibana ) and in more detail about Filebeat - an Open source product that provides a convenient mechanism for delivering logs to ELK .

Three lines about ELK

• Logstash — ,
• Elasticsearch —
• Kibana —

Filebeat?

Filebeat ELK , Logstash .

ELK , Filebeat ( ), ELK.

.

Java 8

ApacheMaven3.6

Spring Boot 2.3.4.RELEASE

Docker

Spring Boot App

Spring Boot , . /

Spring Boot Spring Initalizr

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>6.4</version>
</dependency>

<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
</dependency>

• spring-boot-starter-web — ..
• logstash-logback-encoder —
• lombok — ,

pom.xml

Spring Boot :

@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}

:

@Slf4j
@Service
public class LogGenerator {

    public void generate(int count) {
        log.info("Start generating logs");
        LongStream.range(0, count)
                .forEach(i -> log.info("Log {}", i));
    }
}

0 count

, :

@Slf4j
@RestController
@RequiredArgsConstructor
public class LogController {
    private final LogGenerator generator;

    @GetMapping("/generate")
    public ResponseEntity test(@RequestParam(name = "count", defaultValue = "0") Integer count) {
        log.info("Test request received with count: {}", count);
        generator.generate(count);
        return ResponseEntity.ok("Success!");
    }
}

GET :

http://localhost:8080/generate?count=10

. resources logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

    <appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d [%thread] %-5level  %logger{35} - [%mdc] - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="filebeatAppender" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>./log/application.log</file>
        <append>true</append>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>./log/application.%d.%i.log.gz</fileNamePattern>
            <maxFileSize>10MB</maxFileSize>
        </rollingPolicy>
    </appender>

    <root level="INFO">
        <appender-ref ref="consoleAppender" />
        <appender-ref ref="filebeatAppender" />
    </root>
</configuration>

:

• consoleAppender —
• filebeatAppender — , LogstashEncoder logstash-logback-encoder
  
  — JSON , Logstash. Logstash .

, ./log/application.log log . . maxFileSize

Filebeat .

:

@Slf4j
@Component
public class LogFilter extends OncePerRequestFilter {

    private static final String REQUEST_ID = "requestId";

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                    HttpServletResponse response,
                                    FilterChain filterChain) throws ServletException, IOException {
        String requestId = request.getHeader(REQUEST_ID);
        if (requestId == null) {
            requestId = UUID.randomUUID().toString();
        }

        MDC.put(REQUEST_ID, requestId);

        try {
            log.info("Started process request with {} : {}", REQUEST_ID, requestId);
            filterChain.doFilter(request, response);
        } finally {
            MDC.clear();
        }
    }
}

, ( requestId), MDC (Mapped Diagnostic Context)

MDC.put(REQUEST_ID, requestId);

finally MDC

MDC.clear();

, , . Kibana .

, , :

mvn spring-boot:run

, application.log

curl "localhost:8080/generate?count=10"

Success!, application.log :

{
   "@timestamp":"2020-10-17T22:39:45.595+03:00",
   "@version":"1",
   "message":"Writing [\"Success!\"]",
   "logger_name":"org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor",
   "thread_name":"http-nio-8080-exec-3",
   "level":"INFO",
   "level_value":10000,
   "requestId":"77abe5ac-4458-4dc3-9f4e-a7320979e3ae"
}

Filebeat

Filebeat

7.9.2 macOS

.

Filebeat filebeat.xml

, inputs output:

  inputs:
  - enabled: true
    encoding: utf-8
    exclude_files: ['\.gz$']
    json:
      add_error_key: true
      keys_under_root: true
      overwrite_keys: true
    paths:
    - {     }/*.log
    scan_frequency: 10s
    type: log

, Filebeat . :

• keys_under_root — json json, Filebeat Logstash
• overwrite_keys —
• add_error_key — Filebeat error.message error.type: json json .

output:
  logstash:
    hosts:
    - localhost:5044
    ssl:
      certificate_authorities:
      - {    }/logstash-beats.crt

, Filebeat . Logstash ( )

ssl.certificate_authorities Logstash ( ), .

Filebeat, , .. ELK .

ELK . , docker ELK sebp/elk logstash-beats.crt. certificate_authorities filebeat.xml

docker-compose :

version: '3.7'
services:
  elk:
    image: sebp/elk
    ports:
      - "5601:5601" #kibana
      - "9200:9200" #elastic
      - "5044:5044" #logstash

ELK Filebeat , macOS :

./filebeat -e run

? , LogstashEncoder JSON application.log, Filebeat , Logstash. Kibana.

Kibana :

http://localhost:5601/

Discover:

:

Kibana index ELK . ! Filebeat , . :

curl "localhost:8080/generate?count=100"

:

:

. requestId MDC :

Now, in the Discover tab for our index, you can configure the display of fields and see that all logs within one request are combined by the same requestId . You can expand the JSON field and see the full text of the message received from Filebeat :