Microservices (Part 4) – Circuit Breaker and Log Tracing

July 28, 2020
Posted by: Goutham Ramesh
Category: Application Development

Prerequisite

Friends , Let us recall our reference architecture ( Micro services generic reference architecture ) and also the Eureka Server and Demo Service and Consumer service created in (Part2)

Let us also recall our Gateway Zuul created in Part 3

Recap : Our Solution till now

The solution after Part 3 has the following:

Eureka Service ( Service Registry )
Demo Service ( REST Service Multiple instances registered with Eureka Service )
Demo Consumer Service with REST Template and Load Balancer ( REST Service + REST Template Client to call Demo Service and Ribbon Load-Balancer for Demo Service written in step 1 )
Zuul Gateway Service

Circuit Breaker

In a distributed microservices architecture multiple services can be dependent on each other. Now, what will happen if some services fail or any exception is thrown by some services? Then this exception is propagated to upstream services and finally comes to the end-user. Circuit Breaker pattern prevents failure cascading and gives a default behavior when services fail, with a fall back.

Hystrix

Hystrix is the implementation of Circuit Breaker pattern, which gives a control over latency and failure between distributed services.

The main idea is to stop cascading failures by failing fast and recover as soon as possible — Important aspects of fault-tolerant systems that self-heal.

Let us add Circuit breaker to our Demo Consumer Service

In the pom.xml file, make sure to include these dependencies:

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
<version>2.2.3.RELEASE</version>
</dependency>

in the spring boot main class add annotation @EnableCircuitBreaker like this

@SpringBootApplication
@EnableEurekaClient  
@EnableCircuitBreaker 
public class EjyleSpringRestDemoConsumerServiceApplication {
public static void main(String[] args) {
SpringApplication.run(EjyleSpringRestDemoConsumerServiceApplication.class, args);
}
}
@Configuration
class RestTemplateConfig {
// Create a bean for restTemplate to call services
@Bean
@LoadBalanced// Load balance between service instances running at different ports.
public RestTemplate restTemplate() {
    return new RestTemplate();
}
}

Spring looks for any method annotated with the @HystrixCommandannotation, and wraps that method so that Hystrix can monitor it.

Hystrix watches for failures in that method, and if failures reached a threshold (limit), Hystrix opens the circuit so that subsequent calls will automatically fail. Therefore, and while the circuit is open, Hystrix redirects calls to the fallback method.

So, In the controller class, add annotation and fallback method as shown below:

@HystrixCommand(fallbackMethod = "fallbackHome")
@RequestMapping("/")
public String home() {
logger.info("In home");
// This is useful for debugging
// When having multiple instance of gallery service running at different ports.
// We load balance among them, and display which instance received the request.
String message = restTemplate.getForObject("http://ejyle-demo-service/", String.class);
String response = "Hello from Demo Rest Consumer Service running at port: "
+ env.getProperty("local.server.port") + " Message from demo Service " + message;
return response;
}
@HystrixCommand(fallbackMethod = "fallBackGetConsumer")
@RequestMapping(value = "/getConsumer", method = RequestMethod.GET)
public ResponseEntity<String> getUser(@RequestParam String name) {
logger.info("/getuser  for id = " + name);;
ResponseEntity<String> response = null;
String message = restTemplate.getForObject("http://ejyle-demo-service/getUser?name=" + name + "", String.class);
response = new ResponseEntity<String>(message, HttpStatus.OK);
return response;
}
// a fallback method to be called if failure happened
public String fallbackHome(Throwable hystrixCommand) {
return new String("Service Down !");
}
// a fallback method to be called if failure happened
public ResponseEntity<String> fallBackGetConsumer(String name, Throwable hystrixCommand) {
ResponseEntity<String> response = null;
String message = "Servvice  ot available";
response = new ResponseEntity<String>(message , HttpStatus.OK);
return response;
}

Testing our Circuit Breaker

Run the ‘EjyleEurekaServer’ spring boot app.
Do not run EjyleSpringRestDemo, The idea is to not make this service available so the circuit trips .
Run EjyleSpringRestDemoConsumerService as spring boot app
Run EjyleZuulGateway as Spring boot app

Access url http://localhost:8762/consumer/

Now you can see that response is coming from fall back method

Logging and log Tracing

If you have, let’s say 3 services, A, B and C. We made three different requests.

One request went from A → B, another from A →B →C, and last one went from B →C.

A -> B
A -> B -> C
B -> C

As the number of microservices grow, tracing requests that propagate from one microservice to another and figure out how a requests travels through the application can be quite daunting.

So the answer is logging and Lot Tracing using Log-back and Sleuth

Sleuth makes it possible to trace the requests by adding unique ids to logs.

A trace id (1st) is used for tracking across the microservices; represents the whole journey of a request across all the microservices, while span id (2nd) is used for tracking within the individual microservice.

To use Sleuth add dependency to pom.xml … of following projects

EjyleEurekaServer
EjyleSpringRestDemo,
EjyleSpringRestDemoConsumerService
EjyleZuulGateway

<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
<version>2.2.3.RELEASE</version>
</dependency>

Now let us add logback file to EjyleSpringRestDemo in the same folder as application properties.

<?xml version=”1.0″ encoding=”UTF-8″?>

<configuration>
  <property name="HOME_LOG" value="logs/"/>
  <property name="ARCHIVE_LOG" value="logs/archive"/>
    <appender name="SAVE-TO-FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${HOME_LOG}/restapp.log</file>
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
         <Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %5p [ %X{X-B3-TraceId:-} , %X{X-B3-SpanId:-} ]  %m%n</Pattern>
    </encoder>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
        <fileNamePattern>${ARCHIVE_LOG}/restapp_%i.log</fileNamePattern>
        <minIndex>2</minIndex>
        <maxIndex>3</maxIndex>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
        <maxFileSize>5000KB</maxFileSize>
    </triggeringPolicy>
</appender>
    <root level="info">
        <appender-ref ref="SAVE-TO-FILE" />
    </root>
</configuration>

Now let us add logback file to EjyleSpringRestDemoConsumerService in the same folder as application properties.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <property name="HOME_LOG" value="logs/"/>
  <property name="ARCHIVE_LOG" value="logs/archive"/>
    <appender name="SAVE-TO-FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${HOME_LOG}/restconsumer.log</file>
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
        <Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} %5p [ %X{X-B3-TraceId:-} , %X{X-B3-SpanId:-} ]  %m%n</Pattern>
    </encoder>
    <rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
        <fileNamePattern>${ARCHIVE_LOG}/restconsumer_%i.log</fileNamePattern>
        <minIndex>2</minIndex>
        <maxIndex>3</maxIndex>
    </rollingPolicy>
    <triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
        <maxFileSize>5000KB</maxFileSize>
    </triggeringPolicy>
</appender>
    <root level="info">
        <appender-ref ref="SAVE-TO-FILE" />
    </root>
</configuration>

And In controller of EjyleSpringRestDemo service, use logger to log some info

@RequestMaping("/")
public String home() {
 logger.info("In Home");
--------
--------
}

And In controller of EjyleSpringRestDemoConsumerService service, use logger to log some info

@RequestMapping("/")
public String home() {
 logger.info("In Home");
--------
--------
}

Testing our Logging and log trace

Run the ‘EjyleEurekaServer’ spring boot app.
Run EjyleSpringRestDemo multiple times and you can see multiple Services instances registered under the same service name .
Run EjyleSpringRestDemoConsumerService as spring boot app
Run EjyleZuulGateway as Spring boot app

Now the gateway can be tested using url http://localhost:8762/consumer/

Look at log file of EjyleSpringRestDemo under /logs folder of the project , you will see the following:

2020-07-28 10:24:24.213 INFO [ 47f6af6b2d684b9d , 66736e7ae24029ad ] In Home

Look at log file of EjyleSpringRestDemoConsumerService under /logs folder of the project , you will see the following:

2020-07-28 10:24:23.783 INFO [ 47f6af6b2d684b9d , c0bf271d486965d9 ] In home

Look at the trace id it is same , in this way we can trace requests across microservices

Conclusion

Thank you for reading! If you enjoyed it and If you have any questions or comments you know where to reach me! Will author part-5 shortly .. In which I will cover auth service and add it to gateway