Skip to main content

Supporting OpenTracing jaeger in spring boot applications

This page describes code changes in a typical spring-boot based application to support OpenTracing and jaeger.

Instrumenting a tracer

If you are creating a simple spring boot application that uses spring-boot-starter-web, by default, the application does not support writing traces to jaeger. To support jaeger tracing, the first thing is to modify the build.gradle to add dependency of jaeger:

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'io.opentracing.contrib:opentracing-spring-web-starter:3.0.1'
    // support opentracing jaeger
    implementation 'io.opentracing.contrib:opentracing-spring-jaeger-starter:3.1.2'
    testImplementation('org.springframework.boot:spring-boot-starter-test') {
        exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
    }
}


Then you need to create a @Bean of type io.opentracing.Tracer. https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md has examples of creating the tracer. The following example should be enough in most applications:

package whatever;
 
import org.springframework.context.annotation.Bean;
 
import io.jaegertracing.Configuration;
import io.jaegertracing.Configuration.Propagation;
import io.jaegertracing.internal.propagation.TextMapCodec;
import io.opentracing.Tracer;
import io.opentracing.contrib.java.spring.jaeger.starter.TracerBuilderCustomizer;
import io.opentracing.propagation.Format;
 
@org.springframework.context.annotation.Configuration
public class JaegerTracerConfiguration {
    @Bean
    public Tracer jaegerTracer() {
        // The configuration is fetched from environment, so we need these environment variables:
        // see https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md
        String nameKey = "JAEGER_SERVICE_NAME";
        String serviceName = System.getProperty(nameKey, System.getenv(nameKey));
        if (serviceName == null || serviceName.length() == 0) {
            serviceName = "my-fancy-app";
        }  
        Configuration configuration = Configuration.fromEnv(serviceName);
        return configuration.getTracer();
    }  
}

In this example, the tracer configuration is read from the environment variables.

That's it. Now your application can write traces to the jaeger agent if the environment variables are specified according to https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md#configuration-via-environment.

The OpenTracing frame already has interceptors of HTTP handlers and RestTemplate, so the HTTP headers for tracing are correctly propagated.

Supporting istio

Things gets a little more complicated if istio is used. 


Look at the diagram. The client initiates a request to service A. When A processes the request, it calls service B. Then B calls service C. Istio creates an envoy proxy in every pod. So the request is always passed through the envoy proxy.

In the diagram, when the request comes to Envoy at step 1, Envoy decodes the HTTP header, adds a few HTTP headers for tracing, then forwards the request to App A (step 2).

When App A sends the request to service B in step 3, these injected headers should be propagated. The required headers are listed in https://istio.io/latest/docs/tasks/observability/distributed-tracing/overview/.

Finally, when App A finishes processing the request, the response is sent to Envoy at step 15. Envoy now writes the trace data to Jaeger at the configured rate.

By default, the jaeger Tracer library does not propagate the headers like "x-request-id". So we need to handle it specially. This change is needed for service A and service B. It's not necessary for service C because service C is the last hop.

In the library cnecommon, there are helper classes to extract the istio related headers, and inject the headers to a request. In PCF SM service, this has been processed explicitly. 

Ideally, the header extraction and injection should be transparent to applications. As an application developer, we wish to just use a normal RestTemplate to send HTTP requests.

Luckily, it's possible. We can implement a Codec to perform header extraction and injection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
package whatever;
 
 
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
 
import io.jaegertracing.internal.JaegerSpanContext;
import io.jaegertracing.spi.Codec;
import io.opentracing.propagation.TextMap;
 
/**
 * IstioTracingCodecWrapper wraps an existing codec, and transparently propagates some istio-generated headers.
 *
 */
public class IstioTracingCodecWrapper implements Codec<TextMap> {
    /// The default headers populated by istio
    public static final String ISTIO_HEADERS[] = { "x-request-id", "x-ot-span-context", "x-b3-traceid", "x-b3-spanid",
            "x-b3-parentspanid", "x-b3-sampled", "x-b3-flags" };
    private Codec<TextMap> textMapCodec;
    private Set<String> forwardedHeaders;
 
    public IstioTracingCodecWrapper(Codec<TextMap> textMapCodec) {
        this.textMapCodec = textMapCodec;
        forwardedHeaders = Set.of(ISTIO_HEADERS);
    }
 
    /**
     * Set the headers to forward.
     * @param headers the headers to forward
     */
    public void setForwardedHeaders(String[] headers) {
        forwardedHeaders = Set.of(headers);
    }
 
    @Override
    public void inject(JaegerSpanContext spanContext, TextMap carrier) {
        Map<String, String> newBaggage = new HashMap<>();
        for (Map.Entry<String, String> entry : spanContext.baggageItems()) {
            if (forwardedHeaders.contains(entry.getKey())) {
                carrier.put(entry.getKey(), entry.getValue());
            } else {
                // create a new baggage which does not include the istio headers
                newBaggage.put(entry.getKey(), entry.getValue());
            }
        }
        spanContext = spanContext.withBaggage(newBaggage);
 
        textMapCodec.inject(spanContext, carrier);
    }
 
    @Override
    public JaegerSpanContext extract(TextMap carrier) {
        JaegerSpanContext spanContext = textMapCodec.extract(carrier);
        Map<String, String> baggage = null;
 
        for (Map.Entry<String, String> entry : carrier) {
            // If the header matches the interested pattern, put it into the baggage
            if (forwardedHeaders.contains(entry.getKey())) {
                if (baggage == null) {
                    baggage = new HashMap<String, String>();
                }
 
                baggage.put(entry.getKey(), entry.getValue());
            }
        }
 
        if (baggage != null) {
            if (spanContext == null)
                spanContext = new JaegerSpanContext(0l, 0, 0, 0, (byte) 0);
 
            // put the istio headers in the baggage
            for (Map.Entry<String, String> entry : spanContext.baggageItems()) {
                baggage.put(entry.getKey(), entry.getValue());
            }
            spanContext = spanContext.withBaggage(baggage);
        }
 
        return spanContext;
    }
}

And then when creating the Tracer bean:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
package whatever;
 
import org.springframework.context.annotation.Bean;
 
import io.jaegertracing.Configuration;
import io.jaegertracing.Configuration.Propagation;
import io.jaegertracing.internal.propagation.TextMapCodec;
import io.opentracing.Tracer;
import io.opentracing.contrib.java.spring.jaeger.starter.TracerBuilderCustomizer;
import io.opentracing.propagation.Format;
 
@org.springframework.context.annotation.Configuration
public class JaegerTracerConfiguration {
    @Bean
    public Tracer jaegerTracer() {
        // The configuration is fetched from environment, so we need these environment variables:
        // see https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md
        String nameKey = "JAEGER_SERVICE_NAME";
        String serviceName = System.getProperty(nameKey, System.getenv(nameKey));
        if (serviceName == null || serviceName.length() == 0) {
            serviceName = "noname";
        }
        Configuration configuration = Configuration.fromEnv(serviceName);
        configuration.getCodec().withCodec(Format.Builtin.HTTP_HEADERS, new IstioTracingCodecWrapper(new TextMapCodec(true)));
        configuration.getCodec().withCodec(Format.Builtin.TEXT_MAP, new IstioTracingCodecWrapper(new TextMapCodec(true)));
        return configuration.getTracer();
    }
}

Instrumenting other HTTP clients

If the java applications use RestTemplate to call next hop, then the header propagation is automatically done. However, there are other HTTP client libaries used in our applications:

  • Config service client uses old javax.ws.rs.client.Client
  • Many services use org.eclipse.jetty.client.HttpClient and org.eclipse.jetty.http2.client.HTTP2Client. Such as binding service, policyds, diam-gateway, pcf-smservice...
  • NRF client service uses okhttp3.OkHttpClient
  • Ingress gateway uses many http client libraries, reactor.netty.http.client.HttpClient, okhttp3.OkHttpClient, org.springframework.web.client.RestTemplate, org.eclipse.jetty.client.HttpClient...

So things get complicated. We probably need to explicitly write code to propagate HTTP headers.


Comments

Post a Comment

Popular posts from this blog

Jenkins CI Pipeline for Security Scans

This document describes the set of Jenkins CI Pipeline steps currently in use. Context Diagram Jenkins Workflow OCCNE CI Job Gradle Build  - using gradle, the available source is scanned using OWASP dependency checker, then a number of docker containers are created. Static Scan  - using the McAfee malware scanner container (created in the  Gradle Build  step), all created docker containers are scanned for malware. Verify Build  - Each docker container is loaded and the self test method is executed. Deploy  - An  OCCNE Deployment  job is created and invoked.  ( OCCNE CI  Jobs may run in parallel -  OCCNE Deploy  Jobs are serialized.) OCCNE Deploy Job Prepare Deploy  - Wipe out any old cluster artifacts - get ready for a fresh deploy for (container in  OS_Install, DB_Install,. K8s_install, Cfg_Install ) do: Deploy_{{container}}  - runs the named docker container Test_Deploy_{{container}}  - verifies that the d...
Post a Comment
Read more

GoF patterns

Why learn GoF Design Patterns? Design patterns help you find out patterns in your code. It helps to visualize your code at a higher level and decompose it into logical units.   What are Design Patterns? Design patterns are canonical solutions to recurring problems. They are different from a library that is called from your code. Neither are they framework which is a complicated collection of libraries. Frameworks typically calls your code. The 24 design patterns covered in GoF book can be divided into three categories. Creational Patterns . These patterns seek to answer - "How should objects be created?".  Examples are - Factory, Abstract Factory, Singleton, Builder, Prototype, Dependency Injection. They usually seek to decouple the construction of an object from its use. There are advantages to doing this. Hide implementation of an object, only reveal its interface.  Defer instantiation until run-time.  Allow creation of finite number of instances.  Have f...
Post a Comment
Read more

Git Workflow For Multiple Repositories

Introduction Imagine a situation where you have to work off multiple repositories hosted on multiple hosts. Let us say there are two hosts - ALM and Cloudlab. These repositories on these hosts are managed by two separate teams - ALM and Policy respectively. By setting up a Git workflow across Cloudlab and ALM instances, we can create a development environment for Policy developers working on ATS. This allows them to make changes to the  step files  and approve them within Policy team. Both teams - Policy and ALM, can work independently.  All development and code review is done in the Cloudlab instance. Once changes pass the team's quality assurance review, they are deployed to ALM instance as desired.  This article describes the method for Setting up a Git workflow among multiple repositories across Cloudlab and ALM How to keep those repositories in sync How to use another repo within a given repo Developer Workflow The overall process is as follows: Create a lo...
Post a Comment
Read more