Skip to main content

Supporting OpenTracing jaeger in spring boot applications

This page describes code changes in a typical spring-boot based application to support OpenTracing and jaeger.

Instrumenting a tracer

If you are creating a simple spring boot application that uses spring-boot-starter-web, by default, the application does not support writing traces to jaeger. To support jaeger tracing, the first thing is to modify the build.gradle to add dependency of jaeger:

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'io.opentracing.contrib:opentracing-spring-web-starter:3.0.1'
    // support opentracing jaeger
    implementation 'io.opentracing.contrib:opentracing-spring-jaeger-starter:3.1.2'
    testImplementation('org.springframework.boot:spring-boot-starter-test') {
        exclude group: 'org.junit.vintage', module: 'junit-vintage-engine'
    }
}


Then you need to create a @Bean of type io.opentracing.Tracer. https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md has examples of creating the tracer. The following example should be enough in most applications:

package whatever;
 
import org.springframework.context.annotation.Bean;
 
import io.jaegertracing.Configuration;
import io.jaegertracing.Configuration.Propagation;
import io.jaegertracing.internal.propagation.TextMapCodec;
import io.opentracing.Tracer;
import io.opentracing.contrib.java.spring.jaeger.starter.TracerBuilderCustomizer;
import io.opentracing.propagation.Format;
 
@org.springframework.context.annotation.Configuration
public class JaegerTracerConfiguration {
    @Bean
    public Tracer jaegerTracer() {
        // The configuration is fetched from environment, so we need these environment variables:
        // see https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md
        String nameKey = "JAEGER_SERVICE_NAME";
        String serviceName = System.getProperty(nameKey, System.getenv(nameKey));
        if (serviceName == null || serviceName.length() == 0) {
            serviceName = "my-fancy-app";
        }  
        Configuration configuration = Configuration.fromEnv(serviceName);
        return configuration.getTracer();
    }  
}

In this example, the tracer configuration is read from the environment variables.

That's it. Now your application can write traces to the jaeger agent if the environment variables are specified according to https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md#configuration-via-environment.

The OpenTracing frame already has interceptors of HTTP handlers and RestTemplate, so the HTTP headers for tracing are correctly propagated.

Supporting istio

Things gets a little more complicated if istio is used. 


Look at the diagram. The client initiates a request to service A. When A processes the request, it calls service B. Then B calls service C. Istio creates an envoy proxy in every pod. So the request is always passed through the envoy proxy.

In the diagram, when the request comes to Envoy at step 1, Envoy decodes the HTTP header, adds a few HTTP headers for tracing, then forwards the request to App A (step 2).

When App A sends the request to service B in step 3, these injected headers should be propagated. The required headers are listed in https://istio.io/latest/docs/tasks/observability/distributed-tracing/overview/.

Finally, when App A finishes processing the request, the response is sent to Envoy at step 15. Envoy now writes the trace data to Jaeger at the configured rate.

By default, the jaeger Tracer library does not propagate the headers like "x-request-id". So we need to handle it specially. This change is needed for service A and service B. It's not necessary for service C because service C is the last hop.

In the library cnecommon, there are helper classes to extract the istio related headers, and inject the headers to a request. In PCF SM service, this has been processed explicitly. 

Ideally, the header extraction and injection should be transparent to applications. As an application developer, we wish to just use a normal RestTemplate to send HTTP requests.

Luckily, it's possible. We can implement a Codec to perform header extraction and injection:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
package whatever;
 
 
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
 
import io.jaegertracing.internal.JaegerSpanContext;
import io.jaegertracing.spi.Codec;
import io.opentracing.propagation.TextMap;
 
/**
 * IstioTracingCodecWrapper wraps an existing codec, and transparently propagates some istio-generated headers.
 *
 */
public class IstioTracingCodecWrapper implements Codec<TextMap> {
    /// The default headers populated by istio
    public static final String ISTIO_HEADERS[] = { "x-request-id", "x-ot-span-context", "x-b3-traceid", "x-b3-spanid",
            "x-b3-parentspanid", "x-b3-sampled", "x-b3-flags" };
    private Codec<TextMap> textMapCodec;
    private Set<String> forwardedHeaders;
 
    public IstioTracingCodecWrapper(Codec<TextMap> textMapCodec) {
        this.textMapCodec = textMapCodec;
        forwardedHeaders = Set.of(ISTIO_HEADERS);
    }
 
    /**
     * Set the headers to forward.
     * @param headers the headers to forward
     */
    public void setForwardedHeaders(String[] headers) {
        forwardedHeaders = Set.of(headers);
    }
 
    @Override
    public void inject(JaegerSpanContext spanContext, TextMap carrier) {
        Map<String, String> newBaggage = new HashMap<>();
        for (Map.Entry<String, String> entry : spanContext.baggageItems()) {
            if (forwardedHeaders.contains(entry.getKey())) {
                carrier.put(entry.getKey(), entry.getValue());
            } else {
                // create a new baggage which does not include the istio headers
                newBaggage.put(entry.getKey(), entry.getValue());
            }
        }
        spanContext = spanContext.withBaggage(newBaggage);
 
        textMapCodec.inject(spanContext, carrier);
    }
 
    @Override
    public JaegerSpanContext extract(TextMap carrier) {
        JaegerSpanContext spanContext = textMapCodec.extract(carrier);
        Map<String, String> baggage = null;
 
        for (Map.Entry<String, String> entry : carrier) {
            // If the header matches the interested pattern, put it into the baggage
            if (forwardedHeaders.contains(entry.getKey())) {
                if (baggage == null) {
                    baggage = new HashMap<String, String>();
                }
 
                baggage.put(entry.getKey(), entry.getValue());
            }
        }
 
        if (baggage != null) {
            if (spanContext == null)
                spanContext = new JaegerSpanContext(0l, 0, 0, 0, (byte) 0);
 
            // put the istio headers in the baggage
            for (Map.Entry<String, String> entry : spanContext.baggageItems()) {
                baggage.put(entry.getKey(), entry.getValue());
            }
            spanContext = spanContext.withBaggage(baggage);
        }
 
        return spanContext;
    }
}

And then when creating the Tracer bean:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
package whatever;
 
import org.springframework.context.annotation.Bean;
 
import io.jaegertracing.Configuration;
import io.jaegertracing.Configuration.Propagation;
import io.jaegertracing.internal.propagation.TextMapCodec;
import io.opentracing.Tracer;
import io.opentracing.contrib.java.spring.jaeger.starter.TracerBuilderCustomizer;
import io.opentracing.propagation.Format;
 
@org.springframework.context.annotation.Configuration
public class JaegerTracerConfiguration {
    @Bean
    public Tracer jaegerTracer() {
        // The configuration is fetched from environment, so we need these environment variables:
        // see https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/README.md
        String nameKey = "JAEGER_SERVICE_NAME";
        String serviceName = System.getProperty(nameKey, System.getenv(nameKey));
        if (serviceName == null || serviceName.length() == 0) {
            serviceName = "noname";
        }
        Configuration configuration = Configuration.fromEnv(serviceName);
        configuration.getCodec().withCodec(Format.Builtin.HTTP_HEADERS, new IstioTracingCodecWrapper(new TextMapCodec(true)));
        configuration.getCodec().withCodec(Format.Builtin.TEXT_MAP, new IstioTracingCodecWrapper(new TextMapCodec(true)));
        return configuration.getTracer();
    }
}

Instrumenting other HTTP clients

If the java applications use RestTemplate to call next hop, then the header propagation is automatically done. However, there are other HTTP client libaries used in our applications:

  • Config service client uses old javax.ws.rs.client.Client
  • Many services use org.eclipse.jetty.client.HttpClient and org.eclipse.jetty.http2.client.HTTP2Client. Such as binding service, policyds, diam-gateway, pcf-smservice...
  • NRF client service uses okhttp3.OkHttpClient
  • Ingress gateway uses many http client libraries, reactor.netty.http.client.HttpClient, okhttp3.OkHttpClient, org.springframework.web.client.RestTemplate, org.eclipse.jetty.client.HttpClient...

So things get complicated. We probably need to explicitly write code to propagate HTTP headers.


Comments

Post a Comment

Popular posts from this blog

HOWTO on implementing scanning into a CI Pipeline

Introduction As a part of the Software Security Assurance guidelines, we are required to perform various types of security scanning.  Security scanning is a specialized kind of testing that attempts to identify potential software bugs that could make a deployed application vulnerable to attack and compromise.  As with any testing, the tests can be manual or automatic, and they wan be performed at various points in the development cycle.   We can classify security scanning into three general categories: Static Application Security Testing (SAST)   - performed against source code / configuration data a static scan looks for common logic errors that might lead to system compromise. Dy namic Application Security Testing  (DAST)  - Performed against a running system, a dynamic scan looks for vulnerable software / configurations that might lead to system compromise. Security Monitoring  - Deployed as a part of the system, a security monitor co...
Post a Comment
Read more

Fortify Tooling User Guide

  Introduction The fortify-tools container is located within a shared repository in OCIR and requires a JWT to be able to access.  The variable WF_JWT will need to be set to a valid MAT You will want to choose one of three ways to use the Fortify SCA tooling: Integration Description Using the Fortify Tools Standalone to Scan a Project This is for using the tooling without integration in GitLab CI or Jenkins CI. Using the Fortify Tools In GitLab CI to Scan a Project This is for using the tooling against a project that whose code is hosted in GitLab and whose CI engine is GitLab CI. Using the Fortify Tools In Jenkins CI to Scan a Project This is for using the tooling against a project that whose code is hosted in GitLab and whose CI engine is Jenkins CI. Using the Fortify Tools Standalone to Scan a Project Simple Usage Run the Fortify Tools in a container docker run -t --rm -v <path to project source root directory>:/var/fortify/src phx.ocir.io/oraclegbudevcorp/cn-shared/s...
Post a Comment
Read more