[Industry solution] Use SOFATracer to learn the distributed tracing system Opentracing (1)

[Industry solution] Use SOFATracer to learn the distributed tracing system Opentracing (1)

0x00 summary

SOFA is a financial-grade distributed middleware independently developed by Ant Financial. It contains all the components required to build a financial-grade cloud native architecture. SOFATracer is a component used for distributed system call tracking.

The author has previous experience with zipkin and hopes to expand to Opentracing, so I summarized this article on the basis of studying the official SOFATracer blog and the source code and shared it with you.

Because of the word limit, this article is divided into two parts.

0x01 Reason & Problem

1.1 Selection

Why did you choose to learn from SOFATracer? The reason is simple: there are endorsements by large companies (the best practices tempered in the financial scene), there are official blogs compiled by developers and the community, there are live broadcasts, and the examples are easy to debug. Why not study and use it?

1.2 The problem

Let us use questions to guide the reading.

  • How is spanId generated and what are the rules?
  • How is traceId generated, and what are the rules?
  • Where is the Span generated by the client?
  • Where does ParentSpan come from?
  • ChildSpan was created by ParentSpan. When was it created?
  • How is Trace information transmitted?
  • What does the server do after receiving the request?
  • How to deal with SpanContext on the server side?
  • How to collect link information?

1.3 The scope of this article

Full link tracking is divided into three tracking levels:

  • Cross-process tracking (calling another microservice)
  • Database tracking
  • In-process tracking (in-process) (tracing within a function)

This article only discusses cross-process tracking, because cross-process tracking is the simplest and easy to use^_^. For cross-process tracking, you can write interceptors or filters to track each request, it only requires writing very little code.

0x02 background knowledge

2.1 Trends and challenges

The birth of container and serverless programming methods has greatly improved the efficiency of software delivery and deployment. During the evolution of the architecture, two changes can be seen:

  • The application architecture begins to gradually transform from a monolithic system to microservices, and the business logic therein will subsequently become calls and requests between microservices.
  • From the perspective of resources, the physical unit of traditional servers has gradually faded into an invisible and intangible virtual resource model.

From the above two changes, we can see that behind this flexible and standardized architecture, the original requirements for operation, maintenance and diagnosis have become more and more complex. How to sort out the service dependency call relationship, how to quickly debugtrack the time-consuming service processing in such an environment , find the service performance bottleneck, and reasonably evaluate the capacity of the service has become a tricky task.

2.2 Observability

In order to deal with these problems, Observabilitythe concept of observability ( ) was introduced into the software field. Traditional monitoring and alarming mainly focus on abnormal conditions and failure factors of the system. Observability is more concerned with displaying the operating status of the system from the system itself, which is more like a self-examination of the system. An observable system pays more attention to the state of the application itself, rather than indirect evidence such as the machine or network where it is located. We hope to directly obtain the current throughput and latency information of the application. In order to achieve this goal, we need to reasonably proactively expose more application operating information. In the current application development environment, in the face of complex systems, we will gradually focus on the combination of point to point, line and surface. This allows us to better understand the system, not only knowing What, but also answering Why.

Observability currently mainly includes the following three pillars:

  • Log ( Logging): LoggingMainly records some discrete events. Applications often output log information in a defined format to a file, and then use a log collection program to collect it for analysis and aggregation. Although it is possible to connect all log point events in time, it is difficult to show the complete call relationship path;
  • Measure ( Metrics): Metricoften some aggregation of information, compared to Loggingthe loss of some specific information, but the space occupied by much smaller than the full log can be used to monitor and alarm, in this regard Prometheus has basically become a de facto standard;
  • Distributed trace ( Tracing): Tracingbetween Logging, and Metricamong, to request a series of dimensions to the relationship between service calls and record calls time-consuming, that is to retain the necessary information, in turn dispersed log events by Span series, to help us better To understand the behavior of the system, assist in debugging and troubleshoot performance problems.

The three pillars have the following characteristics:

  • The characteristic of Metric is that it is additive. Atomic, each is a logical measurement unit, or a histogram within a time period. For example: the current depth of the queue can be defined as a measurement unit, which is updated when writing or reading; the number of incoming HTTP requests can be defined as a counter for simple accumulation; the execution time of the request can be defined as A histogram, updated and statistically summarized on a specified time slice.
  • The characteristic of Logging is that it describes some discrete (discontinuous) events. For example: The application outputs debug or error information through a rolling file, and stores it in Elasticsearch through the log collection system; Approval details are stored in the database (BigTable) through Kafka; Or, metadata information for a specific request, from The service request is stripped out and sent to an exception collection service, such as NewRelic.
  • The biggest feature of Tracing is that it processes information within the scope of a single request. Any data and metadata information are bound to a single transaction in the system. For example: a call to the RPC execution process of a remote service; an actual SQL query statement; a business ID of an HTTP request.

2.3 Tracing

Distributed tracing, also known as distributed request tracing, is a method for analyzing and monitoring applications, especially those built using microservices architecture; distributed tracing helps pinpoint where failures occur As well as the causes of poor performance, developers can use distributed tracing to help debug and optimize their code, and IT and DevOps teams can use distributed tracing to monitor applications.

2.3.1 The Birth of Tracing

Tracing is a technology that has appeared in the 90s. But what really made the field popular is a paper by Google "Dapper, a Large-Scale Distributed Systems Tracing Infrastructure", and another paper "Uncertainty in Aggregate Estimates from Sampled Distributed Traces" contains information about sampling. More detailed analysis. A batch of excellent Tracing software was born after the publication of the paper.

2.3.2 Tracing function

  • Fault location-you can see the complete path of the request, which is more convenient to locate the problem than the discrete log (because the sampling rate will be set in the real online environment, you can use the debug switch to achieve full sampling of specific requests);
  • Dependency combing-generating a service dependency graph based on the calling relationship;
  • Performance analysis and optimization-can conveniently record the time-consuming occupation and proportion of different processing units on the system link;
  • Capacity planning and evaluation;
  • Cooperate Loggingand Metricstrengthen monitoring and alarm.

2.4 OpenTracing

In order to solve the problem of API incompatibility between different distributed tracing systems, OpenTracing appeared. OpenTracing aims to standardize Trace data structure and format, and its purpose is to:

  • Interoperability of Trace clients developed in different languages. Clients developed in languages such as Java/.Net/PHP/Python/NodeJs, as long as they follow the OpenTracing standard, can be connected to OpenTracing-compatible monitoring backends.
  • Tracing monitors the interoperability of the backend. As long as they follow the OpenTracing standard, companies can replace specific Tracing monitoring back-end products as needed, such as replacing Zipkin with Jaeger/CAT/Skywalking and other back-ends.

OpenTracing is not a standard. The OpenTracing API provides a standard, vendor-independent framework, which is a highly abstract collection of a series of operations involved in distributed links. This means that if a developer wants to try a different distributed tracking system, the developer only needs to modify the Tracer configuration instead of replacing the entire distributed tracking system.

0x03 OpenTracing data model

Most of the thought models of distributed tracing systems come from Google's Dapper paper, and OpenTracing also uses similar terms . There are several basic concepts that we need to understand clearly in advance:

  • Trace: In a broad sense, a trace represents the execution of a transaction or process in a (distributed) system. In the OpenTracing standard, a trace is a directed acyclic graph (DAG) composed of multiple spans, and each span represents a named and timed continuous execution segment in the trace.

  • Span (span): A span represents a logical operation unit with start time and execution time in the system, that is, a logical operation in the application. A logical causal relationship is established between spans through nesting or sequential arrangement. A span can be understood as a method call, a program block call, or an RPC/database access. As long as it is a program access with a complete time period, it can be considered a span.

  • Logs: Each span can perform multiple Logs operations. Each Logs operation requires a time name with a timestamp and an optional storage structure of any size.

  • Tags: Each span can have multiple tags in the form of key-value pairs (key: value). Tags do not have a timestamp and support simple annotations and supplements to the span.

  • SpanContext: SpanContextMore like a "concept" rather than a useful function of the general OpenTracing layer. It plays an important role in creating Span, Inject(injecting) into the transfer protocol, and Extract(extracting) call chain information from the transfer protocol SpanContext.

3.1 Span

Represents a call unit in the distributed call chain, and its boundary includes a request that enters the service and then exits from the current service by some way (http/dubbo, etc.).

A span generally records some information inside the calling unit, such as each Spancontained operation name, start and end time, additional information Span Tag, can be used to record Spanspecial events in the record Span Log, used to transfer Spancontext SpanContextand define the Spanrelationship between References.

  • An operation name
  • A start timestamp
  • A finish timestamp
  • Tag information: 0 or more Span Tags in the form of keys: values . key must be string, values can be strings, bool, numeric types
  • Log information: 0 or more Span logs
  • A SpanContext
  • By SpanContext can point to zero or more causally related of Span .

3.2 Tracer

Trace describes a "transaction" in a distributed system. A trace is a directed acyclic graph composed of several spans .

Tracer is used to create Spans and understand how to inject (serialize) and extract (deserialize) Spans across process boundaries. It has the following responsibilities:

  1. Create and open a span
  2. Extract/inject a spanContext from a certain medium

From the point of view of graph theory, traces can be considered as a DAG of spans. In other words, the DAG formed by multiple spans is a Trace.

For example, the figure below is a trace formed by eight spans.

  Trace   Span  

        [Span A]   (the root span)
     |             |
 [Span B]      [Span C]  (Span C is a `ChildOf` Span A)
     |             |
 [Span D]      +---+-------+
               |           |
           [Span E]    [Span F] >>> [Span G] >>> [Span H]
                         (Span G `FollowsFrom` Span F)

Sometimes, it is more understandable to visualize in chronological order. The following is an example.

  Trace   Spans  

 | | | | | | | | > time

 [Span A ]
   [Span B ]
      [Span D ]
    [Span C ]
         [Span E ]        [Span F ] [Span G ] [Span H ]

3.3 References between Spans

A span can have a causal relationship with one or more spans. OpenTracing defines two relationships: ChildOf and FollowsFrom. These two reference types represent the direct causal relationship between the child node and the parent node.

ChildOfSpan will become a child, and FollowsFromwill become a parent. These two relationships establish a direct causal relationship between child span and parent span .

3.4 SpanContext

Represents the context corresponding to a span. Span and spanContext are basically in a one-to-one correspondence. This SpanContext can be passed to the downstream of the call chain through some media and methods for some processing (such as child Span id generation, information inheritance printing Logs etc.).

The context stores some information that needs to cross the boundary (needed for propagation tracking), such as:

  • spanId: the id of the current span
  • traceId: The traceId to which this span belongs (that is, the unique id of this call chain).
    • trace_idAnd span_idto distinguish Tracein Span; OpenTraceing any implementation-dependent state (such as trace and span id) needs to be a process of cross- Span contacted.
  • baggage: Other information that can pass across multiple call units, that is, key-value pairs across processes. Baggage ItemsAnd Span Tagthe same structure, the only difference is: Span Tagonly the current Spanpresence, not in the whole traceof the transfer, and Baggage Itemswill vary with the call chain transfer.

SpanContextThe simplified version of the data structure is as follows:

- trace_id: "abc123"
- span_id: "xyz789
- Baggage Items:
	- special_id: "vsid1738"

Straddling the (inter-service or protocol) transmission implemented during transfer and associated call relationship, need to be able to SpanContextinject into the downstream medium and extracted in a downstream transmission medium SpanContext.

It is often possible to use similar HTTP Headersmechanisms provided by the protocol itself to achieve such information transfer, Kafkaand message middleware like this also provides Headersmechanisms to achieve such functions .

OpenTracingImplemented, may be used provided Tracer.Inject api (...) and Tracer.Extract (...) facilitate the realization SpanContextof injection and extraction.

  • "Extarct()" gets the trace context from the medium (usually HTTP header).
  • "Inject()" puts the tracking context into the medium to ensure the continuity of the tracking chain.

3.5 Carrier

Carrier represents a medium that carries spanContext. For example, there will be HttpCarrier in the http call scene, and there will be a corresponding DubboCarrier in the dubbo call scene.

3.6 Formatter

This interface is responsible for the specific logic of the serialization and deserialization context in specific scenarios. For example, there is usually a corresponding HttpFormatter in the use of HttpCarrier. The injection and extraction of Tracer is entrusted to Formatter.

3.7 ScopeManager

This class is a newly added component after version 0.30. The function of this component is to obtain information about the Span enabled in the current thread and enable some spans that are not enabled. In some scenarios, we may create multiple spans in a thread at the same time, but only one span is enabled in the same thread at the same time, and other spans may be in the following states:

  1. Wait for the child span to complete
  2. Waiting for some blocking method
  3. Created but did not start

3.8 Reporter

In addition to the above components, when implementing a distributed full-link monitoring framework, there is also a reporter component, through which some key link information (such as span creation and termination) can be printed or reported, and only this information is carried out. After processing, we can visualize and truly monitor the entire link information.

0x04 SOFATracer

SOFATracer is a component for tracing distributed system calls. It records various network call situations in the call link in a log format through a unified traceId, so as to achieve the purpose of visualizing network calls. These logs can be used for rapid fault discovery, service management, etc.

The SOFATracer team has built a complete Tracer framework kernel for us, including data models, encoders, cross-process transparent traceId transmission, sampling, log placement and reporting, and other core mechanisms. It also provides extended APIs and parts based on open source components. The plug-in provides great convenience for us to build our own Tracer platform based on this framework.

SOFATracer currently does not provide data collector and UI display functions; there are two main considerations:

  • SOFATracer, as a very lightweight component in the SOFA system, is intended to log the span data to the disk so that users can process the data more flexibly
  • In terms of UI display, SOFATracer itself is implemented based on the OpenTracing specification, and can be seamlessly connected to some open source products in the model, which can make up for its own shortcomings in link visualization to a certain extent.

Therefore, in the reporting model, SOFATracer provides extensions for log output and external reporting, so that the access party can handle the reported data in a flexible enough manner. Through SOFARPC + SOFATracer + zipKin, a complete link tracking system can be quickly built, including point burying, collection, analysis and display, etc. The collection and analysis are mainly based on the capabilities of zipKin.

At present, SOFATracer has supported embedding support for the following open source components: Spring MVC, RestTemplate, HttpClient, OkHttp3, JDBC, Dubbo (2.6 2.7), SOFARPC, Redis, MongoDB, Spring Message, Spring Cloud Stream (based on Spring Message embedding Point), RocketMQ, Spring Cloud FeignClient, Hystrix.

OpentracingAll core components will be declared as an interface, for example Tracer, , Span, SpanContext(versionFormat further includes high Scopeand ScopeManager) the like. SOFATracerUse the version 0.22.0 , mainly on Tracer, Span, SpanContextto achieve three conceptual model. For the following combination of several components SOFATracerto analyze.

4.1 Tracer & SofaTracer

TracerIs a simple, generalized interface, its role is to build spanand transport span.

SofaTracerImplements io.opentracing.Traceran interface, and expanded sampling, data reporting capabilities.

public class SofaTracer implements Tracer {
    public static final String ROOT_SPAN_ID = "0";
    private final String tracerType;
    private final Reporter clientReporter;
    private final Reporter serverReporter;
    private final Map<String, Object> tracerTags = new ConcurrentHashMap();
    private final Sampler sampler;

4.2 Span & SofaTracerSpan

SpanIt is a span unit. In the actual application process, it Spanis a complete data packet, which contains the data that the current node needs to report.

SofaTracerSpanImplements io.opentracing.Spanan interface, and expanded to Reference, tagsthreads and asynchronous processing plug-ins and extensions required logTypeand currently produces spanthe Tracerability to handle the type.

Each span contains two important information, span id (span id of the current module) and span parent ID (span id of the last calling module), through which a span can be located in the call chain. These belong to the core information and are stored in SpanContextit.

public class SofaTracerSpan implements Span {
    public static final char                                ARRAY_SEPARATOR      = '|';
    private final SofaTracer                                sofaTracer;
    private final List<SofaTracerSpanReferenceRelationship> spanReferences;
    /** tags for String  */
    private final Map<String, String>                       tagsWithStr          = new LinkedHashMap<>();
    /** tags for Boolean */
    private final Map<String, Boolean>                      tagsWithBool         = new LinkedHashMap<>();
    /** tags for Number  */
    private final Map<String, Number>                       tagsWithNumber       = new LinkedHashMap<>();
    private final List<LogData>                             logs                 = new LinkedList<>();
    private String                                          operationName        = StringUtils.EMPTY_STRING;
    private final SofaTracerSpanContext                     sofaTracerSpanContext;
    private long                                            startTime;
    private long                                            endTime              = -1;

It is divided into ClientSpan and ServerSpan in SOFARPC. ClientSpan records the process from the client sending a request to the server to receiving the response from the server. ServerSpan is the process from the server receiving the client time to sending the response result to the client.

4.3 SpanContext & SofaTracerSpanContext

SpanContextFor the OpenTracingimplementation is crucial, through SpanContextcan be achieved across the process chain Reuters pass, and through SpanContextthe information carried in the entire link series.

Official documents in this sentence: "In the OpenTracingmiddle, we forced SpanContextinstance immutable, in order to avoid Spanin finishand referencethere will be a complex operation life cycle issues." This is understandable, if SpanContexta change in the pass-through process For example tracerId, if it is changed , the link may be broken.

SofaTracerSpanContextImplements SpanContextan interface, expanded building SpanContext, serialization baggageItemsand SpanContextother new capabilities.

public interface SofaTraceContext {
    void push(SofaTracerSpan var1);
    SofaTracerSpan getCurrentSpan();
    SofaTracerSpan pop();
    int getThreadLocalSpanSize();
    void clear();
    boolean isEmpty();

4.3.1 Transmit Trace Information

This section answers how to transmit Trace information?

In OpenTracing, Trace information is passed through SpanContext.

SpanContext stores some information that needs to cross boundaries, such as trace Id, span id, and Baggage. This information will be serialized and transmitted by different components according to their own characteristics, for example, serialized into the http header and then transmitted. Then associate the current node to the entire Tracer link through the information carried by this SpanContext.

Simply put, it is to use the HTTP header as a medium (Carrier) to transmit trace information (traceID). Whether the microservices are gRPC or RESTFul, they all use the HTTP protocol. If it is a message queue (Message Queue), put the trace information (traceID) in the message header.

The SofaTracerSpanContext class includes and implements "some information that needs to cross the boundary".

public class SofaTracerSpanContext implements SpanContext {

    //spanId separator
    public static final String        RPC_ID_SEPARATOR       = ".";

    //======= The following is the key for serializing data ========================

    private static final String       TRACE_ID_KET           = "tcid";

    private static final String       SPAN_ID_KET            = "spid";

    private static final String       PARENT_SPAN_ID_KET     = "pspid";

    private static final String       SAMPLE_KET             = "sample";

     * The serialization system transparently passes the prefix of the attribute key
    private static final String       SYS_BAGGAGE_PREFIX_KEY = "_sys_";

    private String                    traceId                = StringUtils.EMPTY_STRING;

    private String                    spanId                 = StringUtils.EMPTY_STRING;

    private String                    parentId               = StringUtils.EMPTY_STRING;

     * Default will not be sampled
    private boolean                   isSampled              = false;

     * The system transparently transmits data,
     * mainly refers to the transparent transmission data of the system dimension.
     * Note that this field cannot be used for transparent transmission of business.
    private final Map<String, String> sysBaggage             = new ConcurrentHashMap<String, String>();

     * Transparent transmission of data, mainly refers to the transparent transmission data of the business
    private final Map<String, String> bizBaggage             = new ConcurrentHashMap<String, String>();

     * sub-context counter
    private AtomicInteger             childContextIndex      = new AtomicInteger(0);

4.3.2 Thread Storage

In each node of the link, the SpanContext is thread-related, and the details are stored in the thread ThreadLocal.

The implementation is the SofaTracerThreadLocalTraceContext function. We can see that ThreadLocal is used, because Context is related to thread context.

public class SofaTracerThreadLocalTraceContext implements SofaTraceContext {
    private final ThreadLocal<SofaTracerSpan> threadLocal = new ThreadLocal();

    public void push(SofaTracerSpan span) {
        if (span != null) {

    public SofaTracerSpan getCurrentSpan() throws EmptyStackException {
        return this.isEmpty() ? null : (SofaTracerSpan)this.threadLocal.get();

    public SofaTracerSpan pop() throws EmptyStackException {
        if (this.isEmpty()) {
            return null;
        } else {
            SofaTracerSpan sofaTracerSpan = (SofaTracerSpan)this.threadLocal.get();
            return sofaTracerSpan;

    public int getThreadLocalSpanSize() {
        SofaTracerSpan sofaTracerSpan = (SofaTracerSpan)this.threadLocal.get();
        return sofaTracerSpan == null ? 0 : 1;

    public boolean isEmpty() {
        SofaTracerSpan sofaTracerSpan = (SofaTracerSpan)this.threadLocal.get();
        return sofaTracerSpan == null;

    public void clear() {

4.4 Reporter

Log placement is divided into summary log placement and statistical log placement;

  • The summary log is a log that will land on the disk every time it is called;
  • The statistical log is a log for statistical output at regular intervals.

Data reporting is a function implemented by SofaTracer based on the OpenTracing Tracer interface extension; the Reporter instance exists as an attribute of the SofaTracer, and the Reporter instance is initialized when the SofaTracer instance is constructed.

In addition to the core reporting function, the design of the Reporter interface also provides the ability to obtain the Reporter type. This is because the burying mechanism solution currently provided by SOFATracer needs to rely on this implementation.

public interface Reporter {

    // Reporter  
    String getReporterType();
    // span
    void report(SofaTracerSpan span);
    // span  
    void close();

There are two implementation classes for Reporter, SofaTracerCompositeDigestReporterImpl and DiskReporterImpl:

  • SofaTracerCompositeDigestReporterImpl: Implementation of combined summary log reporting. When reporting, it will traverse all the Reporters in the current SofaTracerCompositeDigestReporterImpl and perform the report operation one by one; it can be extended to use by external users.
  • DiskReporterImpl: The core implementation class of data storage disk, and it is also the reporter used by default in SOFATracer.

0x05 sample code

5.1 RestTemplate

We are using the RestTemplate example

import com.sofa.alipay.tracer.plugins.rest.SofaTracerRestTemplateBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.http.ResponseEntity;
import org.springframework.util.concurrent.ListenableFuture;
import org.springframework.web.client.AsyncRestTemplate;
import org.springframework.web.client.RestTemplate;

public class RestTemplateDemoApplication {
    private static Logger logger = LoggerFactory.getLogger(RestTemplateDemoApplication.class);

    public static void main(String[] args) throws Exception {
        SpringApplication.run(RestTemplateDemoApplication.class, args);
        RestTemplate restTemplate = SofaTracerRestTemplateBuilder.buildRestTemplate();
        ResponseEntity<String> responseEntity = restTemplate.getForEntity(
            "http://localhost:8801/rest", String.class);
        logger.info("Response is {}", responseEntity.getBody());

        AsyncRestTemplate asyncRestTemplate = SofaTracerRestTemplateBuilder
        ListenableFuture<ResponseEntity<String>> forEntity = asyncRestTemplate.getForEntity(
            "http://localhost:8801/asyncrest", String.class);
        logger.info("Async Response is {}", forEntity.get().getBody());

        logger.info("test finish .......");

0x06 start

Here we must first mention SOFATracer's burying mechanism. Different components have different application scenarios and extension points. Therefore, the implementation of plug-ins must be adapted to local conditions. The SOFATracer burying method is generally implemented through the Filter and Interceptor mechanisms. Therefore, the Client startup/Server startup we mentioned below is mainly to create the Filter and Interceptor mechanisms.

Let's take RestTemplate as an example to see the start of SofaTracer.

6.1 Spring SPI

Only the SofaTracerRestTemplateBuilder is used in the code, how can a complete link tracking be achieved? The secret is in the pom.xml file.


Many classes are defined in the spring.factories file of tracer-sofa-boot-starter.


There is a very decoupled extension mechanism in Spring Boot: Spring Factories. This kind of extension mechanism is actually implemented by imitating SPI extension mechanism in Java.

The full name of SPI is Service Provider Interface, which is a service discovery mechanism to find a service implementation for an interface. The service can be specified dynamically when the module is assembled. It is somewhat similar to the idea of IOC, which is to move the control of the assembly outside the program.

Spring Factories is to configure the implementation class name of the interface in the META-INF/spring.factories file, and then read these configuration files in the program and instantiate them. This custom SPI mechanism is the basis of Spring Boot Starter's implementation.

For the SpringBoot project, after introducing the tracer-sofa-boot-starter, the Spring program directly reads and instantiates the classes in the spring.factories file of the tracer-sofa-boot-starter. Users can directly use many SOFA functions in the program.

Take Reporter as an example. The automatic configuration class SofaTracerAutoConfiguration will save all current bean instances of type SpanReportListener to the List object of SpanReportListenerHolder. The SpanReportListener type of Bean will be injected into the current Ioc container in the ZipkinSofaTracerAutoConfiguration automatic configuration class. In this way, when invokeReportListeners is called, you can get the report class of zipkin, so that the report can be realized.

For non-SpringBoot application reporting support, in essence, you need to instantiate the ZipkinSofaTracerSpanRemoteReporter object and place this object in the List object of SpanReportListenerHolder. So SOFATracer provides a ZipkinReportRegisterBean in the zipkin plug-in, and by implementing the bean life cycle interface InitializingBean provided by Spring, a ZipkinSofaTracerSpanRemoteReporter instance is constructed after ZipkinReportRegisterBean is initialized and handed over to the SpanReportListenerHolder class for management.

6.2 Client start

This part of the code is SofaTracerRestTemplateConfiguration. The main function is to generate a RestTemplateInterceptor.

The function of RestTemplateInterceptor is to process the request first.

First of all, the role of SofaTracerRestTemplateConfiguration is to generate a SofaTracerRestTemplateEnhance.

@ConditionalOnProperty(prefix = "com.alipay.sofa.tracer.resttemplate", value = "enable", matchIfMissing = true)
public class SofaTracerRestTemplateConfiguration {

    public SofaTracerRestTemplateBeanPostProcessor sofaTracerRestTemplateBeanPostProcessor() {
        return new SofaTracerRestTemplateBeanPostProcessor(sofaTracerRestTemplateEnhance());

    public SofaTracerRestTemplateEnhance sofaTracerRestTemplateEnhance() {
        return new SofaTracerRestTemplateEnhance();

Secondly, SofaTracerRestTemplateEnhance will generate a RestTemplateInterceptor so that it can be processed before the request.

public class SofaTracerRestTemplateEnhance {

    private final RestTemplateInterceptor restTemplateInterceptor;

    public SofaTracerRestTemplateEnhance() {
        AbstractTracer restTemplateTracer = SofaTracerRestTemplateBuilder.getRestTemplateTracer();
        this.restTemplateInterceptor = new RestTemplateInterceptor(restTemplateTracer);

    public void enhanceRestTemplateWithSofaTracer(RestTemplate restTemplate) {
        //check interceptor
        if (checkRestTemplateInterceptor(restTemplate)) {
        List<ClientHttpRequestInterceptor> interceptors = new ArrayList<>(
        interceptors.add(0, this.restTemplateInterceptor);

    private boolean checkRestTemplateInterceptor(RestTemplate restTemplate) {
        for (ClientHttpRequestInterceptor interceptor : restTemplate.getInterceptors()) {
            if (interceptor instanceof RestTemplateInterceptor) {
                return true;
        return false;

6.3 Server start

This part of the code is OpenTracingSpringMvcAutoConfiguration. The main function is to register SpringMvcSofaTracerFilter. When the Spring Filter is used to intercept a Servlet program, it can decide whether to continue the request to the Servlet program, and whether to modify the request and response messages.

@EnableConfigurationProperties({ OpenTracingSpringMvcProperties.class, SofaTracerProperties.class })
@ConditionalOnProperty(prefix = "com.alipay.sofa.tracer.springmvc", value = "enable", matchIfMissing = true)
public class OpenTracingSpringMvcAutoConfiguration {

    private OpenTracingSpringMvcProperties openTracingSpringProperties;

    @ConditionalOnWebApplication(type = ConditionalOnWebApplication.Type.SERVLET)
    public class SpringMvcDelegatingFilterProxyConfiguration {
        public FilterRegistrationBean springMvcDelegatingFilterProxy() {
            FilterRegistrationBean filterRegistrationBean = new FilterRegistrationBean();
            SpringMvcSofaTracerFilter filter = new SpringMvcSofaTracerFilter();
            List<String> urlPatterns = openTracingSpringProperties.getUrlPatterns();
            if (urlPatterns == null || urlPatterns.size() <= 0) {
            } else {
            return filterRegistrationBean;

    @ConditionalOnWebApplication(type = ConditionalOnWebApplication.Type.REACTIVE)
    public class WebfluxSofaTracerFilterConfiguration {
        @Order(Ordered.HIGHEST_PRECEDENCE + 10)
        public WebFilter webfluxSofaTracerFilter() {
            return new WebfluxSofaTracerFilter();

The following describes the burying mechanism and the overall request process, etc., so stay tuned.

0xEE personal information

Thinking about life and technology

WeChat public account: Rossi s thinking

If you want to get the news of personally written articles in time, or want to see personally recommended technical materials, stay tuned.

0xFF reference

Distributed tracing system - Opentracing

Open Distributed Tracing (OpenTracing) entry and Jaeger implementation

OpenTracing semantic description

Overview of distributed tracking system and comparison of mainstream open source systems

Skywalking Distributed Tracking and Monitoring: Getting Started

Distributed full link monitoring-opentracing small test

opentracing actual combat

Detailed explanation of Go microservice full link tracking

OpenTracing Java Library tutorial (3)-Passing SpanContext across services

OpenTracing Java Library tutorial (1)-introduction to trace and span

Ant Financial Distributed Link Tracking Component SOFATracer Overview|Analysis

Ant Financial Open Source Distributed Link Tracking Component SOFATracer Link Transparent Transmission Principle and SLF4J MDC Expansion Capability Analysis

Ant Financial open source distributed link tracking component SOFATracer sampling strategy and source code analysis


The OpenTracing Semantic Specification

Ant Financial Distributed Link Tracking Component SOFATracer Data Reporting Mechanism and Source Code Analysis

Analysis of the burying mechanism of the open source distributed link tracking component SOFATracer of Ant Financial

Analysis of SOFATracer burying mechanism of distributed link components

[Analysis | SOFARPC Framework] SOFARPC Link Tracking Analysis