Skip to content

Enhance Go applications with OTel SDK

This page contains instructions on how to set up OpenTelemetry enhancements in a Go application.

OpenTelemetry, also known simply as OTel, is an open-source observability framework that helps generate and collect telemetry data: traces, metrics, and logs in Go apps.

Enhance Go apps with the OpenTelemetry SDK

Dependencies related to the OpenTelemetry exporter and SDK must be installed first. If you are using another request router, please refer to request routing. After switching/going into the application source folder run the following command:

go get go.opentelemetry.io/otel@v1.8.0 \
  go.opentelemetry.io/otel/trace@v1.8.0 \
  go.opentelemetry.io/otel/sdk@v1.8.0 \
  go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin@v0.33.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace@v1.7.0 \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@v1.4.1

Create an initialization feature using the OpenTelemetry SDK

In order for an application to be able to send data, a feature is required to initialize OpenTelemetry. Add the following code snippet to the main.go file:

import (
    "context"
    "os"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.7.0"
    "go.uber.org/zap"
    "google.golang.org/grpc"
)

var tracerExp *otlptrace.Exporter

func retryInitTracer() func() {
    var shutdown func()
    go func() {
        for {
            // otel will reconnected and re-send spans when otel col recover. so, we don't need to re-init tracer exporter.
            if tracerExp == nil {
                shutdown = initTracer()
            } else {
                break
            }
            time.Sleep(time.Minute * 5)
        }
    }()
    return shutdown
}

func initTracer() func() {
    // temporarily set timeout to 10s
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    serviceName, ok := os.LookupEnv("OTEL_SERVICE_NAME")
    if !ok {
        serviceName = "server_name"
        os.Setenv("OTEL_SERVICE_NAME", serviceName)
    }
    otelAgentAddr, ok := os.LookupEnv("OTEL_EXPORTER_OTLP_ENDPOINT")
    if !ok {
        otelAgentAddr = "http://localhost:4317"
        os.Setenv("OTEL_EXPORTER_OTLP_ENDPOINT", otelAgentAddr)
    }
    zap.S().Infof("OTLP Trace connect to: %s with service name: %s", otelAgentAddr, serviceName)

    traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithInsecure(), otlptracegrpc.WithDialOption(grpc.WithBlock()))
    if err != nil {
        handleErr(err, "OTLP Trace gRPC Creation")
        return nil
    }

    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(traceExporter),
        sdktrace.WithSampler(sdktrace.AlwaysSample()),
    sdktrace.WithResource(resource.NewWithAttributes(semconv.SchemaURL)))

    otel.SetTracerProvider(tracerProvider)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

    tracerExp = traceExporter
    return func() {
        // Shutdown will flush any remaining spans and shut down the exporter.
        handleErr(tracerProvider.Shutdown(ctx), "failed to shutdown TracerProvider")
    }
}

func handleErr(err error, message string) {
    if err != nil {
        zap.S().Errorf("%s: %v", message, err)
    }
}

Initialize tracker in main.go

Modify the main feature to initialize the tracker in main.go. Also when your service shuts down, you should call TracerProvider.Shutdown() to ensure all spans are exported. The service makes the call as a deferred feature in the main function:

func main() {
    // start otel tracing
    if shutdown := retryInitTracer(); shutdown != nil {
            defer shutdown()
        }
    ......
}

Add OpenTelemetry Gin middleware to the application

Configure Gin to use the middleware by adding the following line to main.go:

import (
    ....
  "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
)

func main() {
    ......
    r := gin.Default()
    r.Use(otelgin.Middleware("my-app"))
    ......
}

Run the application

  • Local debugging and running

    Note: This step is only used for local development and debugging. In the production environment, the Operator will automatically complete the injection of the following environment variables.

    The above steps have completed the work of initializing the SDK. Now if you need to develop and debug locally, you need to obtain the address of insight-agent-opentelemerty-collector in the insight-system namespace in advance, assuming: insight-agent-opentelemetry-collector .insight-system.svc.cluster.local:4317.

    Therefore, you can add the following environment variables when you start the application locally:

    OTEL_SERVICE_NAME=my-golang-app OTEL_EXPORTER_OTLP_ENDPOINT=http://insight-agent-opentelemetry-collector.insight-system.svc.cluster.local:4317 go run main.go...

  • Production environment running

Please refer to the introduction of Only injecting environment variable annotations in Achieving non-intrusive enhancement of applications through Operators to add annotations to deployment yaml:

instrumentation.opentelemetry.io/inject-sdk: "insight-system/insight-opentelemetry-autoinstrumentation"

If you cannot use annotations, you can manually add the following environment variables to the deployment yaml:

······
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: 'http://insight-agent-opentelemetry-collector.insight-system.svc.cluster.local:4317'
  - name: OTEL_SERVICE_NAME
    value: "your depolyment name" # modify it.
  - name: OTEL_K8S_NAMESPACE
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: metadata.namespace
  - name: OTEL_RESOURCE_ATTRIBUTES_NODE_NAME
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: spec.nodeName
  - name: OTEL_RESOURCE_ATTRIBUTES_POD_NAME
    valueFrom:
      fieldRef:
        apiVersion: v1
        fieldPath: metadata.name
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: 'k8s.namespace.name=$(OTEL_K8S_NAMESPACE),k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME)'
······

Request Routing

OpenTelemetry gin/gonic enhancements

# Add one line to your import() stanza depending upon your request router:
middleware "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

Then inject the OpenTelemetry middleware:

router. Use(middleware. Middleware("my-app"))

OpenTelemetry gorillamux enhancements

# Add one line to your import() stanza depending upon your request router:
middleware "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"

Then inject the OpenTelemetry middleware:

router. Use(middleware. Middleware("my-app"))

gRPC enhancements

Likewise, OpenTelemetry can help you auto-detect gRPC requests. To detect any gRPC server you have, add the interceptor to the server's instantiation.

import (
  grpcotel "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)
func main() {
  [...]

    s := grpc.NewServer(
        grpc.UnaryInterceptor(grpcotel.UnaryServerInterceptor()),
        grpc.StreamInterceptor(grpcotel.StreamServerInterceptor()),
    )
}

It should be noted that if your program uses Grpc Client to call third-party services, you also need to add an interceptor to Grpc Client:

    [...]

    conn, err := grpc.Dial(addr, grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
        grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()),
    )

If not using request routing

import (
  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

Everywhere you pass http.Handler to ServeMux you will wrap the handler function. For example, the following replacements would be made:

- mux.Handle("/path", h)
+ mux.Handle("/path", otelhttp.NewHandler(h, "description of path"))
---
- mux.Handle("/path", http.HandlerFunc(f))
+ mux.Handle("/path", otelhttp.NewHandler(http.HandlerFunc(f), "description of path"))

In this way, you can ensure that each feature wrapped with othttp will automatically collect its metadata and start the corresponding trace.

Custom Span

In many cases, the middleware provided by OpenTelemetry cannot help us record more internally called features, and we need to customize Span to record

 ······
    _, span := otel.Tracer("GetServiceDetail").Start(ctx,
        "spanMetricDao.GetServiceDetail",
        trace.WithSpanKind(trace.SpanKindInternal))
    defer span.End()
  ······

Add custom properties and custom events to span

It is also possible to set a custom attribute or tag as a span. To add custom properties and events, follow these steps:

Import Tracking and Property Libraries

import (
    ...
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/trace"
)

Get the current Span from the context

span := trace.SpanFromContext(c.Request.Context())

Set properties in the current Span

span.SetAttributes(attribute. String("controller", "books"))

Add an Event to the current Span

Adding span events is done using AddEvent on the span object.

span.AddEvent(msg)

Log errors and exceptions

import "go.opentelemetry.io/otel/codes"

// Get the current span
span := trace.SpanFromContext(ctx)

// RecordError will automatically convert an error into a span even
span.RecordError(err)

// Flag this span as an error
span.SetStatus(codes.Error, "internal error")

References

For the Demo presentation, please refer to:

Comments