tell

This library include metric and trace helper functions with opentelemetry.

go get github.com/rakunlabs/tell

# if empty, metrics and trace providers and create noop provider to continue to work same as code perspective.
# default is empty
OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector:4317
# inteval duration so send new metrics to otel collector (using time.Parseduration) default 5s

Otel Environment Values

Metric and trace checking some special environment values for collector. We should fallow to opentelemetry schemas.

Our environment already give these informations not need to do anything.

Local testing more than one service with metrics, you should also provide this informations to prevent mixing.

# OTEL_SERVICE_NAME=transaction_api
OTEL_RESOURCE_ATTRIBUTES=service.name=transaction_api,service.instance.id=xyz123

# Example in swarm
OTEL_RESOURCE_ATTRIBUTES=service.name={{.Service.Name}},service.instance.id={{.Task.ID}},host.id={{.Node.ID}},host.name={{.Node.Hostname}}

Check much more details of attributes in here opentelemetry-semconv

You can also add your own values after that, these are global values for all services need to have.

Initialize

Add this configuration in your application config struct:

type Config struct {
    // Telemetry configurations
    Telemetry tell.Config
}

After that in main of program pass the telemetry config to create new collector which is connection collector and initialize telemetry and trace providers with common attributes.

collector, err := tell.New(ctx, cfg.Telemetry)
if err != nil {
    return fmt.Errorf("failed to init telemetry; %w", err)
}

defer collector.Shutdown()

Now you initialized and connected to our collector. You can send some metrics and trace data.

tell.New function also set the global values so next time when you need you can get from the global package.

These global get using by third-party libraries.

// to get tracer provider // go.opentelemetry.io/otel
otel.GetTracerProvider()
// to get meter provider // go.opentelemetry.io/otel/metric/global
global.MeterProvider()

Metric

To add some metric, use collector's MeterProvider to create a metric entry and add some values to that entry.

Hold this meters in a struct to reach easily. Check example

// to get meter provider in collector
collector.MeterProvider
// to get meter provider in global
otel.GetMeterProvider()

Counter:

successCounter, err = collector.MeterProvider.Meter("").
    Int64Counter("request_success", metric.WithDescription("number of success count"))
if err != nil {
    log.Panic().Msgf("failed to initialize successCounter; %w", err)
}

// use counter, add attributes here to give much meaning to your counter.
successCounter.Add(c.Request().Context(), 1, attribute.Key("special").String("X"))

Up/Down Counter: this is same as counter but it can also decrese.

counterUpDown, err = collector.MeterProvider.Meter("").
    Int64UpDownCounter("request_success", metric.WithDescription("number of success count"))
if err != nil {
    log.Panic().Msgf("failed to initialize successCounter; %w", err)
}

// use counter, add attributes here to give much meaning to your counter.
counterUpDown.Add(c.Request().Context(), 1, attribute.Key("special").String("X"))

Histogram:

valuehistogram, err = collector.MeterProvider.Meter("").
    Float64Histogram("request_histogram", metric.WithDescription("value histogram"))
if err != nil {
    log.Panic().Msgf("failed to initialize valuehistogram; %w", err)
}

// use histogram, add attributes here to give much meaning to your counter
valuehistogram.Record(c.Request().Context(), float64(countInt), attribute.Key("special").String("X"))

Gauge: this is special and it need to be run with async and we need to register to callback. It is like background operation.

meter := collector.MeterProvider.Meter("")

up, err := meter.Int64ObservableGauge("up", metric.WithDescription("application up status"))
if err != nil {
    log.Error().Err(err).Msg("failed to set up gauge metric")
}

regUp, err := meter.RegisterCallback(func(ctx context.Context, o metric.Observer) error {
    o.ObserveInt64(up, c.isUp) // value to observe
    return nil
}, up)

if err != nil {
    log.Error().Err(err).Msg("failed to register up gauge metric")
}

// shutdown will deregister
c.AddRegister(regUp)

Example Usage

Add to the project this package _example/telemetry/metric.go to hold the custom metrics.

package main

//
// config loaded
//

// open telemetry
collector, err := tell.New(ctx, cnf.Telemetry)
if err != nil {
    log.Fatal().Err(err).Msg("failed to init telemetry")
}

defer collector.Shutdown()

telemetry.AddGlobalAttr(attribute.Key("channel").String(cnf.Channel))
if err := telemetry.SetGlobalMeter(); err != nil {
    log.Fatal().Err(err).Msg("failed to set metric")
}

After that use your metrics

// in somewhere use your metrics
telemetry.GlobalMeter.Success.Add(ctx, 1, telemetry.GlobalAttr...)

Echo

Metric & Trace

Trace is not ready for finops, we will add details later.

go get go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho

// add otel tracing
e.Use(otelecho.Middleware(ServiceName))

Runtime

Runtime added as default but need to enable in the configuration.

go get go.opentelemetry.io/contrib/instrumentation/runtime

if err := runtime.Start(); err != nil {
    return fmt.Errorf("failed to start runtime metrics; %w", err)
}

Others

Check the open telemetry's registry page, new instruments can add here.

https://opentelemetry.io/registry/?language=go

Trace

Create a collector, also our collector will create trace provider.

collector, err := tell.New(ctx, cfg.Telemetry)

Use to trace provider to create some trace data.

Context is important to trace data, it will give you the parent-child relationship.
But also if you have cancellation in context maybe you need to use new context based on parent without cancellation.

ctx := context.WithoutCancel(c.Request().Context())

Use SetStatus before end the span, it will give you more information about the span and good to service graph.

spanCall.SetStatus(codes.Error, err.Error())

Custom Internal Trace

Start a trace with using previous context. After the start it will create new context and use that context for next trace.
If you not use context or not give previous one tracer, it will start as root and not good for view.

// set otel.Tracer("") in a value to use again and again
// set always spankind to internal for internal operations
ctx, span := otel.Tracer("").Start(c.Request().Context(), "PostCount", trace.WithSpanKind(trace.SpanKindInternal))
defer span.End()

// add extra values to your trace data
// our collector adds extra fields automatically as servicename, containerid
span.SetAttributes(attribute.Key("request.count.set").Int64(countInt))

Http Request

Create new span to measure http time but don't forget to add span kind as client. This is important for generating service-graph!

ctx, spanCall := tracer.Start(ctx, "GetTransaction", trace.WithSpanKind(trace.SpanKindClient))
defer spanCall.End()

// add context propagation
otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(request.Header))

Database

Important to have span kind as client and db.name attribute. It will help to generate service-graph with virtual nodes.

ctx, span := otel.Tracer("").Start(ctx,
    "add_product",
    trace.WithSpanKind(trace.SpanKindClient),
    trace.WithAttributes(attribute.String("db.name", "postgres")),
)
defer span.End()

Resources

https://github.com/open-telemetry/opentelemetry-go
https://opentelemetry.io/registry/