# OpenTelemetry v2 integration

> For the complete documentation index, see [llms.txt](https://docs.temporal.io/llms.txt).
> Any documentation page is available as raw Markdown by appending `.md` to its URL.

> Configure trace propagation, automatic tracing, custom tracing, and metrics with the Go SDK OpenTelemetry v2 plugin.

Temporal's OpenTelemetry integration lets you understand the internal state
of Temporal applications across Clients, Workflows, Activities, and Nexus
Operations by instrumenting them with
[OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/).

Temporal provides [durable execution](/temporal#durable-execution). OpenTelemetry
is the vendor-neutral framework for generating and exporting telemetry
to your backend.

The OpenTelemetry plugin is what connects the two. It propagates OpenTelemetry
context across Temporal boundaries. It can also create spans and emit metrics
for Temporal SDK operations.

> **Pre-release**

All code snippets in this guide are taken from the
[OpenTelemetry v2 sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2).
Refer to the sample for complete code.

## Install

Add the OpenTelemetry v2 integration to your Go module:

```bash
go get go.temporal.io/sdk/contrib/opentelemetry-v2@latest
```

Also add the OpenTelemetry SDK packages and the exporter or metric reader your
backend requires.

## Set up the tracer provider

A [Tracer Provider](https://opentelemetry.io/docs/concepts/signals/traces/#tracer-provider)
is the factory for Tracers. Create Temporal's replay-safe Tracer Provider and
install it as the OpenTelemetry global before you create the plugin or call
`Tracer`:

<!--SNIPSTART samples-go-opentelemetry-v2-tracer-provider {"selectedLines": ["14-22"]}-->
[opentelemetry-v2/setup.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/setup.go)
```go
// ...
	provider := temporalotel.NewReplaySafeTracerProvider(
		// WithBatcher performs exporter I/O outside the Workflow goroutine.
		sdktrace.WithBatcher(exporter),
		sdktrace.WithResource(resource.NewWithAttributes(
			semconv.SchemaURL,
			semconv.ServiceName(serviceName),
		)),
	)
	otel.SetTracerProvider(provider)
```
<!--SNIPEND-->

`NewReplaySafeTracerProvider` keeps span IDs stable across retries and replay when
instrumenting Workflows. A standard OpenTelemetry Tracer Provider is not safe for
creating spans in Workflows.

Your application owns the Tracer Provider for the life of the process. Shut it
down before exit so remaining spans can flush through the
[trace exporter](https://opentelemetry.io/docs/concepts/signals/traces/#trace-exporters).

## Add the plugin

Pass the plugin to your Temporal Client when you create it. Workers made from
that Client get the plugin automatically.

<!--SNIPSTART samples-go-opentelemetry-v2-plugin-client-->
[opentelemetry-v2/workflow-activity-propagation/worker/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/worker/main.go)
```go
plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}

c, err := client.Dial(client.Options{Plugins: []client.Plugin{plugin}})
if err != nil {
	return fmt.Errorf("unable to create client: %w", err)
}
defer c.Close()
```
<!--SNIPEND-->

By default the plugin only performs
[context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
so [Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context)
can cross Temporal boundaries.

## Add custom spans

### In Workflows

A [Tracer](https://opentelemetry.io/docs/concepts/signals/traces/#tracer)
creates spans. In Workflows, use `Tracer` instead of `otel.Tracer`. It keeps
span IDs and start times accurate across retries and replay. A standard
OpenTelemetry Tracer is not safe for creating spans in Workflows.

As in
[OpenTelemetry Go](https://opentelemetry.io/docs/languages/go/instrumentation/),
`Start` returns a context that contains the active span. Pass that
`workflow.Context` to downstream Temporal calls so later spans nest under it as
children:

<!--SNIPSTART samples-go-opentelemetry-v2-application-spans {"selectedLines": ["3-18"]}-->
[opentelemetry-v2/workflow-activity-propagation/opentelemetry.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/opentelemetry.go)
```go
// ...
func Workflow(ctx workflow.Context, name string) (string, error) {
	tracer := temporalotel.Tracer(instrumentationName)
	ctx, span := tracer.Start(ctx, "workflow-operation")
	defer span.End()

	ctx = workflow.WithActivityOptions(ctx, workflow.ActivityOptions{
		StartToCloseTimeout: 10 * time.Second,
	})

	var result string
	if err := workflow.ExecuteActivity(ctx, Activity, name).Get(ctx, &result); err != nil {
		return "", err
	}

	return result, nil
}
```
<!--SNIPEND-->

### Outside Workflows

In Clients, Activities, and other non-Workflow code, use an ordinary OpenTelemetry
[Tracer](https://opentelemetry.io/docs/concepts/signals/traces/#tracer):

<!--SNIPSTART samples-go-opentelemetry-v2-application-spans {"selectedLines": ["20-25"]}-->
[opentelemetry-v2/workflow-activity-propagation/opentelemetry.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/workflow-activity-propagation/opentelemetry.go)
```go
// ...
func Activity(ctx context.Context, name string) (string, error) {
	_, span := otel.Tracer(instrumentationName).Start(ctx, "activity-operation")
	defer span.End()

	return fmt.Sprintf("Hello, %s!", name), nil
}
```
<!--SNIPEND-->

## Enable automatic instrumentation

Set options on `PluginOptions` to create spans and emit metrics for Temporal
SDK operations:

<!--SNIPSTART samples-go-opentelemetry-v2-metrics-plugin-->
[opentelemetry-v2/automatic-instrumentation/worker/main.go](https://github.com/temporalio/samples-go/blob/main/opentelemetry-v2/automatic-instrumentation/worker/main.go)
```go
plugin, err := temporalotel.NewPlugin(temporalotel.PluginOptions{
	TracerOptions: tracing.TracerOptions{
		AddTemporalSpans: true,
	},
	MetricsHandlerOptions: &temporalotel.MetricsHandlerOptions{
		UseMonotonicCounters: true,
	},
})
if err != nil {
	return fmt.Errorf("unable to create plugin: %w", err)
}
```
<!--SNIPEND-->

### `AddTemporalSpans`

Set `AddTemporalSpans` to `true` to create spans for Temporal SDK operations
across Clients, Workflows, Activities, and Nexus Operations.

### `MetricsHandlerOptions`

Set `MetricsHandlerOptions` to a non-`nil` value to emit
[Temporal SDK metrics](/references/sdk-metrics) through OpenTelemetry.
`UseMonotonicCounters` controls whether counters are monotonic.

By default the handler uses a Meter from the global
[Meter Provider](https://opentelemetry.io/docs/concepts/signals/metrics/#meter-provider).
Set `MetricsHandlerOptions.Meter` to use a specific Meter.

## Configure context propagation

[Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
is how OpenTelemetry moves context across process boundaries: inject on the way
out, extract on the way in.

The plugin propagates
[Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context),
which keeps spans linked into one trace, and
[baggage](https://opentelemetry.io/docs/concepts/signals/baggage/): optional
key-value data that travels with the context.

### `TextMapPropagator`

The plugin injects and extracts both with a
[TextMapPropagator](https://opentelemetry.io/docs/specs/otel/context/api-propagators/#textmap-propagator).
By default that propagator supports
[W3C Trace Context](https://www.w3.org/TR/trace-context/) and
[W3C Baggage](https://www.w3.org/TR/baggage/). Set
`PluginOptions.TextMapPropagator` to override it.

### `HeaderKey`

Propagated values are stored in the Temporal header under `_tracer-data`. Set
`TracerOptions.HeaderKey` to use a different key.

### `DisableBaggage`

Set `DisableBaggage` to `true` to stop propagating baggage.

### `AllowInvalidParentSpans`

Set `AllowInvalidParentSpans` to `true` to ignore errors when extracting
[Span Context](https://opentelemetry.io/docs/concepts/signals/traces/#span-context)
from Temporal headers. Use this when migrating between tracing libraries
while Workflows or Activities are still in progress.

## Resources

- [OpenTelemetry v2 sample](https://github.com/temporalio/samples-go/tree/main/opentelemetry-v2)
- [OpenTelemetry v2 Go package](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry-v2)
- [Traces](https://opentelemetry.io/docs/concepts/signals/traces/)
- [Metrics](https://opentelemetry.io/docs/concepts/signals/metrics/)
- [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/)
- [Context propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
- [Go SDK observability guide](/develop/go/platform/observability)