# OpenTelemetry Agent SDK sends traces, metrics, and logs to an OTLP collector when you set the `OTEL_*` variables. Record your own with `@opentelemetry/api`. ```bash export OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.com export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20$OTEL_TOKEN" export OTEL_SERVICE_NAME=incident-agent agent-sdk serve --dir . --dev ``` - The default protocol is `http/protobuf`. `http/json` also works, and `grpc` logs a warning and falls back to `http/protobuf`. - Metrics use delta temporality. Grafana Cloud and Prometheus reject delta, so set `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative` when you export to either one. - If you don't set an endpoint, Agent SDK doesn't export anything. - You can create a meter, tracer, or logger at the top of `bot/agent.ts`, and it will use Agent SDK's providers. ## Variables | Variable | Effect | | --------------------------------------------------- | ------------------------------------------------------------------------ | | `OTEL_EXPORTER_OTLP_ENDPOINT` | Base URL. Appends `/v1/traces`, `/v1/metrics`, `/v1/logs`. | | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Traces URL, used as written. Overrides the base URL. | | `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | Metrics URL, used as written. Overrides the base URL. | | `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Logs URL, used as written. Overrides the base URL. | | `OTEL_EXPORTER_OTLP_HEADERS` | Sent on every export. URL-encode values. Reread on each export. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` (default) or `http/json`. | | `OTEL_SERVICE_NAME` | `service.name`. Defaults to the agent name, then `unknown_service:node`. | | `OTEL_RESOURCE_ATTRIBUTES` | `service.namespace`, `service.version`, `deployment.environment.name`. | | `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` | `delta` (default), `cumulative`, or `lowmemory`. | | `OTEL_LOGS_EXPORTER` | `otlp` exports logs. Unset, `none`, and `console` do not. | | `OTEL_SDK_DISABLED` | `true` turns export off. | | `OTEL_TRACES_EXPORTER` | `none` turns traces off. | | `OTEL_METRICS_EXPORTER` | `none` turns metrics off. | | `OTEL_LOG_USER_PROMPTS` | `1` adds prompt text to log records. | | `OTEL_LOG_TOOL_CONTENT` | `1` adds tool payloads to log records. | ## Hosted agents ```ts export default defineAgent({ hosting: { egressDomains: ["otlp-gateway-prod-us-east-0.grafana.net"], secretNames: [ "OTEL_EXPORTER_OTLP_ENDPOINT", "OTEL_EXPORTER_OTLP_HEADERS", "OTEL_SERVICE_NAME", ], }, }); ``` `service.name` is fixed by the first export, so redeploy to change it. ## Record and tag ```ts import { metrics, trace } from "@opentelemetry/api"; import { logs } from "@opentelemetry/api-logs"; import { currentConversationId } from "@grok/agent-sdk/otel"; const tracer = trace.getTracer("incident-agent"); const lookups = metrics.getMeter("incident-agent").createCounter("incident.lookups"); const logger = logs.getLogger("incident-agent"); lookups.add(1, { "cursor.conversation.id": currentConversationId() }); ``` Spans started during a turn get `cursor.conversation.id`, and a metric gets it only when you set it. To tag a whole turn, call `ctx.telemetry` from a tool. Spans in the turn get the trace keys, Agent SDK's log records get the log keys, and `cursor.token.usage` gets the metric keys. Outside a tool, use `telemetry` from `@grok/agent-sdk/telemetry`. ```ts async execute({ key }, ctx) { ctx.telemetry.setTraceAttributes({ "work.item": key }); ctx.telemetry.setMetricAttributes({ "work.item": key }); ctx.telemetry.setLogAttributes({ "work.item": key }); return await jira.getIssue(key); } ``` Grafana renames a `.` in a metric key to `_`. ```promql sum by (work_item) (cursor_token_usage_total) ``` - Each distinct metric value is a new series, so never use a per-turn ID, a secret, an email, or a raw user ID. - A session holds up to 32 keys of each kind, with keys up to 128 characters and values up to 256. ## Limits - Agent SDK doesn't support custom span processors, samplers, views, your own OpenTelemetry SDK, or library auto-instrumentation. It reads `OTEL_TRACES_SAMPLER` and `OTEL_METRIC_EXPORT_INTERVAL` once at startup, so restart the process after you change either one. - Import Agent SDK before any other OpenTelemetry setup. Otherwise the turn's tags may not reach your spans. ## Migrating from serve(\{ otel }) Delete `serve({ otel })`, `bot/otel.ts`, `bot/instrumentation.ts`, and any `defineOtel` call. Set these variables instead. | Old option | Set this instead | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `serviceName` | `OTEL_SERVICE_NAME` | | `serviceVersion` | `service.version` in `OTEL_RESOURCE_ATTRIBUTES` | | `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES`, comma-separated, URL-encoded values | | `exporters[].url` | `OTEL_EXPORTER_OTLP_ENDPOINT` (appends `/v1/traces`, `/v1/metrics`, `/v1/logs`) | | `exporters[].url` with `signals` | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`, or `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`, used as written | | `exporters[].headers` | `OTEL_EXPORTER_OTLP_HEADERS`, comma-separated, URL-encoded values | | `exporters[].protocol` | `OTEL_EXPORTER_OTLP_PROTOCOL`, or `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`, `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL`, `OTEL_EXPORTER_OTLP_LOGS_PROTOCOL` | | `exporters[].signals` | The per-signal URL. `OTEL_TRACES_EXPORTER=none`, `OTEL_METRICS_EXPORTER=none`, or `OTEL_LOGS_EXPORTER=none` stops that signal. | | `traces: false` | `OTEL_TRACES_EXPORTER=none` | | `traces: true` | Leave `OTEL_TRACES_EXPORTER` unset. Traces start when an endpoint is set. | | `traces.sampleRatio` | `OTEL_TRACES_SAMPLER=parentbased_traceidratio` and `OTEL_TRACES_SAMPLER_ARG`, before process start | | `metrics: false` | `OTEL_METRICS_EXPORTER=none` | | `metrics: true` | Leave `OTEL_METRICS_EXPORTER` unset. Metrics start when a metrics URL is set. | | `metrics.exportIntervalMillis` | `OTEL_METRIC_EXPORT_INTERVAL`, milliseconds, before process start | | `logs: false` | Unset `OTEL_LOGS_EXPORTER`, or set `none` | | `logs: true` | `OTEL_LOGS_EXPORTER=otlp` | | `logs.userPrompts` | `OTEL_LOGS_EXPORTER=otlp` and `OTEL_LOG_USER_PROMPTS=1` | | `logs.toolContent` | `OTEL_LOGS_EXPORTER=otlp` and `OTEL_LOG_TOOL_CONTENT=1` | | `otel: false` | No endpoint, or `OTEL_SDK_DISABLED=true` | Set the hosted values after deploy: ```bash agent-sdk secret set my-agent OTEL_SERVICE_NAME OTEL_EXPORTER_OTLP_ENDPOINT OTEL_EXPORTER_OTLP_HEADERS ``` --- ## Sitemap [Overview of all docs pages](/llms.txt)