1# Telemetry Configuration
2
3Export traces, metrics, and logs from Gordon via OpenTelemetry (OTLP/HTTP).
4
5When enabled, Gordon ships all three observability signals to an OTLP-compatible backend such as OpenObserve, Jaeger, or Grafana Cloud. Existing zerolog output continues unchanged; the OTel log bridge runs alongside it.
6
7## Configuration
8
9```toml
10[telemetry]
11enabled = false
12endpoint = ""
13auth_token = ""
14traces = true
15metrics = true
16logs = true
17trace_sample_rate = 1.0
18```
19
20## Options
21
22| Option | Type | Default | Description |
23|--------|------|---------|-------------|
24| `enabled` | bool | `false` | Enable OTLP telemetry export |
25| `endpoint` | string | `""` | OTLP HTTP endpoint URL (e.g. `http://localhost:4318/api/default`) |
26| `auth_token` | string | `""` | Base64-encoded `user:password` for Basic auth |
27| `traces` | bool | `true` | Export distributed traces |
28| `metrics` | bool | `true` | Export metrics (deploy counters, container lifecycle, registry, events) |
29| `logs` | bool | `true` | Bridge zerolog output to OTLP logs |
30| `trace_sample_rate` | float | `1.0` | Fraction of traces to sample (`0.0` = none, `1.0` = all) |
31
32## How It Works
33
34Gordon initializes an OTel provider at startup. When telemetry is enabled:
35
361. **Traces** -- Spans wrap critical operations: container deploy, image pull, registry manifest push, and proxy target resolution. The `otelhttp` middleware adds a span to every HTTP request on both the proxy and registry servers.
372. **Metrics** -- Gordon records custom counters and histograms for deploys, container restarts, crash loops, managed container count, registry pushes, and event bus throughput.
383. **Logs** -- A zerowrap/otel hook bridges all structured log output to the OTLP log pipeline. Every log line automatically carries `trace_id` and `span_id` when emitted inside a traced request.
39
40When telemetry is disabled (the default), all OTel instruments are noop -- zero overhead.
41
42## Endpoint URL
43
44The `endpoint` field accepts a full URL. Gordon parses it to extract:
45
46- **Host and port** for the OTLP exporter
47- **Base path** appended with `/v1/traces`, `/v1/metrics`, or `/v1/logs`
48- **Scheme** -- `http://` disables TLS; `https://` enables it
49
50For OpenObserve, the endpoint typically includes the organization path:
51
52```text
53http://localhost:5080/api/default
54```
55
56This produces export URLs like `http://localhost:5080/api/default/v1/traces`.
57
58## Authentication
59
60Set `auth_token` to the Base64-encoded `user:password` string. Gordon sends it as a `Basic` authorization header on every OTLP export request.
61
62For OpenObserve, copy the token from **Ingestion > OTLP** in the web UI.
63
64Since Gordon itself is the platform (not a managed container), it does not use `gordon secrets set`. Store the token with one of these methods:
65
66| Method | How |
67|--------|-----|
68| Environment variable | `GORDON_TELEMETRY_AUTH_TOKEN=<token> gordon serve` |
69| Plain text in config | Set `auth_token = "<token>"` (restrict file permissions) |
70
71## Trace Sampling
72
73| Value | Behavior |
74|-------|----------|
75| `1.0` | Sample every trace (default) |
76| `0.5` | Sample 50% of traces |
77| `0.0` | Drop all traces (tracing pipeline stays initialized) |
78
79At low traffic volumes (< 1000 req/s), keep the rate at `1.0`. Reduce it if storage costs or export bandwidth become a concern.
80
81## Metrics Reference
82
83Gordon exports these custom metrics (all prefixed with `gordon.`):
84
85### Deployments
86
87| Metric | Type | Unit | Description |
88|--------|------|------|-------------|
89| `gordon.deploy.total` | Counter | - | Total deployments attempted |
90| `gordon.deploy.duration_seconds` | Histogram | s | Time from deploy start to completion |
91| `gordon.deploy.errors` | Counter | - | Deployments that failed |
92
93Attributes: `domain`, `image`
94
95### Container Lifecycle
96
97| Metric | Type | Unit | Description |
98|--------|------|------|-------------|
99| `gordon.container.restarts` | Counter | - | Container restart count |
100| `gordon.container.crash_loops` | Counter | - | Crash loop detections |
101| `gordon.container.managed` | UpDownCounter | - | Currently tracked containers |
102
103Attributes: `source` (restarts only: `monitor` or `api`); `gordon.container.managed` is a global gauge with no attributes
104
105### Registry
106
107| Metric | Type | Unit | Description |
108|--------|------|------|-------------|
109| `gordon.registry.push.total` | Counter | - | Manifest pushes received |
110| `gordon.registry.push.bytes` | Counter | By | Total manifest bytes pushed |
111
112Attributes: `name`, `reference`
113
114### Event Bus
115
116| Metric | Type | Unit | Description |
117|--------|------|------|-------------|
118| `gordon.events.processed` | Counter | - | Events handled successfully |
119| `gordon.events.dropped` | Counter | - | Events dropped (channel full) |
120
121Attributes: `event_type`
122
123### HTTP (via otelhttp)
124
125The `otelhttp` middleware automatically records standard HTTP server metrics on both the proxy and registry servers:
126
127- `http.server.request.duration`
128- `http.server.request.body.size`
129- `http.server.response.body.size`
130
131## Trace Spans
132
133| Span Name | Package | Description |
134|-----------|---------|-------------|
135| `container.deploy` | container | Full deploy lifecycle (root span) |
136| `container.create_and_start` | container | Container creation, start, and readiness |
137| `container.ensure_image` | container | Image pull and tagging |
138| `registry.put_manifest` | registry | Manifest storage and event publish |
139| `proxy.get_target` | proxy | Proxy target resolution and caching |
140
141The `otelhttp` middleware adds an HTTP-level span to every request on both servers.
142
143## Examples
144
145### OpenObserve (self-hosted)
146
147```toml
148[telemetry]
149enabled = true
150endpoint = "http://localhost:5080/api/default"
151auth_token = "YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0"
152traces = true
153metrics = true
154logs = true
155trace_sample_rate = 1.0
156```
157
158### Grafana Cloud (OTLP)
159
160```toml
161[telemetry]
162enabled = true
163endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
164auth_token = "<instance-id>:<api-key>"
165traces = true
166metrics = true
167logs = true
168trace_sample_rate = 0.1
169```
170
171### Traces Only (minimal overhead)
172
173```toml
174[telemetry]
175enabled = true
176endpoint = "http://localhost:4318"
177traces = true
178metrics = false
179logs = false
180trace_sample_rate = 0.5
181```
182
183## Environment Variable Override
184
185```bash
186GORDON_TELEMETRY_ENABLED=true
187GORDON_TELEMETRY_ENDPOINT=http://localhost:5080/api/default
188GORDON_TELEMETRY_AUTH_TOKEN=YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0
189GORDON_TELEMETRY_TRACES=true
190GORDON_TELEMETRY_METRICS=true
191GORDON_TELEMETRY_LOGS=true
192GORDON_TELEMETRY_TRACE_SAMPLE_RATE=1.0
193```
194
195## Related
196
197- [Configuration Overview](./index.md)
198- [Logging](./logging.md)
199- [Configuration Reference](./reference.md)