telemetry.md Raw History Folder

Telemetry Configuration

Export traces, metrics, and logs from Gordon via OpenTelemetry (OTLP/HTTP).

When enabled, Gordon ships all three observability signals to an OTLP-compatible backend such as OpenObserve, Jaeger, or Grafana Cloud. Existing zerolog output continues unchanged; the OTel log bridge runs alongside it.

Configuration

[telemetry]
enabled = false
endpoint = ""
auth_token = ""
traces = true
metrics = true
logs = true
trace_sample_rate = 1.0

Options

Option	Type	Default	Description
`enabled`	bool	`false`	Enable OTLP telemetry export
`endpoint`	string	`""`	OTLP HTTP endpoint URL (e.g. `http://localhost:4318/api/default`)
`auth_token`	string	`""`	Base64-encoded `user:password` for Basic auth
`traces`	bool	`true`	Export distributed traces
`metrics`	bool	`true`	Export metrics (deploy counters, container lifecycle, registry, events)
`logs`	bool	`true`	Export Gordon, proxy access, and app container logs to OTLP
`trace_sample_rate`	float	`1.0`	Fraction of traces to sample (`0.0` = none, `1.0` = all)

How It Works

Gordon initializes an OTel provider at startup. When telemetry is enabled:

Traces -- Spans wrap critical operations: container deploy, image pull, registry manifest push, and proxy target resolution. The otelhttp middleware adds a span to every HTTP request on both the proxy and registry servers.
Metrics -- Gordon records custom counters and histograms for deploys, container restarts, crash loops, managed container count, registry pushes, and event bus throughput.
Logs -- Gordon exports three log families (see Logs).

When telemetry is disabled (the default), all OTel instruments are noop -- zero overhead.

Logs

With logs = true, Gordon exports:

Family	Source	`service.name`
Gordon process logs	zerowrap output, with `trace_id`/`span_id` inside traced requests	`gordon`
Proxy access logs	Every proxied request, attributed to the app service owning the host	`<app>.<service>` (`gordon` for registry and unknown hosts)
App container logs	stdout and stderr of every running app service container	`<app>.<service>`

Every app record carries these resource attributes:

Attribute	Example	Use
`service.name`	`blog.web`	One exact service; unique across apps
`service.namespace`	`blog`	All services of one app
`gordon.app`	`blog`	Filter by app
`gordon.service`	`web`	Filter by service name across apps

Record attributes:

log.type: access or container
log.iostream: stdout or stderr (container logs)
Access logs: http.request.method, http.response.status_code, server.address, url.path, client.address, request.id, and more. Severity follows the status: 5xx is ERROR, 4xx is WARN, otherwise INFO.

Loki indexes service.name and service.namespace as stream labels by default; the other attributes are structured metadata, filtered after the stream selector. Example queries in Grafana:

{service_namespace="blog"}                          # every log of the blog app
{service_name="blog.web"} | log_type="access"       # access logs of one service
{service_namespace=~".+"} | gordon_service="web"    # every "web" service, all apps

Container output is exported from the moment Gordon starts, including the first lines of every container deployed afterwards. Output emitted while Gordon is stopped is not exported. Lines longer than 64 KiB are truncated. Export is buffered and never slows the proxy or the app: on overload, the oldest records are dropped.

Per-app opt-out

App container logs are exported by default. Opt out in the app manifest, per app or per service; a service setting overrides the app setting:

[telemetry]
logs = true          # app default

[services.db.telemetry]
logs = false         # only "db" stops exporting

The opt-out covers container output only; access logs of the app's hosts are still exported.

Endpoint URL

The endpoint field accepts a full URL. Gordon parses it to extract:

Host and port for the OTLP exporter
Base path appended with /v1/traces, /v1/metrics, or /v1/logs
Scheme -- http:// disables TLS; https:// enables it

For OpenObserve, the endpoint typically includes the organization path:

http://localhost:5080/api/default

This produces export URLs like http://localhost:5080/api/default/v1/traces.

Authentication

Set auth_token to the Base64-encoded user:password string. Gordon sends it as a Basic authorization header on every OTLP export request.

For OpenObserve, copy the token from Ingestion > OTLP in the web UI.

Gordon itself is the platform, not an app, so the token does not go through gordon apps secrets. Store it with one of these methods:

Method	How
Environment variable	`GORDON_TELEMETRY_AUTH_TOKEN=<token> gordon serve`
Plain text in config	Set `auth_token = "<token>"` (restrict file permissions)

Trace Sampling

Value	Behavior
`1.0`	Sample every trace (default)
`0.5`	Sample 50% of traces
`0.0`	Drop all traces (tracing pipeline stays initialized)

At low traffic volumes (< 1000 req/s), keep the rate at 1.0. Reduce it if storage costs or export bandwidth become a concern.

Metrics Reference

Gordon exports these custom metrics (all prefixed with gordon.):

Deployments

Metric	Type	Unit	Description
`gordon.deploy.total`	Counter	-	Total deployments attempted
`gordon.deploy.duration_seconds`	Histogram	s	Time from deploy start to completion
`gordon.deploy.errors`	Counter	-	Deployments that failed

Attributes: domain, image

Container Lifecycle

Metric	Type	Unit	Description
`gordon.container.restarts`	Counter	-	Container restart count
`gordon.container.crash_loops`	Counter	-	Crash loop detections
`gordon.container.managed`	Gauge	-	Containers of running app services, read from ACTIVE app state at each export

Attributes: source (restarts only: monitor or api). gordon.container.managed has no attributes; tell instances apart with the host.name resource attribute.

Registry

Metric	Type	Unit	Description
`gordon.registry.push.total`	Counter	-	Manifest pushes received
`gordon.registry.push.bytes`	Counter	By	Total manifest bytes pushed

Attributes: name, reference

Event Bus

Metric	Type	Unit	Description
`gordon.events.processed`	Counter	-	Events handled successfully
`gordon.events.dropped`	Counter	-	Events dropped (channel full)

Attributes: event_type

HTTP (via otelhttp)

The otelhttp middleware automatically records standard HTTP server metrics on both the proxy and registry servers:

http.server.request.duration
http.server.request.body.size
http.server.response.body.size

Trace Spans

Span Name	Package	Description
`container.deploy`	container	Full deploy lifecycle (root span)
`container.create_and_start`	container	Container creation, start, and readiness
`container.ensure_image`	container	Image pull and tagging
`registry.put_manifest`	registry	Manifest storage and event publish
`proxy.get_target`	proxy	Proxy target resolution and caching

The otelhttp middleware adds an HTTP-level span to every request on both servers.

Examples

OpenObserve (self-hosted)

[telemetry]
enabled = true
endpoint = "http://localhost:5080/api/default"
auth_token = "YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0"
traces = true
metrics = true
logs = true
trace_sample_rate = 1.0

Grafana Cloud (OTLP)

[telemetry]
enabled = true
endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
auth_token = "<instance-id>:<api-key>"
traces = true
metrics = true
logs = true
trace_sample_rate = 0.1

Traces Only (minimal overhead)

[telemetry]
enabled = true
endpoint = "http://localhost:4318"
traces = true
metrics = false
logs = false
trace_sample_rate = 0.5

Environment Variable Override

GORDON_TELEMETRY_ENABLED=true
GORDON_TELEMETRY_ENDPOINT=http://localhost:5080/api/default
GORDON_TELEMETRY_AUTH_TOKEN=YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0
GORDON_TELEMETRY_TRACES=true
GORDON_TELEMETRY_METRICS=true
GORDON_TELEMETRY_LOGS=true
GORDON_TELEMETRY_TRACE_SAMPLE_RATE=1.0

source

1# Telemetry Configuration
2
3Export traces, metrics, and logs from Gordon via OpenTelemetry (OTLP/HTTP).
4
5When enabled, Gordon ships all three observability signals to an OTLP-compatible backend such as OpenObserve, Jaeger, or Grafana Cloud. Existing zerolog output continues unchanged; the OTel log bridge runs alongside it.
6
7## Configuration
8
9```toml
10[telemetry]
11enabled = false
12endpoint = ""
13auth_token = ""
14traces = true
15metrics = true
16logs = true
17trace_sample_rate = 1.0
18```
19
20## Options
21
22| Option | Type | Default | Description |
23|--------|------|---------|-------------|
24| `enabled` | bool | `false` | Enable OTLP telemetry export |
25| `endpoint` | string | `""` | OTLP HTTP endpoint URL (e.g. `http://localhost:4318/api/default`) |
26| `auth_token` | string | `""` | Base64-encoded `user:password` for Basic auth |
27| `traces` | bool | `true` | Export distributed traces |
28| `metrics` | bool | `true` | Export metrics (deploy counters, container lifecycle, registry, events) |
29| `logs` | bool | `true` | Export Gordon, proxy access, and app container logs to OTLP |
30| `trace_sample_rate` | float | `1.0` | Fraction of traces to sample (`0.0` = none, `1.0` = all) |
31
32## How It Works
33
34Gordon initializes an OTel provider at startup. When telemetry is enabled:
35
361. **Traces** -- Spans wrap critical operations: container deploy, image pull, registry manifest push, and proxy target resolution. The `otelhttp` middleware adds a span to every HTTP request on both the proxy and registry servers.
372. **Metrics** -- Gordon records custom counters and histograms for deploys, container restarts, crash loops, managed container count, registry pushes, and event bus throughput.
383. **Logs** -- Gordon exports three log families (see [Logs](#logs)).
39
40When telemetry is disabled (the default), all OTel instruments are noop -- zero overhead.
41
42## Logs
43
44With `logs = true`, Gordon exports:
45
46| Family | Source | `service.name` |
47|--------|--------|----------------|
48| Gordon process logs | zerowrap output, with `trace_id`/`span_id` inside traced requests | `gordon` |
49| Proxy access logs | Every proxied request, attributed to the app service owning the host | `<app>.<service>` (`gordon` for registry and unknown hosts) |
50| App container logs | stdout and stderr of every running app service container | `<app>.<service>` |
51
52Every app record carries these resource attributes:
53
54| Attribute | Example | Use |
55|-----------|---------|-----|
56| `service.name` | `blog.web` | One exact service; unique across apps |
57| `service.namespace` | `blog` | All services of one app |
58| `gordon.app` | `blog` | Filter by app |
59| `gordon.service` | `web` | Filter by service name across apps |
60
61Record attributes:
62
63- `log.type`: `access` or `container`
64- `log.iostream`: `stdout` or `stderr` (container logs)
65- Access logs: `http.request.method`, `http.response.status_code`, `server.address`, `url.path`, `client.address`, `request.id`, and more. Severity follows the status: `5xx` is `ERROR`, `4xx` is `WARN`, otherwise `INFO`.
66
67Loki indexes `service.name` and `service.namespace` as stream labels by default; the other attributes are structured metadata, filtered after the stream selector. Example queries in Grafana:
68
69```text
70{service_namespace="blog"}                          # every log of the blog app
71{service_name="blog.web"} | log_type="access"       # access logs of one service
72{service_namespace=~".+"} | gordon_service="web"    # every "web" service, all apps
73```
74
75Container output is exported from the moment Gordon starts, including the first lines of every container deployed afterwards. Output emitted while Gordon is stopped is not exported. Lines longer than 64 KiB are truncated. Export is buffered and never slows the proxy or the app: on overload, the oldest records are dropped.
76
77### Per-app opt-out
78
79App container logs are exported by default. Opt out in the [app manifest](./apps.md#telemetry), per app or per service; a service setting overrides the app setting:
80
81```toml
82[telemetry]
83logs = true          # app default
84
85[services.db.telemetry]
86logs = false         # only "db" stops exporting
87```
88
89The opt-out covers container output only; access logs of the app's hosts are still exported.
90
91## Endpoint URL
92
93The `endpoint` field accepts a full URL. Gordon parses it to extract:
94
95- **Host and port** for the OTLP exporter
96- **Base path** appended with `/v1/traces`, `/v1/metrics`, or `/v1/logs`
97- **Scheme** -- `http://` disables TLS; `https://` enables it
98
99For OpenObserve, the endpoint typically includes the organization path:
100
101```text
102http://localhost:5080/api/default
103```
104
105This produces export URLs like `http://localhost:5080/api/default/v1/traces`.
106
107## Authentication
108
109Set `auth_token` to the Base64-encoded `user:password` string. Gordon sends it as a `Basic` authorization header on every OTLP export request.
110
111For OpenObserve, copy the token from **Ingestion > OTLP** in the web UI.
112
113Gordon itself is the platform, not an app, so the token does not go through `gordon apps secrets`. Store it with one of these methods:
114
115| Method | How |
116|--------|-----|
117| Environment variable | `GORDON_TELEMETRY_AUTH_TOKEN=<token> gordon serve` |
118| Plain text in config | Set `auth_token = "<token>"` (restrict file permissions) |
119
120## Trace Sampling
121
122| Value | Behavior |
123|-------|----------|
124| `1.0` | Sample every trace (default) |
125| `0.5` | Sample 50% of traces |
126| `0.0` | Drop all traces (tracing pipeline stays initialized) |
127
128At low traffic volumes (< 1000 req/s), keep the rate at `1.0`. Reduce it if storage costs or export bandwidth become a concern.
129
130## Metrics Reference
131
132Gordon exports these custom metrics (all prefixed with `gordon.`):
133
134### Deployments
135
136| Metric | Type | Unit | Description |
137|--------|------|------|-------------|
138| `gordon.deploy.total` | Counter | - | Total deployments attempted |
139| `gordon.deploy.duration_seconds` | Histogram | s | Time from deploy start to completion |
140| `gordon.deploy.errors` | Counter | - | Deployments that failed |
141
142Attributes: `domain`, `image`
143
144### Container Lifecycle
145
146| Metric | Type | Unit | Description |
147|--------|------|------|-------------|
148| `gordon.container.restarts` | Counter | - | Container restart count |
149| `gordon.container.crash_loops` | Counter | - | Crash loop detections |
150| `gordon.container.managed` | Gauge | - | Containers of running app services, read from ACTIVE app state at each export |
151
152Attributes: `source` (restarts only: `monitor` or `api`). `gordon.container.managed` has no attributes; tell instances apart with the `host.name` resource attribute.
153
154### Registry
155
156| Metric | Type | Unit | Description |
157|--------|------|------|-------------|
158| `gordon.registry.push.total` | Counter | - | Manifest pushes received |
159| `gordon.registry.push.bytes` | Counter | By | Total manifest bytes pushed |
160
161Attributes: `name`, `reference`
162
163### Event Bus
164
165| Metric | Type | Unit | Description |
166|--------|------|------|-------------|
167| `gordon.events.processed` | Counter | - | Events handled successfully |
168| `gordon.events.dropped` | Counter | - | Events dropped (channel full) |
169
170Attributes: `event_type`
171
172### HTTP (via otelhttp)
173
174The `otelhttp` middleware automatically records standard HTTP server metrics on both the proxy and registry servers:
175
176- `http.server.request.duration`
177- `http.server.request.body.size`
178- `http.server.response.body.size`
179
180## Trace Spans
181
182| Span Name | Package | Description |
183|-----------|---------|-------------|
184| `container.deploy` | container | Full deploy lifecycle (root span) |
185| `container.create_and_start` | container | Container creation, start, and readiness |
186| `container.ensure_image` | container | Image pull and tagging |
187| `registry.put_manifest` | registry | Manifest storage and event publish |
188| `proxy.get_target` | proxy | Proxy target resolution and caching |
189
190The `otelhttp` middleware adds an HTTP-level span to every request on both servers.
191
192## Examples
193
194### OpenObserve (self-hosted)
195
196```toml
197[telemetry]
198enabled = true
199endpoint = "http://localhost:5080/api/default"
200auth_token = "YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0"
201traces = true
202metrics = true
203logs = true
204trace_sample_rate = 1.0
205```
206
207### Grafana Cloud (OTLP)
208
209```toml
210[telemetry]
211enabled = true
212endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
213auth_token = "<instance-id>:<api-key>"
214traces = true
215metrics = true
216logs = true
217trace_sample_rate = 0.1
218```
219
220### Traces Only (minimal overhead)
221
222```toml
223[telemetry]
224enabled = true
225endpoint = "http://localhost:4318"
226traces = true
227metrics = false
228logs = false
229trace_sample_rate = 0.5
230```
231
232## Environment Variable Override
233
234```bash
235GORDON_TELEMETRY_ENABLED=true
236GORDON_TELEMETRY_ENDPOINT=http://localhost:5080/api/default
237GORDON_TELEMETRY_AUTH_TOKEN=YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0
238GORDON_TELEMETRY_TRACES=true
239GORDON_TELEMETRY_METRICS=true
240GORDON_TELEMETRY_LOGS=true
241GORDON_TELEMETRY_TRACE_SAMPLE_RATE=1.0
242```
243
244## Related
245
246- [Configuration Overview](./index.md)
247- [Logging](./logging.md)
248- [Configuration Reference](./reference.md)

telemetry.md RawHistoryFolder