telemetry.md Raw History Folder

Telemetry Configuration

Export traces, metrics, and logs from Gordon via OpenTelemetry (OTLP/HTTP).

When enabled, Gordon ships all three observability signals to an OTLP-compatible backend such as OpenObserve, Jaeger, or Grafana Cloud. Existing zerolog output continues unchanged; the OTel log bridge runs alongside it.

Configuration

[telemetry]
enabled = false
endpoint = ""
auth_token = ""
traces = true
metrics = true
logs = true
trace_sample_rate = 1.0

Options

Option	Type	Default	Description
`enabled`	bool	`false`	Enable OTLP telemetry export
`endpoint`	string	`""`	OTLP HTTP endpoint URL (e.g. `http://localhost:4318/api/default`)
`auth_token`	string	`""`	Base64-encoded `user:password` for Basic auth
`traces`	bool	`true`	Export distributed traces
`metrics`	bool	`true`	Export metrics (deploy counters, container lifecycle, registry, events, backups)
`logs`	bool	`true`	Export Gordon, proxy access, and app container logs to OTLP
`trace_sample_rate`	float	`1.0`	Fraction of traces to sample (`0.0` = none, `1.0` = all)

How It Works

Gordon initializes an OTel provider at startup. When telemetry is enabled:

Traces -- Spans wrap critical operations: container deploy, image pull, registry manifest push, and proxy target resolution. The otelhttp middleware adds a span to every HTTP request on both the proxy and registry servers.
Metrics -- Gordon records custom counters and histograms for deploys, container restarts, crash loops, managed container count, registry pushes, and event bus throughput.
Logs -- Gordon exports three log families (see Logs).

When telemetry is disabled (the default), all OTel instruments are noop -- zero overhead.

Logs

With logs = true, Gordon exports:

Family	Source	`service.name`
Gordon process logs	zerowrap output, with `trace_id`/`span_id` inside traced requests	`gordon`
Proxy access logs	Every proxied request, attributed to the app service owning the host	`<app>.<service>` (`gordon` for registry and unknown hosts)
App container logs	stdout and stderr of every running app service container	`<app>.<service>`

Every app record carries these resource attributes:

Attribute	Example	Use
`service.name`	`blog.web`	One exact service; unique across apps
`service.namespace`	`blog`	All services of one app
`gordon.app`	`blog`	Filter by app
`gordon.service`	`web`	Filter by service name across apps

Record attributes:

log.type: access or container
log.iostream: stdout or stderr (container logs)
Access logs: http.request.method, http.response.status_code, server.address, url.path, client.address, request.id, and more. Severity follows the status: 5xx is ERROR, 4xx is WARN, otherwise INFO.

Loki indexes service.name and service.namespace as stream labels by default; the other attributes are structured metadata, filtered after the stream selector. Example queries in Grafana:

{service_namespace="blog"}                          # every log of the blog app
{service_name="blog.web"} | log_type="access"       # access logs of one service
{service_namespace=~".+"} | gordon_service="web"    # every "web" service, all apps

Container output is exported from the moment Gordon starts, including the first lines of every container deployed afterwards. Output emitted while Gordon is stopped is not exported. Lines longer than 64 KiB are truncated. Export is buffered and never slows the proxy or the app: on overload, the oldest records are dropped.

Per-app opt-out

App container logs are exported by default. Opt out in the app manifest, per app or per service; a service setting overrides the app setting:

[telemetry]
logs = true          # app default

[services.db.telemetry]
logs = false         # only "db" stops exporting

The opt-out covers container output only; access logs of the app's hosts are still exported.

Endpoint URL

The endpoint field accepts a full URL. Gordon parses it to extract:

Host and port for the OTLP exporter
Base path appended with /v1/traces, /v1/metrics, or /v1/logs
Scheme -- http:// disables TLS; https:// enables it

For OpenObserve, the endpoint typically includes the organization path:

http://localhost:5080/api/default

This produces export URLs like http://localhost:5080/api/default/v1/traces.

Authentication

Set auth_token to the Base64-encoded user:password string. Gordon sends it as a Basic authorization header on every OTLP export request.

For OpenObserve, copy the token from Ingestion > OTLP in the web UI.

Gordon itself is the platform, not an app, so the token does not go through gordon apps secrets. Store it with one of these methods:

Method	How
Environment variable	`GORDON_TELEMETRY_AUTH_TOKEN=<token> gordon serve`
Plain text in config	Set `auth_token = "<token>"` (restrict file permissions)

Trace Sampling

Value	Behavior
`1.0`	Sample every trace (default)
`0.5`	Sample 50% of traces
`0.0`	Drop all traces (tracing pipeline stays initialized)

At low traffic volumes (< 1000 req/s), keep the rate at 1.0. Reduce it if storage costs or export bandwidth become a concern.

Metrics Reference

Gordon exports these custom metrics (all prefixed with gordon.):

Deployments

Metric	Type	Unit	Description
`gordon.deploy.total`	Counter	-	Total deployments attempted
`gordon.deploy.duration_seconds`	Histogram	s	Time from deploy start to completion
`gordon.deploy.errors`	Counter	-	Deployments that failed

Attributes: domain, image

Container Lifecycle

Metric	Type	Unit	Description
`gordon.container.restarts`	Counter	-	Container restart count
`gordon.container.crash_loops`	Counter	-	Crash loop detections
`gordon.container.managed`	Gauge	-	Containers of running app services, read from ACTIVE app state at each export

Attributes: source (restarts only: monitor or api). gordon.container.managed has no attributes; tell instances apart with the host.name resource attribute.

Registry

Metric	Type	Unit	Description
`gordon.registry.push.total`	Counter	-	Manifest pushes received
`gordon.registry.push.bytes`	Counter	By	Total manifest bytes pushed

Attributes: name, reference

Backups

Metric	Type	Unit	Description
`gordon.backup.runs`	Counter	-	Finished backup runs (scheduled and manual)
`gordon.backup.duration_seconds`	Histogram	s	Duration of each backup run
`gordon.backup.last_success_timestamp_seconds`	Gauge	s	Start time (Unix) of the last successful backup per target
`gordon.backup.last_size_bytes`	Gauge	By	Size of the last successful backup per target
`gordon.backup.next_run_timestamp_seconds`	Gauge	s	Unix time of the next scheduled run

Attributes: kind (database or volume), app, service, target (database name or declared volume name), and result (success or failure, runs only). next_run_timestamp_seconds only carries kind.

The last success is the start time of the run, the same timestamp the stored dump or archive is named after. At startup Gordon restores it from storage for every declared target and drops targets that are no longer declared, so the gauges survive restarts and removed apps stop being reported. A run missed while Gordon is down never appears as a failure: alert on the age of the last success instead.

With the OpenTelemetry Collector Prometheus exporter, names become gordon_backup_runs_total, gordon_backup_last_success_timestamp_seconds, and so on. Example alerts:

# No successful backup of a target for more than 26 hours
time() - gordon_backup_last_success_timestamp_seconds > 26 * 3600

# A host stopped exporting its volume backup schedule (one rule per expected host and kind).
# host_name requires resource_to_telemetry_conversion on the Collector Prometheus exporter.
absent_over_time(gordon_backup_next_run_timestamp_seconds{host_name="my-host", kind="volume"}[15m])

The age alert only covers targets that succeeded at least once: a target that has never succeeded has no last-success series. Watch gordon_backup_runs_total{result="failure"} to catch those.

Event Bus

Metric	Type	Unit	Description
`gordon.events.processed`	Counter	-	Events handled successfully
`gordon.events.dropped`	Counter	-	Events dropped (channel full)

Attributes: event_type

HTTP (via otelhttp)

The otelhttp middleware automatically records standard HTTP server metrics on both the proxy and registry servers:

http.server.request.duration
http.server.request.body.size
http.server.response.body.size

Trace Spans

Span Name	Package	Description
`container.deploy`	container	Full deploy lifecycle (root span)
`container.create_and_start`	container	Container creation, start, and readiness
`container.ensure_image`	container	Image pull and tagging
`registry.put_manifest`	registry	Manifest storage and event publish
`proxy.get_target`	proxy	Proxy target resolution and caching

The otelhttp middleware adds an HTTP-level span to every request on both servers.

Examples

OpenObserve (self-hosted)

[telemetry]
enabled = true
endpoint = "http://localhost:5080/api/default"
auth_token = "YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0"
traces = true
metrics = true
logs = true
trace_sample_rate = 1.0

Grafana Cloud (OTLP)

[telemetry]
enabled = true
endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
auth_token = "<instance-id>:<api-key>"
traces = true
metrics = true
logs = true
trace_sample_rate = 0.1

Traces Only (minimal overhead)

[telemetry]
enabled = true
endpoint = "http://localhost:4318"
traces = true
metrics = false
logs = false
trace_sample_rate = 0.5

Environment Variable Override

GORDON_TELEMETRY_ENABLED=true
GORDON_TELEMETRY_ENDPOINT=http://localhost:5080/api/default
GORDON_TELEMETRY_AUTH_TOKEN=YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0
GORDON_TELEMETRY_TRACES=true
GORDON_TELEMETRY_METRICS=true
GORDON_TELEMETRY_LOGS=true
GORDON_TELEMETRY_TRACE_SAMPLE_RATE=1.0

source

1# Telemetry Configuration
2
3Export traces, metrics, and logs from Gordon via OpenTelemetry (OTLP/HTTP).
4
5When enabled, Gordon ships all three observability signals to an OTLP-compatible backend such as OpenObserve, Jaeger, or Grafana Cloud. Existing zerolog output continues unchanged; the OTel log bridge runs alongside it.
6
7## Configuration
8
9```toml
10[telemetry]
11enabled = false
12endpoint = ""
13auth_token = ""
14traces = true
15metrics = true
16logs = true
17trace_sample_rate = 1.0
18```
19
20## Options
21
22| Option | Type | Default | Description |
23|--------|------|---------|-------------|
24| `enabled` | bool | `false` | Enable OTLP telemetry export |
25| `endpoint` | string | `""` | OTLP HTTP endpoint URL (e.g. `http://localhost:4318/api/default`) |
26| `auth_token` | string | `""` | Base64-encoded `user:password` for Basic auth |
27| `traces` | bool | `true` | Export distributed traces |
28| `metrics` | bool | `true` | Export metrics (deploy counters, container lifecycle, registry, events, backups) |
29| `logs` | bool | `true` | Export Gordon, proxy access, and app container logs to OTLP |
30| `trace_sample_rate` | float | `1.0` | Fraction of traces to sample (`0.0` = none, `1.0` = all) |
31
32## How It Works
33
34Gordon initializes an OTel provider at startup. When telemetry is enabled:
35
361. **Traces** -- Spans wrap critical operations: container deploy, image pull, registry manifest push, and proxy target resolution. The `otelhttp` middleware adds a span to every HTTP request on both the proxy and registry servers.
372. **Metrics** -- Gordon records custom counters and histograms for deploys, container restarts, crash loops, managed container count, registry pushes, and event bus throughput.
383. **Logs** -- Gordon exports three log families (see [Logs](#logs)).
39
40When telemetry is disabled (the default), all OTel instruments are noop -- zero overhead.
41
42## Logs
43
44With `logs = true`, Gordon exports:
45
46| Family | Source | `service.name` |
47|--------|--------|----------------|
48| Gordon process logs | zerowrap output, with `trace_id`/`span_id` inside traced requests | `gordon` |
49| Proxy access logs | Every proxied request, attributed to the app service owning the host | `<app>.<service>` (`gordon` for registry and unknown hosts) |
50| App container logs | stdout and stderr of every running app service container | `<app>.<service>` |
51
52Every app record carries these resource attributes:
53
54| Attribute | Example | Use |
55|-----------|---------|-----|
56| `service.name` | `blog.web` | One exact service; unique across apps |
57| `service.namespace` | `blog` | All services of one app |
58| `gordon.app` | `blog` | Filter by app |
59| `gordon.service` | `web` | Filter by service name across apps |
60
61Record attributes:
62
63- `log.type`: `access` or `container`
64- `log.iostream`: `stdout` or `stderr` (container logs)
65- Access logs: `http.request.method`, `http.response.status_code`, `server.address`, `url.path`, `client.address`, `request.id`, and more. Severity follows the status: `5xx` is `ERROR`, `4xx` is `WARN`, otherwise `INFO`.
66
67Loki indexes `service.name` and `service.namespace` as stream labels by default; the other attributes are structured metadata, filtered after the stream selector. Example queries in Grafana:
68
69```text
70{service_namespace="blog"}                          # every log of the blog app
71{service_name="blog.web"} | log_type="access"       # access logs of one service
72{service_namespace=~".+"} | gordon_service="web"    # every "web" service, all apps
73```
74
75Container output is exported from the moment Gordon starts, including the first lines of every container deployed afterwards. Output emitted while Gordon is stopped is not exported. Lines longer than 64 KiB are truncated. Export is buffered and never slows the proxy or the app: on overload, the oldest records are dropped.
76
77### Per-app opt-out
78
79App container logs are exported by default. Opt out in the [app manifest](./apps.md#telemetry), per app or per service; a service setting overrides the app setting:
80
81```toml
82[telemetry]
83logs = true          # app default
84
85[services.db.telemetry]
86logs = false         # only "db" stops exporting
87```
88
89The opt-out covers container output only; access logs of the app's hosts are still exported.
90
91## Endpoint URL
92
93The `endpoint` field accepts a full URL. Gordon parses it to extract:
94
95- **Host and port** for the OTLP exporter
96- **Base path** appended with `/v1/traces`, `/v1/metrics`, or `/v1/logs`
97- **Scheme** -- `http://` disables TLS; `https://` enables it
98
99For OpenObserve, the endpoint typically includes the organization path:
100
101```text
102http://localhost:5080/api/default
103```
104
105This produces export URLs like `http://localhost:5080/api/default/v1/traces`.
106
107## Authentication
108
109Set `auth_token` to the Base64-encoded `user:password` string. Gordon sends it as a `Basic` authorization header on every OTLP export request.
110
111For OpenObserve, copy the token from **Ingestion > OTLP** in the web UI.
112
113Gordon itself is the platform, not an app, so the token does not go through `gordon apps secrets`. Store it with one of these methods:
114
115| Method | How |
116|--------|-----|
117| Environment variable | `GORDON_TELEMETRY_AUTH_TOKEN=<token> gordon serve` |
118| Plain text in config | Set `auth_token = "<token>"` (restrict file permissions) |
119
120## Trace Sampling
121
122| Value | Behavior |
123|-------|----------|
124| `1.0` | Sample every trace (default) |
125| `0.5` | Sample 50% of traces |
126| `0.0` | Drop all traces (tracing pipeline stays initialized) |
127
128At low traffic volumes (< 1000 req/s), keep the rate at `1.0`. Reduce it if storage costs or export bandwidth become a concern.
129
130## Metrics Reference
131
132Gordon exports these custom metrics (all prefixed with `gordon.`):
133
134### Deployments
135
136| Metric | Type | Unit | Description |
137|--------|------|------|-------------|
138| `gordon.deploy.total` | Counter | - | Total deployments attempted |
139| `gordon.deploy.duration_seconds` | Histogram | s | Time from deploy start to completion |
140| `gordon.deploy.errors` | Counter | - | Deployments that failed |
141
142Attributes: `domain`, `image`
143
144### Container Lifecycle
145
146| Metric | Type | Unit | Description |
147|--------|------|------|-------------|
148| `gordon.container.restarts` | Counter | - | Container restart count |
149| `gordon.container.crash_loops` | Counter | - | Crash loop detections |
150| `gordon.container.managed` | Gauge | - | Containers of running app services, read from ACTIVE app state at each export |
151
152Attributes: `source` (restarts only: `monitor` or `api`). `gordon.container.managed` has no attributes; tell instances apart with the `host.name` resource attribute.
153
154### Registry
155
156| Metric | Type | Unit | Description |
157|--------|------|------|-------------|
158| `gordon.registry.push.total` | Counter | - | Manifest pushes received |
159| `gordon.registry.push.bytes` | Counter | By | Total manifest bytes pushed |
160
161Attributes: `name`, `reference`
162
163### Backups
164
165| Metric | Type | Unit | Description |
166|--------|------|------|-------------|
167| `gordon.backup.runs` | Counter | - | Finished backup runs (scheduled and manual) |
168| `gordon.backup.duration_seconds` | Histogram | s | Duration of each backup run |
169| `gordon.backup.last_success_timestamp_seconds` | Gauge | s | Start time (Unix) of the last successful backup per target |
170| `gordon.backup.last_size_bytes` | Gauge | By | Size of the last successful backup per target |
171| `gordon.backup.next_run_timestamp_seconds` | Gauge | s | Unix time of the next scheduled run |
172
173Attributes: `kind` (`database` or `volume`), `app`, `service`, `target` (database name or declared volume name), and `result` (`success` or `failure`, runs only). `next_run_timestamp_seconds` only carries `kind`.
174
175The last success is the start time of the run, the same timestamp the stored dump or archive is named after. At startup Gordon restores it from storage for every declared target and drops targets that are no longer declared, so the gauges survive restarts and removed apps stop being reported. A run missed while Gordon is down never appears as a failure: alert on the age of the last success instead.
176
177With the OpenTelemetry Collector Prometheus exporter, names become `gordon_backup_runs_total`, `gordon_backup_last_success_timestamp_seconds`, and so on. Example alerts:
178
179```promql
180# No successful backup of a target for more than 26 hours
181time() - gordon_backup_last_success_timestamp_seconds > 26 * 3600
182
183# A host stopped exporting its volume backup schedule (one rule per expected host and kind).
184# host_name requires resource_to_telemetry_conversion on the Collector Prometheus exporter.
185absent_over_time(gordon_backup_next_run_timestamp_seconds{host_name="my-host", kind="volume"}[15m])
186```
187
188The age alert only covers targets that succeeded at least once: a target that has never succeeded has no last-success series. Watch `gordon_backup_runs_total{result="failure"}` to catch those.
189
190### Event Bus
191
192| Metric | Type | Unit | Description |
193|--------|------|------|-------------|
194| `gordon.events.processed` | Counter | - | Events handled successfully |
195| `gordon.events.dropped` | Counter | - | Events dropped (channel full) |
196
197Attributes: `event_type`
198
199### HTTP (via otelhttp)
200
201The `otelhttp` middleware automatically records standard HTTP server metrics on both the proxy and registry servers:
202
203- `http.server.request.duration`
204- `http.server.request.body.size`
205- `http.server.response.body.size`
206
207## Trace Spans
208
209| Span Name | Package | Description |
210|-----------|---------|-------------|
211| `container.deploy` | container | Full deploy lifecycle (root span) |
212| `container.create_and_start` | container | Container creation, start, and readiness |
213| `container.ensure_image` | container | Image pull and tagging |
214| `registry.put_manifest` | registry | Manifest storage and event publish |
215| `proxy.get_target` | proxy | Proxy target resolution and caching |
216
217The `otelhttp` middleware adds an HTTP-level span to every request on both servers.
218
219## Examples
220
221### OpenObserve (self-hosted)
222
223```toml
224[telemetry]
225enabled = true
226endpoint = "http://localhost:5080/api/default"
227auth_token = "YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0"
228traces = true
229metrics = true
230logs = true
231trace_sample_rate = 1.0
232```
233
234### Grafana Cloud (OTLP)
235
236```toml
237[telemetry]
238enabled = true
239endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
240auth_token = "<instance-id>:<api-key>"
241traces = true
242metrics = true
243logs = true
244trace_sample_rate = 0.1
245```
246
247### Traces Only (minimal overhead)
248
249```toml
250[telemetry]
251enabled = true
252endpoint = "http://localhost:4318"
253traces = true
254metrics = false
255logs = false
256trace_sample_rate = 0.5
257```
258
259## Environment Variable Override
260
261```bash
262GORDON_TELEMETRY_ENABLED=true
263GORDON_TELEMETRY_ENDPOINT=http://localhost:5080/api/default
264GORDON_TELEMETRY_AUTH_TOKEN=YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0
265GORDON_TELEMETRY_TRACES=true
266GORDON_TELEMETRY_METRICS=true
267GORDON_TELEMETRY_LOGS=true
268GORDON_TELEMETRY_TRACE_SAMPLE_RATE=1.0
269```
270
271## Related
272
273- [Configuration Overview](./index.md)
274- [Logging](./logging.md)
275- [Configuration Reference](./reference.md)

telemetry.md RawHistoryFolder