detached febe319

feat(telemetry): add OpenTelemetry instrumentation for traces, metrics, and logs

Wire OTel SDK with OTLP/HTTP exporters into Gordon's bootstrap. Add
telemetry provider, 10 custom metrics (deploy, container lifecycle,
registry push, event bus), trace spans on critical paths (deploy,
image pull, manifest push, proxy resolution), otelhttp middleware on
both proxy and registry servers, and a zerolog-to-OTel log bridge.

Includes [telemetry] config section with Viper defaults, docs for
all options/metrics/spans, and updates to config reference pages.
commit
febe319ce025776d9ac281fa98c6e44ebf188565
author
brice · · 2026-02-10 07:14 UTC
parent
4bbf7c1
signature
unsigned
15 files · +789 -9
Unified
UnifiedSplit
M docs/config/index.md +19 -0
@@ -82,6 +82,16 @@ max_size = 100
8282 max_backups = 3
8383 max_age = 28
8484
85+# Telemetry (OpenTelemetry)
86+[telemetry]
87+enabled = true # Enable OTLP export (default: false)
88+endpoint = "http://localhost:5080/api/default" # OTLP HTTP endpoint
89+auth_token = "" # Base64 user:password for Basic auth
90+traces = true # Export traces
91+metrics = true # Export metrics
92+logs = true # Bridge zerolog to OTLP logs
93+trace_sample_rate = 1.0 # 0.0 = none, 1.0 = all
94+
8595 # Environment variables
8696 [env]
8797 dir = "~/.gordon/env" # Default location
@@ -144,6 +154,7 @@ keep_last = 3
144154 | `[api.rate_limit]` | Rate limiting configuration | [Rate Limiting](./rate-limiting.md) |
145155 | `[deploy]` | Deployment behavior | [Deploy](./deploy.md) |
146156 | `[logging]` | Logging configuration | [Logging](./logging.md) |
157+| `[telemetry]` | OpenTelemetry observability export | [Telemetry](./telemetry.md) |
147158 | `[env]` | Environment variable settings | [Environment](./env.md) |
148159 | `[volumes]` | Volume management | [Volumes](./volumes.md) |
149160 | `[network_isolation]` | Network isolation settings | [Network Isolation](./network-isolation.md) |
@@ -194,6 +205,13 @@ keep_last = 3
194205 | `images.prune.enabled` | `false` |
195206 | `images.prune.schedule` | `"daily"` |
196207 | `images.prune.keep_last` | `3` |
208+| `telemetry.enabled` | `false` |
209+| `telemetry.endpoint` | `""` |
210+| `telemetry.auth_token` | `""` |
211+| `telemetry.traces` | `true` |
212+| `telemetry.metrics` | `true` |
213+| `telemetry.logs` | `true` |
214+| `telemetry.trace_sample_rate` | `1.0` |
197215
198216 When `auth.enabled=false`, Gordon runs in local-only mode: `/admin/*` is disabled and `/v2/*` is loopback-only.
199217
@@ -236,5 +254,6 @@ Pattern: `GORDON_SECTION_KEY` (uppercase, underscores instead of dots)
236254 - [Routes Configuration](./routes.md)
237255 - [External Routes](./external-routes.md)
238256 - [Authentication](./auth.md)
257+- [Telemetry](./telemetry.md)
239258 - [Backups](./backups.md)
240259 - [Images](./images.md)
@@ -82,6 +82,16 @@ max_size = 100
82 max_backups = 382 max_backups = 3
83 max_age = 2883 max_age = 28
84 84
85+# Telemetry (OpenTelemetry)
86+[telemetry]
87+enabled = true # Enable OTLP export (default: false)
88+endpoint = "http://localhost:5080/api/default" # OTLP HTTP endpoint
89+auth_token = "" # Base64 user:password for Basic auth
90+traces = true # Export traces
91+metrics = true # Export metrics
92+logs = true # Bridge zerolog to OTLP logs
93+trace_sample_rate = 1.0 # 0.0 = none, 1.0 = all
94+
85 # Environment variables95 # Environment variables
86 [env]96 [env]
87 dir = "~/.gordon/env" # Default location97 dir = "~/.gordon/env" # Default location
@@ -144,6 +154,7 @@ keep_last = 3
144 | `[api.rate_limit]` | Rate limiting configuration | [Rate Limiting](./rate-limiting.md) |154 | `[api.rate_limit]` | Rate limiting configuration | [Rate Limiting](./rate-limiting.md) |
145 | `[deploy]` | Deployment behavior | [Deploy](./deploy.md) |155 | `[deploy]` | Deployment behavior | [Deploy](./deploy.md) |
146 | `[logging]` | Logging configuration | [Logging](./logging.md) |156 | `[logging]` | Logging configuration | [Logging](./logging.md) |
157+| `[telemetry]` | OpenTelemetry observability export | [Telemetry](./telemetry.md) |
147 | `[env]` | Environment variable settings | [Environment](./env.md) |158 | `[env]` | Environment variable settings | [Environment](./env.md) |
148 | `[volumes]` | Volume management | [Volumes](./volumes.md) |159 | `[volumes]` | Volume management | [Volumes](./volumes.md) |
149 | `[network_isolation]` | Network isolation settings | [Network Isolation](./network-isolation.md) |160 | `[network_isolation]` | Network isolation settings | [Network Isolation](./network-isolation.md) |
@@ -194,6 +205,13 @@ keep_last = 3
194 | `images.prune.enabled` | `false` |205 | `images.prune.enabled` | `false` |
195 | `images.prune.schedule` | `"daily"` |206 | `images.prune.schedule` | `"daily"` |
196 | `images.prune.keep_last` | `3` |207 | `images.prune.keep_last` | `3` |
208+| `telemetry.enabled` | `false` |
209+| `telemetry.endpoint` | `""` |
210+| `telemetry.auth_token` | `""` |
211+| `telemetry.traces` | `true` |
212+| `telemetry.metrics` | `true` |
213+| `telemetry.logs` | `true` |
214+| `telemetry.trace_sample_rate` | `1.0` |
197 215
198 When `auth.enabled=false`, Gordon runs in local-only mode: `/admin/*` is disabled and `/v2/*` is loopback-only.216 When `auth.enabled=false`, Gordon runs in local-only mode: `/admin/*` is disabled and `/v2/*` is loopback-only.
199 217
@@ -236,5 +254,6 @@ Pattern: `GORDON_SECTION_KEY` (uppercase, underscores instead of dots)
236 - [Routes Configuration](./routes.md)254 - [Routes Configuration](./routes.md)
237 - [External Routes](./external-routes.md)255 - [External Routes](./external-routes.md)
238 - [Authentication](./auth.md)256 - [Authentication](./auth.md)
257+- [Telemetry](./telemetry.md)
239 - [Backups](./backups.md)258 - [Backups](./backups.md)
240 - [Images](./images.md)259 - [Images](./images.md)
M docs/config/reference.md +20 -0
@@ -63,6 +63,18 @@ max_size = 100 # Max size in MB before rotation
6363 max_backups = 3 # Number of old files to keep
6464 max_age = 28 # Days to keep old files
6565
66+# =============================================================================
67+# TELEMETRY (OpenTelemetry)
68+# =============================================================================
69+[telemetry]
70+enabled = false # Enable OTLP telemetry export
71+endpoint = "" # OTLP HTTP endpoint URL
72+auth_token = "" # Base64-encoded user:password for Basic auth
73+traces = true # Export distributed traces
74+metrics = true # Export metrics
75+logs = true # Bridge zerolog output to OTLP logs
76+trace_sample_rate = 1.0 # Fraction of traces to sample (0.0–1.0)
77+
6678 # =============================================================================
6779 # ENVIRONMENT
6880 # =============================================================================
@@ -186,6 +198,13 @@ keep_last = 3 # Keep N newest tags per repository
186198 | `logging.container_logs.max_size` | `100` | 100 MB |
187199 | `logging.container_logs.max_backups` | `3` | Keep 3 old files |
188200 | `logging.container_logs.max_age` | `28` | 28 days |
201+| `telemetry.enabled` | `false` | Enable OTLP telemetry export |
202+| `telemetry.endpoint` | `""` | OTLP HTTP endpoint URL |
203+| `telemetry.auth_token` | `""` | Base64 `user:password` for Basic auth |
204+| `telemetry.traces` | `true` | Export distributed traces |
205+| `telemetry.metrics` | `true` | Export metrics |
206+| `telemetry.logs` | `true` | Bridge zerolog to OTLP logs |
207+| `telemetry.trace_sample_rate` | `1.0` | Fraction of traces to sample (0.0–1.0) |
189208 | `deploy.pull_policy` | `"if-tag-changed"` | Pull on tag change |
190209 | `deploy.readiness_mode` | `"auto"` | Readiness strategy (`auto`, `docker-health`, `delay`) |
191210 | `deploy.health_timeout` | `"90s"` | Max wait for health-based readiness before deploy fails |
@@ -280,6 +299,7 @@ gordon serve
280299
281300 - [Configuration Overview](./index.md)
282301 - [Authentication](./auth.md)
302+- [Telemetry](./telemetry.md)
283303 - [Network Isolation](./network-isolation.md)
284304 - [Volumes](./volumes.md)
285305 - [Images](./images.md)
@@ -63,6 +63,18 @@ max_size = 100 # Max size in MB before rotation
63 max_backups = 3 # Number of old files to keep63 max_backups = 3 # Number of old files to keep
64 max_age = 28 # Days to keep old files64 max_age = 28 # Days to keep old files
65 65
66+# =============================================================================
67+# TELEMETRY (OpenTelemetry)
68+# =============================================================================
69+[telemetry]
70+enabled = false # Enable OTLP telemetry export
71+endpoint = "" # OTLP HTTP endpoint URL
72+auth_token = "" # Base64-encoded user:password for Basic auth
73+traces = true # Export distributed traces
74+metrics = true # Export metrics
75+logs = true # Bridge zerolog output to OTLP logs
76+trace_sample_rate = 1.0 # Fraction of traces to sample (0.0–1.0)
77+
66 # =============================================================================78 # =============================================================================
67 # ENVIRONMENT79 # ENVIRONMENT
68 # =============================================================================80 # =============================================================================
@@ -186,6 +198,13 @@ keep_last = 3 # Keep N newest tags per repository
186 | `logging.container_logs.max_size` | `100` | 100 MB |198 | `logging.container_logs.max_size` | `100` | 100 MB |
187 | `logging.container_logs.max_backups` | `3` | Keep 3 old files |199 | `logging.container_logs.max_backups` | `3` | Keep 3 old files |
188 | `logging.container_logs.max_age` | `28` | 28 days |200 | `logging.container_logs.max_age` | `28` | 28 days |
201+| `telemetry.enabled` | `false` | Enable OTLP telemetry export |
202+| `telemetry.endpoint` | `""` | OTLP HTTP endpoint URL |
203+| `telemetry.auth_token` | `""` | Base64 `user:password` for Basic auth |
204+| `telemetry.traces` | `true` | Export distributed traces |
205+| `telemetry.metrics` | `true` | Export metrics |
206+| `telemetry.logs` | `true` | Bridge zerolog to OTLP logs |
207+| `telemetry.trace_sample_rate` | `1.0` | Fraction of traces to sample (0.0–1.0) |
189 | `deploy.pull_policy` | `"if-tag-changed"` | Pull on tag change |208 | `deploy.pull_policy` | `"if-tag-changed"` | Pull on tag change |
190 | `deploy.readiness_mode` | `"auto"` | Readiness strategy (`auto`, `docker-health`, `delay`) |209 | `deploy.readiness_mode` | `"auto"` | Readiness strategy (`auto`, `docker-health`, `delay`) |
191 | `deploy.health_timeout` | `"90s"` | Max wait for health-based readiness before deploy fails |210 | `deploy.health_timeout` | `"90s"` | Max wait for health-based readiness before deploy fails |
@@ -280,6 +299,7 @@ gordon serve
280 299
281 - [Configuration Overview](./index.md)300 - [Configuration Overview](./index.md)
282 - [Authentication](./auth.md)301 - [Authentication](./auth.md)
302+- [Telemetry](./telemetry.md)
283 - [Network Isolation](./network-isolation.md)303 - [Network Isolation](./network-isolation.md)
284 - [Volumes](./volumes.md)304 - [Volumes](./volumes.md)
285 - [Images](./images.md)305 - [Images](./images.md)
A docs/config/telemetry.md +199 -0
@@ -0,0 +1,199 @@
1+# Telemetry Configuration
2+
3+Export traces, metrics, and logs from Gordon via OpenTelemetry (OTLP/HTTP).
4+
5+When enabled, Gordon ships all three observability signals to an OTLP-compatible backend such as OpenObserve, Jaeger, or Grafana Cloud. Existing zerolog output continues unchanged; the OTel log bridge runs alongside it.
6+
7+## Configuration
8+
9+```toml
10+[telemetry]
11+enabled = false
12+endpoint = ""
13+auth_token = ""
14+traces = true
15+metrics = true
16+logs = true
17+trace_sample_rate = 1.0
18+```
19+
20+## Options
21+
22+| Option | Type | Default | Description |
23+|--------|------|---------|-------------|
24+| `enabled` | bool | `false` | Enable OTLP telemetry export |
25+| `endpoint` | string | `""` | OTLP HTTP endpoint URL (e.g. `http://localhost:4318/api/default`) |
26+| `auth_token` | string | `""` | Base64-encoded `user:password` for Basic auth |
27+| `traces` | bool | `true` | Export distributed traces |
28+| `metrics` | bool | `true` | Export metrics (deploy counters, container lifecycle, registry, events) |
29+| `logs` | bool | `true` | Bridge zerolog output to OTLP logs |
30+| `trace_sample_rate` | float | `1.0` | Fraction of traces to sample (`0.0` = none, `1.0` = all) |
31+
32+## How It Works
33+
34+Gordon initializes an OTel provider at startup. When telemetry is enabled:
35+
36+1. **Traces** -- Spans wrap critical operations: container deploy, image pull, registry manifest push, and proxy target resolution. The `otelhttp` middleware adds a span to every HTTP request on both the proxy and registry servers.
37+2. **Metrics** -- Gordon records custom counters and histograms for deploys, container restarts, crash loops, managed container count, registry pushes, and event bus throughput.
38+3. **Logs** -- A zerowrap/otel hook bridges all structured log output to the OTLP log pipeline. Every log line automatically carries `trace_id` and `span_id` when emitted inside a traced request.
39+
40+When telemetry is disabled (the default), all OTel instruments are noop -- zero overhead.
41+
42+## Endpoint URL
43+
44+The `endpoint` field accepts a full URL. Gordon parses it to extract:
45+
46+- **Host and port** for the OTLP exporter
47+- **Base path** appended with `/v1/traces`, `/v1/metrics`, or `/v1/logs`
48+- **Scheme** -- `http://` disables TLS; `https://` enables it
49+
50+For OpenObserve, the endpoint typically includes the organization path:
51+
52+```
53+http://localhost:5080/api/default
54+```
55+
56+This produces export URLs like `http://localhost:5080/api/default/v1/traces`.
57+
58+## Authentication
59+
60+Set `auth_token` to the Base64-encoded `user:password` string. Gordon sends it as a `Basic` authorization header on every OTLP export request.
61+
62+For OpenObserve, copy the token from **Ingestion > OTLP** in the web UI.
63+
64+Since Gordon itself is the platform (not a managed container), it does not use `gordon secrets set`. Store the token with one of these methods:
65+
66+| Method | How |
67+|--------|-----|
68+| Environment variable | `GORDON_TELEMETRY_AUTH_TOKEN=<token> gordon serve` |
69+| Plain text in config | Set `auth_token = "<token>"` (restrict file permissions) |
70+
71+## Trace Sampling
72+
73+| Value | Behavior |
74+|-------|----------|
75+| `1.0` | Sample every trace (default) |
76+| `0.5` | Sample 50% of traces |
77+| `0.0` | Drop all traces (tracing pipeline stays initialized) |
78+
79+At low traffic volumes (< 1000 req/s), keep the rate at `1.0`. Reduce it if storage costs or export bandwidth become a concern.
80+
81+## Metrics Reference
82+
83+Gordon exports these custom metrics (all prefixed with `gordon.`):
84+
85+### Deployments
86+
87+| Metric | Type | Unit | Description |
88+|--------|------|------|-------------|
89+| `gordon.deploy.total` | Counter | - | Total deployments attempted |
90+| `gordon.deploy.duration_seconds` | Histogram | s | Time from deploy start to completion |
91+| `gordon.deploy.errors` | Counter | - | Deployments that failed |
92+
93+Attributes: `domain`, `image`
94+
95+### Container Lifecycle
96+
97+| Metric | Type | Unit | Description |
98+|--------|------|------|-------------|
99+| `gordon.container.restarts` | Counter | - | Container restart count |
100+| `gordon.container.crash_loops` | Counter | - | Crash loop detections |
101+| `gordon.container.managed` | UpDownCounter | - | Currently tracked containers |
102+
103+Attributes: `domain`, `source` (restarts only: `monitor` or `api`)
104+
105+### Registry
106+
107+| Metric | Type | Unit | Description |
108+|--------|------|------|-------------|
109+| `gordon.registry.push.total` | Counter | - | Manifest pushes received |
110+| `gordon.registry.push.bytes` | Counter | By | Total manifest bytes pushed |
111+
112+Attributes: `name`, `reference`
113+
114+### Event Bus
115+
116+| Metric | Type | Unit | Description |
117+|--------|------|------|-------------|
118+| `gordon.events.processed` | Counter | - | Events handled successfully |
119+| `gordon.events.dropped` | Counter | - | Events dropped (channel full) |
120+
121+Attributes: `event_type`
122+
123+### HTTP (via otelhttp)
124+
125+The `otelhttp` middleware automatically records standard HTTP server metrics on both the proxy and registry servers:
126+
127+- `http.server.request.duration`
128+- `http.server.request.body.size`
129+- `http.server.response.body.size`
130+
131+## Trace Spans
132+
133+| Span Name | Package | Description |
134+|-----------|---------|-------------|
135+| `container.deploy` | container | Full deploy lifecycle (root span) |
136+| `container.create_and_start` | container | Container creation, start, and readiness |
137+| `container.ensure_image` | container | Image pull and tagging |
138+| `registry.put_manifest` | registry | Manifest storage and event publish |
139+| `proxy.get_target` | proxy | Proxy target resolution and caching |
140+
141+The `otelhttp` middleware adds an HTTP-level span to every request on both servers.
142+
143+## Examples
144+
145+### OpenObserve (self-hosted)
146+
147+```toml
148+[telemetry]
149+enabled = true
150+endpoint = "http://localhost:5080/api/default"
151+auth_token = "YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0"
152+traces = true
153+metrics = true
154+logs = true
155+trace_sample_rate = 1.0
156+```
157+
158+### Grafana Cloud (OTLP)
159+
160+```toml
161+[telemetry]
162+enabled = true
163+endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
164+auth_token = "<instance-id>:<api-key>"
165+traces = true
166+metrics = true
167+logs = true
168+trace_sample_rate = 0.1
169+```
170+
171+### Traces Only (minimal overhead)
172+
173+```toml
174+[telemetry]
175+enabled = true
176+endpoint = "http://localhost:4318"
177+traces = true
178+metrics = false
179+logs = false
180+trace_sample_rate = 0.5
181+```
182+
183+## Environment Variable Override
184+
185+```bash
186+GORDON_TELEMETRY_ENABLED=true
187+GORDON_TELEMETRY_ENDPOINT=http://localhost:5080/api/default
188+GORDON_TELEMETRY_AUTH_TOKEN=YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0
189+GORDON_TELEMETRY_TRACES=true
190+GORDON_TELEMETRY_METRICS=true
191+GORDON_TELEMETRY_LOGS=true
192+GORDON_TELEMETRY_TRACE_SAMPLE_RATE=1.0
193+```
194+
195+## Related
196+
197+- [Configuration Overview](./index.md)
198+- [Logging](./logging.md)
199+- [Configuration Reference](./reference.md)
@@ -0,0 +1,199 @@
1+# Telemetry Configuration
2+
3+Export traces, metrics, and logs from Gordon via OpenTelemetry (OTLP/HTTP).
4+
5+When enabled, Gordon ships all three observability signals to an OTLP-compatible backend such as OpenObserve, Jaeger, or Grafana Cloud. Existing zerolog output continues unchanged; the OTel log bridge runs alongside it.
6+
7+## Configuration
8+
9+```toml
10+[telemetry]
11+enabled = false
12+endpoint = ""
13+auth_token = ""
14+traces = true
15+metrics = true
16+logs = true
17+trace_sample_rate = 1.0
18+```
19+
20+## Options
21+
22+| Option | Type | Default | Description |
23+|--------|------|---------|-------------|
24+| `enabled` | bool | `false` | Enable OTLP telemetry export |
25+| `endpoint` | string | `""` | OTLP HTTP endpoint URL (e.g. `http://localhost:4318/api/default`) |
26+| `auth_token` | string | `""` | Base64-encoded `user:password` for Basic auth |
27+| `traces` | bool | `true` | Export distributed traces |
28+| `metrics` | bool | `true` | Export metrics (deploy counters, container lifecycle, registry, events) |
29+| `logs` | bool | `true` | Bridge zerolog output to OTLP logs |
30+| `trace_sample_rate` | float | `1.0` | Fraction of traces to sample (`0.0` = none, `1.0` = all) |
31+
32+## How It Works
33+
34+Gordon initializes an OTel provider at startup. When telemetry is enabled:
35+
36+1. **Traces** -- Spans wrap critical operations: container deploy, image pull, registry manifest push, and proxy target resolution. The `otelhttp` middleware adds a span to every HTTP request on both the proxy and registry servers.
37+2. **Metrics** -- Gordon records custom counters and histograms for deploys, container restarts, crash loops, managed container count, registry pushes, and event bus throughput.
38+3. **Logs** -- A zerowrap/otel hook bridges all structured log output to the OTLP log pipeline. Every log line automatically carries `trace_id` and `span_id` when emitted inside a traced request.
39+
40+When telemetry is disabled (the default), all OTel instruments are noop -- zero overhead.
41+
42+## Endpoint URL
43+
44+The `endpoint` field accepts a full URL. Gordon parses it to extract:
45+
46+- **Host and port** for the OTLP exporter
47+- **Base path** appended with `/v1/traces`, `/v1/metrics`, or `/v1/logs`
48+- **Scheme** -- `http://` disables TLS; `https://` enables it
49+
50+For OpenObserve, the endpoint typically includes the organization path:
51+
52+```
53+http://localhost:5080/api/default
54+```
55+
56+This produces export URLs like `http://localhost:5080/api/default/v1/traces`.
57+
58+## Authentication
59+
60+Set `auth_token` to the Base64-encoded `user:password` string. Gordon sends it as a `Basic` authorization header on every OTLP export request.
61+
62+For OpenObserve, copy the token from **Ingestion > OTLP** in the web UI.
63+
64+Since Gordon itself is the platform (not a managed container), it does not use `gordon secrets set`. Store the token with one of these methods:
65+
66+| Method | How |
67+|--------|-----|
68+| Environment variable | `GORDON_TELEMETRY_AUTH_TOKEN=<token> gordon serve` |
69+| Plain text in config | Set `auth_token = "<token>"` (restrict file permissions) |
70+
71+## Trace Sampling
72+
73+| Value | Behavior |
74+|-------|----------|
75+| `1.0` | Sample every trace (default) |
76+| `0.5` | Sample 50% of traces |
77+| `0.0` | Drop all traces (tracing pipeline stays initialized) |
78+
79+At low traffic volumes (< 1000 req/s), keep the rate at `1.0`. Reduce it if storage costs or export bandwidth become a concern.
80+
81+## Metrics Reference
82+
83+Gordon exports these custom metrics (all prefixed with `gordon.`):
84+
85+### Deployments
86+
87+| Metric | Type | Unit | Description |
88+|--------|------|------|-------------|
89+| `gordon.deploy.total` | Counter | - | Total deployments attempted |
90+| `gordon.deploy.duration_seconds` | Histogram | s | Time from deploy start to completion |
91+| `gordon.deploy.errors` | Counter | - | Deployments that failed |
92+
93+Attributes: `domain`, `image`
94+
95+### Container Lifecycle
96+
97+| Metric | Type | Unit | Description |
98+|--------|------|------|-------------|
99+| `gordon.container.restarts` | Counter | - | Container restart count |
100+| `gordon.container.crash_loops` | Counter | - | Crash loop detections |
101+| `gordon.container.managed` | UpDownCounter | - | Currently tracked containers |
102+
103+Attributes: `domain`, `source` (restarts only: `monitor` or `api`)
104+
105+### Registry
106+
107+| Metric | Type | Unit | Description |
108+|--------|------|------|-------------|
109+| `gordon.registry.push.total` | Counter | - | Manifest pushes received |
110+| `gordon.registry.push.bytes` | Counter | By | Total manifest bytes pushed |
111+
112+Attributes: `name`, `reference`
113+
114+### Event Bus
115+
116+| Metric | Type | Unit | Description |
117+|--------|------|------|-------------|
118+| `gordon.events.processed` | Counter | - | Events handled successfully |
119+| `gordon.events.dropped` | Counter | - | Events dropped (channel full) |
120+
121+Attributes: `event_type`
122+
123+### HTTP (via otelhttp)
124+
125+The `otelhttp` middleware automatically records standard HTTP server metrics on both the proxy and registry servers:
126+
127+- `http.server.request.duration`
128+- `http.server.request.body.size`
129+- `http.server.response.body.size`
130+
131+## Trace Spans
132+
133+| Span Name | Package | Description |
134+|-----------|---------|-------------|
135+| `container.deploy` | container | Full deploy lifecycle (root span) |
136+| `container.create_and_start` | container | Container creation, start, and readiness |
137+| `container.ensure_image` | container | Image pull and tagging |
138+| `registry.put_manifest` | registry | Manifest storage and event publish |
139+| `proxy.get_target` | proxy | Proxy target resolution and caching |
140+
141+The `otelhttp` middleware adds an HTTP-level span to every request on both servers.
142+
143+## Examples
144+
145+### OpenObserve (self-hosted)
146+
147+```toml
148+[telemetry]
149+enabled = true
150+endpoint = "http://localhost:5080/api/default"
151+auth_token = "YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0"
152+traces = true
153+metrics = true
154+logs = true
155+trace_sample_rate = 1.0
156+```
157+
158+### Grafana Cloud (OTLP)
159+
160+```toml
161+[telemetry]
162+enabled = true
163+endpoint = "https://otlp-gateway-prod-us-east-0.grafana.net/otlp"
164+auth_token = "<instance-id>:<api-key>"
165+traces = true
166+metrics = true
167+logs = true
168+trace_sample_rate = 0.1
169+```
170+
171+### Traces Only (minimal overhead)
172+
173+```toml
174+[telemetry]
175+enabled = true
176+endpoint = "http://localhost:4318"
177+traces = true
178+metrics = false
179+logs = false
180+trace_sample_rate = 0.5
181+```
182+
183+## Environment Variable Override
184+
185+```bash
186+GORDON_TELEMETRY_ENABLED=true
187+GORDON_TELEMETRY_ENDPOINT=http://localhost:5080/api/default
188+GORDON_TELEMETRY_AUTH_TOKEN=YWRtaW5AZXhhbXBsZS5jb206c2VjcmV0
189+GORDON_TELEMETRY_TRACES=true
190+GORDON_TELEMETRY_METRICS=true
191+GORDON_TELEMETRY_LOGS=true
192+GORDON_TELEMETRY_TRACE_SAMPLE_RATE=1.0
193+```
194+
195+## Related
196+
197+- [Configuration Overview](./index.md)
198+- [Logging](./logging.md)
199+- [Configuration Reference](./reference.md)
M docs/reference/index.md +1 -0
@@ -6,6 +6,7 @@ Technical reference documentation for Gordon.
66
77 - [Docker Labels](./docker-labels.md) - Container and image labels used by Gordon
88 - [Environment Variables](./env-variables.md) - Environment variable syntax and interpolation
9+- [Telemetry & Metrics](/docs/config/telemetry.md) - OpenTelemetry configuration, custom metrics, and trace spans
910 - [Troubleshooting](./troubleshooting.md) - Common issues and solutions
1011
1112 ## Related
@@ -6,6 +6,7 @@ Technical reference documentation for Gordon.
6 6
7 - [Docker Labels](./docker-labels.md) - Container and image labels used by Gordon7 - [Docker Labels](./docker-labels.md) - Container and image labels used by Gordon
8 - [Environment Variables](./env-variables.md) - Environment variable syntax and interpolation8 - [Environment Variables](./env-variables.md) - Environment variable syntax and interpolation
9+- [Telemetry & Metrics](/docs/config/telemetry.md) - OpenTelemetry configuration, custom metrics, and trace spans
9 - [Troubleshooting](./troubleshooting.md) - Common issues and solutions10 - [Troubleshooting](./troubleshooting.md) - Common issues and solutions
10 11
11 ## Related12 ## Related
M go.mod +20 -5
@@ -14,6 +14,11 @@ require (
1414 github.com/spf13/cobra v1.10.2
1515 github.com/spf13/viper v1.21.0
1616 github.com/stretchr/testify v1.11.1
17+ go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.16.0
18+ go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.40.0
19+ go.opentelemetry.io/otel/sdk v1.40.0
20+ go.opentelemetry.io/otel/sdk/log v0.16.0
21+ go.opentelemetry.io/otel/sdk/metric v1.40.0
1722 golang.org/x/crypto v0.47.0
1823 golang.org/x/mod v0.32.0
1924 golang.org/x/term v0.39.0
@@ -23,6 +28,7 @@ require (
2328 require (
2429 github.com/atotto/clipboard v0.1.4 // indirect
2530 github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
31+ github.com/cenkalti/backoff/v5 v5.0.3 // indirect
2632 github.com/cespare/xxhash/v2 v2.3.0 // indirect
2733 github.com/charmbracelet/colorprofile v0.4.1 // indirect
2834 github.com/charmbracelet/x/ansi v0.11.5 // indirect
@@ -32,6 +38,7 @@ require (
3238 github.com/clipperhouse/stringish v0.1.1 // indirect
3339 github.com/clipperhouse/uax29/v2 v2.5.0 // indirect
3440 github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
41+ github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.7 // indirect
3542 github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
3643 github.com/mattn/go-localereader v0.0.1 // indirect
3744 github.com/mattn/go-runewidth v0.0.19 // indirect
@@ -41,7 +48,15 @@ require (
4148 github.com/rivo/uniseg v0.4.7 // indirect
4249 github.com/rs/zerolog v1.34.0 // indirect
4350 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
51+ go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0 // indirect
52+ go.opentelemetry.io/otel/log v0.16.0 // indirect
53+ go.opentelemetry.io/proto/otlp v1.9.0 // indirect
4454 go.yaml.in/yaml/v3 v3.0.4 // indirect
55+ golang.org/x/net v0.49.0 // indirect
56+ google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409 // indirect
57+ google.golang.org/genproto/googleapis/rpc v0.0.0-20260128011058-8636f8732409 // indirect
58+ google.golang.org/grpc v1.78.0 // indirect
59+ google.golang.org/protobuf v1.36.11 // indirect
4560 )
4661
4762 require (
@@ -81,11 +96,11 @@ require (
8196 github.com/spf13/pflag v1.0.10 // indirect
8297 github.com/subosito/gotenv v1.6.0 // indirect
8398 go.opentelemetry.io/auto/sdk v1.2.1 // indirect
84- go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0 // indirect
85- go.opentelemetry.io/otel v1.40.0 // indirect
86- go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0 // indirect
87- go.opentelemetry.io/otel/metric v1.40.0 // indirect
88- go.opentelemetry.io/otel/trace v1.40.0 // indirect
99+ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0
100+ go.opentelemetry.io/otel v1.40.0
101+ go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0
102+ go.opentelemetry.io/otel/metric v1.40.0
103+ go.opentelemetry.io/otel/trace v1.40.0
89104 golang.org/x/sys v0.40.0 // indirect
90105 golang.org/x/text v0.33.0 // indirect
91106 gopkg.in/yaml.v3 v3.0.1 // indirect
@@ -14,6 +14,11 @@ require (
14 github.com/spf13/cobra v1.10.214 github.com/spf13/cobra v1.10.2
15 github.com/spf13/viper v1.21.015 github.com/spf13/viper v1.21.0
16 github.com/stretchr/testify v1.11.116 github.com/stretchr/testify v1.11.1
17+ go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.16.0
18+ go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.40.0
19+ go.opentelemetry.io/otel/sdk v1.40.0
20+ go.opentelemetry.io/otel/sdk/log v0.16.0
21+ go.opentelemetry.io/otel/sdk/metric v1.40.0
17 golang.org/x/crypto v0.47.022 golang.org/x/crypto v0.47.0
18 golang.org/x/mod v0.32.023 golang.org/x/mod v0.32.0
19 golang.org/x/term v0.39.024 golang.org/x/term v0.39.0
@@ -23,6 +28,7 @@ require (
23 require (28 require (
24 github.com/atotto/clipboard v0.1.4 // indirect29 github.com/atotto/clipboard v0.1.4 // indirect
25 github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect30 github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
31+ github.com/cenkalti/backoff/v5 v5.0.3 // indirect
26 github.com/cespare/xxhash/v2 v2.3.0 // indirect32 github.com/cespare/xxhash/v2 v2.3.0 // indirect
27 github.com/charmbracelet/colorprofile v0.4.1 // indirect33 github.com/charmbracelet/colorprofile v0.4.1 // indirect
28 github.com/charmbracelet/x/ansi v0.11.5 // indirect34 github.com/charmbracelet/x/ansi v0.11.5 // indirect
@@ -32,6 +38,7 @@ require (
32 github.com/clipperhouse/stringish v0.1.1 // indirect38 github.com/clipperhouse/stringish v0.1.1 // indirect
33 github.com/clipperhouse/uax29/v2 v2.5.0 // indirect39 github.com/clipperhouse/uax29/v2 v2.5.0 // indirect
34 github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect40 github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
41+ github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.7 // indirect
35 github.com/lucasb-eyer/go-colorful v1.3.0 // indirect42 github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
36 github.com/mattn/go-localereader v0.0.1 // indirect43 github.com/mattn/go-localereader v0.0.1 // indirect
37 github.com/mattn/go-runewidth v0.0.19 // indirect44 github.com/mattn/go-runewidth v0.0.19 // indirect
@@ -41,7 +48,15 @@ require (
41 github.com/rivo/uniseg v0.4.7 // indirect48 github.com/rivo/uniseg v0.4.7 // indirect
42 github.com/rs/zerolog v1.34.0 // indirect49 github.com/rs/zerolog v1.34.0 // indirect
43 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect50 github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
51+ go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0 // indirect
52+ go.opentelemetry.io/otel/log v0.16.0 // indirect
53+ go.opentelemetry.io/proto/otlp v1.9.0 // indirect
44 go.yaml.in/yaml/v3 v3.0.4 // indirect54 go.yaml.in/yaml/v3 v3.0.4 // indirect
55+ golang.org/x/net v0.49.0 // indirect
56+ google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409 // indirect
57+ google.golang.org/genproto/googleapis/rpc v0.0.0-20260128011058-8636f8732409 // indirect
58+ google.golang.org/grpc v1.78.0 // indirect
59+ google.golang.org/protobuf v1.36.11 // indirect
45 )60 )
46 61
47 require (62 require (
@@ -81,11 +96,11 @@ require (
81 github.com/spf13/pflag v1.0.10 // indirect96 github.com/spf13/pflag v1.0.10 // indirect
82 github.com/subosito/gotenv v1.6.0 // indirect97 github.com/subosito/gotenv v1.6.0 // indirect
83 go.opentelemetry.io/auto/sdk v1.2.1 // indirect98 go.opentelemetry.io/auto/sdk v1.2.1 // indirect
84- go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0 // indirect99+ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0
85- go.opentelemetry.io/otel v1.40.0 // indirect100+ go.opentelemetry.io/otel v1.40.0
86- go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0 // indirect101+ go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0
87- go.opentelemetry.io/otel/metric v1.40.0 // indirect102+ go.opentelemetry.io/otel/metric v1.40.0
88- go.opentelemetry.io/otel/trace v1.40.0 // indirect103+ go.opentelemetry.io/otel/trace v1.40.0
89 golang.org/x/sys v0.40.0 // indirect104 golang.org/x/sys v0.40.0 // indirect
90 golang.org/x/text v0.33.0 // indirect105 golang.org/x/text v0.33.0 // indirect
91 gopkg.in/yaml.v3 v3.0.1 // indirect106 gopkg.in/yaml.v3 v3.0.1 // indirect
M go.sum +16 -0
@@ -72,6 +72,8 @@ github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlnd
7272 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
7373 github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
7474 github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
75+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
76+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
7577 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
7678 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
7779 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@@ -165,20 +167,32 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0 h1:7iP2uCb
165167 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0/go.mod h1:c7hN3ddxs/z6q9xwvfLPk+UHlWRQyaeR1LdgfL/66l0=
166168 go.opentelemetry.io/otel v1.40.0 h1:oA5YeOcpRTXq6NN7frwmwFR0Cn3RhTVZvXsP4duvCms=
167169 go.opentelemetry.io/otel v1.40.0/go.mod h1:IMb+uXZUKkMXdPddhwAHm6UfOwJyh4ct1ybIlV14J0g=
170+go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.16.0 h1:djrxvDxAe44mJUrKataUbOhCKhR3F8QCyWucO16hTQs=
171+go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.16.0/go.mod h1:dt3nxpQEiSoKvfTVxp3TUg5fHPLhKtbcnN3Z1I1ePD0=
172+go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.40.0 h1:9y5sHvAxWzft1WQ4BwqcvA+IFVUJ1Ya75mSAUnFEVwE=
173+go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.40.0/go.mod h1:eQqT90eR3X5Dbs1g9YSM30RavwLF725Ris5/XSXWvqE=
168174 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0 h1:QKdN8ly8zEMrByybbQgv8cWBcdAarwmIPZ6FThrWXJs=
169175 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0/go.mod h1:bTdK1nhqF76qiPoCCdyFIV+N/sRHYXYCTQc+3VCi3MI=
170176 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0 h1:wVZXIWjQSeSmMoxF74LzAnpVQOAFDo3pPji9Y4SOFKc=
171177 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0/go.mod h1:khvBS2IggMFNwZK/6lEeHg/W57h/IX6J4URh57fuI40=
178+go.opentelemetry.io/otel/log v0.16.0 h1:DeuBPqCi6pQwtCK0pO4fvMB5eBq6sNxEnuTs88pjsN4=
179+go.opentelemetry.io/otel/log v0.16.0/go.mod h1:rWsmqNVTLIA8UnwYVOItjyEZDbKIkMxdQunsIhpUMes=
172180 go.opentelemetry.io/otel/metric v1.40.0 h1:rcZe317KPftE2rstWIBitCdVp89A2HqjkxR3c11+p9g=
173181 go.opentelemetry.io/otel/metric v1.40.0/go.mod h1:ib/crwQH7N3r5kfiBZQbwrTge743UDc7DTFVZrrXnqc=
174182 go.opentelemetry.io/otel/sdk v1.40.0 h1:KHW/jUzgo6wsPh9At46+h4upjtccTmuZCFAc9OJ71f8=
175183 go.opentelemetry.io/otel/sdk v1.40.0/go.mod h1:Ph7EFdYvxq72Y8Li9q8KebuYUr2KoeyHx0DRMKrYBUE=
184+go.opentelemetry.io/otel/sdk/log v0.16.0 h1:e/b4bdlQwC5fnGtG3dlXUrNOnP7c8YLVSpSfEBIkTnI=
185+go.opentelemetry.io/otel/sdk/log v0.16.0/go.mod h1:JKfP3T6ycy7QEuv3Hj8oKDy7KItrEkus8XJE6EoSzw4=
186+go.opentelemetry.io/otel/sdk/log/logtest v0.16.0 h1:/XVkpZ41rVRTP4DfMgYv1nEtNmf65XPPyAdqV90TMy4=
187+go.opentelemetry.io/otel/sdk/log/logtest v0.16.0/go.mod h1:iOOPgQr5MY9oac/F5W86mXdeyWZGleIx3uXO98X2R6Y=
176188 go.opentelemetry.io/otel/sdk/metric v1.40.0 h1:mtmdVqgQkeRxHgRv4qhyJduP3fYJRMX4AtAlbuWdCYw=
177189 go.opentelemetry.io/otel/sdk/metric v1.40.0/go.mod h1:4Z2bGMf0KSK3uRjlczMOeMhKU2rhUqdWNoKcYrtcBPg=
178190 go.opentelemetry.io/otel/trace v1.40.0 h1:WA4etStDttCSYuhwvEa8OP8I5EWu24lkOzp+ZYblVjw=
179191 go.opentelemetry.io/otel/trace v1.40.0/go.mod h1:zeAhriXecNGP/s2SEG3+Y8X9ujcJOTqQ5RgdEJcawiA=
180192 go.opentelemetry.io/proto/otlp v1.9.0 h1:l706jCMITVouPOqEnii2fIAuO3IVGBRPV5ICjceRb/A=
181193 go.opentelemetry.io/proto/otlp v1.9.0/go.mod h1:xE+Cx5E/eEHw+ISFkwPLwCZefwVjY+pqKg1qcK03+/4=
194+go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
195+go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
182196 go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
183197 go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
184198 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
@@ -205,6 +219,8 @@ golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE=
205219 golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8=
206220 golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
207221 golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
222+gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
223+gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
208224 google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409 h1:merA0rdPeUV3YIIfHHcH4qBkiQAc1nfCKSI7lB4cV2M=
209225 google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409/go.mod h1:fl8J1IvUjCilwZzQowmw2b7HQB2eAuYBabMXzWurF+I=
210226 google.golang.org/genproto/googleapis/rpc v0.0.0-20260128011058-8636f8732409 h1:H86B94AW+VfJWDqFeEbBPhEtHzJwJfTbgE2lZa54ZAQ=
@@ -72,6 +72,8 @@ github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlnd
72 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=72 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
73 github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=73 github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
74 github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=74 github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
75+github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
76+github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
75 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=77 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
76 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=78 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
77 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=79 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@@ -165,20 +167,32 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0 h1:7iP2uCb
165 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0/go.mod h1:c7hN3ddxs/z6q9xwvfLPk+UHlWRQyaeR1LdgfL/66l0=167 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.65.0/go.mod h1:c7hN3ddxs/z6q9xwvfLPk+UHlWRQyaeR1LdgfL/66l0=
166 go.opentelemetry.io/otel v1.40.0 h1:oA5YeOcpRTXq6NN7frwmwFR0Cn3RhTVZvXsP4duvCms=168 go.opentelemetry.io/otel v1.40.0 h1:oA5YeOcpRTXq6NN7frwmwFR0Cn3RhTVZvXsP4duvCms=
167 go.opentelemetry.io/otel v1.40.0/go.mod h1:IMb+uXZUKkMXdPddhwAHm6UfOwJyh4ct1ybIlV14J0g=169 go.opentelemetry.io/otel v1.40.0/go.mod h1:IMb+uXZUKkMXdPddhwAHm6UfOwJyh4ct1ybIlV14J0g=
170+go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.16.0 h1:djrxvDxAe44mJUrKataUbOhCKhR3F8QCyWucO16hTQs=
171+go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.16.0/go.mod h1:dt3nxpQEiSoKvfTVxp3TUg5fHPLhKtbcnN3Z1I1ePD0=
172+go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.40.0 h1:9y5sHvAxWzft1WQ4BwqcvA+IFVUJ1Ya75mSAUnFEVwE=
173+go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.40.0/go.mod h1:eQqT90eR3X5Dbs1g9YSM30RavwLF725Ris5/XSXWvqE=
168 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0 h1:QKdN8ly8zEMrByybbQgv8cWBcdAarwmIPZ6FThrWXJs=174 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0 h1:QKdN8ly8zEMrByybbQgv8cWBcdAarwmIPZ6FThrWXJs=
169 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0/go.mod h1:bTdK1nhqF76qiPoCCdyFIV+N/sRHYXYCTQc+3VCi3MI=175 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.40.0/go.mod h1:bTdK1nhqF76qiPoCCdyFIV+N/sRHYXYCTQc+3VCi3MI=
170 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0 h1:wVZXIWjQSeSmMoxF74LzAnpVQOAFDo3pPji9Y4SOFKc=176 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0 h1:wVZXIWjQSeSmMoxF74LzAnpVQOAFDo3pPji9Y4SOFKc=
171 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0/go.mod h1:khvBS2IggMFNwZK/6lEeHg/W57h/IX6J4URh57fuI40=177 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.40.0/go.mod h1:khvBS2IggMFNwZK/6lEeHg/W57h/IX6J4URh57fuI40=
178+go.opentelemetry.io/otel/log v0.16.0 h1:DeuBPqCi6pQwtCK0pO4fvMB5eBq6sNxEnuTs88pjsN4=
179+go.opentelemetry.io/otel/log v0.16.0/go.mod h1:rWsmqNVTLIA8UnwYVOItjyEZDbKIkMxdQunsIhpUMes=
172 go.opentelemetry.io/otel/metric v1.40.0 h1:rcZe317KPftE2rstWIBitCdVp89A2HqjkxR3c11+p9g=180 go.opentelemetry.io/otel/metric v1.40.0 h1:rcZe317KPftE2rstWIBitCdVp89A2HqjkxR3c11+p9g=
173 go.opentelemetry.io/otel/metric v1.40.0/go.mod h1:ib/crwQH7N3r5kfiBZQbwrTge743UDc7DTFVZrrXnqc=181 go.opentelemetry.io/otel/metric v1.40.0/go.mod h1:ib/crwQH7N3r5kfiBZQbwrTge743UDc7DTFVZrrXnqc=
174 go.opentelemetry.io/otel/sdk v1.40.0 h1:KHW/jUzgo6wsPh9At46+h4upjtccTmuZCFAc9OJ71f8=182 go.opentelemetry.io/otel/sdk v1.40.0 h1:KHW/jUzgo6wsPh9At46+h4upjtccTmuZCFAc9OJ71f8=
175 go.opentelemetry.io/otel/sdk v1.40.0/go.mod h1:Ph7EFdYvxq72Y8Li9q8KebuYUr2KoeyHx0DRMKrYBUE=183 go.opentelemetry.io/otel/sdk v1.40.0/go.mod h1:Ph7EFdYvxq72Y8Li9q8KebuYUr2KoeyHx0DRMKrYBUE=
184+go.opentelemetry.io/otel/sdk/log v0.16.0 h1:e/b4bdlQwC5fnGtG3dlXUrNOnP7c8YLVSpSfEBIkTnI=
185+go.opentelemetry.io/otel/sdk/log v0.16.0/go.mod h1:JKfP3T6ycy7QEuv3Hj8oKDy7KItrEkus8XJE6EoSzw4=
186+go.opentelemetry.io/otel/sdk/log/logtest v0.16.0 h1:/XVkpZ41rVRTP4DfMgYv1nEtNmf65XPPyAdqV90TMy4=
187+go.opentelemetry.io/otel/sdk/log/logtest v0.16.0/go.mod h1:iOOPgQr5MY9oac/F5W86mXdeyWZGleIx3uXO98X2R6Y=
176 go.opentelemetry.io/otel/sdk/metric v1.40.0 h1:mtmdVqgQkeRxHgRv4qhyJduP3fYJRMX4AtAlbuWdCYw=188 go.opentelemetry.io/otel/sdk/metric v1.40.0 h1:mtmdVqgQkeRxHgRv4qhyJduP3fYJRMX4AtAlbuWdCYw=
177 go.opentelemetry.io/otel/sdk/metric v1.40.0/go.mod h1:4Z2bGMf0KSK3uRjlczMOeMhKU2rhUqdWNoKcYrtcBPg=189 go.opentelemetry.io/otel/sdk/metric v1.40.0/go.mod h1:4Z2bGMf0KSK3uRjlczMOeMhKU2rhUqdWNoKcYrtcBPg=
178 go.opentelemetry.io/otel/trace v1.40.0 h1:WA4etStDttCSYuhwvEa8OP8I5EWu24lkOzp+ZYblVjw=190 go.opentelemetry.io/otel/trace v1.40.0 h1:WA4etStDttCSYuhwvEa8OP8I5EWu24lkOzp+ZYblVjw=
179 go.opentelemetry.io/otel/trace v1.40.0/go.mod h1:zeAhriXecNGP/s2SEG3+Y8X9ujcJOTqQ5RgdEJcawiA=191 go.opentelemetry.io/otel/trace v1.40.0/go.mod h1:zeAhriXecNGP/s2SEG3+Y8X9ujcJOTqQ5RgdEJcawiA=
180 go.opentelemetry.io/proto/otlp v1.9.0 h1:l706jCMITVouPOqEnii2fIAuO3IVGBRPV5ICjceRb/A=192 go.opentelemetry.io/proto/otlp v1.9.0 h1:l706jCMITVouPOqEnii2fIAuO3IVGBRPV5ICjceRb/A=
181 go.opentelemetry.io/proto/otlp v1.9.0/go.mod h1:xE+Cx5E/eEHw+ISFkwPLwCZefwVjY+pqKg1qcK03+/4=193 go.opentelemetry.io/proto/otlp v1.9.0/go.mod h1:xE+Cx5E/eEHw+ISFkwPLwCZefwVjY+pqKg1qcK03+/4=
194+go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
195+go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
182 go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=196 go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
183 go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=197 go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
184 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=198 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
@@ -205,6 +219,8 @@ golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE=
205 golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8=219 golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8=
206 golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=220 golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
207 golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=221 golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
222+gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
223+gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
208 google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409 h1:merA0rdPeUV3YIIfHHcH4qBkiQAc1nfCKSI7lB4cV2M=224 google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409 h1:merA0rdPeUV3YIIfHHcH4qBkiQAc1nfCKSI7lB4cV2M=
209 google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409/go.mod h1:fl8J1IvUjCilwZzQowmw2b7HQB2eAuYBabMXzWurF+I=225 google.golang.org/genproto/googleapis/api v0.0.0-20260128011058-8636f8732409/go.mod h1:fl8J1IvUjCilwZzQowmw2b7HQB2eAuYBabMXzWurF+I=
210 google.golang.org/genproto/googleapis/rpc v0.0.0-20260128011058-8636f8732409 h1:H86B94AW+VfJWDqFeEbBPhEtHzJwJfTbgE2lZa54ZAQ=226 google.golang.org/genproto/googleapis/rpc v0.0.0-20260128011058-8636f8732409 h1:H86B94AW+VfJWDqFeEbBPhEtHzJwJfTbgE2lZa54ZAQ=
M internal/adapters/in/http/middleware/logging.go +13 -2
@@ -12,6 +12,7 @@ import (
1212 "time"
1313
1414 "github.com/bnema/zerowrap"
15+ "go.opentelemetry.io/otel/trace"
1516
1617 "github.com/bnema/gordon/internal/adapters/dto"
1718 )
@@ -95,8 +96,18 @@ func RequestLogger(log zerowrap.Logger, trustedNets ...[]*net.IPNet) func(http.H
9596 // IP spoofing via X-Forwarded-For from untrusted sources.
9697 clientIP := GetClientIP(r, nets)
9798
98- // Attach the logger to the request context for downstream handlers
99- ctx := zerowrap.WithCtx(r.Context(), log)
99+ // Attach the logger to the request context for downstream handlers.
100+ // If an OTel span is active (from otelhttp middleware), inject
101+ // trace_id and span_id into the logger for log↔trace correlation.
102+ reqLog := log
103+ spanCtx := trace.SpanFromContext(r.Context()).SpanContext()
104+ if spanCtx.HasTraceID() {
105+ reqLog = reqLog.WithFields(map[string]any{
106+ "trace_id": spanCtx.TraceID().String(),
107+ "span_id": spanCtx.SpanID().String(),
108+ })
109+ }
110+ ctx := zerowrap.WithCtx(r.Context(), reqLog)
100111 r = r.WithContext(ctx)
101112
102113 // Call the next handler
@@ -12,6 +12,7 @@ import (
12 "time"12 "time"
13 13
14 "github.com/bnema/zerowrap"14 "github.com/bnema/zerowrap"
15+ "go.opentelemetry.io/otel/trace"
15 16
16 "github.com/bnema/gordon/internal/adapters/dto"17 "github.com/bnema/gordon/internal/adapters/dto"
17 )18 )
@@ -95,8 +96,18 @@ func RequestLogger(log zerowrap.Logger, trustedNets ...[]*net.IPNet) func(http.H
95 // IP spoofing via X-Forwarded-For from untrusted sources.96 // IP spoofing via X-Forwarded-For from untrusted sources.
96 clientIP := GetClientIP(r, nets)97 clientIP := GetClientIP(r, nets)
97 98
98- // Attach the logger to the request context for downstream handlers99+ // Attach the logger to the request context for downstream handlers.
99- ctx := zerowrap.WithCtx(r.Context(), log)100+ // If an OTel span is active (from otelhttp middleware), inject
101+ // trace_id and span_id into the logger for log↔trace correlation.
102+ reqLog := log
103+ spanCtx := trace.SpanFromContext(r.Context()).SpanContext()
104+ if spanCtx.HasTraceID() {
105+ reqLog = reqLog.WithFields(map[string]any{
106+ "trace_id": spanCtx.TraceID().String(),
107+ "span_id": spanCtx.SpanID().String(),
108+ })
109+ }
110+ ctx := zerowrap.WithCtx(r.Context(), reqLog)
100 r = r.WithContext(ctx)111 r = r.WithContext(ctx)
101 112
102 // Call the next handler113 // Call the next handler
M internal/adapters/out/eventbus/inmemory.go +25 -0
@@ -9,7 +9,10 @@ import (
99
1010 "github.com/bnema/zerowrap"
1111 "github.com/google/uuid"
12+ "go.opentelemetry.io/otel/attribute"
13+ "go.opentelemetry.io/otel/metric"
1214
15+ "github.com/bnema/gordon/internal/adapters/out/telemetry"
1316 "github.com/bnema/gordon/internal/boundaries/out"
1417 "github.com/bnema/gordon/internal/domain"
1518 )
@@ -24,6 +27,14 @@ type InMemory struct {
2427 cancel context.CancelFunc
2528 bufferSize int
2629 log zerowrap.Logger
30+ metrics *telemetry.Metrics
31+}
32+
33+// SetMetrics sets the telemetry metrics for the event bus.
34+func (bus *InMemory) SetMetrics(m *telemetry.Metrics) {
35+ bus.mu.Lock()
36+ bus.metrics = m
37+ bus.mu.Unlock()
2738 }
2839
2940 // NewInMemory creates a new in-memory event bus.
@@ -83,6 +94,13 @@ func (bus *InMemory) Publish(eventType domain.EventType, payload any) error {
8394 Str("image_name", event.ImageName).
8495 Str("tag", event.Tag).
8596 Msg("event channel is full, dropping event after 5s timeout")
97+
98+ // Record dropped event metric
99+ if bus.metrics != nil {
100+ bus.metrics.EventsDropped.Add(context.Background(), 1, metric.WithAttributes(
101+ attribute.String("event_type", string(event.Type)),
102+ ))
103+ }
86104 return fmt.Errorf("event channel is full, dropping event %s", event.ID)
87105 }
88106 }
@@ -217,6 +235,13 @@ func (bus *InMemory) handleEvent(event domain.Event) {
217235 Str(zerowrap.FieldHandler, fmt.Sprintf("%T", h)).
218236 Dur(zerowrap.FieldDuration, time.Since(start)).
219237 Msg("event handled successfully")
238+
239+ // Record processed event metric
240+ if bus.metrics != nil {
241+ bus.metrics.EventsProcessed.Add(context.Background(), 1, metric.WithAttributes(
242+ attribute.String("event_type", string(event.Type)),
243+ ))
244+ }
220245 }
221246 case <-ctx.Done():
222247 cancel()
@@ -9,7 +9,10 @@ import (
9 9
10 "github.com/bnema/zerowrap"10 "github.com/bnema/zerowrap"
11 "github.com/google/uuid"11 "github.com/google/uuid"
12+ "go.opentelemetry.io/otel/attribute"
13+ "go.opentelemetry.io/otel/metric"
12 14
15+ "github.com/bnema/gordon/internal/adapters/out/telemetry"
13 "github.com/bnema/gordon/internal/boundaries/out"16 "github.com/bnema/gordon/internal/boundaries/out"
14 "github.com/bnema/gordon/internal/domain"17 "github.com/bnema/gordon/internal/domain"
15 )18 )
@@ -24,6 +27,14 @@ type InMemory struct {
24 cancel context.CancelFunc27 cancel context.CancelFunc
25 bufferSize int28 bufferSize int
26 log zerowrap.Logger29 log zerowrap.Logger
30+ metrics *telemetry.Metrics
31+}
32+
33+// SetMetrics sets the telemetry metrics for the event bus.
34+func (bus *InMemory) SetMetrics(m *telemetry.Metrics) {
35+ bus.mu.Lock()
36+ bus.metrics = m
37+ bus.mu.Unlock()
27 }38 }
28 39
29 // NewInMemory creates a new in-memory event bus.40 // NewInMemory creates a new in-memory event bus.
@@ -83,6 +94,13 @@ func (bus *InMemory) Publish(eventType domain.EventType, payload any) error {
83 Str("image_name", event.ImageName).94 Str("image_name", event.ImageName).
84 Str("tag", event.Tag).95 Str("tag", event.Tag).
85 Msg("event channel is full, dropping event after 5s timeout")96 Msg("event channel is full, dropping event after 5s timeout")
97+
98+ // Record dropped event metric
99+ if bus.metrics != nil {
100+ bus.metrics.EventsDropped.Add(context.Background(), 1, metric.WithAttributes(
101+ attribute.String("event_type", string(event.Type)),
102+ ))
103+ }
86 return fmt.Errorf("event channel is full, dropping event %s", event.ID)104 return fmt.Errorf("event channel is full, dropping event %s", event.ID)
87 }105 }
88 }106 }
@@ -217,6 +235,13 @@ func (bus *InMemory) handleEvent(event domain.Event) {
217 Str(zerowrap.FieldHandler, fmt.Sprintf("%T", h)).235 Str(zerowrap.FieldHandler, fmt.Sprintf("%T", h)).
218 Dur(zerowrap.FieldDuration, time.Since(start)).236 Dur(zerowrap.FieldDuration, time.Since(start)).
219 Msg("event handled successfully")237 Msg("event handled successfully")
238+
239+ // Record processed event metric
240+ if bus.metrics != nil {
241+ bus.metrics.EventsProcessed.Add(context.Background(), 1, metric.WithAttributes(
242+ attribute.String("event_type", string(event.Type)),
243+ ))
244+ }
220 }245 }
221 case <-ctx.Done():246 case <-ctx.Done():
222 cancel()247 cancel()
A internal/adapters/out/telemetry/metrics.go +81 -0
@@ -0,0 +1,81 @@
1+package telemetry
2+
3+import (
4+ "go.opentelemetry.io/otel"
5+ "go.opentelemetry.io/otel/metric"
6+)
7+
8+// Metrics holds Gordon-specific OTel metrics instruments.
9+type Metrics struct {
10+ // Deployments
11+ DeployTotal metric.Int64Counter
12+ DeployDuration metric.Float64Histogram
13+ DeployErrors metric.Int64Counter
14+
15+ // Container lifecycle
16+ ContainerRestarts metric.Int64Counter
17+ ContainerCrashLoops metric.Int64Counter
18+ ManagedContainers metric.Int64UpDownCounter
19+
20+ // Registry
21+ ImagePushTotal metric.Int64Counter
22+ ImagePushSize metric.Int64Counter // bytes
23+
24+ // Events
25+ EventsProcessed metric.Int64Counter
26+ EventsDropped metric.Int64Counter
27+}
28+
29+// NewMetrics creates and registers all Gordon metric instruments.
30+// Returns a noop-safe Metrics struct — all fields are always initialized
31+// (OTel returns noop instruments when no MeterProvider is set).
32+func NewMetrics() (*Metrics, error) {
33+ meter := otel.Meter("gordon")
34+ m := &Metrics{}
35+ var err error
36+
37+ if m.DeployTotal, err = meter.Int64Counter("gordon.deploy.total",
38+ metric.WithDescription("Total number of deployments")); err != nil {
39+ return nil, err
40+ }
41+ if m.DeployDuration, err = meter.Float64Histogram("gordon.deploy.duration_seconds",
42+ metric.WithDescription("Deploy duration in seconds"),
43+ metric.WithUnit("s")); err != nil {
44+ return nil, err
45+ }
46+ if m.DeployErrors, err = meter.Int64Counter("gordon.deploy.errors",
47+ metric.WithDescription("Total deploy errors")); err != nil {
48+ return nil, err
49+ }
50+ if m.ContainerRestarts, err = meter.Int64Counter("gordon.container.restarts",
51+ metric.WithDescription("Total container restarts")); err != nil {
52+ return nil, err
53+ }
54+ if m.ContainerCrashLoops, err = meter.Int64Counter("gordon.container.crash_loops",
55+ metric.WithDescription("Total crash loop detections")); err != nil {
56+ return nil, err
57+ }
58+ if m.ManagedContainers, err = meter.Int64UpDownCounter("gordon.container.managed",
59+ metric.WithDescription("Currently managed containers")); err != nil {
60+ return nil, err
61+ }
62+ if m.ImagePushTotal, err = meter.Int64Counter("gordon.registry.push.total",
63+ metric.WithDescription("Total image pushes")); err != nil {
64+ return nil, err
65+ }
66+ if m.ImagePushSize, err = meter.Int64Counter("gordon.registry.push.bytes",
67+ metric.WithDescription("Total bytes pushed to registry"),
68+ metric.WithUnit("By")); err != nil {
69+ return nil, err
70+ }
71+ if m.EventsProcessed, err = meter.Int64Counter("gordon.events.processed",
72+ metric.WithDescription("Total events processed")); err != nil {
73+ return nil, err
74+ }
75+ if m.EventsDropped, err = meter.Int64Counter("gordon.events.dropped",
76+ metric.WithDescription("Total events dropped")); err != nil {
77+ return nil, err
78+ }
79+
80+ return m, nil
81+}
@@ -0,0 +1,81 @@
1+package telemetry
2+
3+import (
4+ "go.opentelemetry.io/otel"
5+ "go.opentelemetry.io/otel/metric"
6+)
7+
8+// Metrics holds Gordon-specific OTel metrics instruments.
9+type Metrics struct {
10+ // Deployments
11+ DeployTotal metric.Int64Counter
12+ DeployDuration metric.Float64Histogram
13+ DeployErrors metric.Int64Counter
14+
15+ // Container lifecycle
16+ ContainerRestarts metric.Int64Counter
17+ ContainerCrashLoops metric.Int64Counter
18+ ManagedContainers metric.Int64UpDownCounter
19+
20+ // Registry
21+ ImagePushTotal metric.Int64Counter
22+ ImagePushSize metric.Int64Counter // bytes
23+
24+ // Events
25+ EventsProcessed metric.Int64Counter
26+ EventsDropped metric.Int64Counter
27+}
28+
29+// NewMetrics creates and registers all Gordon metric instruments.
30+// Returns a noop-safe Metrics struct — all fields are always initialized
31+// (OTel returns noop instruments when no MeterProvider is set).
32+func NewMetrics() (*Metrics, error) {
33+ meter := otel.Meter("gordon")
34+ m := &Metrics{}
35+ var err error
36+
37+ if m.DeployTotal, err = meter.Int64Counter("gordon.deploy.total",
38+ metric.WithDescription("Total number of deployments")); err != nil {
39+ return nil, err
40+ }
41+ if m.DeployDuration, err = meter.Float64Histogram("gordon.deploy.duration_seconds",
42+ metric.WithDescription("Deploy duration in seconds"),
43+ metric.WithUnit("s")); err != nil {
44+ return nil, err
45+ }
46+ if m.DeployErrors, err = meter.Int64Counter("gordon.deploy.errors",
47+ metric.WithDescription("Total deploy errors")); err != nil {
48+ return nil, err
49+ }
50+ if m.ContainerRestarts, err = meter.Int64Counter("gordon.container.restarts",
51+ metric.WithDescription("Total container restarts")); err != nil {
52+ return nil, err
53+ }
54+ if m.ContainerCrashLoops, err = meter.Int64Counter("gordon.container.crash_loops",
55+ metric.WithDescription("Total crash loop detections")); err != nil {
56+ return nil, err
57+ }
58+ if m.ManagedContainers, err = meter.Int64UpDownCounter("gordon.container.managed",
59+ metric.WithDescription("Currently managed containers")); err != nil {
60+ return nil, err
61+ }
62+ if m.ImagePushTotal, err = meter.Int64Counter("gordon.registry.push.total",
63+ metric.WithDescription("Total image pushes")); err != nil {
64+ return nil, err
65+ }
66+ if m.ImagePushSize, err = meter.Int64Counter("gordon.registry.push.bytes",
67+ metric.WithDescription("Total bytes pushed to registry"),
68+ metric.WithUnit("By")); err != nil {
69+ return nil, err
70+ }
71+ if m.EventsProcessed, err = meter.Int64Counter("gordon.events.processed",
72+ metric.WithDescription("Total events processed")); err != nil {
73+ return nil, err
74+ }
75+ if m.EventsDropped, err = meter.Int64Counter("gordon.events.dropped",
76+ metric.WithDescription("Total events dropped")); err != nil {
77+ return nil, err
78+ }
79+
80+ return m, nil
81+}
A internal/adapters/out/telemetry/provider.go +213 -0
@@ -0,0 +1,213 @@
1+// Package telemetry provides OpenTelemetry initialization for Gordon.
2+// It configures trace, metric, and log providers that export via OTLP/HTTP.
3+package telemetry
4+
5+import (
6+ "context"
7+ "fmt"
8+ "net/url"
9+
10+ "go.opentelemetry.io/otel"
11+ "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
12+ "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
13+ "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
14+ otellog "go.opentelemetry.io/otel/sdk/log"
15+ "go.opentelemetry.io/otel/sdk/metric"
16+ "go.opentelemetry.io/otel/sdk/resource"
17+ "go.opentelemetry.io/otel/sdk/trace"
18+ semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
19+)
20+
21+// Config holds telemetry configuration.
22+type Config struct {
23+ Enabled bool `mapstructure:"enabled"`
24+ Endpoint string `mapstructure:"endpoint"` // OTLP HTTP endpoint, e.g. "http://localhost:4318"
25+ AuthToken string `mapstructure:"auth_token"` // Basic auth token (base64 encoded user:pass)
26+ Traces bool `mapstructure:"traces"` // Enable trace export
27+ Metrics bool `mapstructure:"metrics"` // Enable metric export
28+ Logs bool `mapstructure:"logs"` // Bridge zerowrap logs to OTLP
29+ TraceSampleRate float64 `mapstructure:"trace_sample_rate"` // 0.0-1.0, default 1.0
30+}
31+
32+// Provider holds the initialized OTel providers.
33+type Provider struct {
34+ TracerProvider *trace.TracerProvider
35+ MeterProvider *metric.MeterProvider
36+ LogProvider *otellog.LoggerProvider
37+}
38+
39+// endpointConfig holds parsed endpoint details shared across exporter setup.
40+type endpointConfig struct {
41+ host string
42+ basePath string
43+ insecure bool
44+ headers map[string]string
45+}
46+
47+// NewProvider creates and configures OTel providers based on the given config.
48+// Returns a noop provider if telemetry is disabled.
49+// The returned shutdown function must be called on application exit.
50+func NewProvider(ctx context.Context, cfg Config, serviceName, version string) (*Provider, func(context.Context), error) {
51+ noop := func(context.Context) {}
52+
53+ if !cfg.Enabled || cfg.Endpoint == "" {
54+ return &Provider{}, noop, nil
55+ }
56+
57+ res, err := resource.New(ctx,
58+ resource.WithAttributes(
59+ semconv.ServiceName(serviceName),
60+ semconv.ServiceVersion(version),
61+ ),
62+ resource.WithOS(),
63+ resource.WithHost(),
64+ )
65+ if err != nil {
66+ return nil, noop, fmt.Errorf("create resource: %w", err)
67+ }
68+
69+ ep, err := parseEndpoint(cfg)
70+ if err != nil {
71+ return nil, noop, err
72+ }
73+
74+ var shutdowns []func(context.Context) error
75+ p := &Provider{}
76+
77+ if cfg.Traces {
78+ if err := p.setupTracing(ctx, ep, cfg, res, &shutdowns); err != nil {
79+ return nil, noop, err
80+ }
81+ }
82+
83+ if cfg.Metrics {
84+ if err := p.setupMetrics(ctx, ep, res, &shutdowns); err != nil {
85+ return nil, noop, err
86+ }
87+ }
88+
89+ if cfg.Logs {
90+ if err := p.setupLogging(ctx, ep, res, &shutdowns); err != nil {
91+ return nil, noop, err
92+ }
93+ }
94+
95+ shutdown := func(ctx context.Context) {
96+ for _, fn := range shutdowns {
97+ _ = fn(ctx)
98+ }
99+ }
100+
101+ return p, shutdown, nil
102+}
103+
104+// parseEndpoint extracts host, path, and scheme from the configured endpoint URL.
105+func parseEndpoint(cfg Config) (*endpointConfig, error) {
106+ parsedURL, err := url.Parse(cfg.Endpoint)
107+ if err != nil {
108+ return nil, fmt.Errorf("parse endpoint URL: %w", err)
109+ }
110+
111+ headers := make(map[string]string)
112+ if cfg.AuthToken != "" {
113+ headers["Authorization"] = "Basic " + cfg.AuthToken
114+ }
115+
116+ return &endpointConfig{
117+ host: parsedURL.Host,
118+ basePath: parsedURL.Path,
119+ insecure: parsedURL.Scheme == "http",
120+ headers: headers,
121+ }, nil
122+}
123+
124+// setupTracing initializes the trace provider with an OTLP/HTTP exporter.
125+func (p *Provider) setupTracing(ctx context.Context, ep *endpointConfig, cfg Config, res *resource.Resource, shutdowns *[]func(context.Context) error) error {
126+ traceOpts := []otlptracehttp.Option{
127+ otlptracehttp.WithEndpoint(ep.host),
128+ otlptracehttp.WithHeaders(ep.headers),
129+ }
130+ if ep.basePath != "" {
131+ traceOpts = append(traceOpts, otlptracehttp.WithURLPath(ep.basePath+"/v1/traces"))
132+ }
133+ if ep.insecure {
134+ traceOpts = append(traceOpts, otlptracehttp.WithInsecure())
135+ }
136+
137+ traceExp, err := otlptracehttp.New(ctx, traceOpts...)
138+ if err != nil {
139+ return fmt.Errorf("create trace exporter: %w", err)
140+ }
141+
142+ sampler := trace.AlwaysSample()
143+ if cfg.TraceSampleRate == 0 {
144+ sampler = trace.NeverSample()
145+ } else if cfg.TraceSampleRate < 1.0 {
146+ sampler = trace.TraceIDRatioBased(cfg.TraceSampleRate)
147+ }
148+
149+ tp := trace.NewTracerProvider(
150+ trace.WithBatcher(traceExp),
151+ trace.WithResource(res),
152+ trace.WithSampler(sampler),
153+ )
154+ otel.SetTracerProvider(tp)
155+ p.TracerProvider = tp
156+ *shutdowns = append(*shutdowns, tp.Shutdown)
157+ return nil
158+}
159+
160+// setupMetrics initializes the meter provider with an OTLP/HTTP exporter.
161+func (p *Provider) setupMetrics(ctx context.Context, ep *endpointConfig, res *resource.Resource, shutdowns *[]func(context.Context) error) error {
162+ metricOpts := []otlpmetrichttp.Option{
163+ otlpmetrichttp.WithEndpoint(ep.host),
164+ otlpmetrichttp.WithHeaders(ep.headers),
165+ }
166+ if ep.basePath != "" {
167+ metricOpts = append(metricOpts, otlpmetrichttp.WithURLPath(ep.basePath+"/v1/metrics"))
168+ }
169+ if ep.insecure {
170+ metricOpts = append(metricOpts, otlpmetrichttp.WithInsecure())
171+ }
172+
173+ metricExp, err := otlpmetrichttp.New(ctx, metricOpts...)
174+ if err != nil {
175+ return fmt.Errorf("create metric exporter: %w", err)
176+ }
177+
178+ mp := metric.NewMeterProvider(
179+ metric.WithReader(metric.NewPeriodicReader(metricExp)),
180+ metric.WithResource(res),
181+ )
182+ otel.SetMeterProvider(mp)
183+ p.MeterProvider = mp
184+ *shutdowns = append(*shutdowns, mp.Shutdown)
185+ return nil
186+}
187+
188+// setupLogging initializes the log provider with an OTLP/HTTP exporter.
189+func (p *Provider) setupLogging(ctx context.Context, ep *endpointConfig, res *resource.Resource, shutdowns *[]func(context.Context) error) error {
190+ logOpts := []otlploghttp.Option{
191+ otlploghttp.WithEndpoint(ep.host),
192+ otlploghttp.WithHeaders(ep.headers),
193+ }
194+ if ep.basePath != "" {
195+ logOpts = append(logOpts, otlploghttp.WithURLPath(ep.basePath+"/v1/logs"))
196+ }
197+ if ep.insecure {
198+ logOpts = append(logOpts, otlploghttp.WithInsecure())
199+ }
200+
201+ logExp, err := otlploghttp.New(ctx, logOpts...)
202+ if err != nil {
203+ return fmt.Errorf("create log exporter: %w", err)
204+ }
205+
206+ lp := otellog.NewLoggerProvider(
207+ otellog.WithProcessor(otellog.NewBatchProcessor(logExp)),
208+ otellog.WithResource(res),
209+ )
210+ p.LogProvider = lp
211+ *shutdowns = append(*shutdowns, lp.Shutdown)
212+ return nil
213+}
@@ -0,0 +1,213 @@
1+// Package telemetry provides OpenTelemetry initialization for Gordon.
2+// It configures trace, metric, and log providers that export via OTLP/HTTP.
3+package telemetry
4+
5+import (
6+ "context"
7+ "fmt"
8+ "net/url"
9+
10+ "go.opentelemetry.io/otel"
11+ "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
12+ "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
13+ "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
14+ otellog "go.opentelemetry.io/otel/sdk/log"
15+ "go.opentelemetry.io/otel/sdk/metric"
16+ "go.opentelemetry.io/otel/sdk/resource"
17+ "go.opentelemetry.io/otel/sdk/trace"
18+ semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
19+)
20+
21+// Config holds telemetry configuration.
22+type Config struct {
23+ Enabled bool `mapstructure:"enabled"`
24+ Endpoint string `mapstructure:"endpoint"` // OTLP HTTP endpoint, e.g. "http://localhost:4318"
25+ AuthToken string `mapstructure:"auth_token"` // Basic auth token (base64 encoded user:pass)
26+ Traces bool `mapstructure:"traces"` // Enable trace export
27+ Metrics bool `mapstructure:"metrics"` // Enable metric export
28+ Logs bool `mapstructure:"logs"` // Bridge zerowrap logs to OTLP
29+ TraceSampleRate float64 `mapstructure:"trace_sample_rate"` // 0.0-1.0, default 1.0
30+}
31+
32+// Provider holds the initialized OTel providers.
33+type Provider struct {
34+ TracerProvider *trace.TracerProvider
35+ MeterProvider *metric.MeterProvider
36+ LogProvider *otellog.LoggerProvider
37+}
38+
39+// endpointConfig holds parsed endpoint details shared across exporter setup.
40+type endpointConfig struct {
41+ host string
42+ basePath string
43+ insecure bool
44+ headers map[string]string
45+}
46+
47+// NewProvider creates and configures OTel providers based on the given config.
48+// Returns a noop provider if telemetry is disabled.
49+// The returned shutdown function must be called on application exit.
50+func NewProvider(ctx context.Context, cfg Config, serviceName, version string) (*Provider, func(context.Context), error) {
51+ noop := func(context.Context) {}
52+
53+ if !cfg.Enabled || cfg.Endpoint == "" {
54+ return &Provider{}, noop, nil
55+ }
56+
57+ res, err := resource.New(ctx,
58+ resource.WithAttributes(
59+ semconv.ServiceName(serviceName),
60+ semconv.ServiceVersion(version),
61+ ),
62+ resource.WithOS(),
63+ resource.WithHost(),
64+ )
65+ if err != nil {
66+ return nil, noop, fmt.Errorf("create resource: %w", err)
67+ }
68+
69+ ep, err := parseEndpoint(cfg)
70+ if err != nil {
71+ return nil, noop, err
72+ }
73+
74+ var shutdowns []func(context.Context) error
75+ p := &Provider{}
76+
77+ if cfg.Traces {
78+ if err := p.setupTracing(ctx, ep, cfg, res, &shutdowns); err != nil {
79+ return nil, noop, err
80+ }
81+ }
82+
83+ if cfg.Metrics {
84+ if err := p.setupMetrics(ctx, ep, res, &shutdowns); err != nil {
85+ return nil, noop, err
86+ }
87+ }
88+
89+ if cfg.Logs {
90+ if err := p.setupLogging(ctx, ep, res, &shutdowns); err != nil {
91+ return nil, noop, err
92+ }
93+ }
94+
95+ shutdown := func(ctx context.Context) {
96+ for _, fn := range shutdowns {
97+ _ = fn(ctx)
98+ }
99+ }
100+
101+ return p, shutdown, nil
102+}
103+
104+// parseEndpoint extracts host, path, and scheme from the configured endpoint URL.
105+func parseEndpoint(cfg Config) (*endpointConfig, error) {
106+ parsedURL, err := url.Parse(cfg.Endpoint)
107+ if err != nil {
108+ return nil, fmt.Errorf("parse endpoint URL: %w", err)
109+ }
110+
111+ headers := make(map[string]string)
112+ if cfg.AuthToken != "" {
113+ headers["Authorization"] = "Basic " + cfg.AuthToken
114+ }
115+
116+ return &endpointConfig{
117+ host: parsedURL.Host,
118+ basePath: parsedURL.Path,
119+ insecure: parsedURL.Scheme == "http",
120+ headers: headers,
121+ }, nil
122+}
123+
124+// setupTracing initializes the trace provider with an OTLP/HTTP exporter.
125+func (p *Provider) setupTracing(ctx context.Context, ep *endpointConfig, cfg Config, res *resource.Resource, shutdowns *[]func(context.Context) error) error {
126+ traceOpts := []otlptracehttp.Option{
127+ otlptracehttp.WithEndpoint(ep.host),
128+ otlptracehttp.WithHeaders(ep.headers),
129+ }
130+ if ep.basePath != "" {
131+ traceOpts = append(traceOpts, otlptracehttp.WithURLPath(ep.basePath+"/v1/traces"))
132+ }
133+ if ep.insecure {
134+ traceOpts = append(traceOpts, otlptracehttp.WithInsecure())
135+ }
136+
137+ traceExp, err := otlptracehttp.New(ctx, traceOpts...)
138+ if err != nil {
139+ return fmt.Errorf("create trace exporter: %w", err)
140+ }
141+
142+ sampler := trace.AlwaysSample()
143+ if cfg.TraceSampleRate == 0 {
144+ sampler = trace.NeverSample()
145+ } else if cfg.TraceSampleRate < 1.0 {
146+ sampler = trace.TraceIDRatioBased(cfg.TraceSampleRate)
147+ }
148+
149+ tp := trace.NewTracerProvider(
150+ trace.WithBatcher(traceExp),
151+ trace.WithResource(res),
152+ trace.WithSampler(sampler),
153+ )
154+ otel.SetTracerProvider(tp)
155+ p.TracerProvider = tp
156+ *shutdowns = append(*shutdowns, tp.Shutdown)
157+ return nil
158+}
159+
160+// setupMetrics initializes the meter provider with an OTLP/HTTP exporter.
161+func (p *Provider) setupMetrics(ctx context.Context, ep *endpointConfig, res *resource.Resource, shutdowns *[]func(context.Context) error) error {
162+ metricOpts := []otlpmetrichttp.Option{
163+ otlpmetrichttp.WithEndpoint(ep.host),
164+ otlpmetrichttp.WithHeaders(ep.headers),
165+ }
166+ if ep.basePath != "" {
167+ metricOpts = append(metricOpts, otlpmetrichttp.WithURLPath(ep.basePath+"/v1/metrics"))
168+ }
169+ if ep.insecure {
170+ metricOpts = append(metricOpts, otlpmetrichttp.WithInsecure())
171+ }
172+
173+ metricExp, err := otlpmetrichttp.New(ctx, metricOpts...)
174+ if err != nil {
175+ return fmt.Errorf("create metric exporter: %w", err)
176+ }
177+
178+ mp := metric.NewMeterProvider(
179+ metric.WithReader(metric.NewPeriodicReader(metricExp)),
180+ metric.WithResource(res),
181+ )
182+ otel.SetMeterProvider(mp)
183+ p.MeterProvider = mp
184+ *shutdowns = append(*shutdowns, mp.Shutdown)
185+ return nil
186+}
187+
188+// setupLogging initializes the log provider with an OTLP/HTTP exporter.
189+func (p *Provider) setupLogging(ctx context.Context, ep *endpointConfig, res *resource.Resource, shutdowns *[]func(context.Context) error) error {
190+ logOpts := []otlploghttp.Option{
191+ otlploghttp.WithEndpoint(ep.host),
192+ otlploghttp.WithHeaders(ep.headers),
193+ }
194+ if ep.basePath != "" {
195+ logOpts = append(logOpts, otlploghttp.WithURLPath(ep.basePath+"/v1/logs"))
196+ }
197+ if ep.insecure {
198+ logOpts = append(logOpts, otlploghttp.WithInsecure())
199+ }
200+
201+ logExp, err := otlploghttp.New(ctx, logOpts...)
202+ if err != nil {
203+ return fmt.Errorf("create log exporter: %w", err)
204+ }
205+
206+ lp := otellog.NewLoggerProvider(
207+ otellog.WithProcessor(otellog.NewBatchProcessor(logExp)),
208+ otellog.WithResource(res),
209+ )
210+ p.LogProvider = lp
211+ *shutdowns = append(*shutdowns, lp.Shutdown)
212+ return nil
213+}
M internal/app/run.go +52 -2
@@ -25,6 +25,7 @@ import (
2525 "time"
2626
2727 "github.com/bnema/zerowrap"
28+ zerowrapotel "github.com/bnema/zerowrap/otel"
2829 "github.com/fsnotify/fsnotify"
2930 "github.com/spf13/viper"
3031
@@ -38,8 +39,12 @@ import (
3839 "github.com/bnema/gordon/internal/adapters/out/logwriter"
3940 "github.com/bnema/gordon/internal/adapters/out/ratelimit"
4041 "github.com/bnema/gordon/internal/adapters/out/secrets"
42+ "github.com/bnema/gordon/internal/adapters/out/telemetry"
4143 "github.com/bnema/gordon/internal/adapters/out/tokenstore"
4244
45+ // OTel
46+ "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
47+
4348 // Adapters - Input
4449 "github.com/bnema/gordon/internal/adapters/dto"
4550 "github.com/bnema/gordon/internal/adapters/in/http/admin"
@@ -153,6 +158,8 @@ type Config struct {
153158 KeepLast int `mapstructure:"keep_last"`
154159 } `mapstructure:"prune"`
155160 } `mapstructure:"images"`
161+
162+ Telemetry telemetry.Config `mapstructure:"telemetry"`
156163 }
157164
158165 // services holds all the services used by the application.
@@ -201,6 +208,24 @@ func Run(ctx context.Context, configPath string) error {
201208 }
202209
203210 ctx = zerowrap.WithCtx(ctx, log)
211+
212+ // Initialize OpenTelemetry
213+ telProvider, telShutdown, err := telemetry.NewProvider(ctx, cfg.Telemetry, "gordon", "dev")
214+ if err != nil {
215+ log.Warn().Err(err).Msg("failed to initialize telemetry, continuing without it")
216+ } else {
217+ defer telShutdown(ctx)
218+ if cfg.Telemetry.Enabled {
219+ // Bridge zerowrap logs to OTel if log export is enabled
220+ if cfg.Telemetry.Logs && telProvider.LogProvider != nil {
221+ otelHook := zerowrapotel.NewHookWithProvider(telProvider.LogProvider, "gordon")
222+ log = zerowrap.WithHook(log, otelHook)
223+ ctx = zerowrap.WithCtx(ctx, log)
224+ }
225+ log.Info().Str("endpoint", cfg.Telemetry.Endpoint).Msg("telemetry initialized")
226+ }
227+ }
228+
204229 log.Info().Msg("Gordon starting")
205230
206231 if err := ensureTLSConfig(&cfg, log); err != nil {
@@ -470,6 +495,16 @@ func createServices(ctx context.Context, v *viper.Viper, cfg Config, log zerowra
470495 svc.registrySvc = registrySvc.NewService(svc.blobStorage, svc.manifestStorage, svc.eventBus)
471496 svc.imageSvc = images.NewService(svc.runtime, svc.manifestStorage, svc.blobStorage, log)
472497
498+ // Initialize and inject telemetry metrics into services
499+ gordonMetrics, err := telemetry.NewMetrics()
500+ if err != nil {
501+ log.Warn().Err(err).Msg("failed to create telemetry metrics, continuing without metrics")
502+ } else {
503+ svc.containerSvc.SetMetrics(gordonMetrics)
504+ svc.registrySvc.SetMetrics(gordonMetrics)
505+ svc.eventBus.SetMetrics(gordonMetrics)
506+ }
507+
473508 proxyCfg, err := buildProxyConfig(cfg, log)
474509 if err != nil {
475510 return nil, err
@@ -1328,7 +1363,10 @@ func createHTTPHandlers(svc *services, cfg Config, log zerowrap.Logger) (http.Ha
13281363 middleware.SecurityHeaders,
13291364 }
13301365
1331- proxyWithMiddleware := middleware.Chain(proxyMiddlewares...)(svc.proxySvc)
1366+ proxyWithMiddleware := otelhttp.NewHandler(
1367+ middleware.Chain(proxyMiddlewares...)(svc.proxySvc),
1368+ "gordon.proxy",
1369+ )
13321370 proxyMux := http.NewServeMux()
13331371 proxyMux.Handle("/", proxyWithMiddleware)
13341372
@@ -1360,7 +1398,11 @@ func buildRegistryHandlerWithMiddleware(
13601398
13611399 appendRegistryAuthMiddleware(&registryMiddlewares, svc, cfg, log)
13621400
1363- return middleware.Chain(registryMiddlewares...)(registryHandler), cidrAllowlistMiddleware, rateLimitMiddleware
1401+ registryWithOtel := otelhttp.NewHandler(
1402+ middleware.Chain(registryMiddlewares...)(registryHandler),
1403+ "gordon.registry",
1404+ )
1405+ return registryWithOtel, cidrAllowlistMiddleware, rateLimitMiddleware
13641406 }
13651407
13661408 func buildRegistryCIDRAllowlistMiddleware(cfg Config, trustedNets []*net.IPNet, log zerowrap.Logger) func(http.Handler) http.Handler {
@@ -2219,6 +2261,14 @@ func loadConfig(v *viper.Viper, configPath string) error {
22192261 v.SetDefault("images.prune.enabled", false)
22202262 v.SetDefault("images.prune.schedule", string(domain.ScheduleDaily))
22212263 v.SetDefault("images.prune.keep_last", domain.DefaultImagePruneKeepLast)
2264+ v.SetDefault("telemetry.enabled", false)
2265+ v.SetDefault("telemetry.endpoint", "")
2266+ v.SetDefault("telemetry.auth_token", "")
2267+ v.SetDefault("telemetry.traces", true)
2268+ v.SetDefault("telemetry.metrics", true)
2269+ v.SetDefault("telemetry.logs", true)
2270+ v.SetDefault("telemetry.trace_sample_rate", 1.0)
2271+
22222272 v.SetDefault("server.max_concurrent_connections", -1) // -1 = use default (10000), 0 = no limit
22232273 v.SetDefault("server.registry_allowed_ips", []string{})
22242274 v.SetDefault("deploy.readiness_delay", "5s")
@@ -25,6 +25,7 @@ import (
25 "time"25 "time"
26 26
27 "github.com/bnema/zerowrap"27 "github.com/bnema/zerowrap"
28+ zerowrapotel "github.com/bnema/zerowrap/otel"
28 "github.com/fsnotify/fsnotify"29 "github.com/fsnotify/fsnotify"
29 "github.com/spf13/viper"30 "github.com/spf13/viper"
30 31
@@ -38,8 +39,12 @@ import (
38 "github.com/bnema/gordon/internal/adapters/out/logwriter"39 "github.com/bnema/gordon/internal/adapters/out/logwriter"
39 "github.com/bnema/gordon/internal/adapters/out/ratelimit"40 "github.com/bnema/gordon/internal/adapters/out/ratelimit"
40 "github.com/bnema/gordon/internal/adapters/out/secrets"41 "github.com/bnema/gordon/internal/adapters/out/secrets"
42+ "github.com/bnema/gordon/internal/adapters/out/telemetry"
41 "github.com/bnema/gordon/internal/adapters/out/tokenstore"43 "github.com/bnema/gordon/internal/adapters/out/tokenstore"
42 44
45+ // OTel
46+ "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
47+
43 // Adapters - Input48 // Adapters - Input
44 "github.com/bnema/gordon/internal/adapters/dto"49 "github.com/bnema/gordon/internal/adapters/dto"
45 "github.com/bnema/gordon/internal/adapters/in/http/admin"50 "github.com/bnema/gordon/internal/adapters/in/http/admin"
@@ -153,6 +158,8 @@ type Config struct {
153 KeepLast int `mapstructure:"keep_last"`158 KeepLast int `mapstructure:"keep_last"`
154 } `mapstructure:"prune"`159 } `mapstructure:"prune"`
155 } `mapstructure:"images"`160 } `mapstructure:"images"`
161+
162+ Telemetry telemetry.Config `mapstructure:"telemetry"`
156 }163 }
157 164
158 // services holds all the services used by the application.165 // services holds all the services used by the application.
@@ -201,6 +208,24 @@ func Run(ctx context.Context, configPath string) error {
201 }208 }
202 209
203 ctx = zerowrap.WithCtx(ctx, log)210 ctx = zerowrap.WithCtx(ctx, log)
211+
212+ // Initialize OpenTelemetry
213+ telProvider, telShutdown, err := telemetry.NewProvider(ctx, cfg.Telemetry, "gordon", "dev")
214+ if err != nil {
215+ log.Warn().Err(err).Msg("failed to initialize telemetry, continuing without it")
216+ } else {
217+ defer telShutdown(ctx)
218+ if cfg.Telemetry.Enabled {
219+ // Bridge zerowrap logs to OTel if log export is enabled
220+ if cfg.Telemetry.Logs && telProvider.LogProvider != nil {
221+ otelHook := zerowrapotel.NewHookWithProvider(telProvider.LogProvider, "gordon")
222+ log = zerowrap.WithHook(log, otelHook)
223+ ctx = zerowrap.WithCtx(ctx, log)
224+ }
225+ log.Info().Str("endpoint", cfg.Telemetry.Endpoint).Msg("telemetry initialized")
226+ }
227+ }
228+
204 log.Info().Msg("Gordon starting")229 log.Info().Msg("Gordon starting")
205 230
206 if err := ensureTLSConfig(&cfg, log); err != nil {231 if err := ensureTLSConfig(&cfg, log); err != nil {
@@ -470,6 +495,16 @@ func createServices(ctx context.Context, v *viper.Viper, cfg Config, log zerowra
470 svc.registrySvc = registrySvc.NewService(svc.blobStorage, svc.manifestStorage, svc.eventBus)495 svc.registrySvc = registrySvc.NewService(svc.blobStorage, svc.manifestStorage, svc.eventBus)
471 svc.imageSvc = images.NewService(svc.runtime, svc.manifestStorage, svc.blobStorage, log)496 svc.imageSvc = images.NewService(svc.runtime, svc.manifestStorage, svc.blobStorage, log)
472 497
498+ // Initialize and inject telemetry metrics into services
499+ gordonMetrics, err := telemetry.NewMetrics()
500+ if err != nil {
501+ log.Warn().Err(err).Msg("failed to create telemetry metrics, continuing without metrics")
502+ } else {
503+ svc.containerSvc.SetMetrics(gordonMetrics)
504+ svc.registrySvc.SetMetrics(gordonMetrics)
505+ svc.eventBus.SetMetrics(gordonMetrics)
506+ }
507+
473 proxyCfg, err := buildProxyConfig(cfg, log)508 proxyCfg, err := buildProxyConfig(cfg, log)
474 if err != nil {509 if err != nil {
475 return nil, err510 return nil, err
@@ -1328,7 +1363,10 @@ func createHTTPHandlers(svc *services, cfg Config, log zerowrap.Logger) (http.Ha
1328 middleware.SecurityHeaders,1363 middleware.SecurityHeaders,
1329 }1364 }
1330 1365
1331- proxyWithMiddleware := middleware.Chain(proxyMiddlewares...)(svc.proxySvc)1366+ proxyWithMiddleware := otelhttp.NewHandler(
1367+ middleware.Chain(proxyMiddlewares...)(svc.proxySvc),
1368+ "gordon.proxy",
1369+ )
1332 proxyMux := http.NewServeMux()1370 proxyMux := http.NewServeMux()
1333 proxyMux.Handle("/", proxyWithMiddleware)1371 proxyMux.Handle("/", proxyWithMiddleware)
1334 1372
@@ -1360,7 +1398,11 @@ func buildRegistryHandlerWithMiddleware(
1360 1398
1361 appendRegistryAuthMiddleware(&registryMiddlewares, svc, cfg, log)1399 appendRegistryAuthMiddleware(&registryMiddlewares, svc, cfg, log)
1362 1400
1363- return middleware.Chain(registryMiddlewares...)(registryHandler), cidrAllowlistMiddleware, rateLimitMiddleware1401+ registryWithOtel := otelhttp.NewHandler(
1402+ middleware.Chain(registryMiddlewares...)(registryHandler),
1403+ "gordon.registry",
1404+ )
1405+ return registryWithOtel, cidrAllowlistMiddleware, rateLimitMiddleware
1364 }1406 }
1365 1407
1366 func buildRegistryCIDRAllowlistMiddleware(cfg Config, trustedNets []*net.IPNet, log zerowrap.Logger) func(http.Handler) http.Handler {1408 func buildRegistryCIDRAllowlistMiddleware(cfg Config, trustedNets []*net.IPNet, log zerowrap.Logger) func(http.Handler) http.Handler {
@@ -2219,6 +2261,14 @@ func loadConfig(v *viper.Viper, configPath string) error {
2219 v.SetDefault("images.prune.enabled", false)2261 v.SetDefault("images.prune.enabled", false)
2220 v.SetDefault("images.prune.schedule", string(domain.ScheduleDaily))2262 v.SetDefault("images.prune.schedule", string(domain.ScheduleDaily))
2221 v.SetDefault("images.prune.keep_last", domain.DefaultImagePruneKeepLast)2263 v.SetDefault("images.prune.keep_last", domain.DefaultImagePruneKeepLast)
2264+ v.SetDefault("telemetry.enabled", false)
2265+ v.SetDefault("telemetry.endpoint", "")
2266+ v.SetDefault("telemetry.auth_token", "")
2267+ v.SetDefault("telemetry.traces", true)
2268+ v.SetDefault("telemetry.metrics", true)
2269+ v.SetDefault("telemetry.logs", true)
2270+ v.SetDefault("telemetry.trace_sample_rate", 1.0)
2271+
2222 v.SetDefault("server.max_concurrent_connections", -1) // -1 = use default (10000), 0 = no limit2272 v.SetDefault("server.max_concurrent_connections", -1) // -1 = use default (10000), 0 = no limit
2223 v.SetDefault("server.registry_allowed_ips", []string{})2273 v.SetDefault("server.registry_allowed_ips", []string{})
2224 v.SetDefault("deploy.readiness_delay", "5s")2274 v.SetDefault("deploy.readiness_delay", "5s")
M internal/usecase/container/monitor.go +17 -0
@@ -6,6 +6,8 @@ import (
66 "time"
77
88 "github.com/bnema/zerowrap"
9+ "go.opentelemetry.io/otel/attribute"
10+ "go.opentelemetry.io/otel/metric"
911
1012 "github.com/bnema/gordon/internal/domain"
1113 )
@@ -199,6 +201,12 @@ func (m *Monitor) handleCrash(ctx context.Context, log zerowrap.Logger, domainNa
199201 Int("consecutive", rec.consecutive).
200202 Dur("backoff", backoff).
201203 Msg("monitor: crash loop detected, backing off")
204+
205+ // Record crash loop metric
206+ if m.service.metrics != nil {
207+ attrs := metric.WithAttributes(attribute.String("domain", domainName))
208+ m.service.metrics.ContainerCrashLoops.Add(ctx, 1, attrs)
209+ }
202210 return
203211 }
204212 m.mu.Unlock()
@@ -216,6 +224,15 @@ func (m *Monitor) handleCrash(ctx context.Context, log zerowrap.Logger, domainNa
216224 Int("exit_code", exitCode).
217225 Msg("monitor: restarting crashed container")
218226
227+ // Record restart metric
228+ if m.service.metrics != nil {
229+ attrs := metric.WithAttributes(
230+ attribute.String("domain", domainName),
231+ attribute.String("source", "monitor"),
232+ )
233+ m.service.metrics.ContainerRestarts.Add(ctx, 1, attrs)
234+ }
235+
219236 if err := m.service.runtime.StartContainer(ctx, containerID); err != nil {
220237 log.Warn().Err(err).Str("domain", domainName).Msg("monitor: failed to restart container")
221238 }
@@ -6,6 +6,8 @@ import (
6 "time"6 "time"
7 7
8 "github.com/bnema/zerowrap"8 "github.com/bnema/zerowrap"
9+ "go.opentelemetry.io/otel/attribute"
10+ "go.opentelemetry.io/otel/metric"
9 11
10 "github.com/bnema/gordon/internal/domain"12 "github.com/bnema/gordon/internal/domain"
11 )13 )
@@ -199,6 +201,12 @@ func (m *Monitor) handleCrash(ctx context.Context, log zerowrap.Logger, domainNa
199 Int("consecutive", rec.consecutive).201 Int("consecutive", rec.consecutive).
200 Dur("backoff", backoff).202 Dur("backoff", backoff).
201 Msg("monitor: crash loop detected, backing off")203 Msg("monitor: crash loop detected, backing off")
204+
205+ // Record crash loop metric
206+ if m.service.metrics != nil {
207+ attrs := metric.WithAttributes(attribute.String("domain", domainName))
208+ m.service.metrics.ContainerCrashLoops.Add(ctx, 1, attrs)
209+ }
202 return210 return
203 }211 }
204 m.mu.Unlock()212 m.mu.Unlock()
@@ -216,6 +224,15 @@ func (m *Monitor) handleCrash(ctx context.Context, log zerowrap.Logger, domainNa
216 Int("exit_code", exitCode).224 Int("exit_code", exitCode).
217 Msg("monitor: restarting crashed container")225 Msg("monitor: restarting crashed container")
218 226
227+ // Record restart metric
228+ if m.service.metrics != nil {
229+ attrs := metric.WithAttributes(
230+ attribute.String("domain", domainName),
231+ attribute.String("source", "monitor"),
232+ )
233+ m.service.metrics.ContainerRestarts.Add(ctx, 1, attrs)
234+ }
235+
219 if err := m.service.runtime.StartContainer(ctx, containerID); err != nil {236 if err := m.service.runtime.StartContainer(ctx, containerID); err != nil {
220 log.Warn().Err(err).Str("domain", domainName).Msg("monitor: failed to restart container")237 log.Warn().Err(err).Str("domain", domainName).Msg("monitor: failed to restart container")
221 }238 }
M internal/usecase/container/service.go +73 -0
@@ -12,7 +12,13 @@ import (
1212 "time"
1313
1414 "github.com/bnema/zerowrap"
15+ "go.opentelemetry.io/otel"
16+ "go.opentelemetry.io/otel/attribute"
17+ "go.opentelemetry.io/otel/codes"
18+ "go.opentelemetry.io/otel/metric"
19+ "go.opentelemetry.io/otel/trace"
1520
21+ "github.com/bnema/gordon/internal/adapters/out/telemetry"
1622 "github.com/bnema/gordon/internal/boundaries/out"
1723 "github.com/bnema/gordon/internal/domain"
1824 )
@@ -44,6 +50,8 @@ type Config struct {
4450 DrainTimeout time.Duration // Max wait for in-flight requests to drain
4551 }
4652
53+var tracer = otel.Tracer("gordon.container")
54+
4755 const (
4856 PullPolicyAlways = "always"
4957 PullPolicyIfNotPresent = "if-not-present"
@@ -65,6 +73,7 @@ type Service struct {
6573 cacheInvalidator out.ProxyCacheInvalidator
6674 drainWaiter out.ProxyDrainWaiter
6775 config Config
76+ metrics *telemetry.Metrics
6877 containers map[string]*domain.Container
6978 attachments map[string][]string // ownerDomain → []containerIDs
7079 mu sync.RWMutex
@@ -135,6 +144,11 @@ func NewService(
135144 }
136145 }
137146
147+// SetMetrics sets the telemetry metrics for the container service.
148+func (s *Service) SetMetrics(m *telemetry.Metrics) {
149+ s.metrics = m
150+}
151+
138152 // SetProxyCacheInvalidator sets the proxy cache invalidator for synchronous
139153 // cache invalidation during zero-downtime deployments. This must be called
140154 // after construction because the proxy service is created after the container
@@ -216,6 +230,15 @@ func (s *Service) buildContainerConfig(containerDomain, image, actualImageRef st
216230 // Deploy creates and starts a container for the given route.
217231 // Implements zero-downtime deployment: new container starts before old one stops.
218232 func (s *Service) Deploy(ctx context.Context, route domain.Route) (*domain.Container, error) {
233+ ctx, span := tracer.Start(ctx, "container.deploy",
234+ trace.WithAttributes(
235+ attribute.String("domain", route.Domain),
236+ attribute.String("image", route.Image),
237+ ))
238+ defer span.End()
239+
240+ deployStart := time.Now()
241+
219242 // Serialize deploys for the same domain to prevent race conditions
220243 // (e.g. multiple image.pushed events + explicit deploy call from CLI).
221244 unlock, err := s.acquireDomainDeployLock(ctx, route.Domain)
@@ -232,6 +255,26 @@ func (s *Service) Deploy(ctx context.Context, route domain.Route) (*domain.Conta
232255 })
233256 log := zerowrap.FromCtx(ctx)
234257
258+ // Record deploy metrics and trace status
259+ defer func() {
260+ if err != nil {
261+ span.RecordError(err)
262+ span.SetStatus(codes.Error, err.Error())
263+ }
264+ if s.metrics == nil {
265+ return
266+ }
267+ attrs := metric.WithAttributes(
268+ attribute.String("domain", route.Domain),
269+ attribute.String("image", route.Image),
270+ )
271+ s.metrics.DeployTotal.Add(ctx, 1, attrs)
272+ s.metrics.DeployDuration.Record(ctx, time.Since(deployStart).Seconds(), attrs)
273+ if err != nil {
274+ s.metrics.DeployErrors.Add(ctx, 1, attrs)
275+ }
276+ }()
277+
235278 existing, hasExisting := s.getTrackedContainer(route.Domain)
236279
237280 resources, err := s.prepareDeployResources(ctx, route, existing)
@@ -369,6 +412,9 @@ func (s *Service) prepareDeployResources(ctx context.Context, route domain.Route
369412 }
370413
371414 func (s *Service) createStartedContainer(ctx context.Context, route domain.Route, existing *domain.Container, resources *deployResources) (*domain.Container, error) {
415+ ctx, span := tracer.Start(ctx, "container.create_and_start")
416+ defer span.End()
417+
372418 log := zerowrap.FromCtx(ctx)
373419
374420 containerConfig := s.buildContainerConfig(
@@ -408,9 +454,17 @@ func (s *Service) createStartedContainer(ctx context.Context, route domain.Route
408454
409455 func (s *Service) activateDeployedContainer(ctx context.Context, domainName string, container *domain.Container) bool {
410456 s.mu.Lock()
457+ _, wasTracked := s.containers[domainName]
411458 s.containers[domainName] = container
412459 s.mu.Unlock()
413460
461+ // Track managed container count (only increment for new domains, not replacements)
462+ if !wasTracked && s.metrics != nil {
463+ s.metrics.ManagedContainers.Add(ctx, 1, metric.WithAttributes(
464+ attribute.String("domain", domainName),
465+ ))
466+ }
467+
414468 s.publishContainerDeployed(ctx, domainName, container.ID)
415469
416470 s.mu.RLock()
@@ -539,6 +593,14 @@ func (s *Service) Restart(ctx context.Context, domainName string, withAttachment
539593 }
540594 log.Info().Str(zerowrap.FieldEntityID, container.ID).Msg("container restarted")
541595
596+ // Record restart metric
597+ if s.metrics != nil {
598+ s.metrics.ContainerRestarts.Add(ctx, 1, metric.WithAttributes(
599+ attribute.String("domain", domainName),
600+ attribute.String("source", "api"),
601+ ))
602+ }
603+
542604 // Restart attachments if requested
543605 if withAttachments && len(attachmentIDs) > 0 {
544606 for _, attachID := range attachmentIDs {
@@ -626,6 +688,13 @@ func (s *Service) Remove(ctx context.Context, containerID string, force bool) er
626688 }
627689 s.mu.Unlock()
628690
691+ // Decrement managed container count
692+ if removedDomain != "" && s.metrics != nil {
693+ s.metrics.ManagedContainers.Add(ctx, -1, metric.WithAttributes(
694+ attribute.String("domain", removedDomain),
695+ ))
696+ }
697+
629698 // Note: We intentionally do NOT clean up deployMu entries to avoid a race condition:
630699 // If Remove deletes the mutex entry while a concurrent Deploy is acquiring the lock,
631700 // the Deploy would create a fresh mutex, breaking serialization and allowing concurrent
@@ -1161,6 +1230,10 @@ func (s *Service) pullRefForDeploy(ctx context.Context, imageRef string) (string
11611230 // - For digest references (@sha256:...), returns the pullRef since Docker can't tag digests
11621231 // - For tagged images, returns the original imageRef after tagging the pulled image
11631232 func (s *Service) ensureImage(ctx context.Context, imageRef string) (string, error) {
1233+ ctx, span := tracer.Start(ctx, "container.ensure_image",
1234+ trace.WithAttributes(attribute.String("image", imageRef)))
1235+ defer span.End()
1236+
11641237 ctx = zerowrap.CtxWithField(ctx, "image", imageRef)
11651238 log := zerowrap.FromCtx(ctx)
11661239
@@ -12,7 +12,13 @@ import (
12 "time"12 "time"
13 13
14 "github.com/bnema/zerowrap"14 "github.com/bnema/zerowrap"
15+ "go.opentelemetry.io/otel"
16+ "go.opentelemetry.io/otel/attribute"
17+ "go.opentelemetry.io/otel/codes"
18+ "go.opentelemetry.io/otel/metric"
19+ "go.opentelemetry.io/otel/trace"
15 20
21+ "github.com/bnema/gordon/internal/adapters/out/telemetry"
16 "github.com/bnema/gordon/internal/boundaries/out"22 "github.com/bnema/gordon/internal/boundaries/out"
17 "github.com/bnema/gordon/internal/domain"23 "github.com/bnema/gordon/internal/domain"
18 )24 )
@@ -44,6 +50,8 @@ type Config struct {
44 DrainTimeout time.Duration // Max wait for in-flight requests to drain50 DrainTimeout time.Duration // Max wait for in-flight requests to drain
45 }51 }
46 52
53+var tracer = otel.Tracer("gordon.container")
54+
47 const (55 const (
48 PullPolicyAlways = "always"56 PullPolicyAlways = "always"
49 PullPolicyIfNotPresent = "if-not-present"57 PullPolicyIfNotPresent = "if-not-present"
@@ -65,6 +73,7 @@ type Service struct {
65 cacheInvalidator out.ProxyCacheInvalidator73 cacheInvalidator out.ProxyCacheInvalidator
66 drainWaiter out.ProxyDrainWaiter74 drainWaiter out.ProxyDrainWaiter
67 config Config75 config Config
76+ metrics *telemetry.Metrics
68 containers map[string]*domain.Container77 containers map[string]*domain.Container
69 attachments map[string][]string // ownerDomain → []containerIDs78 attachments map[string][]string // ownerDomain → []containerIDs
70 mu sync.RWMutex79 mu sync.RWMutex
@@ -135,6 +144,11 @@ func NewService(
135 }144 }
136 }145 }
137 146
147+// SetMetrics sets the telemetry metrics for the container service.
148+func (s *Service) SetMetrics(m *telemetry.Metrics) {
149+ s.metrics = m
150+}
151+
138 // SetProxyCacheInvalidator sets the proxy cache invalidator for synchronous152 // SetProxyCacheInvalidator sets the proxy cache invalidator for synchronous
139 // cache invalidation during zero-downtime deployments. This must be called153 // cache invalidation during zero-downtime deployments. This must be called
140 // after construction because the proxy service is created after the container154 // after construction because the proxy service is created after the container
@@ -216,6 +230,15 @@ func (s *Service) buildContainerConfig(containerDomain, image, actualImageRef st
216 // Deploy creates and starts a container for the given route.230 // Deploy creates and starts a container for the given route.
217 // Implements zero-downtime deployment: new container starts before old one stops.231 // Implements zero-downtime deployment: new container starts before old one stops.
218 func (s *Service) Deploy(ctx context.Context, route domain.Route) (*domain.Container, error) {232 func (s *Service) Deploy(ctx context.Context, route domain.Route) (*domain.Container, error) {
233+ ctx, span := tracer.Start(ctx, "container.deploy",
234+ trace.WithAttributes(
235+ attribute.String("domain", route.Domain),
236+ attribute.String("image", route.Image),
237+ ))
238+ defer span.End()
239+
240+ deployStart := time.Now()
241+
219 // Serialize deploys for the same domain to prevent race conditions242 // Serialize deploys for the same domain to prevent race conditions
220 // (e.g. multiple image.pushed events + explicit deploy call from CLI).243 // (e.g. multiple image.pushed events + explicit deploy call from CLI).
221 unlock, err := s.acquireDomainDeployLock(ctx, route.Domain)244 unlock, err := s.acquireDomainDeployLock(ctx, route.Domain)
@@ -232,6 +255,26 @@ func (s *Service) Deploy(ctx context.Context, route domain.Route) (*domain.Conta
232 })255 })
233 log := zerowrap.FromCtx(ctx)256 log := zerowrap.FromCtx(ctx)
234 257
258+ // Record deploy metrics and trace status
259+ defer func() {
260+ if err != nil {
261+ span.RecordError(err)
262+ span.SetStatus(codes.Error, err.Error())
263+ }
264+ if s.metrics == nil {
265+ return
266+ }
267+ attrs := metric.WithAttributes(
268+ attribute.String("domain", route.Domain),
269+ attribute.String("image", route.Image),
270+ )
271+ s.metrics.DeployTotal.Add(ctx, 1, attrs)
272+ s.metrics.DeployDuration.Record(ctx, time.Since(deployStart).Seconds(), attrs)
273+ if err != nil {
274+ s.metrics.DeployErrors.Add(ctx, 1, attrs)
275+ }
276+ }()
277+
235 existing, hasExisting := s.getTrackedContainer(route.Domain)278 existing, hasExisting := s.getTrackedContainer(route.Domain)
236 279
237 resources, err := s.prepareDeployResources(ctx, route, existing)280 resources, err := s.prepareDeployResources(ctx, route, existing)
@@ -369,6 +412,9 @@ func (s *Service) prepareDeployResources(ctx context.Context, route domain.Route
369 }412 }
370 413
371 func (s *Service) createStartedContainer(ctx context.Context, route domain.Route, existing *domain.Container, resources *deployResources) (*domain.Container, error) {414 func (s *Service) createStartedContainer(ctx context.Context, route domain.Route, existing *domain.Container, resources *deployResources) (*domain.Container, error) {
415+ ctx, span := tracer.Start(ctx, "container.create_and_start")
416+ defer span.End()
417+
372 log := zerowrap.FromCtx(ctx)418 log := zerowrap.FromCtx(ctx)
373 419
374 containerConfig := s.buildContainerConfig(420 containerConfig := s.buildContainerConfig(
@@ -408,9 +454,17 @@ func (s *Service) createStartedContainer(ctx context.Context, route domain.Route
408 454
409 func (s *Service) activateDeployedContainer(ctx context.Context, domainName string, container *domain.Container) bool {455 func (s *Service) activateDeployedContainer(ctx context.Context, domainName string, container *domain.Container) bool {
410 s.mu.Lock()456 s.mu.Lock()
457+ _, wasTracked := s.containers[domainName]
411 s.containers[domainName] = container458 s.containers[domainName] = container
412 s.mu.Unlock()459 s.mu.Unlock()
413 460
461+ // Track managed container count (only increment for new domains, not replacements)
462+ if !wasTracked && s.metrics != nil {
463+ s.metrics.ManagedContainers.Add(ctx, 1, metric.WithAttributes(
464+ attribute.String("domain", domainName),
465+ ))
466+ }
467+
414 s.publishContainerDeployed(ctx, domainName, container.ID)468 s.publishContainerDeployed(ctx, domainName, container.ID)
415 469
416 s.mu.RLock()470 s.mu.RLock()
@@ -539,6 +593,14 @@ func (s *Service) Restart(ctx context.Context, domainName string, withAttachment
539 }593 }
540 log.Info().Str(zerowrap.FieldEntityID, container.ID).Msg("container restarted")594 log.Info().Str(zerowrap.FieldEntityID, container.ID).Msg("container restarted")
541 595
596+ // Record restart metric
597+ if s.metrics != nil {
598+ s.metrics.ContainerRestarts.Add(ctx, 1, metric.WithAttributes(
599+ attribute.String("domain", domainName),
600+ attribute.String("source", "api"),
601+ ))
602+ }
603+
542 // Restart attachments if requested604 // Restart attachments if requested
543 if withAttachments && len(attachmentIDs) > 0 {605 if withAttachments && len(attachmentIDs) > 0 {
544 for _, attachID := range attachmentIDs {606 for _, attachID := range attachmentIDs {
@@ -626,6 +688,13 @@ func (s *Service) Remove(ctx context.Context, containerID string, force bool) er
626 }688 }
627 s.mu.Unlock()689 s.mu.Unlock()
628 690
691+ // Decrement managed container count
692+ if removedDomain != "" && s.metrics != nil {
693+ s.metrics.ManagedContainers.Add(ctx, -1, metric.WithAttributes(
694+ attribute.String("domain", removedDomain),
695+ ))
696+ }
697+
629 // Note: We intentionally do NOT clean up deployMu entries to avoid a race condition:698 // Note: We intentionally do NOT clean up deployMu entries to avoid a race condition:
630 // If Remove deletes the mutex entry while a concurrent Deploy is acquiring the lock,699 // If Remove deletes the mutex entry while a concurrent Deploy is acquiring the lock,
631 // the Deploy would create a fresh mutex, breaking serialization and allowing concurrent700 // the Deploy would create a fresh mutex, breaking serialization and allowing concurrent
@@ -1161,6 +1230,10 @@ func (s *Service) pullRefForDeploy(ctx context.Context, imageRef string) (string
1161 // - For digest references (@sha256:...), returns the pullRef since Docker can't tag digests1230 // - For digest references (@sha256:...), returns the pullRef since Docker can't tag digests
1162 // - For tagged images, returns the original imageRef after tagging the pulled image1231 // - For tagged images, returns the original imageRef after tagging the pulled image
1163 func (s *Service) ensureImage(ctx context.Context, imageRef string) (string, error) {1232 func (s *Service) ensureImage(ctx context.Context, imageRef string) (string, error) {
1233+ ctx, span := tracer.Start(ctx, "container.ensure_image",
1234+ trace.WithAttributes(attribute.String("image", imageRef)))
1235+ defer span.End()
1236+
1164 ctx = zerowrap.CtxWithField(ctx, "image", imageRef)1237 ctx = zerowrap.CtxWithField(ctx, "image", imageRef)
1165 log := zerowrap.FromCtx(ctx)1238 log := zerowrap.FromCtx(ctx)
1166 1239
M internal/usecase/proxy/service.go +9 -0
@@ -18,12 +18,17 @@ import (
1818 "time"
1919
2020 "github.com/bnema/zerowrap"
21+ "go.opentelemetry.io/otel"
22+ "go.opentelemetry.io/otel/attribute"
23+ "go.opentelemetry.io/otel/trace"
2124
2225 "github.com/bnema/gordon/internal/boundaries/in"
2326 "github.com/bnema/gordon/internal/boundaries/out"
2427 "github.com/bnema/gordon/internal/domain"
2528 )
2629
30+var proxyTracer = otel.Tracer("gordon.proxy")
31+
2732 // proxyTransport is a shared HTTP transport for proxying to application containers.
2833 // ResponseHeaderTimeout is kept short to detect unresponsive backends quickly.
2934 var proxyTransport = &http.Transport{
@@ -156,6 +161,10 @@ func (s *Service) ServeHTTP(w http.ResponseWriter, r *http.Request) {
156161
157162 // GetTarget returns the proxy target for a given domain.
158163 func (s *Service) GetTarget(ctx context.Context, domainName string) (*domain.ProxyTarget, error) {
164+ ctx, span := proxyTracer.Start(ctx, "proxy.get_target",
165+ trace.WithAttributes(attribute.String("domain", domainName)))
166+ defer span.End()
167+
159168 ctx = zerowrap.CtxWithFields(ctx, map[string]any{
160169 zerowrap.FieldLayer: "usecase",
161170 zerowrap.FieldUseCase: "GetTarget",
@@ -18,12 +18,17 @@ import (
18 "time"18 "time"
19 19
20 "github.com/bnema/zerowrap"20 "github.com/bnema/zerowrap"
21+ "go.opentelemetry.io/otel"
22+ "go.opentelemetry.io/otel/attribute"
23+ "go.opentelemetry.io/otel/trace"
21 24
22 "github.com/bnema/gordon/internal/boundaries/in"25 "github.com/bnema/gordon/internal/boundaries/in"
23 "github.com/bnema/gordon/internal/boundaries/out"26 "github.com/bnema/gordon/internal/boundaries/out"
24 "github.com/bnema/gordon/internal/domain"27 "github.com/bnema/gordon/internal/domain"
25 )28 )
26 29
30+var proxyTracer = otel.Tracer("gordon.proxy")
31+
27 // proxyTransport is a shared HTTP transport for proxying to application containers.32 // proxyTransport is a shared HTTP transport for proxying to application containers.
28 // ResponseHeaderTimeout is kept short to detect unresponsive backends quickly.33 // ResponseHeaderTimeout is kept short to detect unresponsive backends quickly.
29 var proxyTransport = &http.Transport{34 var proxyTransport = &http.Transport{
@@ -156,6 +161,10 @@ func (s *Service) ServeHTTP(w http.ResponseWriter, r *http.Request) {
156 161
157 // GetTarget returns the proxy target for a given domain.162 // GetTarget returns the proxy target for a given domain.
158 func (s *Service) GetTarget(ctx context.Context, domainName string) (*domain.ProxyTarget, error) {163 func (s *Service) GetTarget(ctx context.Context, domainName string) (*domain.ProxyTarget, error) {
164+ ctx, span := proxyTracer.Start(ctx, "proxy.get_target",
165+ trace.WithAttributes(attribute.String("domain", domainName)))
166+ defer span.End()
167+
159 ctx = zerowrap.CtxWithFields(ctx, map[string]any{168 ctx = zerowrap.CtxWithFields(ctx, map[string]any{
160 zerowrap.FieldLayer: "usecase",169 zerowrap.FieldLayer: "usecase",
161 zerowrap.FieldUseCase: "GetTarget",170 zerowrap.FieldUseCase: "GetTarget",
M internal/usecase/registry/service.go +31 -0
@@ -9,17 +9,30 @@ import (
99 "strings"
1010
1111 "github.com/bnema/zerowrap"
12+ "go.opentelemetry.io/otel"
13+ "go.opentelemetry.io/otel/attribute"
14+ "go.opentelemetry.io/otel/metric"
15+ "go.opentelemetry.io/otel/trace"
1216
17+ "github.com/bnema/gordon/internal/adapters/out/telemetry"
1318 "github.com/bnema/gordon/internal/boundaries/out"
1419 "github.com/bnema/gordon/internal/domain"
1520 "github.com/bnema/gordon/pkg/validation"
1621 )
1722
23+var registryTracer = otel.Tracer("gordon.registry")
24+
1825 // Service implements the RegistryService interface.
1926 type Service struct {
2027 blobStorage out.BlobStorage
2128 manifestStorage out.ManifestStorage
2229 eventBus out.EventPublisher
30+ metrics *telemetry.Metrics
31+}
32+
33+// SetMetrics sets the telemetry metrics for the registry service.
34+func (s *Service) SetMetrics(m *telemetry.Metrics) {
35+ s.metrics = m
2336 }
2437
2538 // NewService creates a new registry service.
@@ -60,6 +73,14 @@ func (s *Service) GetManifest(ctx context.Context, name, reference string) (*dom
6073
6174 // PutManifest stores a manifest and returns the calculated digest.
6275 func (s *Service) PutManifest(ctx context.Context, manifest *domain.Manifest) (string, error) {
76+ ctx, span := registryTracer.Start(ctx, "registry.put_manifest",
77+ trace.WithAttributes(
78+ attribute.String("name", manifest.Name),
79+ attribute.String("reference", manifest.Reference),
80+ attribute.Int("manifest_size", len(manifest.Data)),
81+ ))
82+ defer span.End()
83+
6384 ctx = zerowrap.CtxWithFields(ctx, map[string]any{
6485 zerowrap.FieldLayer: "usecase",
6586 zerowrap.FieldUseCase: "PutManifest",
@@ -75,6 +96,16 @@ func (s *Service) PutManifest(ctx context.Context, manifest *domain.Manifest) (s
7596 return "", log.WrapErr(err, "failed to store manifest")
7697 }
7798
99+ // Record push metrics
100+ if s.metrics != nil {
101+ attrs := metric.WithAttributes(
102+ attribute.String("name", manifest.Name),
103+ attribute.String("reference", manifest.Reference),
104+ )
105+ s.metrics.ImagePushTotal.Add(ctx, 1, attrs)
106+ s.metrics.ImagePushSize.Add(ctx, int64(len(manifest.Data)), attrs)
107+ }
108+
78109 // Publish image pushed event only for tag references (not digests).
79110 // A docker push sends manifests by both digest and tag; firing only on
80111 // tag prevents duplicate deploy triggers for the same push.
@@ -9,17 +9,30 @@ import (
9 "strings"9 "strings"
10 10
11 "github.com/bnema/zerowrap"11 "github.com/bnema/zerowrap"
12+ "go.opentelemetry.io/otel"
13+ "go.opentelemetry.io/otel/attribute"
14+ "go.opentelemetry.io/otel/metric"
15+ "go.opentelemetry.io/otel/trace"
12 16
17+ "github.com/bnema/gordon/internal/adapters/out/telemetry"
13 "github.com/bnema/gordon/internal/boundaries/out"18 "github.com/bnema/gordon/internal/boundaries/out"
14 "github.com/bnema/gordon/internal/domain"19 "github.com/bnema/gordon/internal/domain"
15 "github.com/bnema/gordon/pkg/validation"20 "github.com/bnema/gordon/pkg/validation"
16 )21 )
17 22
23+var registryTracer = otel.Tracer("gordon.registry")
24+
18 // Service implements the RegistryService interface.25 // Service implements the RegistryService interface.
19 type Service struct {26 type Service struct {
20 blobStorage out.BlobStorage27 blobStorage out.BlobStorage
21 manifestStorage out.ManifestStorage28 manifestStorage out.ManifestStorage
22 eventBus out.EventPublisher29 eventBus out.EventPublisher
30+ metrics *telemetry.Metrics
31+}
32+
33+// SetMetrics sets the telemetry metrics for the registry service.
34+func (s *Service) SetMetrics(m *telemetry.Metrics) {
35+ s.metrics = m
23 }36 }
24 37
25 // NewService creates a new registry service.38 // NewService creates a new registry service.
@@ -60,6 +73,14 @@ func (s *Service) GetManifest(ctx context.Context, name, reference string) (*dom
60 73
61 // PutManifest stores a manifest and returns the calculated digest.74 // PutManifest stores a manifest and returns the calculated digest.
62 func (s *Service) PutManifest(ctx context.Context, manifest *domain.Manifest) (string, error) {75 func (s *Service) PutManifest(ctx context.Context, manifest *domain.Manifest) (string, error) {
76+ ctx, span := registryTracer.Start(ctx, "registry.put_manifest",
77+ trace.WithAttributes(
78+ attribute.String("name", manifest.Name),
79+ attribute.String("reference", manifest.Reference),
80+ attribute.Int("manifest_size", len(manifest.Data)),
81+ ))
82+ defer span.End()
83+
63 ctx = zerowrap.CtxWithFields(ctx, map[string]any{84 ctx = zerowrap.CtxWithFields(ctx, map[string]any{
64 zerowrap.FieldLayer: "usecase",85 zerowrap.FieldLayer: "usecase",
65 zerowrap.FieldUseCase: "PutManifest",86 zerowrap.FieldUseCase: "PutManifest",
@@ -75,6 +96,16 @@ func (s *Service) PutManifest(ctx context.Context, manifest *domain.Manifest) (s
75 return "", log.WrapErr(err, "failed to store manifest")96 return "", log.WrapErr(err, "failed to store manifest")
76 }97 }
77 98
99+ // Record push metrics
100+ if s.metrics != nil {
101+ attrs := metric.WithAttributes(
102+ attribute.String("name", manifest.Name),
103+ attribute.String("reference", manifest.Reference),
104+ )
105+ s.metrics.ImagePushTotal.Add(ctx, 1, attrs)
106+ s.metrics.ImagePushSize.Add(ctx, int64(len(manifest.Data)), attrs)
107+ }
108+
78 // Publish image pushed event only for tag references (not digests).109 // Publish image pushed event only for tag references (not digests).
79 // A docker push sends manifests by both digest and tag; firing only on110 // A docker push sends manifests by both digest and tag; firing only on
80 // tag prevents duplicate deploy triggers for the same push.111 // tag prevents duplicate deploy triggers for the same push.

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 7

Tags 106

↑↓ moveenter openesc close