otel: exported OTLP logs have timeUnixNano=0 (event timestamp never set), causing collectors to drop them

Open 💬 0 comments Opened Jul 2, 2026 by adamrogal-dd

What issue are you seeing?

Native OTLP logs emitted by Codex are being dropped by downstream OTEL collectors/backends because every exported LogRecord has timeUnixNano = 0. The observedTimeUnixNano field is populated correctly, but the event time (timeUnixNano) is always 0.

Many collectors/backends reject or mis-index records whose event time is 0, so the logs silently disappear even though traces and metrics from the same exporter arrive fine. Manually populating timeUnixNano in the POST body makes the collector ingest the records correctly, confirming the zero timestamp is the cause.

Root cause

This is a two-part interaction in the pinned opentelemetry-rust 0.31 stack (codex-rs/Cargo.toml):

  1. The tracing→OTEL bridge never sets the event timestamp. opentelemetry-appender-tracing's on_event builds the LogRecord (target, event name, severity, attributes) but never calls set_timestamp(...). tracing events carry no timestamp of their own, so the record's timestamp field stays None. This is unchanged in 0.30, 0.31, and current main:

https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-appender-tracing/src/layer.rs (the on_event fn)

  1. The OTLP proto transform serializes a missing timestamp as literal 0, with no fallback to observed time:

https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-proto/src/transform/logs.rs
``rust
time_unix_nano: log_record.timestamp().map(to_nanos).unwrap_or_default(), // None -> 0
observed_time_unix_nano: to_nanos(log_record.observed_timestamp().unwrap()), // always set by SDK
``

The SDK stamps observed_timestamp at emit time (hence that field is fine), but timestamp is left unset, so unwrap_or_default() yields 0.

Per the OTLP spec, time_unix_nano = 0 is technically legal ("unknown timestamp"), which is why the Rust SDK tolerates it — but in practice many collectors/backends drop such records.

Relevant Codex code: the log pipeline in codex-rs/otel/src/provider.rs (OpenTelemetryTracingBridge -> SdkLoggerProvider with with_batch_exporter), pinned to opentelemetry-appender-tracing = "0.31.0" in codex-rs/Cargo.toml.

Note: bumping the opentelemetry-* crates will not fix this on its own — current upstream main still does not set the log timestamp.

Steps to reproduce

  1. Start an OTLP collector (fresh Codex install, e.g. in a Docker container) writing logs to a local file.
  2. Configure Codex OTLP log export pointed at the collector ([otel] exporter = { otlp-http = { endpoint = "http://127.0.0.1:4318/v1/logs", protocol = "json" } }).
  3. Run a prompt so Codex emits log events.
  4. Capture the JSON POST body sent to /v1/logs.

Observed: every logRecords[].timeUnixNano is 0 (or "0"); observedTimeUnixNano is a valid nanosecond timestamp. Records are dropped by the collector.

Expected: timeUnixNano is populated with the event time (or, when unavailable, back-filled from the observed timestamp) so records are ingested.

Suggested fixes

  • Codex-side: have the log bridge set the LogRecord timestamp (e.g. from the observed/current time) so exported records carry a non-zero timeUnixNano.
  • Collector-side workaround (for anyone hitting this today): a transform processor that back-fills the event time —

```yaml
processors:
transform:
log_statements:

  • context: log

statements:

  • set(time_unix_nano, observed_time_unix_nano) where time_unix_nano == 0

```

Related

  • #28810 — OTLP logs remaining empty on app-server (shutdown/lifecycle bug). Same crate/area (codex-otel), different root cause: there the logs never export at all; here they export but are dropped for a zero event time.

View original on GitHub ↗