Surface the offered tool + skill/slash-command catalog in the `--json` event stream

Open 💬 8 comments Opened Jul 4, 2026 by geserdugarov

What variant of Codex are you using?

CLI

What feature would you like to see?

Emit a first-class, bounded event describing the effective tools and skills offered to the model through both:

  • codex exec --json, and
  • the equivalent app-server event stream.

The event should be emitted after Codex resolves the effective catalog for the first model sampling request in a turn and before that request is dispatched. If the catalog changes before a later sampling request in the same turn, emit an updated event.

The catalog should include:

  • Tools – the effective tool identities the model can call or discover for that request, including built-in/harness tools, hosted tools such as web search, dynamic or extension-provided tools, and MCP tools.
  • Tool exposure – whether each tool is directly advertised, discoverable through tool search, callable through Code Mode, or available through more than one surface.
  • Tool provenance – enough information to distinguish harness, hosted, dynamic, extension/plugin, and MCP tools, including the MCP server when applicable.
  • Skills – the skill identities actually retained in the model-visible catalog after enablement, filtering, and catalog budgeting across host, executor, and orchestrator sources.

Tool identities should use their effective wire namespace and name rather than assuming a flattened name such as mcp__tavily__search.

Codex TUI slash commands are client-side UI actions rather than capabilities offered to the model, so they are not part of this request. If a model-visible or headless command catalog is introduced later, it could be added separately.

Why

Tools that wrap codex exec --json for observability and analytics can currently reconstruct what the model used, but not what it was offered.

From the stream, a consumer can observe:

  • tools used through item.* events such as command_execution, mcp_tool_call, web_search, and file_change; and
  • skills triggered only heuristically, for example by noticing a command that reads a skills/<name>/SKILL.md file.

There is no supported way to answer:

  • Which tools or skills were offered but never used?
  • Was a skill skipped because it was not present in the model-visible catalog, or because the model chose not to use it?
  • Which MCP tools made it into the effective tool plan?
  • Was a tool directly visible, deferred behind tool search, or available only through Code Mode?
  • Did the effective catalog change between model sampling requests?

Claude Code exposes analogous information through the initial frame of its headless --output-format stream-json stream, allowing dashboards to calculate “offered --> used.” Codex streams contain the used side of that relationship but not the offered side.

Codex now has authoritative internal state at the relevant lifecycle point:

  • #37492 introduced opt-in tool_namespaces_info metadata describing direct, deferred, and Code Mode tool exposure and harness/MCP ownership.
  • #41195 moved the finalized model-specific tool plan onto the request-scoped ToolRouter, alongside the exact model-visible tool specifications used for that sampling request.
  • #37149 unified executor, orchestrator, and host skill catalog rendering under a shared, context-aware budget.

None of these internal representations is currently projected through codex exec --json or app-server lifecycle events. The existing tool_namespaces_info projection is also Responses Lite-specific, does not contain skills, and does not include hosted web search.

This is the same general class of observability gap as #17501, but for the effective model-facing catalog rather than MCP startup state.

Current behavior (what codex exec --json emits today)

Originally reproduced on codex-cli 0.142.5 and source-rechecked against openai/codex main at 6be2a6ca95 on 2026-08-28.

A representative successful codex exec --json turn still begins with:

{"type":"thread.started","thread_id":"019f2b90-0c34-77a1-ad02-b38355965b93"}
{"type":"turn.started"}
{"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"hi there"}}
{"type":"turn.completed","usage":
{"input_tokens":5,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":2,"reasoning_output_tokens"
:0}}

The top-level event vocabulary remains thread.started, turn.started, item.*, turn.completed, turn.failed, and error.

  • thread.started carries only thread_id.
  • turn.started is empty in the codex exec --json projection.
  • App-server's thread/start response, thread/started notification, and turn/started notification contain thread or turn metadata but no effective tool or skill catalog.
  • skills/list is a separate cwd-scoped discovery API.
  • mcpServerStatus/list is a separate MCP inventory/status API whose results do not prove which tools were included in a particular sampling request.

Consequently, consumers must still reimplement Codex configuration, plugin, skill, environment, MCP negotiation, tool exposure, and catalog-budgeting logic to approximate the offered catalog.

Proposed event shape

A dedicated event after effective request-scoped catalog resolution is more accurate than extending thread.started, because thread.started and turn.started are emitted before the final MCP binding, tool plan, and model-visible skill catalog are available.

Illustrative codex exec --json shape:

{
  "type": "turn.catalog.updated",
  "thread_id": "…",
  "turn_id": "…",
  "catalog_id": "…",
  "model": "gpt-5-codex",
  "tools": [
    {
      "namespace": "functions",
      "name": "exec_command",
      "source": {
        "kind": "harness"
      },
      "exposure": {
        "direct": true,
        "deferred": false,
        "code_mode_name": null
      }
    },
    {
      "namespace": "tavily",
      "name": "search",
      "source": {
        "kind": "mcp",
        "server": "tavily"
      },
      "exposure": {
        "direct": false,
        "deferred": true,
        "code_mode_name": "tavily_search"
      }
    },
    {
      "namespace": null,
      "name": "web_search",
      "source": {
        "kind": "hosted"
      },
      "exposure": {
        "direct": true,
        "deferred": false,
        "code_mode_name": null
      }
    }
  ],
  "skills": [
    {
      "id": "…",
      "name": "develop",
      "source": {
        "kind": "host",
        "scope": "repo"
      }
    },
    {
      "id": "…",
      "name": "review",
      "source": {
        "kind": "orchestrator"
      }
    }
  ],
  "skills_report": {
    "total_count": 2,
    "included_count": 2,
    "omitted_count": 0,
    "descriptions_truncated": false
  }
}

The exact event name and field layout are open to maintainer guidance. The important contract is:

  1. The catalog represents the effective model-facing state for the corresponding sampling request.
  2. It is emitted before the request that uses that state.
  3. A new event is emitted when the effective catalog changes.
  4. Consumers can correlate subsequent tool and skill usage with the most recent catalog event.
  5. The same underlying core event is projected consistently into codex exec --json and app-server, using each surface's naming conventions.

A stable catalog_id, revision, or digest would let consumers deduplicate unchanged catalogs and correlate updates without comparing entire arrays.

Constraints

  • Additive event only; existing JSONL parsers must continue to work.
  • No change to human-readable CLI output or model behavior.
  • Mirror the event in app-server rather than requiring clients to reconstruct it through separate list APIs.
  • Emit identity and provenance metadata only:
  • no tool schemas or descriptions;
  • no skill descriptions or SKILL.md contents;
  • no sensitive configuration values.
  • Keep the payload bounded.
  • Skill entries should identify the final model-visible set, not every skill discovered before filtering or budgeting.
  • A small completeness report such as total, included, omitted, or description-truncation status is useful.
  • Detailed catalog budget limits, utilization percentages, and untrimmed-size reporting are related but should be handled separately and should not block the core offered-catalog event.

Alternatives considered

  • Extend thread.started – it is emitted before request-scoped MCP, tool, environment, and skill resolution, so it cannot reliably describe the effective catalog.
  • Attach the catalog to turn.started – the current turn-start lifecycle event is also emitted before the first request-scoped ToolRouter and final skill catalog are constructed.
  • Infer offered tools from used item.* events – this yields only the used subset, cannot identify available-but-unused tools, and reports nothing for tools that were never called.
  • Use skills/list and mcpServerStatus/list – these expose separate discovery/inventory views, not the unified catalog actually offered to a particular model request.
  • Parse local configuration and skill directories out of band – this reimplements Codex's discovery and filtering behavior and can drift from profiles, -c overrides, plugins, environments, MCP negotiation, tool exposure policy, and catalog budgeting.
  • Expose the existing internal tool_namespaces_info unchanged – it is opt-in Responses Lite metadata, omits skills and hosted web search, and is not currently a public lifecycle event.

View original on GitHub ↗

8 Comments

rovboyko · 1 month ago

Very useful functionality especially for orchestrating for role-based harnesses. Nice to have this in the CLI.

geserdugarov · 1 month ago

@jif-oai @bolinfest @etraut-openai would you be willing to sanity-check the proposed event shape here? It crosses skills/MCP catalog discovery, codex exec --json, and app-server stream parity.

geserdugarov · 1 month ago

Rechecked against openai/codex main at bf3c197 (2026-07-20): this is still not implemented in the requested form.

Current state:

  • codex exec --json still defines thread.started with only thread_id, and its JSONL mapper copies only that field from SessionConfiguredEvent.
  • App-server's thread/start / thread/started payloads likewise contain no effective tools or skills catalog.
  • There are partial out-of-band APIs (skills/list and thread-scoped mcpServerStatus/list), but they do not solve stream observability: the former is cwd-scoped, the latter covers only MCP inventory, and neither provides the unified set of built-in/MCP/dynamic tools plus skills or commands actually offered to the model.

Therefore, a JSONL consumer still cannot calculate “offered → used” without independently reconstructing Codex's discovery and configuration logic.

One relevant implementation detail is that SessionConfigured is emitted before MCP initialization, so the earliest thread.started may be too early for the negotiated catalog. A dedicated additive event after catalog resolution and before the first turn - such as session.catalog or turn.configured - may therefore be more accurate than extending thread.started.

Before I prepare an implementation, I need explicit maintainer confirmation that this contribution is wanted and agreement on the public event contract.

@pakrym-oai, since you introduced the current thread.started JSON event shape, could you please confirm:

  1. Would a PR implementing this functionality be welcome?
  2. Should this extend thread.started, or use a dedicated post-initialization event?
  3. Should the catalog be session-scoped or turn-scoped, considering MCP refreshes and dynamic tools?
  4. Should the first PR cover both codex exec --json and app-server parity?

If you are not the appropriate owner, could you point me to whoever should approve the direction? I’ll wait for that confirmation before starting implementation.

geserdugarov · 9 days ago

Rechecked against openai/codex main at 3929c99a97 (2026-08-19). This request is still not implemented.

Current public surfaces:

  • codex exec --json still has no catalog/configured event. thread.started contains only thread_id; the mapper and its focused test still assert exactly that shape.
  • App-server's thread/start response and thread/started notification expose thread/session metadata but no effective tools, skills, or slash-command catalog.
  • skills/list and thread-scoped mcpServerStatus/list remain separate out-of-band inventories. They do not identify the unified catalog actually offered to the model for a particular turn.

One relevant change since bf3c197 is #37492 (2b1811e562), which builds opt-in per-turn tool_namespaces_info for Responses Lite metadata from the effective tool registry. It records model-visible function exposure and harness/MCP ownership, but it is not emitted through codex exec --json or app-server, does not cover skills, and is not the complete requested catalog.

The current lifecycle points more strongly toward a turn-scoped event: SessionConfigured is still emitted before initialized MCP bindings are published, while both the effective skill snapshot and effective tool router are resolved per turn. A dedicated additive event after that resolution, such as turn.configured, would therefore be more accurate than extending the early thread.started.

The issue remains current. Maintainer guidance is still needed on the public event contract and whether CLI JSONL and app-server parity should land together.

geserdugarov · 9 days ago

@rka-oai, since you introduced the effective per-turn tool namespace inventory in #37492, could you sanity-check whether exposing a bounded projection of it through codex exec --json and app-server would be an acceptable public contract, and whether such a PR would be welcome?

@felixxia-oai, for the skills portion, is the per-turn host skills snapshot the authoritative source for the skill names actually offered to the model?

If either of you is not the appropriate owner, a pointer to the right person would be greatly appreciated.

TyceHerrman · 17 hours ago

There is no supported interface that confirms the effective budget, utilization, or whether rendering shortened or omitted anything.

#35597 added useful catalog metrics:

  • codex.thread.skills.enabled_total
  • codex.thread.skills.kept_total
  • codex.thread.skills.description_truncated_chars
  • codex.thread.skills.truncated

#37149 extended metrics and shared budgeting across executor, orchestrator, and host catalogs. However, these primarily reveal pressure after descriptions have already been shortened or entries omitted. They do not expose the untrimmed catalog cost or percentage of the allowance currently used.

Could the proposed per-turn catalog event include the renderer’s effective report, for example:

"skill_catalog_render": {
"budget_unit": "tokens",
"budget_limit": 21000,
"untrimmed_size": 5600,
"budget_fraction_used": 0.267,
"total_count": 87,
"included_count": 87,
"omitted_count": 0,
"truncated_description_chars": 0,
"catalog_surfaces": [
"executor_world_state",
"orchestrator_world_state",
"host_world_state"
]
}

This should ideally be emitted after the final per-turn catalog is resolved. That would let Desktop, CLI, hooks, and external dashboards warn at a chosen threshold without independently recreating Codex’s discovery, priority, tokenization, and rendering logic.

geserdugarov · 5 hours ago

Thanks - this is adjacent, but I’d prefer to keep #31088 focused on exposing the effective per-turn tool/skill/command identities for offered-to-used analysis. A small omission/truncation indicator may fit, but detailed budget and utilization reporting seems better suited to a separate follow-up and shouldn’t block the core catalog event.

geserdugarov · 5 hours ago

Rechecked against openai/codex main at 6be2a6ca95 (2026-08-28). The public gap remains: neither codex exec --json nor app-server lifecycle events expose the effective tool and skill catalog.

A relevant change since the previous audit is #41195: the finalized tool plan now lives on the request-scoped StepContext, including model-visible specifications, Code Mode names, and optional namespace inventory. This provides a clearer authoritative source, but it remains internal, does not cover the final budgeted skill identities, and the existing namespace projection omits hosted web search.

I updated the issue description to frame this as a bounded, request-scoped catalog event emitted after resolution and before model sampling, with another event when the effective catalog changes. Detailed budget utilization remains out of scope; a small omission or truncation indicator is sufficient.