Proposal: render top-level visualization fences inline in the TUI

What variant of Codex are you using?

CLI / TUI

What feature would you like to see?

Summary

I would like to discuss a render-only path for terminal-native previews of assistant-authored visualization fences in the Codex TUI.

The prototype recognizes supported top-level fenced blocks after streaming finishes, renders them asynchronously, and replaces only the TUI presentation with terminal images. The raw assistant Markdown remains canonical in rollout history, so resume, copy, and non-image fallbacks do not depend on persisted PNG or escape-sequence data.

The current prototype supports:

• display LaTeX through RaTeX;
• a subset of Mermaid through mermaid-rs-renderer;
• an intentionally bounded subset of DOT through layout-rs;
• Kitty Graphics Protocol output, including nested tmux passthrough;
• a trusted local [open/zoom] link below each preview.

Unsupported formats or failed renders stay visible as their original fenced source. D2 is deliberately not supported by this prototype.

Demo

<img width="864" alt="Codex TUI rendering DOT, Mermaid, and LaTeX fences inline" src="https://raw.githubusercontent.com/yanyaoer/codex/issue-assets-inline-viz-terminal-demo-20260803/inline-viz-terminal-demo.png" />

The screenshot shows simple DOT, Mermaid, and LaTeX blocks rendered inline with terminal-aware foreground/background colors. Each artifact also has an [open/zoom] fallback.

Why

Codex increasingly works with diagrams, formulas, screenshots, and other visual artifacts. Opening every generated artifact in a separate application breaks the terminal review loop, while storing rendered output in model-visible history would make transcripts larger and less portable.

This prototype treats the image as a derived view of canonical Markdown:

1. The model emits ordinary fenced source.
2. Codex keeps that source unchanged in history.
3. A separate Rust renderer sidecar creates a content-addressed PNG after stream finalization.
4. The source-backed history cell reflows when the PNG becomes available, on resize, and on resume.
5. If rendering or terminal image support is unavailable, Codex shows the original source.

This also builds on the existing inline visualization link work in #33925 rather than replacing the HTML directive path.

Prototype implementation

Branch: https://github.com/yanyaoer/codex/tree/feat/inline-viz-rust-renderer

Compare: https://github.com/openai/codex/compare/main...yanyaoer:codex:feat/inline-viz-rust-renderer

High-level shape:

• packages a separate codex-inline-viz-renderer binary so renderer dependencies do not enter the main Codex process;
• discovers and validates only supported top-level Markdown fences;
• optionally exposes a dynamic preflight tool so compiler diagnostics can reach the model for at most two correction attempts;
• performs final rendering in the background and requests transcript reflow when artifacts are ready;
• stores content-addressed per-thread PNGs and verifies canonical containment, PNG structure, dimensions, decode limits, and immutable prepared-image cache entries;
• adapts diagram colors to the terminal palette and prepares formulas by alpha-cropping, recoloring, and fitting them to roughly 3–6 terminal rows;
• keeps terminal graphics bytes out of raw Markdown and replayable text surfaces.

The current branch contains the full vertical slice so the interaction can be evaluated end to end. It is larger than an ideal review unit; if the direction is useful, I would split it into renderer/packaging, TUI presentation/reflow, and optional preflight pieces before review.

Important limitations

This is a working prototype, not a claim of complete Mermaid, DOT, or TeX compatibility.

• Mermaid is incomplete. mermaid-rs-renderer implements only a subset of Mermaid syntax and features. The simple flowchart in the screenshot works, but many diagram kinds, directives, themes, and advanced constructs do not.
• DOT is incomplete by design. The layout-rs path accepts a bounded subset and is not Graphviz-compatible. Subgraphs/clusters, HTML labels, ports, URLs/images, and many attributes are rejected.
• D2 is unsupported. D2 fences remain raw source.
• LaTeX is bounded display math, not TeX. File I/O, links, user macros, and unsafe or unsupported control sequences are rejected.
• The bundled deterministic Noto Sans font does not cover CJK or emoji; input requiring missing glyphs is rejected instead of silently producing tofu.
• Terminal behavior still varies. The prototype has been exercised in Kitty, Ghostty, and nested tmux, but broader terminal and multiplexer coverage is needed.

One architectural question is whether Codex should accept deliberately bounded native renderers, start with the RaTeX-only surface, or rely on optional external tools for fuller Mermaid/DOT compatibility.

Safety bounds

• top-level supported fences only;
• at most 8 artifacts per message;
• 256 KiB source limit;
• 30-second renderer timeout;
• 20 MiB output and bounded image dimensions/decode work;
• format-specific rejection of imports, URLs, file I/O, macros, and directives;
• cleared renderer environment and parent-owned output path;
• canonical-path containment before display or hyperlink creation.

Validation

• just test -p codex-inline-viz-renderer — 9 passed;
• INSTA_WORKSPACE_ROOT="$PWD" just test -p codex-tui -E 'test(/inline_visualization/)' — 40 passed;
• python3 -m unittest discover -s scripts/codex_package -p 'test_*.py' — 15 passed;
• manual end-to-end rendering in Ghostty and Kitty, including nested tmux and [open/zoom] links.

Related discussions

• #29451 — terminal-native previews for image artifacts;
• #36233 — RaTeX/Kitty LaTeX rendering;
• #33925 — existing TUI inline visualization links.

Questions

1. Does keeping raw Markdown canonical while treating PNGs as disposable terminal presentation fit the intended TUI/history architecture?
2. Is a separately packaged Rust renderer sidecar an acceptable dependency boundary?
3. Would a narrower RaTeX-only first step be preferable to the incomplete Mermaid/DOT support?
4. If diagram rendering is desirable, should Codex prefer bounded native subsets or optional external renderers with broader compatibility?

Additional information

No additional information.