codex crash while running a node integration test
Open 💬 1 comment Opened Jul 9, 2026 by mikem-coalesce
What version of Codex CLI is running?
0.144.0
What subscription do you have?
Pro
Which model were you using?
gpt-5.5-xhigh
What platform is your computer?
Darwin 25.5.0 arm64 arm
What terminal emulator and version are you using (if applicable)?
Ghostty
Codex doctor report
What issue are you seeing?
let's build out a test harness as you state build out:
1. Zod microbenchmark: Replay sanitized raw V1 Firestore node documents through NodeDataTransform.fromStorage, using realistic column/source-reference counts. Measure process.cpuUsage(), wall time,
RSS,
and event-loop delay; support node --cpu-prof.
2. Endpoint load harness: Seed 100–1,000 representative V1 nodes, then issue concurrent GetNodes requests using a matrix of detail and skipParsing. Comparing skipParsing=true against false separates
repository/Zod cost from Strumenta cost.
For this task, write yourself a new goal and spawn agents in parallel — as many as needed to do it better and faster. Split the work into independent pieces, dispatch them concurrently, and synthesize
the results as they return. Give each agent its own dedicated /goal
• I’ll create a root goal and durable ledger first, then split the work into independent tracks: microbenchmark design/implementation, endpoint load harness design/implementation, and fixture/runtime
integration review. Each agent will create and maintain its own dedicated goal; I’ll integrate, test, and resolve shared-interface conflicts in the root checkout.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The root ledger now exists. I’m tightening its finishing criteria before any harness edits: both harnesses must execute the actual V1 production paths, produce machine-readable and human-readable metrics,
support scale/concurrency controls, include profiling instructions, and pass focused correctness plus smoke validation without committing customer data.
• Started `/root/zod_microbench`
• Started `/root/endpoint_load`
• Started `/root/harness_review`
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The agents are running with independent ledgers and file scopes. I’m using the root track to settle integration constraints now: executable format, Firestore emulator lifecycle, fixture ownership, CLI
output contracts, and which checks belong in CI versus on-demand profiling.
• Interacted with `/root/zod_microbench`
• Interacted with `/root/endpoint_load`
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• While implementation proceeds, I’m deriving non-sensitive shape statistics from the already-fetched staging V1 sample. These counts will set realistic defaults for columns, source references, transforms,
and document sizes without committing any raw customer or staging documents.
• Interacted with `/root/zod_microbench`
• Interacted with `/root/endpoint_load`
• Interacted with `/root/harness_review`
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The representative defaults are now calibrated from 259 staging V1 documents without retaining their contents: median 14 columns and 12.4 KB per node; p95 55 columns and 42.9 KB; p99 91 columns and 67.5
KB. The endpoint harness is being built as an opt-in integration workload so 100–1,000-node runs never enter ordinary CI, while pure generator/metrics tests remain part of normal focused validation.
• Interacted with `/root/harness_review`
• Interacted with `/root/zod_microbench`
• Interacted with `/root/endpoint_load`
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• I’m recording the workload calibration and execution split in the root ledger now, so the benchmark defaults and attribution math survive integration or interruption. No raw staging documents will enter
the repository; only aggregate shape statistics and deterministic generators will.
• Context compacted
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The corpus calibration now includes transform diversity, which matters because repeated SQL can overstate Strumenta cache performance. I’m feeding that into both harness implementations and the
independent review before I accept their workload model.
• Interacted with `/root/zod_microbench`
• Interacted with `/root/endpoint_load`
• Interacted with `/root/harness_review`
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The microbenchmark is now landing on the exact NodeDataTransform.fromStorage chain, with fixture construction and warmup kept outside the timed region. I’m checking the invocation surface and metric
semantics next, while the endpoint worker finishes the real scheduler/Firestore path.
• Interacted with `/root/zod_microbench`
• Interacted with `/root/harness_review`
• Interacted with `/root/endpoint_load`
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The independent review has found one measurement issue worth fixing before acceptance: peak-memory sampling currently runs inside the aggregate timed loop, so it can inflate the reported Zod wall/CPU
cost. I’m recording that review checkpoint in the root ledger and will require the final output to expose or exclude harness overhead clearly.
• Interacted with `/root/endpoint_load`
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The endpoint harness is now wired to the actual HTTP handler and Firestore repository, not a mocked handler seam. Its matrix reports the base path, parser delta, ordinary detail mapping, and the detail-
plus-lineage interaction separately; I’m reviewing its seeding and request rotation before running the 100-node smoke.
• Interacted with `/root/harness_review`
• Interacted with `/root/endpoint_load`
• Interacted with `/root/endpoint_load`
• Interacted with `/root/zod_microbench`
• Interacted with `/root/endpoint_load`
› continue
• Continuing from the current state. Both implementations are present; I’m moving into live validation now, starting with fast helper suites while the workers finish the last fidelity corrections.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The fast suites now pass: 17 tests across fixture generation, metrics, exact storage validation, endpoint concurrency, and attribution. I’m adding stable task entry points next so the harnesses can be
rerun without reconstructing long commands, then I’ll run the real emulator-backed endpoint matrix.
■ Conversation interrupted - tell the model what to do differently. Something went wrong? Hit `/feedback` to report the issue.
• Context compacted
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• I’m resuming from the active goal state and will verify the current worktree before deciding what remains. I’ll also keep the goal ledger current as I finish validation and report timings.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• I’m using the goal-ledger skill because this goal is explicitly ledger-backed. First pass is current-state verification: files, task wiring, and whether the parallel agents have final results or remaining
blockers.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The live agent registry now only shows the root agent, so I’m treating the subagent work as incorporated into the shared worktree and the saved summary. I’ll finish the remaining wiring and validation
directly from the current files.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The harnesses and READMEs are present; the missing piece is a short task entry point so the commands are discoverable alongside the parser workbench tasks. I’m adding that wiring now, then I’ll run the
narrow validation suite and the real endpoint smoke.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The endpoint task can use the existing integration runner and explicitly disable metadata service because this harness stays in Firestore-backed NodeHandlers paths. That keeps the smoke smaller without
changing the API path under test.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• I added the Taskfile entries and am updating the README snippets to point at those stable entry points while preserving the raw commands for profiling and JSON cases.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The Taskfile command appends optional Vitest args after the harness file; I’m checking the repo’s test runner before I rely on the CPU-prof example. If the runner treats those as positional, I’ll move the
args before the file.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The task entries are discoverable. I’m moving into validation now: focused unit tests first, then formatter/type gates, then the opt-in endpoint matrix smoke.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The focused node suite passes, including the endpoint workload helper tests. The validation-error log is intentional from the negative benchmark test that proves missing metadata still fails the
production transform.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The benchmark task works for the human report. Because this repo’s Taskfile uses prefixed output globally, the direct CLI remains the right documented path for clean JSON; I’m correcting that README
detail.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The microbenchmark’s main timing run completed: 500 transformed documents in about 87 ms wall time with 170 ms CPU and 5.75k docs/sec on this machine. I’m now checking clean JSON output and profiler
invocation.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The CPU-profile smoke produced .cpuprofile files. One JSON validation shell line accidentally used the broken Homebrew node after the benchmark process, so I’m rerunning that parse with the pinned Node
path; the benchmark output itself was generated by Node 20.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• Clean JSON output verifies as versioned schema with Zod version and document count. Next I’m checking formatting on only the files touched for this harness, then type checking.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• git diff --check is clean. The root-level pnpm exec prettier is not available in this monorepo layout, so I’m rerunning Prettier from the javascript/ package with paths adjusted.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• Formatting is clean. I’m starting TypeScript now; this can take longer because it checks the whole JavaScript package.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• TypeScript is still running and has not emitted errors yet. I’m letting that finish before starting the heavier endpoint integration smoke so failures are easier to attribute.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The type check is still active after a minute with no output. I’ll keep waiting; once it returns, I’ll either fix compile issues or move to the endpoint matrix.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The full TypeScript check hit the default V8 heap limit around 4 GB, which is a repo-scale check issue rather than a type error. I’m rerunning the same command with the integration runner’s 12 GB heap
setting so it can complete.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The high-heap type check is still running cleanly with no output so far. After it exits, I’ll run the endpoint harness through the scheduler/emulator path.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• TypeScript found three harness issues: an unused helper-test variable and an event-loop histogram field that exists at runtime in some Node builds but is not in this repo’s types. I’m fixing those
directly rather than weakening the type check.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• I’m changing sampleCount to a harness-owned timer tick count instead of depending on a nonportable IntervalHistogram.count property. The delay values still come from Node’s event-loop histogram; the count
is just evidence that the monitor had time to sample.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The compile errors are patched. I’m rerunning the focused tests and the high-heap type check together now; after that I’ll run the endpoint integration smoke.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• Focused tests and the high-heap TypeScript check pass. I’m starting the real integration smoke now with 100 V1 nodes, both endpoint variants, concurrency 2, and the full detail x skipParsing matrix.
■ Conversation interrupted - tell the model what to do differently. Something went wrong? Hit `/feedback` to report the issue.
■ Conversation interrupted - tell the model what to do differently. Something went wrong? Hit `/feedback` to report the issue.
• Context compacted
› the last session crashed codex while running tests .. thoughts?
• Using the goal-ledger flow again because this is still the active PYX-153 harness goal. I’m going to treat the crash as a test-run stability issue first: check for orphaned test/emulator processes, see
whether the endpoint JSON was written, then decide whether to rerun narrowly instead of restarting the whole suite.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• No matching harness, Vitest, scheduler, or emulator process is still alive, so the crash did not leave an obvious orphan chewing CPU. Next I’m checking whether the interrupted endpoint smoke produced a
JSON report; if it did, we can use that instead of rerunning the whole integration path.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
• The interrupted run did not write /tmp/pyx153-endpoint-load.json, so there is no endpoint timing artifact to salvage. My read is that the crash risk is the integration harness shape, not the
microbenchmark: it starts the repo’s common integration dependencies and exercises the real scheduler endpoint, while the earlier full TypeScript check also needed a larger heap. I’m going to rerun only a
tiny endpoint smoke with 100 seeded V1 nodes, one worker, and capped output.
■ Conversation interrupted - tell the model what to do differently. Something went wrong? Hit `/feedback` to report the issue.
› what was running in the background before codex crashed?
• Before Codex crashed, the active command was the endpoint load harness smoke:
PYX153_ENDPOINT_LOAD_REQUESTS=3
PYX153_ENDPOINT_LOAD_WARMUP=1
PYX153_ENDPOINT_LOAD_CONCURRENCY=2
PYX153_ENDPOINT_LOAD_NODES=100
PYX153_ENDPOINT_LOAD_PAGE_SIZE=100
PYX153_ENDPOINT_LOAD_JSON=/tmp/pyx153-endpoint-load.json
task nodes-endpoint-load-harness
That task expands to the repo integration runner for:
./javascript/test.py --flavor=integration --library=vitest --passWithNoTests --disable-metadata-service \
src/scheduler/handlers/NodeHandlers/endpoint-harness/endpoint-load.integration.vitest.ts
So the background/child work was most likely the integration test stack: Vitest/node worker processes, common test dependencies, local service/emulator setup, seeding 100 V1 nodes, then issuing concurrent
GetNodes requests across the detail / skipParsing matrix.
I checked after the crash: no matching test.py, vitest, firebase, scheduler, tsx, or PYX-153 harness process is still alive, and /tmp/pyx153-endpoint-load.json was not written. The already-completed
pieces before that were the Zod microbenchmark, CPU profile smoke, focused unit tests, Prettier check, and high-heap TypeScript check.What steps can reproduce the bug?
Uploaded thread: 019f48a3-1300-7ac2-a4f8-70acc82e1f69
What is the expected behavior?
not have codex crash while running integration tests in a shell in the background
Additional information
This was what was running when crashed. 'task' is the Gotask utility.
› what was running in the background before codex crashed?
• Before Codex crashed, the active command was the endpoint load harness smoke:
PYX153_ENDPOINT_LOAD_REQUESTS=3
PYX153_ENDPOINT_LOAD_WARMUP=1
PYX153_ENDPOINT_LOAD_CONCURRENCY=2
PYX153_ENDPOINT_LOAD_NODES=100
PYX153_ENDPOINT_LOAD_PAGE_SIZE=100
PYX153_ENDPOINT_LOAD_JSON=/tmp/pyx153-endpoint-load.json
task nodes-endpoint-load-harness
That task expands to the repo integration runner for:
./javascript/test.py --flavor=integration --library=vitest --passWithNoTests --disable-metadata-service \
src/scheduler/handlers/NodeHandlers/endpoint-harness/endpoint-load.integration.vitest.ts
So the background/child work was most likely the integration test stack: Vitest/node worker processes, common test dependencies, local service/emulator setup, seeding 100 V1 nodes, then issuing concurrent
GetNodes requests across the detail / skipParsing matrix.
I checked after the crash: no matching test.py, vitest, firebase, scheduler, tsx, or PYX-153 harness process is still alive, and /tmp/pyx153-endpoint-load.json was not written. The already-completed
pieces before that were the Zod microbenchmark, CPU profile smoke, focused unit tests, Prettier check, and high-heap TypeScript check.This issue has 1 comment on GitHub. Read the full discussion on GitHub ↗