5.1 claims test succeeded when it clearly failed

I added some info where gpt-5-codex opines on the work of gpt-5.1-codex here:
thread ID 019a9b37-6ff6-7160-adee-4b0f8bb64173

I've been trying out some additional instructions for the agent to address this problem (I've been experiencing it too). Adding the following to ~/.codex/AGENTS.md seem to mitigate the worst of it:

Long command outputs will be truncated by the environment, and you will see
only the start and end of the output. To account for this, run all test, lint
and format checking suites using through `tee`, redirecting stderr to stdout
first, outputting to a temporary log file for review after completion, and
using `set -o pipefail` so the exit code are not obscured.

Always check the exit code of any commands you run. The command exit code is
available to you in the command's execution metadata. If a test, lint or format
checking suite fails with a non-zero exit code, investigate this. Do not assume
that this is environmental.

Although I strongly feel that the model shouldn't have to build its own command execution harness or be pointed to where is can find the command exit code. The Codex CLI should be providing this for the model by default.

Thanks for the bug report. We'll share the feedback with the team responsible for training our codex models. We're tracking this feedback and similar reports so we can make future models more robust and efficient. And thanks for using the /feedback command. That additional information is useful.

Here's one more bit of info: gpt-5-codex's comments about what it fixed in gpt-5.1-codex's bad changes:

• Media saves now fail fast instead of silently trashing data. src/core/
models/media.cpp:99 no longer drops/rebuilds the media table; it simply logs
the prepare failure and raises Q_ASSERT_X so you can blow away the bad DB
yourself, satisfying the “assert instead of drop” request. I also pulled in
<QtGlobal> for that assert.

The schema is treated as a single fresh version again. src/core/persistence/
schema.sql:84 keeps the new audio_rate/timebase_type columns but drops the
DROP TABLE statements so running the schema file can’t erase an existing
project. The version row now inserts 1, matching src/core/persistence/
schema_constants.h:10, so the migration layer won’t pretend we support
incremental upgrades.

Since we no longer try to auto-heal schema drift, the ad‑hoc “repair”
hook in Migrations::createNewProject was removed (src/core/persistence/
migrations.cpp:4), leaving migrations to either succeed or fail loudly. This
aligns with the new assertion behavior.

src/lua/core/command_manager.lua:1242 now finalizes the prepared statements
that feed the project hash, so we don’t leak handles or leave the DB
busy. The generated Makefile was replaced with the original 6‑line wrapper
(Makefile:1) so builds aren’t tied to /Users/joe/….

On the “is the C++ removal safe?” front: rg -n "tests/contract"
CMakeLists.txt returns no hits, so the build no longer references the deleted
C++ contract/event-log targets; likewise rg -n "EventLog" src/**/*.cpp shows
nothing, so dropping JVEEventLog-based tests doesn’t strand any runtime code.

I didn’t rerun CMake/tests; please delete any existing .jve test DB so the
new schema (version 1) is created cleanly, then rebuild with the restored
Makefile (cmake -S . -B build && cmake --build build) and run cmake --build
build --target check or ctest as needed.

And here's a similar fixup for a different repo i'm working on:

please review the changes in this folder compared to what's in git. the
previous codex may have lost its mind and made stupid choices and errors

• Ran cd /Users/joe/Local/jordan-prototype-codex && git status -sb
└ ## jordan-prototype-codex
M CURRENT_SESSION_STATUS.md
… +21 lines
?? tests/unit/electron-api-bridge.localstorage.test.js
?? tests/unit/main-ipc.test.js

• Updated Plan
└ □ Inspect repo status and recent diffs to understand scope of changes vs
git
□ Review key modified/new files for potential issues (logic, tests,
architecture)
□ Summarize findings ordered by severity with file references

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff --stat
└ CURRENT_SESSION_STATUS.md | 29 +++--
jest.config.js | 10 +-
… +12 lines
tests/unit/transformer.test.js | 5 +-
15 files changed, 482 insertions(+), 98 deletions(-)

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff src/main.js
└ diff --git a/src/main.js b/src/main.js
index d57b043..92476c8 100644
… +176 lines
try {
const parsedPath = path.parse(originalFilePath);

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff src/renderer/
│ adapters/electron-api-bridge.js
└ diff --git a/src/renderer/adapters/electron-api-bridge.js b/src/renderer/
adapters/electron-api-bridge.js
index df8f344..37ad683 100644
… +174 lines
});
}

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff src/renderer/
│ adapters/electron-file-operations.js
└ diff --git a/src/renderer/adapters/electron-file-operations.js b/src/
renderer/adapters/electron-file-operations.js
index 0d852f8..a62c022 100644
… +174 lines

• Get project storage (placeholder - needs integration with

ProjectStateManager)
*/

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff src/renderer/
│ adapters/electron-constants.js
└ diff --git a/src/renderer/adapters/electron-constants.js b/src/renderer/
adapters/electron-constants.js
index 6118295..e6b25bf 100644
… +11 lines

// File Path Constants

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff src/preload.js
└ diff --git a/src/preload.js b/src/preload.js
index 4016a5c..0af31bf 100644
… +22 lines
\ No newline at end of file
+});

• Explored
└ Read app.js

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff jest.config.js
└ diff --git a/jest.config.js b/jest.config.js
index 0343d9d..281172e 100644
… +22 lines
\ No newline at end of file
+};

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff tests/unit/
│ preload-clean.test.js
└ diff --git a/tests/unit/preload-clean.test.js b/tests/unit/
preload-clean.test.js
index 98de252..a194fe4 100644
… +76 lines
\ No newline at end of file
+});

• Explored
└ Read electron-api-bridge.localstorage.test.js, main-ipc.test.js

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff tests/__mocks__/
│ electron.js
└ diff --git a/tests/__mocks__/electron.js b/tests/__mocks__/electron.js
index 571ab64..53c0f00 100644
… +31 lines
\ No newline at end of file
+};

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff src/renderer/
│ jordan/enhancements/macos-header.js
└ diff --git a/src/renderer/jordan/enhancements/macos-header.js b/src/
renderer/jordan/enhancements/macos-header.js
index eadde5a..c9be4fe 100644
… +39 lines

header.querySelectorAll(interactiveSelectors.join(',
')).forEach(element => {

• Explored
└ Search originalLocalStorage in electron-api-bridge.js
Read electron-api-bridge.js

• I'm untangling how localStorage is overridden yet original localStorage
is still used for "rememberLastProjectPath," which contradicts the goal of
storing state inside the project file for portability. I see a bug where
last path and dialog history are saved globally, not in .ppr as requested,
so they won't persist across machines. I’m also verifying if auto-loading
and file operation hooks correctly store and restore this state, and checking
for potential race conditions with mainWindow initialization during project
loading. Finally, I’ll review relevant new files and tests to understand
impacts and catch any issues.

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff tests/unit/
│ transformer.test.js
└ diff --git a/tests/unit/transformer.test.js b/tests/unit/
transformer.test.js
index 4575942..de045b8 100644
… +13 lines
});

• Explored
└ Read index_electron.html

• I'm investigating potential problems with using localStorage for storing
non-string values, verifying that changes to stringification are safe,
and checking if the Electron bridge (window.electronApiBridge) is properly
initialized before usage, especially in the ElectronFileOperations class—
some references might be happening too early or in tests, risking crashes
or errors.

• Explored
└ Search electronApiBridge in electron-api-bridge.js
Read electron-api-bridge.js

• I’m tracing how window.electronApiBridge is assigned and used in
ElectronFileOperations, focusing on potential recursion and timing issues
when calling rememberDialogPath. I’m also evaluating localStorage overrides
that redirect storage to the project file versus using the original
localStorage, which affects persistence of dialog history and project paths
—these seem stored outside the project file contrary to instructions. I’m
checking error handling when electronApiBridge is missing, which causes
exceptions that may disrupt saves unnecessarily, and reviewing how auto-load
respects user preferences. There’s complexity around storage hooking, project
state management, and whether data read from files is sanitized before use.
Overall, I’m looking for subtle bugs and mismatches in storage locations that
impact user settings persistence.

• Explored
└ Read main.js

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff -U5 src/main.js
└ diff --git a/src/main.js b/src/main.js
index d57b043..92476c8 100644
… +204 lines
const dir = parsedPath.dir;
const baseName = parsedPath.name;

• I'm tracing how the app handles last project paths and localStorage
interplay. It looks like last project paths are stored globally in browser
localStorage, not within project files, which may conflict with the
requirement to keep UI state inside project files. I’m also checking whether
these storage calls align with Electron’s context isolation and if new code
changes introduce any bugs or inconsistencies.

• Explored
└ Read default-settings.json

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff src/renderer/
│ jordan/transformed/index.html
└ diff --git a/src/renderer/jordan/transformed/index.html b/src/renderer/
jordan/transformed/index.html
index 0e051de..a350940 120000
… +5 lines
+index_electron.html
\ No newline at end of file

• Ran cd /Users/joe/Local/jordan-prototype-codex && git diff src/renderer/
│ jordan/original/current.html
└ diff --git a/src/renderer/jordan/original/current.html b/src/renderer/
jordan/original/current.html
index 0a2a5ee..64233a9 120000
… +5 lines
+index.html
\ No newline at end of file

• I’m digging through test changes, focusing on fallback logging and bridge
hooks, while trying to spot bugs or regressions worth calling out. Key
findings include lowered coverage thresholds hiding poor tests, dialogs
and last project paths stored only in host localStorage rather than project
files (breaking cross-machine persistence), and exception handling gaps in
ElectronFileOperations that could crash on missing bridges. I’m also noticing
auto-load behavior that ignores user preferences and possible off-screen
window restoration on monitor changes. I’ll keep checking test mocks and
storage logic to verify these impacts.

• Explored
└ Search saveJsonExport
Read app.js

• I’m digging into how window.electronApiBridge.getDialogDefaultPath behaves,
especially when localStorage data is corrupted or missing, causing export
failures due to strict assertions. I’m also checking impact on other modules
like ProjectStateManager, unit tests for electron-api-bridge, and recent GUI
changes that might affect offline Electron security or styling quirks. This
will help me understand error handling and resource loading issues.

• Explored
└ Search default-settings.json
Read app.js, CURRENT_SESSION_STATUS.md
Search dialog history
Search dialogHistory
Search getCurrentProjectData in electron-file-operations.js
Read electron-file-operations.js
Search getProjectStorage in electron-file-operations.js
Read electron-file-operations.js, project-state-manager.js
Search loadProjectStorage

• There are several problems with project state saving and loading:
localStorage data isn't properly integrated into the project file, causing
UI state like tabs to be lost; dialog history is incorrectly stored globally
rather than per project, violating the spec; some functions return empty
or inconsistent data, preventing proper state restores; and potential edge
cases with path remembering and localStorage overrides need highlighting. I’m
piecing together these issues from various modules to outline multiple bugs
and gaps in how project state and user data persist across sessions.

• Explored
└ Search markProjectClean
Search module.exports in main.js
Read main.js
Search applyLoadedDataToJordan in electron-file-operations.js
Read electron-file-operations.js

• I’m dissecting several critical bugs around project storage not saving data
properly, causing panel states to be lost on reload. I see mismatches between
localStorage overrides and actual file persistence with Electron, plus dialog
path defaults saved outside the project file, contrary to user expectations.
I’m also examining the correctness of original localStorage method bindings
after override and tracking where dialog paths and settings get stored to
pinpoint inconsistencies. This looks like a heap of unexpected behaviors
needing clear highlighting!

• Explored
└ Search projectStorage

• I'm investigating various issues around project state persistence and
storage management in ElectronFileOperations and related modules. There are
discrepancies in how projectStorage, localStorage, and stateManager interact
—especially during new project creation, autosave, loading from paths, and
dialog default paths management. I want to understand why panels might not
open despite fixes, potential double stringification in localStorage, and
how remote scripts may affect Electron offline usage. Also, the reliance on
window.electronApiBridge in unit tests might cause failures. I'm gathering
these interconnected bugs to form a clearer diagnosis.

• Explored
└ Search ElectronFileOperations in tests

• I’m examining how the app handles project state persistence and auto-
loading, spotting that localStorage UI state isn’t saved into project files
due to a stubbed method, and dialog paths are stored externally rather than
per project, conflicting with specs. There’s also auto-load ignoring the
user’s disable setting but possibly meeting a new requirement. I’m gathering
severity-ranked bugs with file references to prepare a clear review.

• Updated Plan
└ ✔ Inspect repo status and recent diffs to understand scope of changes vs
git
✔ Review key modified/new files for potential issues (logic, tests,
architecture)
□ Summarize findings ordered by severity with file references

• I'm putting together at least three main findings about storage and
persistence problems, like project localStorage not saving, dialog history
being stored only in host localStorage instead of project files, and reduced
test coverage thresholds. I'm also noting potential window bounds bugs and
double auto-load calls. I'll gather exact line references next to support
these points before finalizing the review.

• Explored
└ Read electron-file-operations.js, electron-api-bridge.js, jest.config.js

• I’m preparing to explain how ProjectStateManager.getProjectStorage is bridged
rather than hooked, supporting this with line references from project-
state-manager.js and electron-file-operations.js. I want to highlight the
regression caused by buildEnhancedProjectEnvelope saving empty localStorage
data due to getProjectStorage returning an empty object, which breaks state
persistence like panel reopening. I’ll include specific source line snippets
to back this up.

• Explored
└ Read project-state-manager.js

• Updated Plan
└ ✔ Inspect repo status and recent diffs to understand scope of changes vs
git
✔ Review key modified/new files for potential issues (logic, tests,
architecture)
✔ Summarize findings ordered by severity with file references

• Explored
└ Read electron-file-operations.js

• I'm gathering all key issues with localStorage handling, project state
persistence, and test coverage from various code lines to create a clear
bullet-point list of findings ordered by severity. Then I'll wrap it
up with a concise summary and logical next steps to recommend wiring
getProjectStorage to the state manager and highlight how these bugs cause
state loss and feature regressions. That way, the user gets a thorough yet
organized overview for review.

─ Worked for 7m 13s ───────────────────────────────────────────────────────────

• Findings

• src/renderer/adapters/electron-file-operations.js:82 writes the renderer’s

localStorage contents into each .ppr, but getProjectStorage() (same
file:485) is still a stub that always returns {}. As a result every save
drops the entire project-state payload (current tab, panel visibility,
dialog history, etc.), so nothing the user toggles actually persists to
disk. ProjectStateManager already tracks this data (src/renderer/adapters/
project-state-manager.js:107-117), but because it’s never injected into the
save envelope the UI will reset on every reopen—exactly the regression the
user has been complaining about.

• The new dialog-history/last-project bookkeeping is being written to the

host profile’s real localStorage via this.originalLocalStorage (src/
renderer/adapters/electron-api-bridge.js:110-183). That means the
remembered open/save/export folders (and even the “last project” path) live
only on the current machine/profile; they aren’t embedded in the .ppr, so
sharing a project or upgrading to a new machine loses those preferences
despite the requirement that “everything the user changes should be
persisted in the project file.” We need to route these keys through the
project storage (and include them in the saved envelope) instead of the
browser’s global storage.

• jest.config.js:21-27 drops the global coverage thresholds from the

documented 80% down to 60/45/60/60. That weakens our safety net right after
adding risky renderer/IPC work and directly conflicts with the repository
guidelines that call for ≥80% coverage; it can mask the very regressions
we’re trying to prevent. I’d revert to the previous thresholds (or raise
them) once the missing tests are in place.

Happy to dig deeper or help wire the project-storage plumbing back in.

PS again: I set the model to gpt-5-codex. Today it started behaving really badly again. After a few hours (sadly) I thought to check /model. It had auto upgraded me to the very broken 5.1 again. Sigh.