Windows PowerShell 5.1 can mojibake BOM-less UTF-8 Markdown in tool calls
This is a focused follow-up to #13755 and the duplicate report #15422.
Summary
On Windows PowerShell 5.1, Codex shell tool calls can read BOM-less UTF-8 Markdown files as mojibake when the file contains non-ASCII text such as Turkish characters. This can make the agent context disagree with the actual file bytes and can cause generated patches to fail or edit the wrong text.
Root-cause hypothesis
The current PowerShell command prefix pins [Console]::OutputEncoding, but Windows PowerShell can still decode BOM-less UTF-8 files through the active ANSI code page unless the whole tool-call shell process is consistently forced to UTF-8.
In practice, commands such as Get-Content and Select-String can return corrupted text for Turkish characters on Windows PowerShell 5.1 even when the underlying file is valid UTF-8.
Reproduction shape
- Use Windows PowerShell 5.1.
- Create a BOM-less UTF-8 Markdown file containing Turkish characters, for example
Hızlı Geçiş Sistemi — doğru çğışöü. - Read it through shell commands used by Codex, especially
Get-Content/Select-String. - Observe mojibake in the agent-visible output, which can then cause patch mismatches.
PowerShell 7 appears to avoid this in the reported cases, but Windows users still commonly have PowerShell 5.1 as the default shell.
Patch outline
I prepared a focused branch with a potential fix here:
https://github.com/evrenonur/codex/tree/codex/powershell-utf8-text-io
The approach is to update the Windows PowerShell startup prefix used by codex-shell-command so that it:
- runs
chcp.com 65001 - sets
[Console]::InputEncoding,[Console]::OutputEncoding, and$OutputEncodingto UTF-8 without BOM - defaults
Get-ContentandSelect-Stringto UTF-8 inside the tool-call shell process - adds a Windows-only regression test for BOM-less UTF-8 Markdown containing Turkish characters
Local validation
I ran:
cargo fmt --package codex-shell-command
cargo test -p codex-shell-command powershell --lib
Both passed locally.
Contribution note
The current contributing guide says external PRs are invitation-only, so I am sharing the analysis, reproduction details, and branch/patch outline here instead of relying on an unsolicited PR.
This issue has 2 comments on GitHub. Read the full discussion on GitHub ↗