Cannot force UTF-8

Oh, I forgot to mention it also does stuff like this all the time which causes problems instead of adding proper new lines, it add literal new lines:

if (pointIndex !== null) {\r\n hook.selectedIndex = pointIndex;\r\n hook._isDraggingPoint = true; // Start dragging the selected point\r\n // Clear other selections when selecting a point\r\n hook.selectedLine = null;\r\n hook.selectedShape = null;\r\n hook._hoverLightHandle = null;\r\n if (typeof hook._updateCursor === "function") {\r\n hook._updateCursor();\r\n }\r\n hook._renderPoints();

same

Same here, it broke my whole project adding those ðà on every file

I often encounter this issue as well. Observation shows that it seems to be implemented by writing Python scripts to edit files, sometimes altering the UTF-8 file header to BOM.

Same here

same here, UTF-8 and /n issues

It's incredibly frustrating. Windows, Codex.

This should be fixed in the latest versions of codex.

Same problem here

Same problem

sam problem

same!!
constrant in agents.md helps

@etraut-openai this is definitely not fixed.

To reproduce, create a file with "→↑↓" in it and ask codex what it sees.
With version 0.77.0, on Windows, Codex tells me:

it contains three control characters: \^Z\^X\^Y (displayed as ^Z^X^Y).

It also fails to create files or apply_patch to files with non-ascii characters and can cause mayhem in some cases (deleting files and recreating them because the patch fails). It can also replace non-ascii unicode characters with control symbols and when reviewing it's easy to miss this, so can break an app in quite a subtle way.

If using the VS Code extension, setting it to run in WSL works.
Also, running the same commands as Codex (Get-Content, etc) works fine, it's something in the Codex codebase that doesn't preserve certain Unicode characters.

Reopening

Maybe useful: if I tell Codex to read the file with rg instead of cat or Get-Content it's able to see non-ascii characters (but it still can't use apply_patch because I guess that reads at some point and scrambles the non-ascii characters).

FWIW, my AGENTS.md addition for this issue:

If you read a file and see control characters that look out of place, assume this is an issue reading non-ascii characters in the file. Any attempt to use apply_patch on a line with these control-characters will fail, often with an error like "Failed to find expected lines". You MUST NOT try to bypass this issue by deleting and recreating the file. You MUST MOT write those control glyphs back into the file. You SHOULD hand control back to the user and tell them where you're seeing these glyphs and that they prevent you from modifying certain lines.

I might look into this if there's no objections. It'll be my first Codex PR and I don't know Rust so I might not be fast...

For others facing this, a workaround is to make your terminal use UTF-8 as the code page (assuming that doesn't break other things for you).

You can run chcp 65001 before starting Codex or put it in your PowerShell profile, or just tell Codex to do this for you (e.g. in the VS Code extension).

› Read testing.txt tell me what you see

• I see:

  ASCII: ABCxyz 123 !@#
  Latin1: „” Ž™š á ¤
  COMBINING: eï où aø
  pointers: ␦
  Symbols: ? ? V ? õ æ O p
  CJK: ?? ?? ??
  Emoji: ?? ?? ?? ?? ??


› Set the active code page to utf-8

• Active code page set to 65001 (UTF-8).

› Read testing.txt tell me what you see now

• I see:

  ASCII: ABCxyz 123 !@#
  Latin1: äöü ÄÖÜ ß ñ
  COMBINING: é ö å
  pointers: → ↑ ↓ ← ↔
  Symbols: ● ★ ✓ ✗ § µ Ω π
  CJK: 漢字 かな カナ
  Emoji: 😀 😅 🧪 🚀 🧵

@etraut-openai I see a few options to fix this issue:

1. When Codex is running in PowerShell, switch the code page to UTF-8. This could be controllable via config (and work for other shells that have similar issues). Although for Codex in VS Code this seems like a sensible default (famous last words!).
2. When codex loads, try to work out the active code page of the shell (e.g. if in PowerShell, run chcp) and store this. Then when decoding text (bytes_to_string_smart in text_encoding.rs), use the active code page if known, else fall back to the existing heuristic logic in detect_encoding. Maybe bytes_to_string_smart is too general, since this only applies to stdout/stderr bytes. Thoughts?
3. Similar to 2, but only detect the code page when required and cache it for performance. Less plumbing, no global variable.
4. I notice some work on a read_file tool. Seems like that would ease a lot of these encoding issues and workarounds.

Options 1-3 all have flaws (I'm pretty sure), my vote would be for a tool to read files and save a lot of OS-specific encoding logic. Maybe in the interim log a warning to users if their active code page isn't something that bytes_to_string_smart can handle.

OK I've looked into this, it's quite interesting!

Summary of my findings (noting that I am not an expert in encodings, PowerShell or Rust):

• The issue is that PowerShell cmdlets (like Get-Content) write their text output as bytes to stdout by encoding with a particular encoding
• Common encodings ('Code Pages' in Windows speak) are CP850 (Western Europe, Aus) and CP437 (US)
• Codex runs PowerShell commands and then reads bytes from stdout to get the results of the command, but it doesn't know what encoding was used to produce those bytes
• The code uses chardetng to infer the encoding (see bytes_to_string_smart)
• But chardetng was written for browsers and focuses on web encodings, it can't detect common console encodings like CP850 and CP437
• You could extend the logic of chardetng to also detect these console encodings, but that's not simple.
• You could check which encoding PowerShell will use by calling the Windows GetConsoleOutputCP() API or chcp, but only a subset of shell commands actually use these settings. E.g. rg commands don't. So you'd need to actually look at the shell command to work out which setting it will use, which is probably too brittle.
• The reason that apply_patch fails is because the agent reads the file to change with Get-Content and gets mojibake (wrongly-decoded non-ascii characters), and passes that to apply_patch, but that function reads the file with Rust and gets the correct contents. The two don't match, so the patch fails.

It seems to me the best (least worst) solution is a two pronged approach:

1. Minimise PowerShell cmdlet use by providing alternatives (e.g. a read_file tool to replace Get-Content calls which is probably the lion's share of cases).
2. Attempt to read the shell's encoding settings at startup, and if it isn't 65001 (UTF-8) print a warning that the user may experience some issues, with a link to something showing how to change this (and the option to suppress the warning of course).

But detecting whether the user is likely to have trouble isn't simple because there are three settings that define the encoding, and different commands use different settings! I _think_ the best advice to the average user is to set this in their PowerShell profile:

chcp 65001 | Out-Null
[Console]::OutputEncoding = [System.Text.UTF8Encoding]::new()
$OutputEncoding = [System.Text.UTF8Encoding]::new()

I don't think it's possible to do this in the Codex codebase since they need to be set in the parent shell so they take effect when Codex spawns a shell to run individual commands.

BTW, even with all this fixed, apply_patch can still fail on multi-byte characters (kanji, emojis), because a logging preview function can slice multi-byte characters in half, but that's another issue.

So I'll park work on this now until I hear back...

P.S I now have a lot more sympathy for Windows not being a first class citizen from day one!

Thanks for your analysis and thoughtful summary.

@davidgilbertson thanks for the in-depth conversation here. We actually already merged an experimental fix for this in https://github.com/openai/codex/pull/7902, which you can test out by running codex --enable powershell_utf8 or adding features.powershell_utf8 = true to your config.toml.

It is largely similar to (2) but with minimal changes to the user's shell environment. Please test it out and let us know if this works for you, and if there are specific commands that are still hitting unicode issues (reporting with /feedback is ideal).

How did I miss that! Yep that works for me and should cover the majority of (my) cases.

I'm sure you know, but it will still have trouble with commands that use the chcp encoding (if it's not UTF8), like running a .cmd file with (some) non-ascii characters:

@echo off
rem A note from AI — so it includes an em dash 
echo Hello world

Yep, thanks for flagging that edge case!

Codex just replaced all ≈ symbols of my codebase by ≈

It actually replaces any special character, whether it's accented letters, symbols, anything...