readme_toc.py misparses tilde and longer Markdown fences
Summary
The README TOC generator only recognizes backtick fences and treats every line beginning with three backticks as a close. It therefore includes headings from tilde fences and from content inside a longer backtick fence.
Reproduction
Run the README TOC generator on:
~~~markdown
Before
~~~markdown
Not a heading
~~~
After
~~~
Also try a four-backtick fence containing a literal triple-backtick line.
Expected: headings inside either fenced block are excluded, and the longer fence remains open until a closing fence with the same marker and at least the same length.
Actual: headings inside tilde fences are treated as headings, and a triple-backtick content line can prematurely close a longer backtick fence.
Root cause
The parser used a boolean toggled by line.strip().startswith("`"); it did not track fence marker type or length.
PR-ready reference
I prepared a tested reference branch containing a minimal parser fix and regression tests:
- Commit: https://github.com/Hughhhhcoder/codex/commit/57516831cf353aab3a73c4e99136ab987a00d1db
- Branch:
codex/codex-readme-fence
The patch tracks the fence character and opening length, ignores backtick openers whose info string contains a backtick, and accepts only valid matching closers.
Validation
python -m pytest -c /tmp/codex-empty.ini --noconftest -q scripts/test_readme_toc.py — 3 passed.
Because the current contribution policy does not accept external code contributions or pull requests, I am submitting this analysis and reference commit here instead of opening a PR. Please feel free to implement or adapt the fix.