asciicheck.py flags CRLF line endings as invalid characters (false positive on Windows checkouts)

Open 💬 0 comments Opened Aug 27, 2026 by xu-kai-quan

Description

scripts/asciicheck.py (run in CI via .github/workflows/repo-checks.yml as ./scripts/asciicheck.py README.md) reads a file, decodes it as UTF-8, and flags every character outside 0x20-0x7E (plus a small allowlist) as an error:

for lineno, line in enumerate(text.splitlines(keepends=True), 1):
    for colno, char in enumerate(line, 1):
        codepoint = ord(char)
        if char == "\n":
            continue
        if (
            not (0x20 <= codepoint <= 0x7E)
            and codepoint not in allowed_unicode_codepoints
        ):
            errors.append((lineno, colno, char, codepoint))

Only \n is exempted — \r (U+000D) is not. If the file being checked has CRLF line endings (e.g. checked out on Windows with core.autocrlf=true, a common Git-for-Windows default), every single line is reported as containing an "invalid character". --fix doesn't help either: \r isn't in the substitutions map, so the rewritten file still contains \r and the script still exits non-zero.

The repo has no .gitattributes line forcing eol=lf for the files this script checks, so a Windows contributor who checks out the repo with the common core.autocrlf=true setting and runs this script locally against README.md gets a wall of false-positive errors, even though the intent of the script (per its own docstring) is to catch non-breaking spaces/smart quotes/dashes, not legitimate line endings.

Steps to reproduce

from pathlib import Path
p = Path("crlf_test.md")
p.write_bytes(b"hello world\r\nsecond line\r\n")
$ python scripts/asciicheck.py crlf_test.md
Invalid character at line 1, column 12: U+000D (\r)
Invalid character at line 2, column 12: U+000D (\r)
$ echo $?
1

Expected behavior

CRLF line endings should not be flagged as invalid characters.

Actual behavior

Every line of a CRLF file produces a spurious "Invalid character ... U+000D (\r)" error, and --fix cannot resolve it since \r isn't in the substitution table.

Environment

  • Commit: 7c37479 (main, 2026-08-27), Python 3.14, Windows 11
  • File: scripts/asciicheck.py:93-94
  • No existing test file covers this script.

Suggested fix

-            if char == "\n":
+            if char in ("\n", "\r"):
                 continue

View original on GitHub ↗