Non-ASCII log lines are silently dropped on Windows consoles (cp1252), including production JSON logs #7592

Title

Non-ASCII log lines are silently dropped on Windows consoles (cp1252), including production JSON logs

Environment

• livekit-agents 1.8.1
• Python 3.12.10
• Windows, console code page cp1252
• Non-English agent (ar-EG)

Description

On Windows, sys.stdout uses the cp1252 code page, which cannot encode Arabic, CJK, emoji, or accented Latin. The CLI's root log handler does not set an encoding or an error handler, so every log line containing non-ASCII characters raises UnicodeEncodeError inside logging and the message is discarded — replaced by a --- Logging error --- traceback.

The relevant code:

• livekit/agents/cli/log.py:208 — handler = logging.StreamHandler(sys.stdout), no errors= and no encoding reconfiguration anywhere in the CLI.
• livekit/agents/cli/log.py:147 — production logs are serialized with json.dumps(..., ensure_ascii=False), so raw (unescaped) non-ASCII is written to the stream.

This affects both dev and production output. In production the JSON log line is silently lost, which means agent transcripts in a non-English language can go missing from logs entirely.

Notably, this got worse after #2280 / PR #2622. Before that change, ensure_ascii=True escaped non-ASCII to \uXXXX, which is representable in cp1252 and therefore safe on Windows. Emitting raw characters now requires the stream to actually support UTF-8.

Reproduction

import sys, logging

logging.basicConfig(level=logging.INFO, stream=sys.stdout, force=True)
print("stream encoding:", sys.stdout.encoding)
logging.getLogger("demo").info("عاوز واحد شاورما فراخ")
print("done")

Actual output:

stream encoding: cp1252
done
--- Logging error ---
Traceback (most recent call last):
  File "...\logging\__init__.py", line 1163, in emit
    stream.write(msg + self.terminator)
  File "...\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 7-10: character maps to <undefined>
Message: '\u0639\u0627\u0648\u0632 \u0648\u0627\u062d\u062f \u0634\u0627\u0648\u0631\u0645\u0627 \u0641\u0631\u0627\u062e'

Note that the log message never appears, and only "done" is printed. The failure is per-line, so a single Arabic-heavy session produces a flood of these tracebacks and buries any genuinely useful log output.

Suggested fixes

Any one of these would resolve it:

1. Set errors="replace" on the handler so encoding failures degrade instead of dropping the line:

   handler = logging.StreamHandler(sys.stdout, errors="replace")

2. Reconfigure the streams to UTF-8 at CLI startup:

   for stream in (sys.stdout, sys.stderr):
       stream.reconfigure(encoding="utf-8", errors="replace")

3. Honour the PYTHONIOENCODING / PYTHONUTF8 environment variables.

Option 1 is the smallest change and is safe when stdout has been replaced with a non-TextIOWrapper object. Note that the job subprocess needs the same treatment, so a fix applied only in the parent process would not be sufficient.

Impact

Anyone building a non-English voice agent on Windows. Voice agents are frequently multilingual, and transcripts are the primary debugging surface for this SDK, so the affected logs are exactly the ones needed to debug STT or turn-detection issues.

Workaround

Call sys.stdout.reconfigure(encoding="utf-8", errors="replace") at module import time in your agent.py (not inside if __name__ == "__main__", so job subprocesses inherit it), or run with PYTHONUTF8=1.