Changelog

Note

Entries marked “via Claude” were developed with Claude Code. Dan directed the design, reviewed all output, and takes responsibility for the result. Unmarked entries by Dan were written without AI assistance.

7.7.0 (2026-10-17)

Features:

  • Added a -m/--mime-type flag to the chardetect CLI, which shows the detected MIME type next to the encoding, like file -i. (Dan Blanchard via Claude, #384)

Accuracy:

  • Visual-order Hebrew is now detected. The ISO-8859-8 model trains on both bidi conventions, so 1990s visual-order pages come back as ISO-8859-8 while logical-order text still prefers Windows-1255. (Dan Blanchard via Claude)

  • cp864 Arabic now trains on contextually shaped text in visual order, the way real cp864 systems stored it, instead of isolated letter forms in logical order. (Dan Blanchard via Claude)

Performance:

  • Large inputs no longer need max_bytes to stay fast. On the compiled build, detect(data, max_bytes=len(data)) on 272 MiB takes about 0.23s for cp1252 and 0.56s for Shift_JIS, down from several seconds. UTF-8 validation and a final decode of the winner still read the whole window, so the answer still decodes it. The other stages stop at a 256 KB evidence cap, which sits above the default max_bytes, so default calls are unchanged. ADR-0006 has the design. (Dan Blanchard via Claude)

  • The UTF-7 + scan is no longer quadratic on line-wrapped base64 like PEM certificates, MIME attachments, and JWTs. 128 KiB of it went from 5.4 seconds to 1.2 milliseconds. (Dan Blanchard via Claude)

  • A head full of unclosed <meta or <?xml tags no longer makes the markup scan quadratic, taking it from about 5 ms to about 20 µs. A literal < inside an attribute value now hides that one tag. (Dan Blanchard via Claude)

  • The pure-Python install on CPython scores bigram models with operator.itemgetter instead of a Python loop, making detect() about 1.25x faster and detect_all() about 1.35x faster across the test corpus. Compiled builds and PyPy keep their existing paths. (Afonso Ribeiro)

Bug Fixes:

  • Language labels on Unicode text no longer tilt toward accent-heavy languages, so a French paragraph with a few accents stops coming back as Hungarian. Models and input are now scored in the same IDF-weighted space. Mislabels on the corpus’s Unicode files fall from 140 to 104 of 1587 (French from 20 to 4), and encodings, confidences, and MIME types do not move. (Dan Blanchard via Claude)

  • Null-separated ASCII records with short fields, like find -print0 output, no longer come back as UTF-16-LE at 0.95 with Chinese as the language. They get the answer longer records already got: binary, or ASCII when the nulls are sparse. (Dan Blanchard via Claude)

  • Every answer now decodes the examined window under the name the caller receives. A BOM, a BOM-less UTF-16/32 null pattern, an ISO-2022 or HZ escape designation, or a <meta charset>/<?xml encoding> declaration was trusted without decoding the rest of the window, so a confident answer could raise further in. Each now has to decode the whole window, and the 5.x/6.x display names EUC-JP, SHIFT_JIS, Big5, and ISO-2022-JP are used only when those narrower codecs decode it too. (Dan Blanchard via Claude; Samina, #394, #396)

  • A charset= inside another attribute, like a refresh URL, no longer beats a real <meta charset>, and a declaration naming an unknown encoding no longer hides a later one that works. A repeated attribute now keeps its first value, and a <meta tag that never closes is ignored. (Sash, #389)

  • The charset parameter of a content attribute may now be quoted or have spaces around its =. (Huanyi Xie, #398)

  • Markup declarations are read the way the WHATWG prescan reads them. Declarations inside <!-- --> comments are ignored, x-user-defined means Windows-1252, and a UTF-16 or UTF-32 label in ASCII markup means UTF-8, so an ASCII page declaring utf-16le no longer comes back as UTF-16-LE at 0.95. (Dan Blanchard via Claude)

  • PEP 263 cookies are resolved the way tokenize.detect_encoding resolves them, so iso-latin-1 and similar spellings now work, utf-8-sig means UTF-8, and a long first line no longer hides a cookie on line 2. (Dan Blanchard via Claude)

  • A PEP 263 cookie on line 2 counts only when line 1 is blank or a comment, as in Python, so a coding= comment after a line of code is ignored. (Huanyi Xie, #391)

  • An EBCDIC <?xml encoding="..."> declaration now reports text/xml instead of text/html, which fixes the MIME type for ten EBCDIC RSS feeds in the test corpus. (Dan Blanchard via Claude)

  • detect_all(prefer_superset=True) no longer lists the same name twice. Greek text came back as Windows-1253 at two confidences; each name now appears once, at its higher confidence. (Dan Blanchard via Claude)

  • prefer_superset=True no longer swaps in a Windows code page that cannot decode the data, like Windows-1252 for ISO-8859-1 text holding 0x81. apply_preferred_superset takes an optional data argument for the check. (Dan Blanchard via Claude)

  • prefer_superset=True now respects include_encodings and exclude_encodings, keeping the detected name when the filters rule out its superset. (00200200, #393)

  • chardetect now reads one byte past its 200 KB cap, so a multi-byte sequence cut at the cap is treated as truncated input instead of the end of the file. (00200200, #392)

  • lookup_encoding’s cache is now capped at 1024 entries. Charset names come from untrusted markup, so a stream of distinct spellings could grow it without limit. (Samina, #390)

  • UniversalDetector no longer treats a buffer filled to exactly max_bytes as the whole stream, which could turn a correct utf-8 answer into a single-byte codec when a multi-byte sequence was cut at the boundary. (Dan Blanchard via Claude)

  • detect_all() no longer puts a demoted niche Latin encoding back near the top. The demoted entry now takes the confidence of the candidate it sits behind, so the ranking is ordered by confidence as documented. (Dan Blanchard via Claude)

  • Language detection now honors max_bytes instead of always reading up to 2 KB. (Dan Blanchard via Claude)

  • The deprecated chardet.equivalences shim is reachable again after a bare import chardet, resolves every name the old module had, and passes patches through to the module that owns each name, so mock.patch.object on it restores cleanly. It warns when a name is used rather than at import, so importing it no longer fails under -W error::DeprecationWarning. (Dan Blanchard via Claude)

  • A mostly-ASCII Windows-1252 file with a single Ö or ö no longer detects as HP-Roman8. The niche Latin demotion now compares the candidate and its replacement on the bytes they read differently, so one Welsh ŵ still keeps ISO-8859-14 and a Kven đ in Finnish still keeps ISO-8859-10. Some corpus files that came back as ISO-8859-1 now come back as Windows-1252, an accepted superset. (António Afonso via Claude, #383)

  • An English file whose only non-ASCII letter is a capital É, À, or Ó no longer detects as MacRoman, which reads those bytes as the ellipsis and curly quotes. The era-prevalence prior now compares the two readings on the bytes they disagree on instead of standing down. (Dan Blanchard via Claude, #383)

Deprecations:

  • EncodingInfo.is_multibyte is now is_multibyte_cjk, since the flag marks CJK byte structure rather than width and is off for UTF-8, UTF-16 and UTF-32. The old attribute still works with a DeprecationWarning and will be removed in 8.0. EncodingInfo is internal and never reaches detect() callers. (Dan Blanchard via Claude)

Documentation:

  • The supported encodings table no longer lists UTF-8, UTF-16, UTF-32 and UTF-7 as single-byte. The multi-byte column used to copy the registry flag that marks CJK byte structure for the structural stage, which is off for the Unicode encodings on purpose. The generator now asks each codec whether any character takes more than one byte. (Dan Blanchard via Claude)

Build:

  • pipeline/markup.py is now compiled by mypyc (15 modules), so the markup superset promotion that moved there no longer runs interpreted in compiled wheels. (Dan Blanchard via Claude)

  • Test coverage is back to 100%, and CI now enforces it. (Dan Blanchard via Claude)

  • The models.bin, rowmax.bin, idf.bin, and langnorm.bin formats now have one owner, chardet.models._format, shared by the trainer and the runtime loader. (Dan Blanchard via Claude)

  • scripts/verify_no_overlap.py --require-provenance fails unless the training metadata records an exclusion set matching the current test data. The shipped models record none until the next full retrain. (Dan Blanchard via Claude)

7.6.0 (2026-08-14)

Performance:

  • Compiled wheels now score bigram profiles through a small Cython kernel alongside mypyc, and the pair is 4.7x faster than the pure wheel on CPython 3.14. _kernel.py stays plain Python (PyPy and pure wheels run it interpreted, unchanged), _kernel.pxd adds C types at build time and ships nothing, and detection output is bit-identical. The kernel declares itself safe without the GIL, so free-threaded CPython scales instead of silently re-enabling the GIL on import: 3.14t runs the whole suite in ~340ms across 8 threads, the fastest configuration measured. Compiled builds now need both hooks:

    HATCH_BUILD_HOOK_ENABLE_MYPYC=true HATCH_BUILD_HOOK_ENABLE_CUSTOM=true uv build

    (Dan Blanchard via Claude)

  • Added support for CPython 3.15, including the free-threaded build. No code changes were needed. (Dan Blanchard via Claude)

Bug Fixes:

  • Fixed delimited ASCII data like |NAME,+LAY| misdetecting as UTF-7, a follow-up to #371. Two new checks: the whole buffer must actually decode as UTF-7 (+| is an illegal shift, so tabular data fails immediately), and a block encoding a single code unit must land in a script range where a lone shifted character plausibly occurs. +LAY decodes to U+2C06, Glagolitic; no genuine lone block in the corpus lands anywhere like it, while em dashes, ellipses, kanji, and accented letters all pass. (Dan Blanchard via Claude)

  • Signed UTF-7 no longer reads as ASCII. The BOM stage recognizes the four UTF-7 signature prefixes (+/v8- and friends) when the rest of the buffer decodes as UTF-7 — the prefix alone is ordinary ASCII (a diff of V8 source paths starts with +/v8). This is a deliberate divergence from WHATWG’s browser-security exclusion of UTF-7: chardet already detects the unsigned form, so refusing only the signed one made no sense. (Dan Blanchard via Claude)

  • detect() no longer returns an encoding that cannot decode the input it was given (#380). When the whole input has been examined and the winner’s only multi-byte evidence is an incomplete trailing sequence, the best candidate that decodes the input completely wins instead. Genuinely truncated data keeps its answer: CJK cut mid-character, or input sliced at max_bytes. (Dan Blanchard via Claude)

  • Short apostrophe-heavy English is no longer labeled Scottish Gaelic or Breton. A rare-language label on an input under 128 bytes now needs a 0.03 lead over the best mainstream language, mirroring the encoding-side arbitration in ADR-0005. (Dan Blanchard via Claude)

  • Hungarian text no longer loses to a Czech reading: confusion rescoring compares tied pairs only under language models both encodings have. (Dan Blanchard via Claude)

  • Space-padded text no longer matches a degenerate Serbian model at high confidence. Statistical scoring now skips repeated-whitespace bigrams, matching the whitespace collapse training already applies. Fixed 21 files plus a long-standing GB2312 known failure. (Dan Blanchard via Claude)

  • EBCDIC text is no longer invisible to the early pipeline. The binary stage treats EBCDIC’s 0x05/0x15 tab and newline as whitespace when the data is high-byte-dominated, and the markup stage reads charset declarations through a cp037 decode of the head. (Dan Blanchard via Claude)

  • Fixed the last two EBCDIC sibling misdetections: a letter reading beating punctuation is no longer evidence by itself, and low-confidence near-ties scan deeper but need the category vote and the bigram rescore to agree. (Dan Blanchard via Claude)

  • Fixed training normalization gaps that starved ISO-8859-16 (legacy cedilla forms) and the 26 pre-euro encodings (the euro sign) at exactly the bytes that discriminate them from their siblings. Cedilla folds to comma-below and the euro to the currency sign wherever the target encoding cannot represent them. (Dan Blanchard via Claude)

Improvements:

  • Retrained every bigram model on a refreshed, deduplicated corpus with the ADR-0004 hardening: whitespace collapses after encoding, retention guards hard-fail mostly unencodable corpora, Serbian gets real Latin-script text, CP1006 gains its sixteen missing Urdu letters, and wiki markup is stripped before bigram counting. Training provenance is now recorded per model, so test data added after a retrain is detectable as such. (Dan Blanchard via Claude)

  • New ANSI-art model. cp437 detection now includes a profile trained on 16,621 text-mode art files from 16colo.rs, keyed under the zxx pseudo-language and reported with language=None. (Dan Blanchard via Claude)

  • Rare-language arbitration: a low-confidence statistical winner from a language with no documented legacy-encoding population (Scottish Gaelic, Welsh, Irish, Breton) yields to a near-tied mainstream candidate. Genuine Celtic text wins by landslides and is unaffected; eight boundary sentinels in the test suite guard the gate. Design and evidence in ADR-0005. (Dan Blanchard via Claude)

  • Confusion-group resolution is context-aware: votes are counted per occurrence, letter readings with no word shape are demoted, and art-model wins are exempt. Fixed twelve EBCDIC and Latin files that were riding single-byte coin flips. (Dan Blanchard via Claude)

  • Statistical dead heats no longer resolve by candidate enumeration order. Three tiebreaks: prefer the Windows superset, prefer the more prevalent era when there is no high-byte evidence, and prefer a classic-Mac candidate when line endings are bare \r. (Dan Blanchard via Claude)

  • Training pipeline hardening after a cache-loss post-mortem: retrains that would silently drop a model now abort loudly, caches filter in place instead of being deleted wholesale, and the artpack fetcher builds into a temporary directory. (Dan Blanchard via Claude)

7.5.1 (2026-08-06)

Bug Fixes:

  • Fixed markup-declared encodings being reported under a name that can’t decode the input. A page declaring Shift_JIS but using CP932 extension characters (like ①) came back as SHIFT_JIS, which fails .decode() on those same bytes. Superset promotion (CP932, CP949) now always fires when the reported name can’t decode the data but the superset can. (Dan Blanchard via Claude)

  • Fixed a lying charset declaration beating genuine UTF-8 content. A UTF-8 page declaring <meta charset="iso-8859-1"> came back as ISO-8859-1, which decodes to mojibake. Valid multi-byte UTF-8 now wins over a conflicting declaration; pure ASCII and real single-byte content still honor it. (Dan Blanchard via Claude)

  • Fixed BOM-less UTF-16 byte-order detection for pure-CJK text. With no ASCII in the sample, the only null bytes come from the low byte of characters like U+4E00 (一), which sit in the wrong parity position, so short Chinese UTF-16 samples came back with reversed endianness at full confidence. Byte order is now chosen by decoding both ways and comparing text quality, with the null signal breaking near-ties. Found by scoring chardet against charset-normalizer’s char-dataset. (Dan Blanchard via Claude)

7.5.0 (2026-08-05)

Bug Fixes:

  • Fixed multi-byte encodings being eliminated when the input ends in an incomplete character. Byte-validity filtering decoded with a one-shot strict decode, which cannot tell a truncated tail from corrupt data, so a single dangling lead byte dropped every CJK candidate and the result came down to input-length parity — a 184-byte GBK sample detected as GB18030, the same sample minus one byte as Windows-1256. This was also reachable on complete, well-formed files, because chardet slices its own input at max_bytes and at _SCAN_LIMIT in _validate_bytes(): a valid 14 kB GBK page with an honest <meta charset="gbk"> lost its declaration, and with it text/html and 0.95 confidence, whenever byte 4096 happened to split a character. Validity checks now decode incrementally with final=False, deferring a partial trailing character while still rejecting corruption anywhere before it. (António Afonso via Claude, #376)

  • Fixed compat_names (the default) leaking internal Python codec names for seven encodings. detect() now returns ISO-8859-2, ISO-8859-6, ISO-8859-13, Windows-1250, Windows-1256, Windows-1257, and CP874 instead of their lowercase codec spellings. These were absent from _COMPAT_NAMES after the 7.1.0 switch to codec-name canonicals, which made default output inconsistent with their siblings (e.g. cp1250 vs Windows-1251) and with the encoding-name table in Usage. (António Afonso via Claude, #374)

  • Fixed compat_names (the default) leaking the internal cp932 codec name. detect() now returns CP932 instead of cp932, matching its Japanese siblings (shift_jis_2004 → SHIFT_JIS) and the value chardet 5.x/6.x returned. (uttam12331, #375)

Performance:

  • Statistical scoring now skips single-byte models that provably can’t beat the current runner-up, using per-model row-maximum tables (rowmax.bin). Results are bit-identical: multi-byte models are always scored in full and detect_all() bypasses pruning. Mean detection time dropped ~2.9x with mypyc, with the largest gains on legacy CJK (p99 from 9.5ms to 3.3ms). (Dan Blanchard via Claude)

  • Model tables are now bytes (native array indexing under mypyc, instead of boxed memoryview calls), and the model blob is decompressed in chunks rather than one shot: peak process memory dropped from 53.9 to 27.4 MiB. (Dan Blanchard via Claude)

  • Confusion-group resolution and post-processing now use bytes.translate prefilters instead of per-byte Python scans, making near-tie resolution cheaper on large inputs. (Dan Blanchard via Claude)

Improvements:

  • prefer_superset=True is now documented as the recommended mode and will become the default in chardet 8.0. Detection examines at most max_bytes of input, so only the superset encoding is guaranteed to decode bytes beyond that window — the same reasoning behind the WHATWG/W3C Encoding Standard’s rule that browsers decode ascii and iso-8859-1 content as windows-1252. Callers that depend on subset names should start passing prefer_superset=False explicitly. (Dan Blanchard via Claude)

  • chardet.equivalences is now a deprecation shim. Accuracy-evaluation predicates (is_correct, is_equivalent_detection, etc.) moved to chardet.evaluation; public-API encoding-name remapping (apply_compat_names, apply_preferred_superset) moved to chardet.output_names. Existing imports keep working with a DeprecationWarning. chardet.equivalences will be removed in 8.0. (Dan Blanchard via Claude)

  • Internal pipeline reorganization: language detection, markup-superset promotion, and post-processing rank corrections moved out of the orchestrator into pipeline/language.py, pipeline/markup.py, and pipeline/postprocess.py respectively. No behavior change. The two new modules are also added to the mypyc compilation list. (Dan Blanchard via Claude)

7.4.3 (2026-04-13)

Bug Fixes:

  • Fixed ValueError: embedded null character crash when input contained a <meta charset> declaration with a null byte in the encoding name (e.g. b'<meta charset="\x00utf-8">'). codecs.lookup() raises ValueError on embedded nulls, and lookup_encoding() was only catching LookupError. Also added defensive ValueError catches in _validate_bytes() and _to_utf8() for completeness. (Dan Blanchard via Claude, #369)

7.4.2 (2026-04-12)

Bug Fixes:

  • Fixed RuntimeError: pipeline must always return at least one result on ~2% of all possible two-byte inputs (e.g. b"\xf9\x92"). Multi-byte encodings like CP932 and Johab could score above the structural confidence threshold on very short inputs, but then statistical scoring would return nothing, leaving the pipeline with an empty result list instead of falling through to the no_match_encoding fallback. (Jason Barnett via Claude, #367, #368)

Improvements:

  • Added ~90 encoding aliases from the WHATWG Encoding Standard and IANA Character Sets registry so that <meta charset> labels like x-cp1252, x-sjis, dos-874, csUTF8, and the cswindows* family all resolve correctly through the markup detection stage. Every alias was driven by a failing spec-compliance test. (Dan Blanchard via Claude, #366)

  • Added a spec-compliance test suite covering Python decode round-trips for all 86 registry encodings, WHATWG web-platform label resolution, IANA preferred MIME names, and Unicode/RFC conformance (BOM sniffing, UTF-8 boundary cases, UTF-16 surrogate pairs). This is the test suite that would have caught the 7.4.1 BOM bug before release. (Dan Blanchard via Claude, #366)

7.4.1 (2026-04-07)

Bug Fixes:

  • BOM-prefixed UTF-16 and UTF-32 input now reports utf-16 and utf-32 instead of the endian-specific variants. Python’s utf-16-le/utf-16-be/utf-32-le/utf-32-be codecs keep the BOM as a U+FEFF in the decoded string, while utf-16/utf-32 strip it, so callers passing the detection result directly to .decode() were getting a stray BOM at the start of their text. BOM-less UTF-16/32 detection (via null-byte patterns) is unchanged and still returns the endian-specific name. (Dan Blanchard via Claude, #364, #365)

7.4.0 (2026-03-26)

Performance:

  • Switched to dense zlib-compressed model format (v2): models are now stored as contiguous memoryview slices of a single decompressed blob, eliminating per-model struct.unpack overhead. Cold start (import + first detect) dropped from ~75ms to ~13ms with mypyc. (Dan Blanchard via Claude, #354)

Accuracy:

  • Accuracy improved from 98.6% to 99.3% (2499/2517 files) through a combination of training and scoring improvements:

    • Eliminated train/test data overlap by content-fingerprinting test suite articles and excluding them from training data (#351)

    • Added MADLAD-400 and Wikipedia as supplemental training sources to fill gaps left by exclusion filtering (#351)

    • Improved non-ASCII bigram scoring: high-byte bigrams are now preserved during training (instead of being crushed by global normalization), and weighted by per-bigram IDF so encoding-specific byte patterns contribute proportionally to how discriminative they are (#352)

    • Added encoding-aware substitution filtering: character substitutions during training now only apply for characters the target encoding cannot represent

    • Increased training samples from 15K to 25K per language/encoding pair (Dan Blanchard via Claude)

Bug Fixes:

  • Added dedicated structural analyzers for CP932, CP949, and Big5-HKSCS: these superset encodings previously shared their base encoding’s byte-range analyzer, missing extended ranges unique to each superset (Dan Blanchard via Claude, #353)

7.3.0 (2026-03-24)

License:

  • 0BSD license — the project license has been changed from MIT to 0BSD, a maximally permissive license with no attribution requirement. All prior 7.x releases should also be considered 0BSD licensed as of this release. (Dan Blanchard via Claude)

Features:

  • Added mime_type field to detection results — identifies file types for both binary (via magic number matching) and text content. Returned in all detect(), detect_all(), and UniversalDetector results. (Dan Blanchard via Claude, #350)

  • New pipeline/magic.py module detects 40+ binary file formats including images, audio/video, archives, documents, executables, and fonts. ZIP-based formats (XLSX, DOCX, JAR, APK, EPUB, wheel, OpenDocument) are distinguished by entry filenames. (Dan Blanchard via Claude, #350)

Bug Fixes:

  • Fixed incorrect equivalence between UTF-16-LE and UTF-16-BE in accuracy testing — these are distinct encodings with different byte order, not interchangeable (Dan Blanchard via Claude)

Performance:

  • Added 4 new modules to mypyc compilation (orchestrator, confusion, magic, ascii), bringing the total to 11 compiled modules (Dan Blanchard via Claude)

  • Capped statistical scoring at 16 KB — bigram models converge quickly, so large files no longer score the full 200 KB. Worst-case detection time dropped from 62ms to 26ms with no accuracy loss. (Dan Blanchard via Claude)

  • Replaced dataclasses.replace() with direct DetectionResult construction on hot paths, eliminating ~354k function calls per full test suite run (Dan Blanchard via Claude)

Build:

  • Added riscv64 to the mypyc wheel build matrix — prebuilt wheels are now published for RISC-V Linux alongside existing architectures (Bruno Verachten, #348)

7.2.0 (2026-03-17)

Features:

  • Added include_encodings and exclude_encodings parameters to detect(), detect_all(), and UniversalDetector — restrict or exclude specific encodings from the candidate set, with corresponding -i/--include-encodings and -x/--exclude-encodings CLI flags (Dan Blanchard via Claude, #343)

  • Added no_match_encoding (default "cp1252") and empty_input_encoding (default "utf-8") parameters — control which encoding is returned when no candidate survives the pipeline or the input is empty, with corresponding CLI flags (Dan Blanchard via Claude, #343)

  • Added -l/--language flag to chardetect CLI — shows the detected language (ISO 639-1 code and English name) alongside the encoding (Dan Blanchard via Claude, #342)

7.1.0 (2026-03-11)

Features:

  • Added PEP 263 encoding declaration detection — # -*- coding: ... -*- and # coding=... declarations on lines 1–2 of Python source files are now recognized with confidence 0.95 (Dan Blanchard via Claude, #249)

  • Added chardet.universaldetector backward-compatibility stub so that from chardet.universaldetector import UniversalDetector works with a deprecation warning (Dan Blanchard via Claude, #341)

Fixes:

  • Fixed false UTF-7 detection of ASCII text containing ++ or +word patterns (Dan Blanchard, #332, #335)

  • Fixed 0.5s startup cost on first detect() call — model norms are now computed during loading instead of lazily iterating 21M entries (Dan Blanchard via Claude, #333, #336)

  • Fixed undocumented encoding name changes between chardet 5.x and 7.0 — detect() now returns chardet 5.x-compatible names by default (Dan Blanchard via Claude, #338)

  • Improved ISO-2022-JP family detection — recognizes ESC sequences for ISO-2022-JP-2004 (JIS X 0213) and ISO-2022-JP-EXT (JIS X 0201 Kana) (Dan Blanchard via Claude)

  • Fixed silent truncation of corrupt model data (iter_unpack yielded fewer tuples instead of raising) (Dan Blanchard via Claude)

  • Fixed incorrect date in LICENSE (Dan Blanchard)

Performance:

  • 5.5x faster first-detect time (~0.42s → ~0.075s) by computing model norms as a side-product of load_models() (Dan Blanchard via Claude)

  • ~40% faster model parsing via struct.iter_unpack for bulk entry extraction (eliminates ~305K individual unpack calls) (Dan Blanchard via Claude)

New API parameters:

  • Added compat_names parameter (default True) to detect(), detect_all(), and UniversalDetector — set to False to get raw Python codec names instead of chardet 5.x/6.x compatible display names (Dan Blanchard via Claude)

  • Added prefer_superset parameter (default False) — remaps legacy ISO/subset encodings to their modern Windows/CP superset equivalents (e.g., ASCII → Windows-1252, ISO-8859-1 → Windows-1252). This will default to ``True`` in the next major version (8.0). (Dan Blanchard via Claude)

  • Deprecated should_rename_legacy in favor of prefer_superset — a deprecation warning is emitted when used (Dan Blanchard via Claude)

Improvements:

  • Switched internal canonical encoding names to Python codec names (e.g., "utf-8" instead of "UTF-8"), with compat_names controlling the public output format. See Usage for the full mapping table. (Dan Blanchard via Claude)

  • Added lookup_encoding() to registry for case-insensitive resolution of arbitrary encoding name input to canonical names (Dan Blanchard via Claude)

  • Achieved 100% line coverage across all source modules (+31 tests) (Dan Blanchard via Claude)

  • Updated benchmark numbers: 98.2% encoding accuracy, 95.2% language accuracy on 2,510 test files (Dan Blanchard via Claude)

  • Pinned test-data cloning to chardet release version tags for reproducible builds (Dan Blanchard via Claude)

7.0.1 (2026-03-04)

Fixes:

  • Fixed false UTF-7 detection of SHA-1 git hashes (Alex Rembish, #324)

  • Fixed _SINGLE_LANG_MAP missing aliases for single-language encoding lookup (e.g., big5 → big5hkscs) (Dan Blanchard)

  • Fixed PyPy TypeError in UTF-7 codec handling (Dan Blanchard)

Improvements:

  • Retrained bigram models — 24 previously failing test cases now pass (Dan Blanchard via Claude)

  • Updated language equivalences for mutual intelligibility (Slovak/Czech, East Slavic + Bulgarian, Malay/Indonesian, Scandinavian languages) (Dan Blanchard via Claude)

7.0.0 (2026-03-02)

Ground-up, 0BSD-licensed rewrite of chardet (Dan Blanchard via Claude, #322). Same package name, same public API — drop-in replacement for chardet 5.x/6.x.

Highlights:

  • 0BSD license (previous versions were LGPL)

  • 96.8% accuracy on 2,179 test files (+2.3pp vs chardet 6.0.0, +7.7pp vs charset-normalizer)

  • 41x faster than chardet 6.0.0 with mypyc (28x pure Python), 7.5x faster than charset-normalizer

  • Language detection for every result (90.5% accuracy across 49 languages)

  • 99 encodings across six eras (MODERN_WEB, LEGACY_ISO, LEGACY_MAC, LEGACY_REGIONAL, DOS, MAINFRAME)

  • 12-stage detection pipeline — BOM, UTF-16/32 patterns, escape sequences, binary detection, markup charset, ASCII, UTF-8 validation, byte validity, CJK gating, structural probing, statistical scoring, post-processing; the markup stage’s PEP 263 declaration sniffing was requested by patrikha in #249

  • Bigram frequency models trained on CulturaX multilingual corpus data for all supported language/encoding pairs

  • Optional mypyc compilation — 1.49x additional speedup on CPython

  • Thread-safe detect() and detect_all() with no measurable overhead; scales on free-threaded Python 3.13t+

  • Negligible import memory (96 B)

  • Zero runtime dependencies

Breaking changes vs 6.0.0:

  • detect() and detect_all() now default to encoding_era=EncodingEra.ALL (6.0.0 defaulted to MODERN_WEB)

  • Internal architecture is completely different (probers replaced by pipeline stages). Only the public API is preserved.

  • LanguageFilter is accepted but ignored (deprecation warning emitted)

  • chunk_size is accepted but ignored (deprecation warning emitted)

6.0.0.post1 (2026-02-22)

  • Fixed __version__ not being set correctly in the package (Dan Blanchard)

6.0.0 (2026-02-22)

Features:

  • Unified single-byte charset detection with proper language-specific bigram models for all single-byte encodings (replaces Latin1Prober and MacRomanProber heuristics) (Dan Blanchard)

  • 38 new languages: Arabic, Belarusian, Breton, Croatian, Czech, Danish, Dutch, English, Esperanto, Estonian, Farsi, Finnish, French, German, Icelandic, Indonesian, Irish, Italian, Kazakh, Latvian, Lithuanian, Macedonian, Malay, Maltese, Norwegian, Polish, Portuguese, Romanian, Scottish Gaelic, Serbian, Slovak, Slovene, Spanish, Swedish, Tajik, Ukrainian, Vietnamese, Welsh (Dan Blanchard)

  • EncodingEra filtering via new encoding_era parameter (Dan Blanchard)

  • max_bytes and chunk_size parameters for detect(), detect_all(), and UniversalDetector; chunked processing was proposed by deedy5 in #284 (Dan Blanchard)

  • -e/--encoding-era CLI flag (Dan Blanchard via Claude)

  • EBCDIC detection (CP037, CP500) (Dan Blanchard)

  • Direct GB18030 support (replaces redundant GB2312 prober) (Dan Blanchard)

  • Binary file detection (Dan Blanchard)

  • Python 3.12, 3.13, and 3.14 support (Hugo van Kemenade, #283)

  • GitHub Codespaces support (oxygen dioxide, #312)

Breaking changes:

  • Dropped Python 3.7, 3.8, and 3.9 (requires Python 3.10+)

  • Removed Latin1Prober and MacRomanProber

  • Removed EUC-TW support

  • Removed LanguageFilter.NONE

  • detect() default changed to encoding_era=EncodingEra.MODERN_WEB

Fixes:

5.2.0 (2023-08-01)

  • Added support for running the CLI via python -m chardet (Dan Blanchard)

5.1.0 (2022-12-01)

5.0.0 (2022-06-25)

4.0.0 (2020-12-10)

  • Added detect_all() function returning all candidate encodings (Damien, #111)

  • Converted single-byte charset probers to nested dicts (performance) (Dan Blanchard, #121)

  • CharsetGroupProber now short-circuits on definite matches (performance) (Dan Blanchard, #203)

  • Added language field to detect_all output (Dan Blanchard)

  • Switched from Travis to GitHub Actions (Dan Blanchard, #204)

  • Dropped Python 2.6, 3.4, 3.5

3.0.4 (2017-06-08)

3.0.3 (2017-05-16)

3.0.2 (2017-04-12)

  • Fixed detect sometimes returning None instead of a result dict (Dan Blanchard, #114)

3.0.1 (2017-04-11)

  • Fixed crash in EUC-TW prober with certain strings (Dan Blanchard)

3.0.0 (2017-04-11)

chardet 2.3.0 (2014-10-07)

  • Added CP932 detection (hashy)

  • Fixed UTF-8 BOM not detected as UTF-8-SIG (atbest, #32)

  • Switched chardetect to use argparse (Dan Blanchard)

chardet 2.2.1 (2013-12-18)

  • Fixed missing parenthesis in chardetect.py (Owen, #12)

chardet 2.2.0 (2013-12-16)

Merged the charade fork back into chardet, unifying Python 2 and Python 3 support under the original package name.

charade 1.0.3 (2013-01-18)

charade 1.0.2 (2013-01-18)

charade 1.0.1 (2012-12-03)

charade 1.0.0 (2012-12-02)

  • Initial release: Python 3 port of chardet, forked as a separate package (Ian Cordasco)

chardet 2.1.1 (2012-10-01)

  • Bumped version past Mark Pilgrim’s last release

  • chardetect can now read from stdin (Erik Rose)

  • Fixed BOM byte strings for UCS-4-2143 and UCS-4-3412 (Toshio Kuratomi)

  • Restored Mark Pilgrim’s original docs and COPYING file (Toshio Kuratomi)

chardet 1.1 (2012-07-27)

chardet 1.0.1 (2008-04-19)

  • Packaging fix, added egg distributions for Python 2.4 and 2.5 (Mark Pilgrim)

chardet 1.0 (2006-12-23)

  • Initial release: Python 2 port of Mozilla’s universal charset detector (Mark Pilgrim)