|0.1.0||Apr 16, 2019|
#1989 in Text processing
41 downloads per month
Used in shift_or_euc_c
A Japanese legacy encoding detector for detecting between Shift_JIS, EUC-JP, and, optionally, ISO-2022-JP given the assumption that the encoding is one of those.
This detector is generally more accurate (but see below about the failure mode on half-width katakana) and decides much sooner than machine learning-based detectors. To decide EUC-JP, machine learning-based detectors try to gain confidence that the input looks like EUC-JP. To decide EUC-JP, this detector instead looks for two simple rule-based signs of the input not being Shift_JIS.
As a consequence of not containing machine learning tables, the binary size
footprint that this crate adds on top of
encoding_rs is tiny.
See the file named COPYRIGHT.
Sample Program Usage
- Install Rust
git clone https://github.com/hsivonen/shift_or_euc
cargo run --example detect PATH_TO_FILE
The program prints one of:
Principle of Operation
The detector is based on two observations:
- The ISO-2022-JP escape sequences don't normally occur in Shift_JIS or EUC-JP, so encountering such an escape sequence (before non-ASCII has been encountered) can be taken as indication of ISO-2022-JP.
- When normal (full-with) kana or common kanji encoded as Shift_JIS is decoded as EUC-JP, or vice versa, the result is either an error or half-width katakana, and it's very uncommon for Japanese HTML to have half-width katakana character before a normal kana or common kanji character. Therefore, if decoding as Shift_JIS results in error or have-width katakana, the detector decides that the content is EUC-JP, and vice versa.
The detector gives the wrong answer if the text has a half-width katakana character before normal kana or common kanji. Some uncommon kanji are undecidable. (All JIS X 0208 Level 1 kanji are decidable.)
The half-width katakana issue is mainly relevant for old 8-bit JIS X 0201-only text files that would decode correctly as Shift_JIS but that the detector detects as EUC-JP.
The undecidable kanji issue does not realistically show up when a full document is fed to the detector, because, realistically, in a full document, there is at least one kana or common kanji. It can occur, though, if the detector is only run on a prefix of a document and the prefix only contains the title of the document. It is possible for document title to consist entirely of undecidable kanji. (Indeed, Japanese Wikipedia has articles with such titles.) If the detector is undecided, falling back to Shift_JIS is typically the Web oriented better guess.