Text Processing in Python

Text from the outside world often arrives in an unknown or broken encoding. charset-normalizer and chardet detect the encoding of raw bytes (Requests uses charset-normalizer), and ftfy repairs mojibake and other Unicode damage after decoding. For comparing strings, RapidFuzz computes fuzzy similarity scores quickly with a C++ core, while the standard library's difflib computes diffs and similarity ratios without dependencies but more slowly.

For structured text, pyparsing and parsy build parsers from small combinable pieces in plain Python, sqlparse splits and formats SQL, phonenumbers parses and validates international phone numbers, and Pygments highlights source code in many languages. Babel formats dates, numbers and currencies by locale and manages translation catalogs. python-slugify and Unidecode turn Unicode text into ASCII for URLs and identifiers, and shortuuid creates short, URL-safe unique IDs.