Natural Language Processing in Python

spaCy is built for production pipelines: fast tokenization, part-of-speech tagging, dependency parsing and named entity recognition, with trained pipelines for many languages and a clean way to add custom components. NLTK is the teaching and research toolkit, with corpora, lexical resources and classic algorithms that are easy to inspect. Stanza, from the Stanford NLP Group, provides neural pipelines across a wide range of languages with accurate syntactic analysis. Gensim focuses on topic modeling and word vectors (LDA, word2vec) and streams large corpora without loading them into memory.

Chinese text needs its own tools. jieba segments text into words, pypinyin converts characters to pinyin, OpenCC converts between Traditional and Simplified Chinese with regional variants, and pangu.py inserts spacing between CJK and Latin characters. For transformer models, embeddings and LLM-based text tasks, see the AI and Agents section, which lists Hugging Face Transformers and Sentence Transformers.