Web Scraping in Python

Scrapy is a complete crawling framework: spiders, scheduling, throttling, retries, item pipelines and export to files or databases. It is the best fit for large crawls of server-rendered HTML. Pages that build their content with JavaScript need a real browser instead. Browser Use and Stagehand put a language model in charge of a browser so you can describe what to extract instead of writing selectors, and Crawl4AI crawls pages and returns clean, LLM-ready Markdown. Scrapling focuses on resilience, with selectors that can relocate elements after a site's layout changes.

Once you have the HTML, Trafilatura extracts the main text and metadata from article pages and drops navigation and boilerplate. markdownify converts HTML to Markdown, and feedparser reads RSS and Atom feeds, which are often an easier source than scraping. For general HTML parsing, Beautiful Soup and lxml are in the HTML Manipulation section, and Playwright and Selenium are listed under Testing.