2024Data & Automation
Twitter / X Data Scraper
A Python-based command-line tool that uses Selenium to scrape tweets from Twitter/X — by user profile, hashtag, or search query. Supports flexible authentication, configurable tweet limits, advanced search queries, and CSV export of scraped data.
Le Défi
- Handling Twitter's dynamic, JavaScript-rendered content reliably with Selenium
- Designing a flexible authentication system supporting environment variables, CLI args, and interactive prompts
- Implementing rate-limit-aware scraping without triggering account bans
La Solution
- Used Selenium WebDriver with explicit waits to reliably handle dynamic page rendering.
- Built a tiered authentication system: CLI args → .env variables → interactive prompt fallback.
- Added configurable tweet limits and optional no-limit mode for large-scale data collection.
Fonctionnalités Clés
- Scrape tweets by user profile, hashtag, or search query
- Flexible authentication: CLI args, .env file, or interactive prompt
- Configurable tweet limit (default 50, or unlimited)
- Support for latest and top tweet sorting
- Advanced search query support (matches Twitter's advanced search syntax)
- CSV export with optional poster metadata (followers, following)
Processus
Phase 01
Set up Selenium with ChromeDriver for browser automation
Phase 02
Designed the CLI interface using argparse with multiple authentication and scraping options
Phase 03
Implemented user profile, hashtag, and query-based scraping modes
Phase 04
Added CSV export with optional extended data fields (poster followers/following)