SHEET 37 · Terrain & GIS
County Data Scraper
Playwright pipeline scraping 716 county portals across 30 parallel browser tabs with month-by-month CSV downloads back to 1970
no public build · runs locally
2024localTerrain & GIS
About
A three-phase pipeline that catalogs 716 US counties from Beacon/Schneider Corp, classifies each portal's site architecture, and downloads month-by-month sales CSVs going back to 1970. Runs headed Chromium in a virtual framebuffer (xvfb) to bypass anti-bot measures, supports up to 30 parallel browser tabs, auto-splits date ranges at the 500-record cap, and includes checkpoint/resume for interrupted runs with a live terminal dashboard.
8 FEATURES
Key features
- 01Three-phase pipeline: extraction, classification, downloading
- 02Concurrent browser workers (up to 30 tabs in parallel)
- 03Checkpoint/resume system for long-running scrapes
- 04Live terminal dashboard with progress bars
- 05Automatic month-splitting for capped result sets
- 06Headed Chromium in virtual framebuffer for anti-bot bypass
- 07Single-county, date-range, and retry-failed modes
- 08Structured logging with timestamped log files
1 NEARBY SHEETS
