Once a dataset passes a few hundred thousand rows, spreadsheets crawl and crash. Processing a million rows or more reliably needs the right tools and a pipeline that can resume after a failure instead of starting over.
Why spreadsheets stop working
Spreadsheets load everything into memory and recalculate constantly. At scale they slow to a crawl, hit row limits, and one wrong formula can corrupt the lot. They are great for thousands of rows, hopeless for millions.
What we use instead
- Python with Pandas for flexible transformation.
- DuckDB for fast analytical queries on large files.
- PostgreSQL for storage and structured querying.
- Batch processing with progress tracking, so a failure does not cost a full re-run.
Make it resumable: a job over millions of rows will eventually hit a bad record or a timeout. A pipeline that checkpoints and resumes turns a disaster into a shrug.
Typical jobs at this scale
Deduplicating and cleaning CRM or lead lists, migrating data between systems, recurring marketing reports, and e-commerce analytics. The pipeline that does it cleanly is the same one described in our ETL guide — just built for volume.