What this prompt does
This prompt makes the model build a stream-based CSV-to-Postgres loader in TypeScript that never loads the whole file into memory. The core idea is correct for large data: read the file as a stream, validate and coerce each row in a Transform, and bulk-insert with Postgres COPY rather than row-by-row inserts. That combination is what lets a multi-million-row import finish without an out-of-memory crash, and it is the pattern that actually survives real data instead of falling over on the first big file.
The [csv_path] variable points at the source file the loader streams from, and [target_table] is the Postgres table COPY writes into. [row_volume] sizes the progress logging — rows/sec and ETA only mean something when the model knows roughly how big the job is — while [batch_size] controls how many rows accumulate before each COPY flush, trading throughput against memory and how much work a crash can lose. A resumable checkpoint file records the last committed offset so a failed run restarts where it stopped instead of from row one.
When to use it
- Importing large CSV exports (millions of rows) where a naive parse OOMs the process.
- Loading data into Postgres fast, using COPY instead of slow per-row INSERTs.
- Building an ingestion step that must validate and quarantine bad rows, not abort the whole run.
- Running long imports that need to survive a crash and resume from a checkpoint.
- Adding progress visibility (rows/sec, ETA) to a batch job that currently runs blind.
- Handling backpressure so a fast reader doesn't outrun the database writes.
Example output
You get the full loader module plus the package.json dependencies. The pipeline is fs.createReadStream through csv-parse, a Transform that validates and coerces each row and routes rejects to an errors file, then pg-copy-streams performing COPY into [target_table] and flushing every [batch_size] rows. Progress logging reports rows/sec and ETA sized for [row_volume], and a checkpoint file plus backpressure handling and a clean shutdown that commits the in-flight batch round it out. It's a working module you point at your file, not a fragment, so you can run it against a sample and scale up.
Pro tips
- Add the checkpoint early — reprocessing nine million good rows because row 9.1M failed is exactly the pain this avoids.
- Tune
[batch_size]to your memory and crash tolerance: bigger flushes are faster but lose more in-flight work and use more memory. - Make sure
[target_table]columns and types match what the Transform coerces to, or COPY will reject batches. - Set
[row_volume]to a realistic estimate so the ETA and rows/sec readout is meaningful from the start. - Point
[csv_path]at a small sample first to confirm the validation rules before unleashing the loader on the full file. - Inspect the rejects file after a run; routing bad rows aside only helps if you actually review what got quarantined.
