Large CSV Exports Taught Me to Design for Interruption

A CSV export often begins as a loop: query posts, build rows, send a file. That approach worked until I ran it against a real content set with custom fields, SEO metadata, relationships, and media references. Memory grew, output buffers delayed the response, browser requests hit practical limits, and escaped schema content exposed differences between parsers. Fixing one bottleneck at a time taught me a broader lesson: large admin tools should assume interruption. The operator needs bounded work, a stable cursor, visible progress, and an output format that survives the messy values already stored in production.

The first failure was memory-shaped

Loading the complete dataset and assembling one giant string multiplies memory use. WordPress objects, metadata caches, transformed values, and the final CSV can all exist at once. I replaced that with batched selection and streamed rows as they were encoded.

Streaming alone was not enough. PHP and web-server output buffers could still retain data, so I explicitly managed the response path and verified that bytes left the process during the export rather than at the end.

A long request was still a fragile request

Even with stable memory, a browser download can run into proxy, host, or request time limits. I added a browser-driven batching workflow that processes a bounded page of records, advances a stable cursor, and combines the result without rereading earlier rows.

I prefer ID-based progression to offsets when content may change during the run. New or edited records can make offsets skip or repeat rows; an ordered last-seen ID gives the operation a clearer resume point.

CSV escaping is a compatibility contract

Real values contained quotes, backslashes, newlines, templated schema fragments, and strings that were not standalone JSON. A “cleanup” step could easily corrupt them. I kept the raw field semantics and made the exporter and importer agree on one CSV dialect.

I tested round trips, not just readable downloads: export the value, parse it through the destination path, and compare the meaningful result. That caught escape behavior that looked plausible in a text editor but changed after a second parser touched it.

The interface needed diagnostics, not a spinner

For a long operation, “working” is not enough. I added batch progress, processed counts, warnings, errors, and a downloadable trace. If a row was partially updated before another field failed, the report said so.

This changed support conversations. Instead of rerunning the whole file and hoping, the operator could identify a specific row, reason, and last successful boundary.

The safeguards I now reuse

• Bound query and processing batch sizes.
• Advance through a stable ordered cursor.
• Stream or persist results instead of holding them all in memory.
• Make every batch safe to retry.
• Preserve raw source values unless a transformation is explicit.
• Separate warnings, hard failures, and partial writes.
• Test with the untouched customer file as well as fixtures.

I also keep export and import compatibility tests together. A format owned by two code paths needs one shared contract.

Scale exposed product decisions

The technical fixes reduced memory and timeouts, but the real improvement was operational. An editor could understand what happened and recover without developer-only access. That makes an internal tool safer and cheaper to use.

Large-data reliability is not only about faster loops. It is about making partial progress trustworthy. Once a process can resume, explain itself, and preserve source meaning, scale becomes a predictable workload rather than a special event.

Import and export had to evolve together. Once the exporter preserved more metadata, the importer needed to understand the same column names, empty-value rules, and escape behavior. I added compatibility coverage across both implementations rather than letting each repository maintain a slightly different dialect. When one side changed, the round-trip test exposed it immediately.

I also resisted treating CSV as a complete site-migration format. Flat rows can carry taxonomies, selected custom fields, and portable identities, but parent-child relationships, cross-row references, serialized editor state, and media dependencies require an explicit remapping strategy. The tool now reports what it supports and warns about values it cannot safely infer. This keeps a useful content workflow from being marketed internally as a universal backup. Format boundaries are part of reliability: operators should know whether they are moving articles, reconstructing a content model, or cloning a site, because those are different jobs.

Security stayed in the design as the operation grew. Export actions require an appropriate capability and nonce, generated files live only as long as needed, and spreadsheet-formula prefixes are handled deliberately so opening the CSV does not execute untrusted cells. Scale should not turn an admin convenience into a data-exposure path.

Interruption is now one of my acceptance tests

Before building another bulk WordPress tool, I would design the cursor and result report first. Then I would force a timeout halfway through a test run and confirm the operation can resume without duplicates or lost rows. If interruption is safe, the happy path usually becomes much easier to trust.

Photo by RDNE Stock project on Pexels.