Make Large WordPress CSV Exports Safe to Resume

Large WordPress CSV exports can fail even when each individual row is easy to generate. Loading all posts and metadata at once uses memory quickly. A long browser request may time out, and a retry may repeat work unless the exporter knows where it stopped. In a CSV workflow with custom fields and SEO data, I addressed those problems with smaller batches, streamed output, and a stable record position. The export also needed clear progress and error reporting, so an editor could distinguish a slow run from a failed one and recover without starting everything again.

Keep memory use bounded

Loading the complete dataset and assembling one giant string multiplies memory use. WordPress objects, metadata caches, transformed values, and the final CSV can all exist at once. I replaced that with batched selection and streamed rows as they were encoded.

Streaming alone was not enough. PHP and web-server output buffers could still retain data, so I explicitly managed the response path and verified that bytes left the process during the export rather than at the end.

Use a stable position for each batch

Even with stable memory, a browser download can run into proxy, host, or request time limits. I added a browser-driven batching workflow that processes a bounded page of records, advances a stable cursor, and combines the result without rereading earlier rows.

I prefer ID-based progression to offsets when content may change during the run. New or edited records can make offsets skip or repeat rows; an ordered last-seen ID gives the operation a clearer resume point.

Preserve values through export and import

Real values contained quotes, backslashes, newlines, templated schema fragments, and strings that were not standalone JSON. A “cleanup” step could easily corrupt them. I kept the raw field semantics and made the exporter and importer agree on one CSV dialect.

I tested round trips, not just readable downloads: export the value, parse it through the destination path, and compare the meaningful result. That caught escape behavior that looked plausible in a text editor but changed after a second parser touched it.

Show progress and row-level failures

For a long operation, “working” is not enough. I added batch progress, processed counts, warnings, errors, and a downloadable trace. If a row was partially updated before another field failed, the report said so.

This changed support conversations. Instead of rerunning the whole file and hoping, the operator could identify a specific row, reason, and last successfully processed record.

Build retries into the workflow

• Bound query and processing batch sizes.
• Advance through a stable ordered cursor.
• Stream or persist results instead of holding them all in memory.
• Make every batch safe to retry.
• Preserve raw source values unless a transformation is explicit.
• Separate warnings, hard failures, and partial writes.
• Test with the untouched customer file as well as fixtures.

I also keep export and import compatibility tests together. A format used by both code paths needs one shared format specification.

Document what the CSV format supports

The technical fixes reduced memory and timeouts, but the real improvement was operational. An editor could understand what happened and recover without developer-only access. That makes an internal tool safer and cheaper to use.

Large-data reliability is not only about faster loops. It is about making partial progress trustworthy. Once a process can resume, explain itself, and preserve source meaning, scale becomes a predictable workload rather than a special event.

Import and export had to evolve together. Once the exporter preserved more metadata, the importer needed to understand the same column names, empty-value rules, and escape behavior. I added compatibility coverage across both implementations rather than letting each repository maintain a slightly different dialect. When one side changed, the round-trip test exposed it immediately.

I also resisted treating CSV as a complete site-migration format. Flat rows can carry taxonomies, selected custom fields, and portable identities, but parent-child relationships, cross-row references, serialized editor state, and media dependencies require an explicit remapping strategy. The tool now reports what it supports and warns about values it cannot safely infer. This keeps a useful content workflow from being marketed internally as a universal backup. Clear format limits help operators use the tool correctly: operators should know whether they are moving articles, reconstructing a content model, or cloning a site, because those are different jobs.

Security stayed in the design as the operation grew. Export actions require an appropriate capability and nonce, generated files live only as long as needed, and spreadsheet-formula prefixes are handled deliberately so opening the CSV does not execute untrusted cells. Scale should not turn an admin convenience into a data-exposure path.

Test an interrupted run before relying on it

Test the export with a realistic file size and values containing quotes, newlines, and backslashes. Interrupt a run halfway through, then resume it and compare the result with the source records. Check for missing rows and duplicates. Import a representative export into a disposable environment to confirm that supported fields retain their meaning. A reliable bulk tool should make progress visible, explain failures, and give the operator a clear recovery step when a request ends early.

Photo by RDNE Stock project on Pexels.