The Challenge of Large Product Catalogs
As agentic commerce scales, merchants with thousands or even millions of products face a critical challenge:
How do AI agents efficiently process massive product catalogs without overwhelming memory or network resources?
Traditional JSON feeds load the entire catalog into memory at once — which works fine for small stores, but becomes problematic as catalogs grow.
A 50,000-product feed can easily exceed 100MB, causing slow loading times, high memory usage, and potential timeouts for AI agents processing these feeds.
Today, we're solving this problem with NDJSON streaming.
What Is NDJSON Streaming?
NDJSON (Newline-Delimited JSON) is a format where each line of a file is a valid, self-contained JSON object.
Instead of a single large JSON array, an NDJSON feed is a stream of individual product records, one per line.
This approach offers several key advantages:
1. Incremental Processing
AI agents can read and process products line-by-line, rather than loading the entire feed into memory.
This means an agent can start processing products immediately, even while the feed is still being downloaded.
2. Memory Efficiency
With traditional JSON, a 100MB catalog must be fully loaded into memory before processing can begin.
With NDJSON streaming, only one product (a few KB) needs to be in memory at any given time.
3. Parallel Processing
Different parts of an NDJSON feed can be processed in parallel, enabling distributed systems to handle massive catalogs more efficiently.
4. Resumable Downloads
If a download is interrupted, processing can resume from the last successfully parsed line, rather than starting over.
How NDJSON Works in GXO Feeds
When you generate a product feed through GXO, you now receive two format options:
JSON Format (Traditional)
The complete catalog as a single JSON array — ideal for smaller catalogs (< 5,000 products) or when full catalog access is needed immediately.
[
{ "id": "product-1", "title": "Product 1", ... },
{ "id": "product-2", "title": "Product 2", ... },
...
]
NDJSON Format (Streaming)
The same catalog as a stream of JSON objects, one per line — ideal for large catalogs (5,000+ products) or when incremental processing is preferred.
{"id": "product-1", "title": "Product 1", ...}
{"id": "product-2", "title": "Product 2", ...}
...
Both formats are ACP-compliant and contain identical product data — the only difference is how they're structured for consumption.
