How to extract nested pagination metadata from complex REST API JSON responses using recursive descent JSONPath syntax

The Flat Dream vs. The Nested Nightmare: A Pagination Story

Last month, I was building a connector for a CRM API that returned customer records. The first endpoint I hit was beautiful — a clean, flat JSON response with pagination metadata sitting right at the top level. $.next_cursor gave me everything I needed. Two minutes. Done. Then I hit their activity logs endpoint, and pagination metadata was buried inside a nested envelope structure, split across different levels, sometimes inside arrays, sometimes renamed depending on the result count. What took two minutes on the first endpoint took me six hours on the second.

Here's what I learned from that experience, broken down step by step.

Step 1: Map the Terrain Before You Descend

The Wrong Approach: Guessing and Hardcoding

I started by hardcoding paths like $.meta.pagination.next_url — which worked for exactly one response shape. The moment the API returned a different structure (which it did when results were empty), my parser broke. I was treating a living, breathing API response like a static schema I'd memorized.

The Right Approach: Recursive Descent Scouting

Instead of assuming where pagination lives, use recursive descent to discover it first. Run $..next_url or $..cursor against a sample response to find every occurrence, regardless of depth.

Here's what a real response from that CRM API looked like:

{
  "data": {
    "results": [...42 items...],
    "metadata": {
      "pagination": {
        "current_page": 3,
        "total_pages": 12,
        "next_url": "https://api.example.com/logs?page=4"
      }
    }
  },
  "envelope": {
    "trace_id": "abc-123",
    "pagination": {
      "has_more": true,
      "cursor": "eyJwYWdlIjozfQ=="
    }
  }
}

Notice how pagination metadata appears in two places? One under data.metadata.pagination and another under envelope.pagination. The first has a URL. The second has a cursor token. Both are valid. Both matter. And unless you use recursive descent JSONPath syntax, you'll only find one of them.

Step 2: Understand What Recursive Descent Actually Does

Without Recursive Descent: The Brittle Chain

A standard JSONPath like $.data.metadata.pagination.next_url is a chain of explicit hops. If any node in that chain changes — say the API team renames metadata to meta or wraps results in an extra array — the path breaks silently. You get null. Your pagination loop stops. Your data pipeline quietly truncates at page 3 of 12.

With Recursive Descent: The Depth-Insensitive Search

The .. operator tells the JSONPath engine to search at every depth. $..next_url will find next_url no matter how deeply it's nested. $..pagination will return every pagination object in the response. This is the difference between navigating with a hand-drawn map for one specific city and navigating with a GPS that adapts to any road network.

But here's the catch I learned the hard way: recursive descent can return multiple matches, and the order isn't guaranteed across all JSONPath implementations. If your response has two next_url fields (as mine did), $..next_url returns an array of results, and you need to know which one to use.

Step 3: Filter and Pinpoint with Predicates

The Naive Query: Too Many Results

When I ran $..pagination against my sample response, I got back two objects. That's manageable with 42 records. But on a different API with a heavier envelope structure, I once got back seven pagination objects — some were from cached sub-resources embedded in the response. Picking the wrong one meant I was paginating a nested comment thread instead of the main resource list.

The Filtered Query: Precision Through Predicates

Use JSONPath filter expressions to narrow down. If the real pagination object always contains a total_pages field, you can write:

$..pagination[?(@.total_pages)]

This returns only pagination objects that have a total_pages property. In my case, it filtered out the envelope-level pagination (which only had has_more and cursor) and returned just the one I needed.

Alternatively, if you need the cursor-based pagination from the envelope:

$..pagination[?(@.cursor)]

Same recursive descent. Different predicate. Different result. The predicate is your scalpel — it turns a blunt search into a surgical extraction.

Step 4: Build a Fallback Chain

The Fragile Pattern: Single Path Dependency

My first version of the connector used exactly one JSONPath expression. When the API team shipped a version update that restructured the response, I got 17 emails from users before I even saw the problem. Single-path dependencies are a liability in production data pipelines.

The Resilient Pattern: Ordered Fallbacks

Build a fallback chain using multiple recursive descent expressions, evaluated in priority order:

1. Try $..next_url[?(@)] — look for a direct URL to the next page
2. If null, try $..cursor[?(@)] — look for a cursor token to append
3. If null, try $..pagination[?(@.has_more == true)] — check if there's more data
4. If all fail, assume no more pages and terminate the loop

to fetch — a case where the URL field was omitted entirely but the has_more flag was still true. That remaining 2% would have been silent data loss without the third fallback.

Step 5: Test Against Multiple Response Shapes

The Trap: Testing Only the Happy Path

I'll be honest — I tested my connector against one response. The big one. Forty-two records, full metadata, clean structure. It worked perfectly. I deployed it. Then the API returned an empty results array on the first page of a filtered query, and my connector crashed because $..next_url returned an empty array, not null. I had assumed "no match" meant null, but in many JSONPath implementations, no match means an empty array [], which is truthy in JavaScript and passes a naive if (result) check.

The Fix: Build a Response Shape Library

Save sample responses for every shape the API can return. I now keep a folder of JSON fixtures for every connector I build:

• full_response.json — standard response with results and complete pagination
• empty_results.json — zero results but has_more is true
• last_page.json — results present but no next_url or cursor
• error_envelope.json — error response that still contains a pagination object from a previous request
• deeply_nested.json — response where pagination is buried five levels deep inside embedded sub-resources

Run every JSONPath expression against every fixture before shipping. If $..pagination[?(@.total_pages)] returns something unexpected on the error envelope, you'll catch it at your desk — not in a 2 a.m. alert.

Step 6: Handle Array Index Ambiguity in Results

When Recursive Descent Returns Too Much

Here's a subtle issue that bit me on a different project. The API embedded related resources inside each record, and those related resources had their own pagination objects. My query $..pagination returned eleven matches — one for the main result set and ten for embedded sub-resources. Even with a predicate like [?(@.total_pages)], I still got three matches because some embedded resources also used total_pages.

Pinpointing with Anchored Descent

When recursive descent returns too many matches, combine it with anchored path segments to scope the search. Instead of starting the descent from the root, start it from a known anchor:

$.data..pagination[?(@.total_pages)]

This tells the engine: start at $.data, then descend recursively from there. It excludes pagination objects living under $.envelope or inside unrelated top-level keys. You're still using recursive descent — you're just constraining where it begins.

If you need to be even more precise, you can chain predicates:

$.data..pagination[?(@.total_pages && @.next_url)]

This returns only pagination objects that have both total_pages and a next_url — effectively filtering out cursor-only pagination blocks and embedded resource pagination that lacks a URL field.

Step 7: Mind the Implementation Differences

Not All JSONPath Engines Are Equal

This is the part nobody tells you. JSONPath is not a single specification with a reference implementation. It's a family of implementations that agree on the basics but diverge on the details — and recursive descent is where the divergences show up most painfully.

Here's what I've encountered across different libraries:

• Return type for no matches: Some return null. Some return an empty array. Some return undefined. Know which one your library uses before writing conditional logic.
• Predicate syntax: Some implementations use [?(@.field)]. Others require [?(@['field'])]. A few support both. If your expression works in testing but breaks in production, check whether the runtime library differs from your test library.
• Recursive descent on arrays: Some engines treat arrays as transparent containers during descent — they search inside array elements automatically. Others require explicit array wildcards like $..[*].pagination.
• Multiple match ordering: Most implementations return matches in document order, but a few return them in an implementation-specific order. Never rely on index [0] being the "first" pagination object without verifying against your specific library.

I standardize on one JSONPath library per project and pin the version. If I need to switch libraries, I re-run the full fixture suite from Step 5.

Step 8: Log What You Found, Not Just What You Used

The Debugging Blind Spot

When my connector failed in production, I could see that pagination had broken, but I couldn't see what the API had actually returned. My logs said "no next_url found" — but they didn't capture what was found. Was the pagination object missing entirely? Was it there but renamed? Was it present but the predicate filtered it out?

Diagnostic Logging for Pagination Extraction

Before returning the resolved pagination value, log the full set of recursive descent matches. Not the entire response — just the pagination-related objects:

When $..pagination returns an array of objects, log that array alongside which one you selected and why. This transforms debugging from archaeology into reading. The next time the API changes its shape, your logs will show you the new structure before you even open the documentation.

I also log the fallback chain results — which expression matched, which didn't, and how long each took. Recursive descent is computationally heavier than a direct path lookup, and on large responses, running four fallback expressions sequentially can add up. Logging execution time per expression helped me reorder the chain to try the most common match first (which I was already doing) and identify when a response was large enough to warrant caching the parsed JSON tree.

The Bottom Line

Nested pagination metadata is one of those problems that looks trivial until you're three hours into debugging a silent data truncation in production. The API documentation says pagination lives at $.meta.pagination.next_url, and it does — 87% of the time. The other 13% is where your pipeline quietly breaks.

Recursive descent JSONPath syntax is the tool that bridges that gap. It trades the precision of a hardcoded path for the resilience of a depth-insensitive search, and when combined with predicates, anchored descent, and a fallback chain, it handles the messy reality of production APIs in a way that fixed paths simply cannot.

The workflow I've settled on after that six-hour debugging session is straightforward: scout with $.. to discover every pagination object, filter with predicates to pinpoint the right one, build a fallback chain to handle response shape variations, test against a library of fixture responses, and log everything you find — not just what you use. That workflow has held up across a dozen connectors and at least as many API "improvements" that restructured response envelopes without updating their changelogs.

The two-minute endpoint with flat pagination is a gift. The six-hour endpoint with nested, split, conditional pagination is the norm. Build for the norm. Use recursive descent.