Extracting and Cleaning URLs from Large API Responses, Safely

An API response with a few thousand nodes in it will happily hide four hundred URLs across a dozen nesting levels, half of them escaped, a third of them duplicated, and a handful pointing at a CDN you did not know you were using. Getting a clean list out of that is a five-minute job — or a thirty-minute one, depending on how you go about it.

The mechanics of extraction are covered in How to Extract All URLs from an API Response. This article is about the part that matters more as the payload gets larger and more real: doing it without handing the payload to someone else, and handling the cases that quietly produce a wrong list.

The risk nobody prices in

Search for a URL extractor and most of the results are server-side. You paste, the browser POSTs, a process somewhere parses it and sends back a list. That is an entirely reasonable architecture, and it means your payload was transmitted to a third party and may have been written to a request log on the way.

For a public API response, who cares. But the responses people actually need to untangle are rarely public ones. They are staging payloads with real customer records in them, internal service responses, Postman exports carrying an environment block, GraphQL results from an endpoint that is not documented yet. A pasted Postman collection frequently contains a bearer token in a header field, and a pasted environment export contains whatever secrets that environment holds.

The practical rule: if you would not put the payload in a public Gist, do not put it in a tool that uploads. Anything running entirely in the browser sidesteps the question — there is no request to log.

Unescaping, which is where most lists go wrong

JSON strings escape forward slashes optionally, and a great many servers take that option. So a URL that renders as https://cdn.example.com/a/b.png may sit in the raw response as:

"https:\/\/cdn.example.com\/a\/b.png"

That is valid JSON and it decodes to the right thing — but only if you decode it. A regex run over the raw text rather than the parsed value will capture the backslashes and give you a list of URLs that 404 when you use them. The same applies to HTML entities: a URL that has been through an HTML-encoding step arrives with & where it should have &, so every query string with more than one parameter breaks.

There is more on why this happens in Why JSON URLs Have Backslashes.

The fix is ordering: parse first, then match. Extract from the decoded values, not from the text of the document.

Filtering that actually narrows things

A raw extraction from a large response is mostly noise — schema URLs, xmlns declarations, tracking pixels, the same avatar CDN four hundred times. Three filters do most of the work:

• By scheme. Keep http and https, drop data:, mailto: and javascript:. Base64 data: URIs in particular can be enormous and will dominate the output.
• By host. Usually you want one origin — your CDN, or the service you are auditing. Everything else is someone else’s problem.
• By path substring. /uploads/, /v2/, .png. This is the filter that turns four hundred results into the eleven you were looking for.

Then deduplicate. Order-preserving deduplication is worth insisting on: it keeps the list in document order, which makes it possible to correlate a URL back to where it came from.

The edge cases worth knowing about

Stringified JSON inside JSON. A field whose value is a JSON document encoded as a string is common in webhook payloads and event logs. Its URLs are double-escaped and one pass will not reach them. You need to detect the nested document and parse it too.

GraphQL array payloads. GraphQL responses nest under data, and lists of objects mean the same key path repeats. Extraction has to walk arrays as well as objects, or you get the first element of each list and nothing else.

Postman and Insomnia exports. These are not API responses at all — they are collection documents where URLs live in structured request.url objects, often split into host, path and query arrays, and sprinkled with {{variable}} placeholders. A URL with an unresolved placeholder is not a usable URL, and whether you want those in your list depends on what you are doing.

Protocol-relative URLs. //cdn.example.com/a.png is a real URL that a scheme filter will discard unless you normalise it first.

A workflow that holds up

1. Paste the raw response — do not pretty-print it first, since that can change escaping.
2. Parse, then extract from decoded values.
3. Filter by scheme, then host, then path.
4. Deduplicate, preserving order.
5. Export as TXT for a shell loop, or CSV if you need it in a spreadsheet.

The API URL Extractor does exactly this sequence in the browser, which means step zero — deciding whether the payload is safe to paste — stops being a question you have to answer.

If you are working from a request you copied out of DevTools rather than a response you already have, the cURL to Code converter will turn it into something you can run and capture properly first.