Conclusion: When JSON extraction drifts for public pages, fix it by separating retrieval quality from parsing logic: confirm response completeness first, then update selectors only with evidence.
Symptoms
A monitoring job that previously extracted stable fields starts returning empty values, partial objects, or inconsistent results across runs. The job may still “succeed” technically, but the dataset becomes unreliable.
Likely causes
Drift often comes from page structure changes, regional variants, partial payloads, or a change in redirect landing pages. Without a minimal evidence record (final URL, body length, and a sentinel), it is hard to know which layer broke.
Troubleshooting order
- 1) Confirm final URL: record the final URL after redirects and compare to the approved allowlist.
- 2) Check completeness baseline: compare body length to a known-good baseline.
- 3) Add one sentinel: track one stable marker that indicates the expected block is present.
- 4) Review drift timeline: correlate the first bad sample to deploys and known source changes.
- 5) Update parsing with evidence: change selectors only after retrieval evidence is stable.
Fixes
Start with the smallest safe change: tighten redirect targets, adjust timeouts only when payloads are truly incomplete, and maintain a small set of known-good pages for regression sampling.
FAQ
Should we change selectors immediately?
No. If the payload is incomplete, selector changes will not help and will hide the real issue.
How do we measure drift without collecting too much data?
Keep a minimal evidence set for diagnostics: final URL, body length, and a short classification. Avoid storing sensitive personal data.
What is a safe way to stabilize the workflow?
Use controlled sampling, baselines, and change control so fixes are repeatable and measurable.