When retrieval and extraction live in one function, replacing the access method also changes how you test the application. For an authorized FastPeopleSearch workflow, a small adapter boundary lets you evaluate an API without coupling your parser to provider-specific responses.
The architecture below is not a working site scraper or a claim of verified compatibility. It is an application design that can be tested locally before any permitted live evaluation.
Make the fetch dependency replaceable
The caller should depend on an access result, not on a particular HTTP library or guessed solver method. Give that result a content classification and a diagnostic reference. Keep storage outside the adapter so an unfamiliar response cannot become a successful database update by accident.
from dataclasses import dataclass
from typing import Literal
@dataclass
class AccessResult:
kind: Literal["content", "unresolved", "denied", "error"]
body: str = ""
def process(url, fetch, accept, parse):
result = fetch(url)
if result.kind != "content" or not accept(result.body):
return {"ready_to_save": False}
return {"ready_to_save": True, "record": parse(result.body)}This Python 3.8+ example defines only an internal contract. A production caller must validate the extracted record before writing it. None of the field names is presented as an actual API response schema.
Turn edge cases into fixtures
Inject a fake fetch function that returns expected content, an unresolved verification response, a denial or a transport failure. Use your own synthetic page structure. Testing the control flow does not require downloading or retaining third-party personal records.
Check that only expected, accepted content reaches parsing. Add a fixture that has a plausible page structure but the wrong test identity. Then exercise a parser failure separately. These cases reveal whether a failure is being incorrectly converted into an empty but apparently successful result.
Normalize at one boundary
The real adapter owns authentication configuration, time limits and documented response interpretation. It should map provider output into the internal contract in one place. Do not spread response-envelope handling or brand-name checks across every business parser.
Read current endpoint and authentication requirements from the API documentation. Keep credentials server-side and validate configuration when the worker starts. A generic request example is not evidence of a special FastPeopleSearch endpoint.
Keep state ownership visible
If a confirmed mode requires a continuing context, make that context belong to the approved task. Do not hide a shared mutable session in a module global. Independent workers should not accidentally overwrite each other’s state, and a completed task should have a clear cleanup path.
Use bounded failures rather than internal infinite retry loops. The scheduler needs to know whether to pause for configuration review, investigate compatibility or stop because the target refuses access.
Evaluate live access after the contract is stable
With fixture tests passing, evaluate Scrapingbypass against a small authorized target and a defined acceptance rule. Keep the parser and fixtures unchanged during the comparison so that you are measuring the access change rather than a moving application contract.
Promotion criteria should include accepted content and correct failure behavior. Passing one happy-path request is not enough if rejected responses still reach storage. This design preserves an existing Python workflow while making provider evaluation, diagnostics and later maintenance more predictable.