A page fetch is not an agent tool

One of the annoying parts of job hunting is that the task sounds small, but the routine eats time.

Open LinkedIn. Open JobStreet. Open Jora. Type the same keywords again. Change the region again. Click into role after role. Copy the ones that look possible into a tracker. After a while it feels less like applying for jobs and more like doing free data entry for platforms.

jobs-scraper is my third attempt at automating that routine. The first attempt used Make to wire an agentic workflow together. The second moved the workflow into an AI agent and let the agent run it. This third attempt pulls out the part that is reusable and less personal: the crawling layer, standardised into a Skill-like tool that different agents, and potentially other people, can use. I still have more personal automation around scoring roles against my strengths, filtering them, and writing cover letters, but that part depends too much on my own positioning. The crawler is different. Collecting job records, cleaning them up, and handing them to the next workflow is generic enough to split out.

The interesting part of jobs-scraper is not that it can fetch a page. That is only the entrance. The harder questions start once an agent is allowed to call it. What authority does the agent get? What shape comes back? Can upstream job text be mistaken for tool instructions? Can a crawl accidentally become a Google Sheet write? If a source returns 403 or 429, does the tool stop cleanly or become more aggressive?

This is not a guide to bypassing platform protections. The repository uses source-specific public or guest-facing data paths, bounded pacing, explicit failure handling, and a local-first boundary around the tool. An agent-ready tool is not simply a script an agent can run. It is a capability the agent can call without being handed the whole room of keys.

At the researched cutoff, the same local workflow is exposed through four surfaces: a Python CLI, a local STDIO MCP server, an Agent Skill, and a portable Google Sheet Job Tracker. Those surfaces all wrap the same job: collect roles, normalise the records, keep useful local state, and write to the tracker only through explicit operations.

Follow one bounded request through the workflow

Start with a plain request:

Find recent LinkedIn product management jobs in Singapore from the last seven days, include full job descriptions, and do not write to my Sheet yet.

A basic scraper might stop at building a URL, fetching the data, and printing the result. That is not enough once the caller is an agent. The tool needs an entrance that can be constrained.

Here the agent calls an MCP tool. You can think of that as a safer button for the agent to press. It is not arbitrary shell access. The tool converts source, region, query, time range, full-JD mode, and page ceilings into bounded CLI arguments. The same request is also a crawl path, not a Sheet sync path. It can read the source, update local cache artefacts, and return a machine-readable summary. It should not mutate the tracker because the model saw a promising job description or produced an enthusiastic intermediate thought.

The result has to be structured as well. Dumping raw HTML back into the model and asking it to work things out is asking for trouble. Job descriptions are data. They should not become instructions for the tool.

This is where the local-first design helps. Credentials, cache, and tracker configuration remain on the user’s machine. The agent receives a defined capability rather than general authority over the environment. When full JD enrichment is enabled, the runtime still has timeouts, page ceilings, retry/backoff behaviour, soft stops, and explicit upstream error handling. None of that is glamorous. It is what keeps the tool boring in the best possible way.

Diagram showing an agent request moving through an MCP contract, local workflow and source adapters into normalised job records, with explicit Sheet sync and downstream scoring kept separate.

Three sources, three adapter paths

jobs-scraper supports LinkedIn, JobStreet, and Jora. They are all job sources, but they are not the same integration problem.

LinkedIn uses guest-facing list and detail paths with geoId, time range, keywords, and pagination offset in the request construction. JobStreet uses a JSON search API for listings and a GraphQL job-detail query for JD enrichment, rather than relying on HTML detail-page parsing. Jora takes a different route again: HTML list and detail parsing, with bounded retry/backoff when 403 responses appear.

Most readers do not need to memorise those paths. The useful point is why they should not be forced through one generic parser. Each source has its own data shape, regional limits, pagination behaviour, rate-limit response, and detail format. A single “parse whatever page comes back” approach would leak all that mess into the agent. The agent would receive source quirks instead of stable records.

The repository keeps that mess inside source-specific adapters. Each adapter handles its own source, then maps the result into a shared job record. The agent does not need to know how LinkedIn, JobStreet, and Jora each behave on a bad day. It needs a consistent output shape and a clear failure when the source cannot be read.

That is also why I would not describe the project as “beating anti-bot systems”. A browser-like request profile, persistent session, bounded pacing, and 403/429 handling are implementation details. They are not evidence of bypassing authentication, CAPTCHAs, private APIs, access controls, or rate limits. The reusable engineering choice is to isolate source differences inside adapters and keep failure explicit.

Agents need stable records, not raw HTML

Raw HTML is a poor interface for an agent. It mixes data, layout, site text, and noise. If the model receives the whole blob, it has to guess what is job information and what is just page furniture.

jobs-scraper turns source output into a common job record: job identifier, title, company, location, posting time, URL, source, and optional JD content. Once the shape is stable, the rest of the workflow can treat different sources consistently.

State matters too. The workflow deduplicates by source and job identity, so the same job should not be added again on every run. Full JD enrichment can also use a local seen-JD cache to avoid fetching descriptions it has already seen. The tool can add work-mode and visa or constraint signals, but those need to stay in their lane: heuristics, not legal classifications and not proof that a role is right for the user.

One small v1.2.1 default captures the point nicely: title skip filtering is opt-in. The default full-JD path no longer silently skips roles because of title keywords. If users want skip keywords, they provide them. It is not a flashy feature. It is the kind of boring default an agent tool needs, because invisible filtering is still a decision.

Separate read paths from write authority

The Google Sheet boundary is the one I care about most.

“Find jobs for me” and “write these jobs into my tracker” are not the same request. A person understands that difference from context. An agent needs the tool boundary to enforce it, so jobs-scraper separates read-only operations such as crawl, audit, and stats from tracker initialisation and regional sync operations that can change Sheet state.

Tracker initialisation must be able to run as a dry run. Real writes should come after preflight checks. Unsupported source and region combinations, missing configuration, absent region pairs, and incompatible schemas should fail closed rather than guessing.

That may sound conservative. I prefer conservative here. Upstream job text cannot choose credentials, select destinations, change configuration, or upgrade a read into a write. A job description is data, not a command channel.

That boundary also makes the repository safer to publish. A public local tool cannot carry the package author’s workbook metadata, private Sheet identifiers, local secret paths, or defaults that quietly write to the wrong place after installation.

Tracking is not ranking

jobs-scraper is careful not to call tracking “ranking”.

The Job Tracker schema has A:AA columns, but scraper sync owns A and C:K. Columns L:AA are reserved for a later scoring and application workflow: total score, verdict, decision, application strategy, role fit, proof, AI/tech leverage, seniority, company quality, domain advantage, ROI, constraints, positioning, risks, and next action are not judgements made by jobs-scraper itself. The safer description is that jobs-scraper collects, normalises, deduplicates, and routes jobs into an agent-ready scoring surface. It does not decide which job is best for me, and it should not package personalised ranking as a scraper feature.

That split matters because agent workflows get messy when one tool tries to do everything: collect data, judge it, rank it, mutate state, and choose the next action. jobs-scraper keeps the boundary narrower. It prepares job data for the next workflow; the application decision belongs somewhere else.

Publicising a private script is product engineering

A private script can be messy and still feel useful. Paths are hard-coded. Sheet names are assumed. Credentials live somewhere on the author’s machine. The author knows which button is safe to press, so the whole thing works well enough.

Hand that to an agent, or to another person installing the repository fresh, and those private assumptions become traps. The agent does not know which Sheet it may write to. Another user should not inherit my local setup. At that point, publishing the repo is not just an upload. The safety boundaries that used to live in my head have to move into the tool.

From v1.0.0 to v1.2.1, the work was broadly about turning a working local multi-source crawler into something more shareable: a portable tracker, region-aware MCP, private metadata cleanup, behaviour-preserving architecture cleanup, and opt-in clean defaults. None of that sounds as exciting as “new source supported”. It is still the difference between a private helper and a tool someone else can actually run.

CI is part of that receipt, not a magic certificate. The exact v1.2.1 candidate CI run completed successfully; the pytest coverage run reported 178 passed, 5 warnings, and 70% coverage. That 70% is reported coverage, not an enforced floor, and CI success does not prove live-source uptime. It does show that, for that commit and that test environment, packaging, interfaces, behaviour, and repository hygiene were checked.

That is why I only pulled the crawler layer into the public repo. Scoring, filtering, and cover-letter writing are too tied to my strengths and positioning. Collecting jobs and shaping them for the next workflow is different. That layer can be standardised. Get the plumbing right once, and every agent does not have to keep rebuilding the same pipework. After doing that a few times, trust me, the pipework gets old.

The first test for Agent-Ready Tools

For this series, jobs-scraper gives me the first test:

An agent-ready tool is not a script an agent can run. It is a local capability an agent can call within explicit boundaries.

That capability needs stable inputs, structured outputs, clear failures, separate write authority, untrusted-content rules, and enough verification to make the boundary believable. jobs-scraper is a useful first case because the real work was not fetching one page. It was turning three inconsistent job sources into a bounded, local-first, testable tool.

It still does not prove that live sources will stay stable forever. It does not settle platform terms. It does not perform personalised ranking by itself. Those claims would need another evidence layer or another workflow.

As Part 01, the practical question is simple: before giving a tool to an agent, do not only ask whether the script runs. Ask where the boundary is, whether the output is stable, whether failure is explicit, whether writes require separate authority, and whether the behaviour can be verified. Those answers matter more than one successful fetch.

References