Crawl the web
with Hermes, pay per call.
Enter a URL and fetch the data. Your agents crawl web pages, extract rich metadata, and build datasets — without you managing scraping infrastructure. Billed per call.
Real Hermes flows that rely on this skill.
Each pattern below is a single Hermes Agent flow with Web Fetch via Locus registered as one skill. The math works because the flow only spends when it actually fetches a page.
Audit and compare pages at scale.
A Hermes research flow receives a list of URLs, fetches each page as a skill call, and assembles a metadata audit — titles, descriptions, canonicals, Open Graph tags, heading structure. Per-call billing keeps each audit proportional to the number of pages checked; idle weeks cost zero.
Fetch web context for any query.
A Hermes RAG flow receives a question, identifies relevant URLs, fetches each page as a skill call, and injects clean text into the LLM context window. No proxy management, no rendering pipeline — the skill returns embedding-ready text in the same call that fetches the page.
Build LLM training data ethically.
A Hermes flow crawls a list of public URLs to build training datasets. robots.txt is respected on every call — data is sourced within the rules the web has set. Clean text output from the skill reduces preprocessing overhead before tokenization.
Track competitor pages for changes.
A scheduled Hermes flow fetches competitor pricing pages, documentation, or news sources on a cron and diffs the structured output. The flow only spends when a fetch fires; Hermes budget caps keep cost predictable across high-frequency monitoring schedules.
Stop building scrapers into your flows.
Register one skill, connect a wallet, and your Hermes Agent flow can fetch any web page on every relevant call. Idle flows cost zero. Scraper migrations cost zero. That's the entire deal.
- Single Hermes skill
- 100% robots.txt compliance
- Flow caps honored
- MCP + pay-per-call native
Hermes Agent specific questions.
If something below doesn't cover your case, ping us — we work directly with Hermes Agent builders, no SDR funnel.
How does this register as a Hermes Agent skill?
+
It's a standard POST endpoint with a typed body. You register it in Hermes the same way you'd register any HTTP skill — endpoint, schema, per-call price. Hermes Agent uses the price to plan flow budgets and to show the user what each fetch will cost before the flow fires.
Does Hermes need a scraping account or proxy service?
+
No. The gateway manages rendering, retries, and delivery entirely. Hermes pays per call from a wallet you connect — no scraping vendor account is created, and there's no proxy credential to rotate or leak in flow logs.
What happens when a Hermes flow hits its budget cap mid-crawl?
+
The skill returns whatever page data it has resolved so far and Hermes stops issuing new fetch calls. The flow can decide whether to request more budget or continue with the partial dataset.
Does the skill respect robots.txt?
+
Yes, always. Every call checks robots.txt before retrieving the page. If crawling is disallowed, the skill returns a clear error. Hermes flows can handle this as a normal tool boundary — skip the URL, log the miss, or surface it to the user.
Can Hermes call this concurrently across many flows?
+
Yes. Per-key rate limits apply with concurrency-fair queueing — a high-volume flow never starves another flow on the same key.
Does the skill handle JavaScript-rendered pages?
+
Yes. The gateway renders JavaScript before returning content, so single-page apps and dynamically loaded pages are captured in the same skill call.