Build a multi-agent web scraper with CrewAI that respects robots.txt
A CrewAI scraper where plain Python fetches pages politely (robots.txt, rate limits) and two agents only summarise and fact-check. Tested code and output.
Most CrewAI scraping demos give the agents a fetch_url tool and let them crawl. It looks good in a notebook. In practice the model decides which links to follow, every page passes through the context window, and nobody checks robots.txt or how fast the requests go.
This post builds the same kind of pipeline the other way round. Plain Python finds the pages, checks robots.txt, waits between requests and parses every field that exists in the HTML. Two CrewAI agents do only the part that needs a model: one writes a one-line summary and topic tags for each page, the other checks those against the source. The target is books.toscrape.com, a fake bookstore built for scraping practice.
The code runs on crewai 1.15.44 (current on PyPI on 22 November 2026) with Python 3.13. There is no API key in this post. The model sits behind a small stub class, and the line where a real model goes is marked. The fetch, parse, robots.txt and rate-limit parts hit the real site, and the output below is from a real run.
The design#
plain Python (no model) CrewAI (model)
┌──────────────────────────────────────────────────────────┐ ┌─────────────────────────────┐
│ robots.txt ─> category page ─> book links ─> book pages │ │ analyst ──> fact checker │
│ │ (1 req/s per host, page budget, retries) │ │ summary + grounded? │
│ └─> can_fetch() before every request │ │ topics per record │
│ parse: title, price, stock, UPC, description (≤1500 ch) │──┤ guardrail: one record/UPC │
└──────────────────────────────────────────────────────────┘ └─────────────┬───────────────┘
v
merge on UPC ─> books.json + books.csv ─> eval
The split follows one rule. If a value is in the HTML, code extracts it. Price, stock count, title and UPC come from CSS selectors, so they are exact and cost nothing. A model only touches free text, where a summary or a label is the point.
This is a sequential pipeline in the terms of multi-agent architecture patterns: the order is fixed in code, each stage does one job, and the cost is one model call per stage. That post covers when a pipeline beats a supervisor or a critic loop, so this one stays on the scraper.
Why not let agents crawl#
Three numbers make the case.
- Context. Say the fetch tool truncates each page to 12,000 characters. At roughly 4 characters per token, 10 pages is about 30,000 tokens arriving in one agent's context, and the next agent gets it again. Ollama's docs say a local model on a machine with less than 24 GiB of VRAM gets a 4k context by default. Most of those pages never reach the model.
- Model calls. In CrewAI's agent loop, each tool call is a separate model turn. Fetching 10 pages through a tool takes at least 11 turns for that one agent. Here, the five pages cost two model calls in total.
- Hallucinated fields. A small model asked to copy a price from 3,000 tokens of text will sometimes get it wrong. A CSS selector does not.
Agent-driven crawling is still useful when a model has to judge which link matters, such as following "pricing" but not "careers" on an unknown site. Even then, put the polite fetcher below the tool so the agent cannot skip it.
Setup#
crewai 1.15.44 declares Requires-Python >=3.10,<3.14, so it will not install on Python 3.14 or newer. On 3.15 pip fails while building tiktoken. Use 3.13:
python3.13 -m venv .venv
.venv/bin/pip install crewai==1.15.44 requests beautifulsoup4
No pandas. The standard library csv module writes the output.
The polite fetcher#
Every request in the crawl goes through one class. It does four things: reads robots.txt once per host, refuses disallowed URLs, waits between requests, and stops at a page budget.
import time
import urllib.robotparser
from urllib.parse import urlparse
import requests
USER_AGENT = "GyaanStackTutorialBot/0.1 (+https://blog.gyaanstack.com)"
class PoliteFetcher:
"""One HTTP client for the whole crawl: robots.txt, a per-host delay, retries, a page budget."""
def __init__(self, min_delay=1.0, max_pages=20, timeout=15):
self.session = requests.Session()
self.session.headers["User-Agent"] = USER_AGENT
self.min_delay = min_delay
self.max_pages = max_pages
self.timeout = timeout
self.pages_fetched = 0
self._robots = {} # host -> RobotFileParser
self._last_hit = {} # host -> time.monotonic() of last request
def _robots_for(self, url):
p = urlparse(url)
host = f"{p.scheme}://{p.netloc}"
if host not in self._robots:
rp = urllib.robotparser.RobotFileParser()
r = self.session.get(f"{host}/robots.txt", timeout=self.timeout)
if r.status_code == 200:
rp.parse(r.text.splitlines())
elif r.status_code in (401, 403):
rp.disallow_all = True # stricter than RFC 9309, same as the stdlib
elif 400 <= r.status_code < 500:
rp.allow_all = True # RFC 9309: 404 and friends mean no rules
else:
rp.disallow_all = True # 5xx or worse: assume everything is off limits
self._robots[host] = rp
print(f"robots.txt for {host}: HTTP {r.status_code}")
return self._robots[host]
def _wait(self, host, rp):
delay = max(self.min_delay, rp.crawl_delay(USER_AGENT) or 0)
elapsed = time.monotonic() - self._last_hit.get(host, 0)
if elapsed < delay:
time.sleep(delay - elapsed)
self._last_hit[host] = time.monotonic()
def get(self, url, retries=3):
if self.pages_fetched >= self.max_pages:
raise RuntimeError(f"page budget of {self.max_pages} used up")
rp = self._robots_for(url)
if not rp.can_fetch(USER_AGENT, url):
raise PermissionError(f"robots.txt disallows {url}")
host = urlparse(url).netloc
for attempt in range(retries):
self._wait(host, rp)
r = self.session.get(url, timeout=self.timeout)
if r.status_code in (429, 503):
ra = r.headers.get("Retry-After", "")
wait = int(ra) if ra.isdigit() else 2 ** (attempt + 1)
print(f"HTTP {r.status_code} on {url}, sleeping {wait}s")
time.sleep(wait)
continue
r.raise_for_status()
self.pages_fetched += 1
return r
raise RuntimeError(f"gave up on {url} after {retries} tries")
A few choices here are deliberate.
robots.txt status codes. RFC 9309 says that if robots.txt returns a 4xx status the file is "unavailable" and a crawler may access anything. If it returns a 5xx, the file is "unreachable" and the crawler should assume everything is disallowed. books.toscrape.com returns 404 for /robots.txt, so the crawl is allowed. Python's own RobotFileParser.read() goes further on 401 and 403 and disallows everything; the class keeps that behaviour because a site that says "forbidden" to the rules file is unlikely to welcome a bot. The fetcher also requests robots.txt through its own session, so the same User-Agent and timeout apply. read() would use urllib with Python's default User-Agent.
Crawl-delay. Crawl-delay is not part of RFC 9309, and Google does not support it, but many sites still set it and urllib.robotparser reads it. The fetcher waits for whichever is longer: its own one second or the site's value.
429 and 503. Status 429 is defined in RFC 6585 as "Too Many Requests", and both 429 and 503 may carry a Retry-After header. That header can be a number of seconds or an HTTP date. The code uses the seconds form and falls back to exponential backoff for a date, which errs on the short side; parse the date if you scrape a site that sends one.
A User-Agent that names you. A browser-imitating string with a contact URL in it is the worst of both. Say who you are and give a page where a site owner can reach you.
Rate limit math. One request per second sounds slow until you count. books.toscrape.com lists 1,000 books across 50 catalogue pages, so a full crawl is about 1,050 requests, or roughly 18 minutes at this rate. That is fine for a nightly job. If you need it in two minutes, ask the site owner, use their API, or accept that you are now running a load test on someone else's server.
Parsing without a model#
import re
from urllib.parse import urljoin, urldefrag, urlparse
from bs4 import BeautifulSoup
MAX_DESCRIPTION_CHARS = 1500
def book_links(fetcher, category_url, limit):
"""Product links on one category page, same host only, no duplicates."""
soup = BeautifulSoup(fetcher.get(category_url).content, "html.parser")
host = urlparse(category_url).netloc
links = []
for a in soup.select("article.product_pod h3 a[href]"):
url, _ = urldefrag(urljoin(category_url, a["href"]))
if urlparse(url).netloc == host and url not in links:
links.append(url)
return links[:limit]
def parse_book(url, html_bytes):
"""Every field that exists in the HTML comes from the HTML, not from a model."""
soup = BeautifulSoup(html_bytes, "html.parser")
main = soup.select_one("div.product_main")
table = {th.get_text(strip=True): td.get_text(strip=True)
for th, td in zip(soup.select("table th"), soup.select("table td"))}
desc = soup.select_one("#product_description + p")
stock = re.search(r"\((\d+) available\)", table.get("Availability", ""))
return {
"url": url,
"upc": table.get("UPC", ""),
"title": main.h1.get_text(strip=True),
"price_gbp": float(main.select_one("p.price_color").get_text().lstrip("£")),
"in_stock": int(stock.group(1)) if stock else 0,
"description": (desc.get_text(" ", strip=True) if desc else "")[:MAX_DESCRIPTION_CHARS],
}
The selector article.product_pod h3 a picks only product links. A generic "every <a> on the page" crawler collects the breadcrumb, the sidebar's 50 category links and the pager first, and a 10-link cap fills up with navigation.
The first version of parse_book received response.text and crashed:
ValueError: could not convert string to float: '£45.17'
The site sends Content-Type: text/html with no charset. In that case Requests follows the HTTP spec and decodes as ISO-8859-1, so the UTF-8 pound sign turns into £. Passing response.content (bytes) lets BeautifulSoup read the <meta charset> in the page instead. A pipeline that hands .text to a model never sees this error. The model gets mojibake and quietly works around it, or does not.
The two agents#
The crew has an analyst that writes a summary and two to five topic words per book, and a fact checker that marks each record as grounded or not. Both tasks return Pydantic models through output_pydantic, and the analyst's task has a function guardrail that rejects the output when a UPC is missing or a summary runs over 30 words. CrewAI feeds the error message back to the agent and retries, up to guardrail_max_retries times.
import csv
import json
import os
os.environ.setdefault("CREWAI_DISABLE_TELEMETRY", "true")
os.environ.setdefault("CREWAI_TRACING_ENABLED", "false")
from crewai import Agent, Crew, Process, Task
from pydantic import BaseModel, Field
from polite import PoliteFetcher
from scrape import book_links, parse_book
from stub_llm import StubLLM
START = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
BOOKS = 5
class Record(BaseModel):
upc: str
summary: str = Field(description="One factual sentence, at most 30 words")
topics: list[str] = Field(description="2 to 5 lowercase topic words")
class Records(BaseModel):
records: list[Record]
class Verdict(BaseModel):
upc: str
grounded: bool
note: str
class Verdicts(BaseModel):
verdicts: list[Verdict]
def make_guardrail(expected_upcs):
def check(output):
try:
recs = Records.model_validate_json(output.raw).records
except ValueError as exc:
return False, f"Output is not valid Records JSON: {exc}"
got = [r.upc for r in recs]
if sorted(got) != sorted(expected_upcs):
return False, f"Return exactly one record per UPC: {expected_upcs}"
for r in recs:
if len(r.summary.split()) > 30:
return False, f"Summary for {r.upc} is over 30 words"
if not 2 <= len(r.topics) <= 5:
return False, f"{r.upc} needs 2 to 5 topics"
return True, output
return check
def build_crew(llm, expected_upcs):
analyst = Agent(
role="Catalogue analyst",
goal="Summarise each book description in one sentence and tag its topics",
backstory="You only use the text you are given. Missing text means an empty summary.",
llm=llm, allow_delegation=False, verbose=False,
)
checker = Agent(
role="Fact checker",
goal="Mark any summary that says something the description does not support",
backstory="You compare a draft against its source and never rewrite the draft.",
llm=llm, allow_delegation=False, verbose=False,
)
summarise = Task(
description="Books as JSON:\n<pages>{pages}</pages>\nOne record per UPC.",
expected_output="A JSON object with a records list",
agent=analyst,
output_pydantic=Records,
guardrail=make_guardrail(expected_upcs),
guardrail_max_retries=2,
)
verify = Task(
description="Source books:\n<pages>{pages}</pages>\nCheck the analyst's records.",
expected_output="A JSON object with a verdicts list",
agent=checker,
context=[summarise],
output_pydantic=Verdicts,
)
return Crew(agents=[analyst, checker], tasks=[summarise, verify],
process=Process.sequential, max_rpm=10, verbose=False)
In 1.15.44 the guardrail runs on the raw output before CrewAI converts it to the Pydantic model, so output.pydantic is None inside the guardrail. The first version read output.pydantic.records and crashed with an AttributeError. Parsing output.raw yourself works, and it turns malformed JSON into one more guardrail failure that the agent gets to fix.
max_rpm=10 on the crew caps model requests per minute, according to the Crews docs. It protects your LLM provider's rate limit. It does nothing for the website, which is why the fetcher has its own delay.
The fact checker is the agent worth keeping when you go to a real model. A summariser told "only use the text you are given" still adds things. A second call that sees the source and the draft side by side, with no job except saying yes or no, catches some of them. It costs one extra call per batch.
The stub model and the run#
CrewAI supports custom LLMs through BaseLLM; the only required method is call. The stub uses that to stand in for a model. It answers in the ReAct format CrewAI's agent loop expects, but it builds the summary from the first sentence of the description and picks topics by word frequency. That keeps the run offline and the output repeatable.
import json
import re
from crewai import BaseLLM
class StubLLM(BaseLLM):
"""Offline stand-in so the pipeline runs with no API key.
It answers like a model would (a ReAct-style 'Final Answer:'), but builds the
answer with plain string rules. Swap it for crewai.LLM(...) to use a real model.
"""
def call(self, messages, tools=None, callbacks=None, available_functions=None,
from_task=None, from_agent=None, response_model=None):
self.calls = getattr(self, "calls", 0) + 1
prompt = messages if isinstance(messages, str) else "\n".join(
str(m.get("content", "")) for m in messages)
pages = json.loads(re.search(r"<pages>(.*?)</pages>", prompt, re.S).group(1))
role = from_agent.role if from_agent else ""
if role == "Catalogue analyst":
out = [{"upc": p["upc"],
"summary": first_sentence(p["description"]),
"topics": keywords(p["description"])} for p in pages]
return "Thought: I have what I need.\nFinal Answer: " + json.dumps({"records": out})
# Fact checker: the stub passes everything; a real model reads both blocks.
draft = json.loads(re.search(r'(\{"records".*\})', prompt, re.S).group(1))
verdicts = [{"upc": r["upc"], "grounded": True, "note": ""} for r in draft["records"]]
return "Thought: Checked.\nFinal Answer: " + json.dumps({"verdicts": verdicts})
def supports_function_calling(self):
return False
def get_context_window_size(self):
return 8192
first_sentence and keywords are short string helpers and are left out here. The stub's fact checker approves everything, so the grounded column in this run proves only that the plumbing works.
The main function crawls, runs the crew, joins model output to parsed fields on UPC and writes both files:
def main():
fetcher = PoliteFetcher(min_delay=1.0, max_pages=BOOKS + 1)
links = book_links(fetcher, START, BOOKS)
books = [parse_book(u, fetcher.get(u).content) for u in links]
print(f"fetched {fetcher.pages_fetched} pages, parsed {len(books)} books")
llm = StubLLM(model="stub") # real run: LLM(model="ollama/llama3.2", ...)
payload = [{"upc": b["upc"], "description": b["description"]} for b in books]
crew = build_crew(llm, [b["upc"] for b in books])
result = crew.kickoff(inputs={"pages": json.dumps(payload)})
notes = {r.upc: r for r in result.tasks_output[0].pydantic.records}
verdicts = {v.upc: v for v in result.pydantic.verdicts}
rows = []
for b in books:
n, v = notes[b["upc"]], verdicts[b["upc"]]
rows.append({**{k: b[k] for k in ("url", "upc", "title", "price_gbp", "in_stock")},
"summary": n.summary, "topics": ";".join(n.topics),
"grounded": v.grounded, "description": b["description"]})
with open("books.json", "w") as f:
json.dump(rows, f, indent=2)
with open("books.csv", "w", newline="", encoding="utf-8") as f:
w = csv.DictWriter(f, fieldnames=[k for k in rows[0] if k != "description"],
extrasaction="ignore")
w.writeheader()
w.writerows(rows)
print(f"LLM calls: {llm.calls}")
for r in rows:
print(f"{r['price_gbp']:>6} {r['in_stock']:>3} {r['title'][:38]:<38} {r['topics']}")
Real output from python crew_app.py, minus two CrewAI warnings that it is estimating token counts for an unknown model name:
robots.txt for https://books.toscrape.com: HTTP 404
fetched 6 pages, parsed 5 books
LLM calls: 2
45.17 19 It's Only the Himalayas africa;south;stupid
49.43 15 Full Moon over Noah’s Ark: An Odyssey ararat;mount;antonson
48.87 14 See America: A Celebration of Our Nati national;parks;posters
36.94 8 Vagabonding: An Uncommon Guide to the life;travel;vagabonding
37.33 7 Under the Tuscan Sun mayes;abandoned;along
Six requests (one category page, five books) at one per second, plus robots.txt, ran in about 7 seconds wall time. The five descriptions add up to 6,391 characters, about 1,600 tokens. Both agents see them, so a real model reads about 3,200 tokens of page text across the two calls. Each call, at about 1,600 tokens of pages plus the prompt, fits in Ollama's default 4k window. Ten books at the 1,500-character cap would be about 3,750 tokens per call and would not, so batch by size or raise the context length.
Plugging in a real model#
Replace the stub line with:
from crewai import LLM
llm = LLM(model="ollama/llama3.2", temperature=0.1)
In 1.15.44 that builds CrewAI's OpenAI-compatible client pointed at http://localhost:11434/v1, and the constructor succeeds with LiteLLM not installed. The LLM docs still say the Ollama provider needs crewai[litellm], so install that extra if your version complains. Run ollama pull llama3.2 first. The Llama 3.2 model page lists a 128K context window, but Ollama's runtime default is what you actually get unless you change it.
Testing it#
Three checks run in test_pipeline.py, all against the same code:
import json
import urllib.robotparser
from crew_app import build_crew
from polite import USER_AGENT
from stub_llm import StubLLM
# 1. robots.txt rules, parsed offline
rp = urllib.robotparser.RobotFileParser()
rp.parse(["User-agent: *", "Disallow: /checkout/", "Crawl-delay: 5"])
assert rp.can_fetch(USER_AGENT, "https://shop.example/catalogue/a.html")
assert not rp.can_fetch(USER_AGENT, "https://shop.example/checkout/cart")
assert rp.crawl_delay(USER_AGENT) == 5
print("robots rules: ok")
# 2. guardrail retry: first answer drops a record, second is complete
class FlakyStub(StubLLM):
def call(self, messages, **kw):
out = super().call(messages, **kw)
if self.calls == 1:
data = json.loads(out.split("Final Answer: ", 1)[1])
data["records"] = data["records"][:1]
return "Final Answer: " + json.dumps(data)
return out
pages = [{"upc": "u1", "description": "A walk across Iceland in winter with two dogs and no map."},
{"upc": "u2", "description": "Recipes from village kitchens in Kerala, with notes on spices."}]
llm = FlakyStub(model="stub")
result = build_crew(llm, ["u1", "u2"]).kickoff(inputs={"pages": json.dumps(pages)})
assert len(result.tasks_output[0].pydantic.records) == 2
print(f"guardrail retry: ok after {llm.calls} LLM calls")
# 3. grounding eval on the real run: topics must appear in the source text
rows = json.load(open("books.json"))
for r in rows:
desc = r["description"].lower()
missing = [t for t in r["topics"].split(";") if t not in desc]
novel = [w for w in r["summary"].lower().split() if w.strip(".,;:!?“”") not in desc]
print(f"{r['upc']} topics missing: {missing or '-'} summary words not in source: {len(novel)}")
robots rules: ok
guardrail retry: ok after 3 LLM calls
a22124811bfa8350 topics missing: - summary words not in source: 0
ce60436f52c5ee68 topics missing: - summary words not in source: 0
f9705c362f070608 topics missing: - summary words not in source: 0
1809259a5a5f1d8d topics missing: - summary words not in source: 0
a94350ee74deaa07 topics missing: - summary words not in source: 0
The retry test shows the guardrail doing its job: the first answer had one record instead of two, CrewAI sent the error back, and the second answer passed. That is three model calls (two for the analyst, one for the checker).
The grounding numbers are all zero because the stub copies text from the source. That is the point of running it: it proves the eval works before a model makes the numbers interesting. With a real model, "summary words not in source" will be non-zero, since paraphrase is allowed. Track it over time and read the records with the highest counts. A topic word that never appears in the description is a stronger signal and deserves a hard failure. For a fuller method, see evaluating an AI agent in Python.
Failure modes to expect#
- Layout change.
div.product_maindisappears andparse_bookraises onNone. That is the right outcome. Fail loudly, alert, and fix the selector. Do not fall back to "send the whole page to the model". - Wrong charset. The
£bug above. Parse bytes, or setresponse.encodingyourself. - Guardrail loop. A small model that cannot satisfy the guardrail burns all retries, then CrewAI raises. Keep batches small and the rules few.
- Stale robots.txt. The fetcher caches robots.txt for the life of the process. RFC 9309 says not to use a cached copy for more than 24 hours, so a long-running crawler needs to refresh it.
- Silent truncation. Description capped at 1,500 characters means a long description loses its end. Log when the cap is hit.
Terms of service and the line robots.txt does not draw#
RFC 9309 says plainly that robots.txt rules "are not a form of access authorization". A site can allow your path in robots.txt and still forbid automated access in its terms of service, or require a login, or offer an API it wants you to use instead. Before pointing this at a real site:
- Read the terms of service and look for clauses on automated access, scraping or bots.
- Prefer an official API or data export when one exists.
- Do not scrape pages behind a login with someone else's account, and do not collect personal data you have no basis to hold.
- Keep the rate low, identify your bot, and stop when asked.
books.toscrape.com exists for practice; its parent site calls it "a fictional bookstore that desperately wants to be scraped". Real sites are not like that, and the code above only covers the technical part of being polite.
Sources#
- CrewAI, Tasks:
output_pydantic,guardrail,guardrail_max_retries. - CrewAI, Crews:
Process.sequential,max_rpm. - CrewAI, Custom LLM implementation:
BaseLLMandcall. - CrewAI, LLMs: Ollama provider configuration.
- CrewAI, Telemetry:
CREWAI_DISABLE_TELEMETRYandCREWAI_TRACING_ENABLED. - PyPI, crewai: version 1.15.44 and its Python range, checked 22 Nov 2026.
- IETF, RFC 9309: Robots Exclusion Protocol, 2022.
- IETF, RFC 6585 (429 Too Many Requests) and RFC 9110, Retry-After.
- Google Search Central, How Google interprets the robots.txt specification.
- Python, urllib.robotparser.
- Requests, Encodings.
- Beautiful Soup, documentation.
- Ollama, Context length and llama3.2.
- Toscrape and books.toscrape.com.