How to Test Web Scrapers With pytest: A Practical Guide

TL;DR:

• Split fetch from parse and the parsing becomes a pure function — HTML string in, records out — testable with no network and no mocking library.
• The offline suite ran 14 tests in 0.16 s; the two live contract tests are deselected by default and take 0.88 s on their own.
• Coverage reported 85%, and the only uncovered lines were fetch() and scrape(). That is the intended shape rather than a gap to close.
• Make field parsers raise. Renaming one CSS class in a copy of the fixture produced ValueError: missing price at a named line instead of 20 rows of nulls.
• A fixture test proves the parser handles the HTML you saved. Only a contract test against the live page detects that the site changed.
• A green suite cannot tell you the target still serves that HTML, still renders server-side, or still returns a page at all.
• Run the live half against real rendered pages on the Scrapeless free plan.

Scrapers break in a way most software does not: nothing in the repository changes, and the code stops working because someone else edited a page. That makes the usual instinct — write tests, watch them go green, ship — necessary but not sufficient, and it changes what the tests should be checking.

The suite below covers a small book-catalogue scraper. It is written in two halves that answer different questions: an offline half that asks whether the parser is correct, and a live half that asks whether the site still matches what the parser expects.

What Scraper Tests Are Actually For

Three failures are worth separating, because only two of them are yours:

Most published scraper-testing advice covers the first row. The second needs a test that talks to the site; the third cannot be caught by a test suite at all, which is worth saying out loud before building one.

Install

python3 -m venv .venv
./.venv/bin/pip install pytest pytest-cov responses parsel requests

The versions this suite ran against:

pytest        9.1.1
pytest-cov    7.1.0
responses     0.26.3
parsel        1.11.0
requests      2.34.2
lxml          6.1.3

responses is included because HTTP mocking is the usual next question. The parsing tests need none of it, and the reason is structural rather than stylistic.

The Split That Makes Parsing Testable

One function touches the network. Everything else takes a string.

import requests
from parsel import Selector

CATEGORY_URL = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}


def fetch(url: str = CATEGORY_URL) -> str:
    """The only function that touches the network."""
    response = requests.get(url, timeout=30)
    response.raise_for_status()
    return response.content.decode("utf-8")


def parse_price(raw: str | None) -> float:
    if not raw:
        raise ValueError("missing price")
    return float(raw.replace("£", "").strip())


def parse_rating(css_class: str | None) -> int:
    word = (css_class or "").replace("star-rating", "").strip()
    if word not in RATINGS:
        raise ValueError(f"unknown rating: {word!r}")
    return RATINGS[word]


def parse(html: str) -> list[dict]:
    """Pure function: HTML in, records out."""
    sel = Selector(text=html)
    return [{
        "title": card.css("h3 a::attr(title)").get(),
        "price": parse_price(card.css("p.price_color::text").get()),
        "rating": parse_rating(card.css("p.star-rating::attr(class)").get()),
        "in_stock": bool(card.css("p.instock.availability").get()),
    } for card in sel.css("article.product_pod")]

parse has no I/O, no clock, and no global state, so testing it needs no mocking at all. The standard library's mocking tools are excellent and mostly unnecessary here — a function that already takes its input as an argument does not need its dependencies patched.

Note that the two field parsers raise rather than returning None. That single decision is what turns a silent markup change into a named failure.

Save a Real Page as a Fixture

Tests need HTML that does not change underneath them, so save a real response once and commit it.

import requests, pathlib

response = requests.get(CATEGORY_URL, timeout=30)
response.raise_for_status()
pathlib.Path("fixtures/mystery.html").write_bytes(response.content)

fixture saved: 50388 bytes

Load it through a session-scoped fixture so the file is read once for the whole run:

# tests/conftest.py
import pathlib
import pytest

FIXTURES = pathlib.Path(__file__).parent.parent / "fixtures"


@pytest.fixture(scope="session")
def mystery_html() -> str:
    return (FIXTURES / "mystery.html").read_text(encoding="utf-8")

Commit the fixture. It is the record of what the page looked like when the parser was written, and a diff against a fresh copy is the fastest way to see what a site changed.

Write Assertions Worth Having

Assert on values and invariants, not on the fact that something came back.

import pytest
from bookscraper import parse, parse_price, parse_rating


def test_parse_returns_every_card(mystery_html):
    assert len(parse(mystery_html)) == 20


def test_record_shape(mystery_html):
    record = parse(mystery_html)[0]
    assert set(record) == {"title", "price", "rating", "in_stock"}
    assert record["title"] == "Sharp Objects"
    assert record["price"] == 47.82
    assert record["rating"] == 4
    assert record["in_stock"] is True


def test_every_price_is_positive(mystery_html):
    assert all(r["price"] > 0 for r in parse(mystery_html))


@pytest.mark.parametrize("raw,expected", [("£47.82", 47.82), ("£9.99", 9.99), ("£100.00", 100.0)])
def test_parse_price(raw, expected):
    assert parse_price(raw) == expected


def test_parse_price_rejects_missing():
    with pytest.raises(ValueError):
        parse_price(None)


def test_parse_rating_rejects_unknown():
    with pytest.raises(ValueError, match="unknown rating"):
        parse_rating("star-rating Eleven")


def test_empty_html_yields_no_records():
    assert parse("<html><body></body></html>") == []

Three kinds of assertion are doing distinct work. Exact values pin one known record. Invariants (all prices > 0, ratings between 1 and 5) hold for records the fixture does not contain yet. And the pytest.raises cases pin the failure behaviour, which is the part a markup change exercises.

Keep the Live Tests Out of the Default Run

Contract tests hit the real site, so they are slow and depend on someone else's uptime. A marker keeps them out of the fast loop without deleting them.

# tests/test_selector_contract.py
import pytest
from bookscraper import fetch, parse

pytestmark = pytest.mark.live


@pytest.fixture(scope="module")
def live_html():
    return fetch()


def test_live_page_still_yields_records(live_html):
    assert len(parse(live_html)) == 20


def test_live_selectors_match_fixture_shape(live_html, mystery_html):
    live, saved = parse(live_html), parse(mystery_html)
    assert {r["title"] for r in live} == {r["title"] for r in saved}

[pytest]
pythonpath = .
testpaths = tests
markers =
    live: hits the real site; excluded from the default run
addopts = -m "not live"

Registering the marker in the config is what stops pytest's marker system from warning about an unknown mark, and addopts makes the exclusion the default rather than something everyone has to remember.

$ pytest -q
..............                                    [100%]
14 passed, 2 deselected in 0.16s

$ pytest -q -m live
..                                                [100%]
2 passed, 14 deselected in 0.88s

The split matters because the two suites belong on different schedules. The offline 14 run on every commit. The live 2 run on a timer, and their failure means the site moved rather than the code — this is the boundary the practical test pyramid draws between fast isolated tests and the small number that cross a real boundary.

Read Coverage as an Architecture Check

$ pytest -q --cov=bookscraper --cov-report=term-missing

Name             Stmts   Miss  Cover   Missing
----------------------------------------------
bookscraper.py      26      4    85%   14-16, 47
----------------------------------------------
TOTAL               26      4    85%
14 passed, 2 deselected in 0.50s

Lines 14-16 are the body of fetch; line 47 is scrape, which composes the two. Every line of parsing logic is covered and every uncovered line is one that talks to the network.

That is the number to want. Chasing 100% here means mocking requests to prove that requests.get was called, which tests the mock. The useful reading of a coverage report on a scraper is which lines are missing, and whether they are the ones you deliberately kept at the edge.

Testing a scraper against a page that renders client-side? The Scrapeless free plan covers enough sessions to capture a rendered fixture worth committing.

What a Markup Change Looks Like

Take the saved fixture, rename one class the way a site redesign would, and run the parser against it:

html = pathlib.Path("fixtures/mystery.html").read_text(encoding="utf-8")
drifted = html.replace("price_color", "product-price")
pathlib.Path("fixtures/mystery_drifted.html").write_text(drifted, encoding="utf-8")
print("price_color occurrences:", html.count("price_color"), "->", drifted.count("price_color"))

price_color occurrences: 20 -> 0

raw = None

    def parse_price(raw: str | None) -> float:
        if not raw:
>           raise ValueError("missing price")
E           ValueError: missing price

bookscraper.py:21: ValueError
=========================== short test summary info ============================
FAILED tests/test_drift_demo.py::test_parse_survives_price_class_rename - Val...
1 failed in 0.11s

The failure names the field and the line. Had parse_price returned None on a missing match, the run would have completed and written twenty records with a null price — and the pipeline would have reported success. The HTML specification's class attribute carries no stability guarantee whatsoever; it is presentational, and treating a class name as a contract means the parser must be loud when the contract is broken.

For the same reason, validating the shape of the record after parsing is worth pairing with these tests — our guide to validating scraped data covers the runtime half of the same problem.

Where the Test Suite Stops

A green suite means the parser handles the HTML in fixtures/. It says nothing about three things that break scrapers in production:

• The page now renders client-side. The HTML a plain client receives is a shell; the selectors are correct and match nothing.
• The response is not the page. A challenge or interstitial arrives with HTTP 200, and a content-only assertion can pass on markup that contains no records.
• The fixture has aged. It still parses cleanly because it is a file, which is exactly why it cannot tell you the site moved.

The first two need a real browser rather than a real request. Capturing the fixture through the Scrapeless Scraping Browser means the saved HTML is the DOM the browser assembled, so the offline suite tests the same document the live run will see. A contract suite is a handful of sessions on a timer rather than a per-commit cost, and pricing lists what that cadence comes to. The third is answered by the contract test comparing live titles against the fixture's — the cheapest early warning available, and the reason those two tests exist at all.

Troubleshooting

fixture 'mystery_html' not found — the fixture lives in tests/conftest.py, and pytest only discovers conftest.py in the test directory or above it.

ModuleNotFoundError: No module named 'bookscraper' — set pythonpath = . in pytest.ini, or install the package in editable mode. Tests run from the rootdir, not from tests/.

PytestUnknownMarkWarning: Unknown pytest.mark.live — register the marker in the markers section of the config.

The live tests fail while the offline suite passes — that is the contract test doing its job. Diff a fresh copy of the page against the committed fixture before touching the parser.

Conclusion

The design decision that makes a scraper testable is not the test framework, it is the split: fetch returns a string, parse takes one, and everything interesting happens in a pure function. Coverage confirms the shape — 85%, with fetch and scrape as the only uncovered lines.

Beyond that, two habits carry most of the value. Make field parsers raise, so a renamed class produces ValueError: missing price at a named line rather than twenty null prices. And keep a small live contract suite behind a marker, because the fixture can only tell you the parser still works on the page you saved.

Ready to test a scraper against pages that render before you parse them? Start with the Scrapeless free plan and capture a fixture from the real DOM.

FAQ

Q: How do you unit test a web scraper without hitting the site?

Separate the fetch from the parse and test the parse. If parse takes an HTML string and returns records, a saved fixture file is the entire test setup — no mocking library, no HTTP interception. The 14 offline tests above ran in 0.16 s because none of them opens a socket.

Q: Do I need a mocking library like responses or unittest.mock?

Only for code that calls the network itself. Once parsing takes a string argument, there is nothing to patch. Reach for HTTP mocking when you want to test the fetch layer's own behaviour — status handling, timeouts, header construction — rather than to test parsing.

Q: How do I detect that a site changed my selectors?

A contract test that fetches the live page and compares it against the committed fixture. test_live_selectors_match_fixture_shape above asserts that the set of titles matches; when it stops matching, the site moved. Keep it behind a marker so it runs on a schedule instead of on every commit.

Q: Should scraper tests run in CI?

The offline ones, on every commit — they are deterministic and fast. Live contract tests should not gate a merge, because a failure means someone else's site changed and the pull request is innocent. Run them on a timer and alert on the result instead.

Q: What coverage should a scraper aim for?

Look at which lines are missing rather than the percentage. 85% with fetch and scrape uncovered is a well-shaped suite; the same 85% with parsing branches uncovered is not. Pushing to 100% usually means asserting that a mock was called, which proves nothing about the data.

Q: Should a parser return None or raise when a field is missing?

Raise. A None propagates into the database as a null and the run reports success, so the failure surfaces days later as missing data. Raising names the field and the line the moment the markup changes, which is what turned one renamed class into ValueError: missing price above.