XPath vs CSS Selectors: Which to Use for Web Scraping

TL;DR:

• CSS and XPath return identical results for ordinary extraction: the same twenty book titles came back from article.product_pod h3 a::attr(title) and //article[@class='product_pod']/h3/a/@title on the same page.
• XPath is the only one of the two that walks up the tree. parent::, ancestor::, preceding-sibling::, and following-sibling:: each matched 20 nodes on that page; CSS has no equivalent for any of them.
• "CSS is faster" is a browser fact, not a Python fact. In parsel the same extraction cost 487.1 ms by CSS against 309.1 ms by XPath over 2,000 iterations, because cssselect compiles every CSS query into XPath before it runs.
• a:contains('Sharp') returns one match in parsel and throws SyntaxError inside a real Chrome page, which is why a selector can work in a scraper and fail in DevTools.
• Neither language matters when the markup never arrives; a selector can only query a DOM that was actually built.
• Both languages read the bytes your fetch layer decoded, so a page served without a charset can hand back £47.82 where the browser shows £47.82.
• Run either syntax against fully rendered pages on the Scrapeless free plan.

Open the inspector on any product listing, copy the selector Chrome offers you, paste it into a Python scraper, and it usually works. The interesting cases are the ones where it does not — where the selector is valid in one engine and a syntax error in another, or where the tidy CSS you wrote turns out to be slower than the XPath you avoided.

The two languages overlap heavily, so the useful comparison sits at the edges: what each one can express, what each one costs, and which engine is actually executing the query.

Every measurement below comes from one public page built for scraping practice — a twenty-book category listing on books.toscrape.com — parsed with lxml 6.0.2, cssselect 1.3.0, and parsel 1.10.0.

The Same Query in Both Languages

For the common targets, the two languages are a straight translation of each other. The table below pairs the queries that select identical node sets on the test page.

Three pairs from that table, run against the live page, returned matching node sets:

from parsel import Selector
import requests

url = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
response = requests.get(url, timeout=30)
response.raise_for_status()
sel = Selector(text=response.content.decode("utf-8"))

pairs = [
    ("article.product_pod h3 a::attr(href)", "//article[@class='product_pod']/h3/a/@href"),
    ("p.star-rating::attr(class)",           "//p[contains(@class,'star-rating')]/@class"),
    ("div.image_container img::attr(alt)",   "//div[@class='image_container']//img/@alt"),
]
for css_query, xpath_query in pairs:
    css_hits = sel.css(css_query).getall()
    xpath_hits = sel.xpath(xpath_query).getall()
    print(len(css_hits), len(xpath_hits), css_hits == xpath_hits)

Each pair printed 20 20 True. Where both languages can express the target, the choice is readability, not capability.

What Only XPath Can Express

CSS selects downward. It moves from an ancestor to a descendant and never back up, so a rule can say "the price inside this card" but not "the card containing this price".

XPath carries axes, and four of them have no CSS equivalent at all. Each matched 20 nodes on the test page:

axes = [
    ("parent::",            "//p[@class='price_color']/parent::div/@class"),
    ("ancestor::",          "//h3/ancestor::article/@class"),
    ("preceding-sibling::", "//div[@class='product_price']/preceding-sibling::h3/a/@title"),
    ("following-sibling::", "//h3/following-sibling::div[@class='product_price']//p[@class='price_color']/text()"),
]
for label, query in axes:
    hits = sel.xpath(query).getall()
    print(f"{label:20s} {len(hits):2d} hits   {hits[0]!r}")

parent::             20 hits   'product_price'
ancestor::           20 hits   'product_pod'
preceding-sibling::  20 hits   'Sharp Objects'
following-sibling::  20 hits   '£47.82'

That last one is the pattern worth keeping. "Find the heading, then take the price that follows it" is a sibling relationship, and expressing it in CSS means selecting the price separately and re-joining the two lists by index — which silently produces wrong pairs the moment one card is missing a price.

The W3C Selectors Level 4 specification defines the CSS grammar, and upward traversal is absent from it by design: the language was built for styling, where a renderer resolves rules top-down. The XPath 1.0 recommendation defines thirteen axes because it was built for addressing arbitrary nodes in a document tree.

The :contains Split

Text matching is where the two languages diverge in a way that produces confusing bug reports.

XPath has always had contains(). CSS had a :contains() pseudo-class in an early draft and it was dropped before the specification stabilised. The catch is that Python's cssselect still implements it.

print(len(sel.css("a:contains('Sharp')").getall()))       # 1
print(len(sel.xpath("//a[contains(., 'Sharp')]").getall()))  # 1

Both print 1. Now the same two queries inside a real browser page, evaluated through the DOM's own engines:

import os
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

JS = """() => {
  const out = {};
  try { out.css_class = document.querySelectorAll('article.product_pod h3 a').length; }
  catch (e) { out.css_class = 'ERR ' + e.name; }
  try { out.css_contains = document.querySelectorAll("a:contains('Sharp')").length; }
  catch (e) { out.css_contains = 'ERR ' + e.name + ': ' + e.message.slice(0, 60); }
  const xp = (q) => document.evaluate(q, document, null,
    XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null).snapshotLength;
  out.xpath_class    = xp("//article[@class='product_pod']/h3/a");
  out.xpath_contains = xp("//a[contains(., 'Sharp')]");
  out.xpath_parent   = xp("//p[@class='price_color']/parent::div");
  out.xpath_ancestor = xp("//h3/ancestor::article");
  return out;
}"""

url = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
endpoint = "wss://browser.scrapeless.com/api/v2/browser?" + urlencode({
    "token": os.environ["SCRAPELESS_API_KEY"], "sessionTTL": 300, "proxyCountry": "US",
})
with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(endpoint)
    page = browser.new_page()
    page.goto(url, wait_until="domcontentloaded")
    for key, value in page.evaluate(JS).items():
        print(f"{key:16s} {value}")
    browser.close()

css_class        20
css_contains     ERR SyntaxError: Failed to execute 'querySelectorAll' on 'Document': 'a:conta
xpath_class      20
xpath_contains   1
xpath_parent     20
xpath_ancestor   20

The CSS text match throws. The XPath text match returns its one node. So a selector that passes tests in a Python scraper raises a syntax error the moment someone pastes it into DevTools to check it — and the reverse holds for anyone who validates selectors in the console before shipping them.

Two things fall out of that run worth noting. Browser XPath is not degraded: parent:: and ancestor:: each resolved 20 nodes through the DOM's document.evaluate interface. And :contains is a library extension rather than a language feature, so it travels only as far as the library does.

If a selector has to work in both places, //a[contains(., 'Sharp')] is the portable form.

Which One Is Actually Faster

The common answer is that CSS is faster. That is true in a browser, where querySelectorAll is a native fast path, and it is backwards in Python.

In lxml there is no CSS engine. Every CSS query is translated into XPath by cssselect and the XPath engine executes it, which the Scrapy selectors documentation states directly. Writing CSS buys the translation step.

import time

N = 2000
start = time.perf_counter()
for _ in range(N):
    sel.css("article.product_pod h3 a::attr(title)").getall()
css_ms = (time.perf_counter() - start) * 1000

start = time.perf_counter()
for _ in range(N):
    sel.xpath("//article[@class='product_pod']/h3/a/@title").getall()
xpath_ms = (time.perf_counter() - start) * 1000

print(f"CSS   {css_ms:8.1f} ms  ({css_ms / N * 1000:6.1f} us/call)")
print(f"XPath {xpath_ms:8.1f} ms  ({xpath_ms / N * 1000:6.1f} us/call)")

CSS      487.1 ms  ( 243.5 us/call)
XPath    309.1 ms  ( 154.5 us/call)

CSS cost 1.58 times the XPath time for identical output. Printing the translation shows where it goes:

from cssselect import GenericTranslator
print(GenericTranslator().css_to_xpath("article.product_pod h3 a"))

descendant-or-self::article[@class and contains(concat(' ', normalize-space(@class), ' '), ' product_pod ')]/descendant-or-self::*/h3/descendant-or-self::*/a

The hand-written //article[@class='product_pod']/h3/a compares one attribute. The generated form normalises whitespace and pads the class list on every candidate node, because it has to be correct for elements carrying several classes. That defensive predicate is the 1.58x.

Put that in proportion before optimising for it: the gap is roughly 89 microseconds per call on a 50 KB page. A single HTTP request costs thousands of times more. Selector language is not where a scraper's time goes, and readability is usually the better trade — the number matters only in a hot parse loop over cached HTML.

The Trap Both Languages Share

A selector returns whatever your fetch layer decoded. When a server sends text/html with no charset, requests falls back to ISO-8859-1 under the HTTP semantics specification's media-type rules, while the bytes on the wire are UTF-8:

import requests
from parsel import Selector

url = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
response = requests.get(url, timeout=30)
response.raise_for_status()

print(response.headers.get("content-type"))
print(response.encoding)
print(response.apparent_encoding)
print(Selector(text=response.text).css("p.price_color::text").get())
print(Selector(text=response.content.decode("utf-8")).css("p.price_color::text").get())

text/html
ISO-8859-1
utf-8
'£47.82'
'£47.82'

The selector was right both times. Decoding response.content explicitly, rather than trusting response.text, is what makes the extracted price usable — and no change of selector language fixes it.

Testing a selector against a page that renders client-side needs a real browser. The Scrapeless free plan covers enough sessions to try both syntaxes against a live DOM.

Where Neither Language Helps

Both languages query a DOM. Neither builds one.

When a listing renders client-side, the HTML a plain HTTP client receives contains the shell and not the records, so a correct selector returns zero nodes and looks like a selector bug. The fix is upstream of the selector: render the page first, then query it. The Scrapeless Scraping Browser exposes a cloud browser over CDP, so the same page the browser assembled is the one your selector runs against — which is exactly how the in-browser numbers above were captured.

import os
from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

url = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
cdp_endpoint = "wss://browser.scrapeless.com/api/v2/browser?" + urlencode({
    "token": os.environ["SCRAPELESS_API_KEY"],
    "sessionTTL": 300,
    "proxyCountry": "US",
})

with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(cdp_endpoint)
    page = browser.new_page()
    page.goto(url, wait_until="domcontentloaded")
    titles = page.eval_on_selector_all(
        "article.product_pod h3 a", "els => els.map(e => e.title)"
    )
    print(len(titles), titles[0])
    browser.close()

20 Sharp Objects

Once the DOM exists, the selector question returns to being a style question. For a walkthrough of the selection layer itself, our parsel guide covers both dialects over one lxml-backed tree, and pricing lists what a rendering session costs.

Choose CSS When, Choose XPath When

Choose CSS when the target is addressable downward and the query will be read by people who write stylesheets. It is shorter, and on the test page it selected every ordinary field with no loss.

Choose XPath when the relationship you need is structural rather than hierarchical — up the tree, across siblings, or keyed on text. It is also the honest default inside lxml and parsel, where it is what actually executes.

Most working scrapers mix the two per field rather than picking a side, and parsel accepts both against the same tree, so the decision is per-selector rather than per-project.

Conclusion

CSS and XPath answer the same question for the ordinary cases, and the twenty titles came back identical from both. The differences that matter are narrower than the usual framing suggests: XPath can traverse upward and match text, CSS cannot; CSS is faster in a browser and slower in lxml, where it compiles to XPath first; and :contains works in one engine while throwing in the other, which is the source of most "works in my scraper, not in the console" confusion.

Pick per selector, keep the portable form when a query has to run in both places, and decode the response before blaming either language for a mangled character.

Ready to test selectors against pages that render before you query them? Start with the Scrapeless free plan and point either syntax at a live DOM.

FAQ

Q: Is XPath faster than CSS selectors?

It depends on the engine. In lxml and parsel, XPath is faster because CSS is compiled into XPath before execution — the measured gap here was 309.1 ms against 487.1 ms over 2,000 iterations of the same extraction. In a browser, querySelectorAll is a native fast path and CSS wins. Either way the difference is microseconds per call, far below the cost of the HTTP request.

Q: Can CSS selectors match element text?

Not in a browser. document.querySelectorAll("a:contains('x')") throws a SyntaxError, because :contains() was dropped before the Selectors specification stabilised. Python's cssselect still implements it, so the same query works in parsel. For a selector that behaves identically in both, use //a[contains(., 'x')].

Q: Can a CSS selector select a parent element?

No. CSS selects downward only, so there is no parent:: or ancestor:: equivalent. XPath handles both, and on the test page //h3/ancestor::article matched all 20 cards. The :has() pseudo-class lets you filter an element by its descendants, which covers some of the same intent, but it still returns the outer element rather than traversing up from an inner one.

Q: Why does my selector work in DevTools but return nothing in Python?

Two usual causes. The page renders its content with JavaScript, so the HTML your HTTP client received never contained the nodes the browser later built — the selector is correct and the DOM is not there. Or the selector uses a browser-only or library-only extension. Compare the raw response body against what the inspector shows before changing the selector.

Q: Should I use CSS or XPath in Scrapy?

Both, per field. Scrapy exposes response.css() and response.xpath() over the same parsel selector, and they can be chained together. Use CSS for straightforward class and attribute matches, and switch to XPath for text matching or upward traversal rather than contorting a CSS rule to fit.

Q: Do XPath selectors work in all browsers?

Yes, through document.evaluate rather than querySelectorAll. It is a separate DOM interface, and axes are fully supported — parent:: and ancestor:: each returned 20 nodes in the run above. The $x() helper in Chrome DevTools wraps the same interface for console use.