Web Scraping with Scrapy

Most of the data on the web was never published as an API. Prices, listings, articles, fixtures and specifications sit inside HTML pages, formatted for people rather than programs. Web scraping is how you turn those pages into rows you can actually analyse, and Scrapy is Python’s industrial-strength tool for doing it at scale.

There is one idea underneath everything here. A web page is a tree, and scraping is the act of describing where in that tree your data lives.

The description is written in one of two query languages, XPath or CSS selectors. Every scraping tool you have heard of, Scrapy, BeautifulSoup, Selenium, Playwright, is ultimately a different way of running those queries against a tree. Learn the queries and the tools become interchangeable.

Scrapy is the right choice when you are crawling thousands or millions of pages, because it handles concurrency, retries, throttling and export pipelines for you. For a single page fetched once, requests plus a parser is simpler and this guide covers that combination too, since it is also the fastest way to prototype selectors before you commit them to a spider.

The pages used throughout

Everything below runs against a fictional bookshop at books.example.com, and both of its page types are printed here so every selector in the guide can be checked against real markup.

The catalogue page at /catalogue repeats this block once per book:

<div id="catalogue">
  <div class="book-card book-card--featured">
    <h3 class="book-card__title">The Long Field</h3>
    <p class="book-card__author">Jane Ashworth</p>
    <p class="book-card__price">12.99</p>
    <a href="/books/the-long-field">View book</a>
  </div>
</div>

Each detail page at /books/<slug> looks like this:

<div class="book-detail">
  <h1 class="book__title">The Long Field</h1>
  <p class="book__blurb">A quiet novel about <em>land</em> and memory.</p>
</div>

If you want a live target to practise against, books.toscrape.com and quotes.toscrape.com are sandbox sites built and maintained specifically for scraping practice, so you can hammer them without bothering anyone.

1. Before you scrape anything

This section is not boilerplate. Getting it wrong is how scrapers get blocked, and occasionally how their authors get letters.

Read robots.txt. Every site can publish one at /robots.txt listing the paths it does not want crawled. It is a convention rather than a technical barrier, but ignoring it is the clearest possible signal of bad faith.

Scrapy can honour it for you, with one catch worth knowing precisely: a project created by scrapy startproject sets ROBOTSTXT_OBEY = True in its settings file, but Scrapy’s own built-in default is False. So if you run a spider through a bare CrawlerProcess() as this guide does, robots.txt is not being checked unless you say so:

process = CrawlerProcess(settings={
    "ROBOTSTXT_OBEY": True,
    "DOWNLOAD_DELAY": 1.0,
    "AUTOTHROTTLE_ENABLED": True,
    "USER_AGENT": "my-research-bot (contact: me@example.com)",
})

Those four settings are close to the minimum for a polite crawler. DOWNLOAD_DELAY waits between requests to the same site, AUTOTHROTTLE_ENABLED adapts that delay upward when the server slows down, and a USER_AGENT naming you with a contact address means an administrator who notices your traffic can ask you to stop instead of just blocking you.

Check the terms of service. Plenty of sites permit scraping, some prohibit it, and a few offer an API that will give you the same data faster and without the fragility. Look for the API first.

Be careful with personal data. Scraping public pages is one thing. Collecting, storing and processing personal information about identifiable people brings GDPR and equivalent regimes into scope, regardless of whether the page was public.

Do not hammer. A crawler with no delay and high concurrency is functionally a small denial of service attack. One request per second is invisible to a real server. Fifty per second is not.

2. HTML as a tree

<div id="catalogue">
  <div class="book-card book-card--featured">
    <h3 class="book-card__title">The Long Field</h3>
    <a href="/books/the-long-field">View book</a>
  </div>
</div>

Every element in that snippet has three things you might want.

The tag name says what kind of element it is: div, h3, a. The attributes carry metadata: id is meant to be unique on the page, class groups related elements and can hold several space-separated values at once, and href says where a link points. The text content is what a human reader sees.

Elements nest, which makes the document a tree. The outer div is a parent, the inner div is its child, and the h3 and a are that child’s children. Every scraping query is a description of a route through this tree, ending at either an element, an attribute value, or a piece of text.

Note the class on the inner div: "book-card book-card--featured" is two class names on one element, not one class name containing a space. This detail causes more scraping bugs than anything else in this guide, and section 4 explains why.

3. XPath

XPath describes a location the way a filesystem path does, by walking down from the root.

'/html/body/div[2]/p[1]'   # absolute path
'//p'                      # every p anywhere in the document
'/html/body/*'             # every direct child of body

Four symbols carry most of the work. A single / means a direct child, so /html/body is the body element that is a direct child of html. A double // means a descendant at any depth, so //p finds every paragraph in the document no matter how deeply buried. Square brackets with a number select by position. An asterisk is a wildcard matching any tag name.

Read a path left to right as a sentence. /html/body/div[2]/p[1] is “start at the root, go into html, then body, then take the second div, then the first paragraph inside it.”

Two things about the bracket notation catch people out. The first is that XPath positions start at 1, not 0, so p[1] is the first paragraph.

The second is subtler and worth slowing down for: //p[1] does not mean “the first paragraph in the document.” It means “every paragraph that is the first paragraph child of its parent”, which on a page with ten sections returns ten elements. To get the first match in the whole document you have to bracket the path itself:

'//p[1]'     # every p that is the first p inside its own parent
'(//p)[1]'   # the first p in the entire document

That distinction silently changes what a scraper collects, and it looks correct either way at a glance.

Filtering by attribute

Tag-name paths almost never survive contact with real markup. Most queries filter on attributes instead, and XPath uses @for that:

'//p[@class="book-card__author"]'              # filter: class equals exactly this
'//div[@id="catalogue"]/div/a/@href'           # navigate into the href value
'//div[contains(@class, "book-card")]'         # filter: class contains this substring

The same @ does two different jobs depending on where it sits. Inside square brackets it is a filter that narrows which elements match. As the final step of a path it is a navigation that extracts an attribute’s value instead of an element.

The contains() version exists because of the multi-class problem. Our book cards carry class="book-card book-card--featured", so [@class="book-card"] matches nothing at all: the attribute’s value is the full string with both names in it. contains(@class, "book-card") matches, which is why it appears in almost every real XPath.

It is also slightly wrong, and worth knowing how. contains is a plain substring test, so contains(@class, "book") also matches class="bookmark" and class="notebook". The precise form pads the attribute with spaces and looks for a padded token:

'//div[contains(concat(" ", normalize-space(@class), " "), " book-card ")]'

That is horrible to read and you will not write it often. Use plain contains() when the class names on the page are distinctive enough that a partial match cannot happen, and reach for the padded version when they are not. Or use a CSS selector, which does the correct thing by default.

4. CSS selectors

CSS selectors are the same syntax web designers use for styling, which makes them shorter and more readable for the common cases.

'div#catalogue > div h3'      # h3 anywhere inside a direct child div of #catalogue
'div.book-card > a'           # a directly inside a div with class book-card
'#catalogue > *'              # every direct child of #catalogue

The symbols map cleanly. A hash means id, so #catalogue is the element with id="catalogue". A dot means class, so .book-cardmatches on class. A greater-than sign means direct child. A plain space means descendant at any depth. An asterisk is the wildcard.

The important advantage over XPath is that CSS class matching understands multiple classes properly. Our card has class="book-card book-card--featured", and .book-card matches it correctly, while .book matches nothing. CSS treats the attribute as a list of whitespace-separated tokens and compares whole tokens, which is exactly the behaviour you wanted from contains() and did not quite get.

What CSS cannot do is navigate upward. There is no CSS expression for “the parent of this element” or “the nearest ancestor with class X”, and there is no way to select an element by the text it contains. XPath does both. That is the whole trade: CSS is shorter for going down the tree by id and class, XPath is more powerful everywhere else.

5. Translating between them

That table is the Rosetta stone, with one asterisk on the second row: as section 3 explained, contains(@class, "book-card") is a substring test and .book-card is a whole-token test, so they agree on tidy markup and diverge on messy markup. They are equivalent in practice and not in principle.

Scrapy exposes both languages through .css() and .xpath() on the same objects, so you never have to commit to one. Real spiders mix them freely, using whichever is shorter for each individual step.

6. Selector objects

A Selector wraps an HTML document and gives you the query methods.

from scrapy import Selector
html = '<html><body><div><p>Hello</p><p>World</p></div></body></html>'
sel = Selector(text=html)
paragraphs = sel.xpath('//p')      # a SelectorList, not strings
paragraphs.getall()                # ['<p>Hello</p>', '<p>World</p>']
paragraphs[0].get()                # '<p>Hello</p>'

The thing to understand is what .xpath() gives back. It is not a list of strings, it is a SelectorList, a list-like collection of Selector objects. That matters because a SelectorList is itself queryable: you can call .xpath() or .css() on it again to drill deeper, which is what makes chaining work.

Queries are cheap and produce more selectors. Extraction is the commit point where you finally turn selectors into text. Two methods do it:

• .getall() returns a list of strings, one per match, and an empty list if nothing matched.
• .get() returns the first match as a string, or None if nothing matched.

You will also see .extract() and .extract_first() throughout older code and tutorials. They are the same methods under older names, they still work, and modern Scrapy documentation recommends .get() and .getall() instead because the pairing is clearer. Use the new names in new code and recognise the old ones when you read them.

The single most useful variant is a default:

title = sel.css('h1.book__title::text').get(default='').strip()

Without that default, a page where the title is missing returns None, and .strip() on None raises AttributeError and kills the callback. This is the most common way a spider that worked on ten pages dies on page eleven, and default='' is the one-word fix.

7. Fetching a page for prototyping

For a single page with no crawling, you do not need a spider:

import requests
from scrapy import Selector
response = requests.get('https://books.example.com/catalogue')
sel = Selector(text=response.text)
links = sel.css('div.book-card > a::attr(href)').getall()

requests.get() fires an HTTP GET and returns a response object. Use .text for the decoded string, which is what you want here, or .content for raw bytes if the page declares an encoding that requests guesses wrong and you need to decode it yourself. Selector(text=...) parses it into a tree, and from there the API is identical to what you will use inside a spider.

This is the right setup for working out your selectors interactively in a notebook. Once they work, move them into a spider.

The limitation is that requests returns exactly what the server sent. If the page builds its content in the browser with JavaScript, that content is not in the HTML you just downloaded. Section 16 covers what to do about that.

8. Extracting text, and the gotcha

Our blurb has an inline tag in the middle of it:

<p class="book__blurb">A quiet novel about <em>land</em> and memory.</p>

That paragraph contains three separate text fragments: 'A quiet novel about ' before the em, 'land' inside it, and ' and memory.'after it. Two XPath expressions ask for different subsets of them:

sel.xpath('//p[@class="book__blurb"]/text()').getall()
# ['A quiet novel about ', ' and memory.']
sel.xpath('//p[@class="book__blurb"]//text()').getall()
# ['A quiet novel about ', 'land', ' and memory.']

A single slash before text() takes only the text nodes that are direct children of the paragraph, which skips anything wrapped in an inline tag. A double slash takes every text node in the subtree.

The failure mode here is quiet. /text() combined with .get() returns 'A quiet novel about ' and looks like a perfectly good result, so a scraper can silently truncate every blurb that happens to contain a link or an emphasis and you will not notice until someone reads the data.

The safe default for any element that might contain inline markup is to take all descendant text and join it:

parts = sel.css('p.book__blurb ::text').getall()
blurb = ''.join(parts).strip()

The CSS equivalents follow the same rule and differ by a single space, which is easy to miss when reading someone else’s code. p.book__blurb::text with no space is direct text only, and p.book__blurb ::text with a space is all descendant text.

Use direct text when you want an element’s own text and deliberately not its children’s, which comes up with nested list items. Use descendant text everywhere else.

9. Extracting attributes

sel.xpath('//a/@href').getall()
sel.css('a::attr(href)').getall()

Those two are equivalent. XPath uses /@href as a final navigation step, CSS uses the ::attr() pseudo-element.

The attributes worth knowing by name: href on links, src on images and scripts, value on form inputs, and any data-*attribute, which is where sites frequently stash structured values for their own JavaScript to read. That last one is a gift when you find it, because a data-price="12.99" attribute is cleaner than parsing “£12.99” out of the text next to it.

10. Chaining, and the bug that comes with it

Chaining builds a query in stages: select the cards, then pull fields out of each one.

cards = sel.css('div.book-card')
titles = cards.css('h3.book-card__title::text').getall()
links = cards.xpath('./a/@href').getall()

CSS chaining behaves the way you expect. cards.css('h3...') searches inside each already-matched card.

XPath chaining does not, and this is the classic Scrapy bug. An XPath starting with / or // is absolute, which means it ignores whatever you chained it onto and starts again from the document root:

cards.xpath('//a/@href')    # every link on the whole page, cards ignored
cards.xpath('.//a/@href')   # links anywhere inside each card
cards.xpath('./a/@href')    # links that are direct children of each card

The leading dot means “relative to the current selection.” Without it, the chain is decorative: your query looks scoped, runs unscoped, and returns far more than you intended, usually including the site’s navigation and footer links. Because it returns plausible-looking data rather than an error, it can survive a long way into a project.

The rule is short. In a chained XPath, always start with ./ or .//.

Mixing the two languages step by step is normal and idiomatic. sel.css('div.book-card').xpath('./a/@href') uses CSS for the part CSS is good at and XPath for the attribute grab, and nobody will look at you strangely.

11. Develop selectors in the Scrapy shell

This is the single biggest productivity tool in Scrapy and it is missing from most tutorials.

scrapy shell "https://books.example.com/catalogue"

That opens an interactive Python session with the page already fetched and a response object waiting. You can try selectors against the live page and see results immediately:

response.css('div.book-card').getall()
response.css('div.book-card > a::attr(href)').getall()
view(response)          # opens the fetched page in your browser

view(response) is worth knowing separately. It opens the exact HTML Scrapy received, which is often not what your browser shows you, and comparing the two is how you diagnose a page that is built by JavaScript.

The workflow that saves hours: work out every selector in the shell first, confirm each one returns what you expect, then paste them into the spider. Writing selectors blind inside a spider and rerunning the whole crawl to test them is the slow way round.

12. Spiders

A spider is a class describing a crawl.

import scrapy
class CatalogueSpider(scrapy.Spider):
    name = "catalogue_spider"
    def start_requests(self):
        urls = ['https://books.example.com/catalogue']
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)
    def parse(self, response):
        authors = response.css('p.book-card__author::text').getall()
        print(authors)

Three pieces are doing the work. name is a unique string that identifies the spider to Scrapy’s command line. start_requests is the entry point and yields the initial requests. parse is the callback that receives a response and pulls data out of it.

The design is generator-based, and that is the part worth understanding. start_requests does not fetch anything. It yields Request objects that describe what to fetch. Scrapy’s engine consumes those yields, fires the actual HTTP requests concurrently, and calls the registered callback as each response arrives. You write what looks like straightforward sequential code and the framework runs it asynchronously, which is where the throughput comes from.

The callback=self.parse argument says which method to run when that URL responds. It has to be self.parse rather than bare parse, because parse is a method on the class: without self Python would look for a module-level function of that name and fail. self.parse is a bound method that Scrapy can hold onto and call later.

For the common case where you just want to fetch some URLs and hand them all to parse, there is a shortcut. Set start_urlsand delete start_requests entirely, because Scrapy generates the requests for you and routes them to the default parse callback:

class CatalogueSpider(scrapy.Spider):
    name = "catalogue_spider"
    start_urls = ['https://books.example.com/catalogue']
    def parse(self, response):
        ...

Inside a callback, response behaves like a Selector with extras. It has .xpath() and .css() with the identical API, plus .url for the address that produced it, .status for the HTTP code, .headers, .text for the decoded body, and .follow() for scheduling more requests. There is no need to wrap it in a Selector yourself.

13. Crawling between pages

Real scrapes are almost always two-stage: a listing page holds links, and the data you want is on the pages behind them.

class BookSpider(scrapy.Spider):
    name = "book_spider"
    start_urls = ['https://books.example.com/catalogue']
    def parse(self, response):
        for link in response.css('div.book-card > a::attr(href)').getall():
            yield response.follow(link, callback=self.parse_book)
    def parse_book(self, response):
        yield {
            'url': response.url,
            'title': response.css('h1.book__title::text').get(default='').strip(),
            'blurb': ''.join(response.css('p.book__blurb ::text').getall()).strip(),
        }

The first callback runs on the listing and yields a new request per link, each pointing at the second callback. The second runs once per book page and yields the data.

response.follow() is preferred over building a raw scrapy.Request for three reasons. It resolves relative URLs against the current page automatically, so the /books/the-long-field in our markup becomes a full absolute URL without you doing string surgery. It carries over the session context from the current response. And it accepts a selector or a link element directly, not just a string.

That last point gives you a tidier version when you are following every link matched by one selector:

def parse(self, response):
    yield from response.follow_all(
        css='div.book-card > a',
        callback=self.parse_book,
    )

Two details in parse_book are deliberate. The default='' on the title means a book page missing its heading yields an empty string instead of crashing the callback with AttributeError on None.strip(). And the blurb uses descendant text joined together, so the <em>land</em> in the middle survives rather than truncating the sentence, as section 8 explained.

14. Yield your data, do not hoard it

You will see spiders that accumulate results into a class attribute:

class BookSpider(scrapy.Spider):
    results = {}          # works, but fights the framework
    def parse_book(self, response):
        self.results[title] = blurb

It works, and for a quick notebook experiment it is fine. It is not how Scrapy is meant to be used, for two reasons. A class-level dictionary is shared state across every instance of the spider, which becomes a real problem the moment you run more than one. And it means nothing is written anywhere until the whole crawl finishes, so a crash at 90% loses everything.

The idiomatic version is to yield a plain dict from the callback, as the previous section does, and let Scrapy’s feed exports write it out. From the command line:

scrapy crawl book_spider -O books.json

Capital -O overwrites the file and lowercase -o appends to it. Scrapy handles json, jsonlines, csv and xml based on the extension you give, and it writes incrementally as items arrive, so an interrupted crawl still leaves you everything it collected up to that point. For anything larger than a toy crawl, jsonlines is the better format because it stays valid when truncated.

15. Running a spider

Two ways. From the command line, scrapy crawl book_spider, which is what you use in production because each invocation is a fresh process.

Programmatically, for notebooks and single-file scripts:

from scrapy.crawler import CrawlerProcess
process = CrawlerProcess(settings={
    "ROBOTSTXT_OBEY": True,
    "DOWNLOAD_DELAY": 1.0,
    "AUTOTHROTTLE_ENABLED": True,
    "USER_AGENT": "my-research-bot (contact: me@example.com)",
    "FEEDS": {"books.jsonl": {"format": "jsonlines"}},
})
process.crawl(BookSpider)
process.start()

.crawl() takes the spider class, not an instance, because Scrapy needs to build it with its own framework hooks attached. .start() launches the event loop and blocks until every spider has finished.

One gotcha will bite you in a notebook. process.start() runs Twisted’s reactor underneath, and the reactor can only be started once per Python process. Call .start() a second time in the same kernel and you get ReactorNotRestartable, with no way to recover other than restarting the kernel. This is a large part of why production Scrapy runs from the command line, where every run is a clean process.

16. When the page is built by JavaScript

Sometimes you write a perfect selector, run it, and get an empty list, while the data is plainly visible in your browser. Almost always the cause is that the content is not in the HTML the server sent. It is fetched and rendered by JavaScript after the page loads, and requests and Scrapy only ever see the initial document.

Confirm it before you do anything else. In scrapy shell, run view(response) and look at what Scrapy actually received, or use your browser’s “view source” rather than the inspector, since the inspector shows the live rendered tree and will happily show you elements that were never in the download.

Once confirmed, try the cheap fix first. Open your browser’s network tab, reload the page, and look at the XHR or fetch requests. Very often the JavaScript is calling a JSON endpoint, and you can call that endpoint directly. That gives you clean structured data with no parsing at all, and it is faster and far more stable than scraping rendered HTML. Always look for this before reaching for a browser.

If there is genuinely no underlying endpoint, you need something that runs JavaScript: Playwright, Selenium, or Scrapy with a rendering middleware such as scrapy-playwright. All of them are dramatically slower and heavier than a plain HTTP request, because you are running a real browser per page, so treat them as the last option rather than the default.

17. Common pitfalls

Forgetting ./ in a chained XPath. The query silently searches the whole document instead of your selection, and returns plausible junk rather than an error.

Assuming //p[1] is the first paragraph on the page. It is every paragraph that is first within its own parent. Use (//p)[1] for the document-wide first match.

Exact class matching. [@class="book-card"] fails on class="book-card book-card--featured". Use CSS .book-card, or contains() and accept its substring behaviour.

contains() matching too much. contains(@class, "book") also matches bookmark. Use the full class name, or CSS.

.get() returning None, then calling a string method on it. Pass default='' whenever the result feeds straight into .strip() or similar.

/text() on elements containing inline tags. It silently truncates at the first nested tag. Use descendant text and join it.

Scraping without a delay. No DOWNLOAD_DELAY and default concurrency will get you blocked, and deserves to.

Assuming CrawlerProcess respects robots.txt. It does not unless you pass ROBOTSTXT_OBEY yourself, because that setting is turned on by the project template rather than by Scrapy’s defaults.

Calling process.start() twice in one kernel. ReactorNotRestartable, and only a kernel restart clears it.

Collecting results in a class attribute. Yield items instead, so feed exports write them incrementally and a crash does not cost you the whole run.

Blaming your selector when the page is JavaScript-rendered. Check view(response) first, then look for the JSON endpoint behind the page before reaching for a headless browser.

Three things carry over to every scraper you will ever write, whatever the tool. The page is a tree and your job is to describe a route through it, so time spent getting comfortable with XPath and CSS pays back across every library. Selectors are cheap and extraction is the commit point, which is why you build the query in stages in the shell and only call .get() when it is right. And a scraper is a program that runs against someone else’s server, so the delay, the user agent and the robots.txt check are part of the job rather than an afterthought.

Work out the selectors interactively, then wrap them in a spider. Scrape politely, and the site stays available for the next person.

Thanks for reading, Andrei.