Spiders — Scrapy 2.11.2 documentation (2024)

scrapy.Spider¶

class scrapy.spiders.Spider¶

class scrapy.Spider¶

This is the simplest spider, and the one from which every other spidermust inherit (including spiders that come bundled with Scrapy, as well as spidersthat you write yourself). It doesn’t provide any special functionality. It justprovides a default start_requests() implementation which sends requests fromthe start_urls spider attribute and calls the spider’s method parsefor each of the resulting responses.

name¶

A string which defines the name for this spider. The spider name is howthe spider is located (and instantiated) by Scrapy, so it must beunique. However, nothing prevents you from instantiating more than oneinstance of the same spider. This is the most important spider attributeand it’s required.

If the spider scrapes a single domain, a common practice is to name thespider after the domain, with or without the TLD. So, for example, aspider that crawls mywebsite.com would often be calledmywebsite.

allowed_domains¶

An optional list of strings containing domains that this spider isallowed to crawl. Requests for URLs not belonging to the domain namesspecified in this list (or their subdomains) won’t be followed ifOffsiteMiddleware isenabled.

Let’s say your target url is https://www.example.com/1.html,then add 'example.com' to the list.

start_urls¶

A list of URLs where the spider will begin to crawl from, when noparticular URLs are specified. So, the first pages downloaded will be thoselisted here. The subsequent Request will be generated successively from datacontained in the start URLs.

custom_settings¶

A dictionary of settings that will be overridden from the project wideconfiguration when running this spider. It must be defined as a classattribute since the settings are updated before instantiation.

For a list of available built-in settings see:Built-in settings reference.

crawler¶

This attribute is set by the from_crawler() class method afterinitializing the class, and links to theCrawler object to which this spider instance isbound.

Crawlers encapsulate a lot of components in the project for their singleentry access (such as extensions, middlewares, signals managers, etc).See Crawler API to know more about them.

settings¶

Configuration for running this spider. This is aSettings instance, see theSettings topic for a detailed introduction on this subject.

logger¶

Python logger created with the Spider’s name. You can use it tosend log messages through it as described onLogging from Spiders.

state¶

A dict you can use to persist some spider state between batches.See Keeping persistent state between batches to know more about it.

from_crawler(crawler, *args, **kwargs)¶

This is the class method used by Scrapy to create your spiders.

You probably won’t need to override this directly because the defaultimplementation acts as a proxy to the __init__() method, callingit with the given arguments args and named arguments kwargs.

Nonetheless, this method sets the crawler and settingsattributes in the new instance so they can be accessed later inside thespider’s code.

Changed in version 2.11: The settings in crawler.settings can now be modified in thismethod, which is handy if you want to modify them based onarguments. As a consequence, these settings aren’t the final valuesas they can be modified later by e.g. add-ons. For the same reason, most of theCrawler attributes aren’t initialized atthis point.

The final settings and the initializedCrawler attributes are available in thestart_requests() method, handlers of theengine_started signal and later.

Parameters

• crawler (Crawler instance) – crawler to which the spider will be bound

• args (list) – arguments passed to the __init__() method

• kwargs (dict) – keyword arguments passed to the __init__() method

classmethod update_settings(settings)¶

The update_settings() method is used to modify the spider’s settingsand is called during initialization of a spider instance.

It takes a Settings object as a parameter andcan add or update the spider’s configuration values. This method is aclass method, meaning that it is called on the Spiderclass and allows all instances of the spider to share the sameconfiguration.

While per-spider settings can be set incustom_settings, using update_settings()allows you to dynamically add, remove or change settings based on othersettings, spider attributes or other factors and use setting prioritiesother than 'spider'. Also, it’s easy to extend update_settings()in a subclass by overriding it, while doing the same withcustom_settings can be hard.

For example, suppose a spider needs to modify FEEDS:

import scrapyclass MySpider(scrapy.Spider): name = "myspider" custom_feed = { "/home/user/documents/items.json": { "format": "json", "indent": 4, } } @classmethod def update_settings(cls, settings): super().update_settings(settings) settings.setdefault("FEEDS", {}).update(cls.custom_feed)

start_requests()¶

This method must return an iterable with the first Requests to crawl forthis spider. It is called by Scrapy when the spider is opened forscraping. Scrapy calls it only once, so it is safe to implementstart_requests() as a generator.

The default implementation generates Request(url, dont_filter=True)for each url in start_urls.

If you want to change the Requests used to start scraping a domain, this isthe method to override. For example, if you need to start by logging in usinga POST request, you could do:

import scrapyclass MySpider(scrapy.Spider): name = "myspider" def start_requests(self): return [ scrapy.FormRequest( "http://www.example.com/login", formdata={"user": "john", "pass": "secret"}, callback=self.logged_in, ) ] def logged_in(self, response): # here you would extract links to follow and return Requests for # each of them, with another callback pass

parse(response)¶

This is the default callback used by Scrapy to process downloadedresponses, when their requests don’t specify a callback.

The parse method is in charge of processing the response and returningscraped data and/or more URLs to follow. Other Requests callbacks havethe same requirements as the Spider class.

This method, as well as any other Request callback, must return aRequest object, an item object, aniterable of Request objects and/or item objects, or None.

See Also

Discover Public API Attack Surface with new API SpyderSpider API ReferenceHow to build a Copilot for Security API Plugin – Part 2How 2 high school teens raised a $500K seed round for their API startup (yes, it’s AI) | TechCrunch

Parameters

response (Response) – the response to parse

log(message[, level, component])¶

Wrapper that sends a log message through the Spider’s logger,kept for backward compatibility. For more information seeLogging from Spiders.

closed(reason)¶

Called when the spider closes. This method provides a shortcut tosignals.connect() for the spider_closed signal.

Let’s see an example:

import scrapyclass MySpider(scrapy.Spider): name = "example.com" allowed_domains = ["example.com"] start_urls = [ "http://www.example.com/1.html", "http://www.example.com/2.html", "http://www.example.com/3.html", ] def parse(self, response): self.logger.info("A response from %s just arrived!", response.url)

Return multiple Requests and items from a single callback:

import scrapyclass MySpider(scrapy.Spider): name = "example.com" allowed_domains = ["example.com"] start_urls = [ "http://www.example.com/1.html", "http://www.example.com/2.html", "http://www.example.com/3.html", ] def parse(self, response): for h3 in response.xpath("//h3").getall(): yield {"title": h3} for href in response.xpath("//a/@href").getall(): yield scrapy.Request(response.urljoin(href), self.parse)

Instead of start_urls you can use start_requests() directly;to give data more structure you can use Item objects:

import scrapyfrom myproject.items import MyItemclass MySpider(scrapy.Spider): name = "example.com" allowed_domains = ["example.com"] def start_requests(self): yield scrapy.Request("http://www.example.com/1.html", self.parse) yield scrapy.Request("http://www.example.com/2.html", self.parse) yield scrapy.Request("http://www.example.com/3.html", self.parse) def parse(self, response): for h3 in response.xpath("//h3").getall(): yield MyItem(title=h3) for href in response.xpath("//a/@href").getall(): yield scrapy.Request(response.urljoin(href), self.parse)

Spider arguments¶

Spiders can receive arguments that modify their behaviour. Some common uses forspider arguments are to define the start URLs or to restrict the crawl tocertain sections of the site, but they can be used to configure anyfunctionality of the spider.

Spider arguments are passed through the crawl command using the-a option. For example:

scrapy crawl myspider -a category=electronics

Spiders can access arguments in their __init__ methods:

import scrapyclass MySpider(scrapy.Spider): name = "myspider" def __init__(self, category=None, *args, **kwargs): super(MySpider, self).__init__(*args, **kwargs) self.start_urls = [f"http://www.example.com/categories/{category}"] # ...

The default __init__ method will take any spider argumentsand copy them to the spider as attributes.The above example can also be written as follows:

import scrapyclass MySpider(scrapy.Spider): name = "myspider" def start_requests(self): yield scrapy.Request(f"http://www.example.com/categories/{self.category}")

If you are running Scrapy from a script, you canspecify spider arguments when callingCrawlerProcess.crawl orCrawlerRunner.crawl:

process = CrawlerProcess()process.crawl(MySpider, category="electronics")

Keep in mind that spider arguments are only strings.The spider will not do any parsing on its own.If you were to set the start_urls attribute from the command line,you would have to parse it on your own into a listusing something like ast.literal_eval() or json.loads()and then set it as an attribute.Otherwise, you would cause iteration over a start_urls string(a very common python pitfall)resulting in each character being seen as a separate url.

A valid use case is to set the http auth credentialsused by HttpAuthMiddlewareor the user agentused by UserAgentMiddleware:

scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword -a user_agent=mybot

Spider arguments can also be passed through the Scrapyd schedule.json API.See Scrapyd documentation.

Generic Spiders¶

Scrapy comes with some useful generic spiders that you can use to subclassyour spiders from. Their aim is to provide convenient functionality for a fewcommon scraping cases, like following all links on a site based on certainrules, crawling from Sitemaps, or parsing an XML/CSV feed.

For the examples used in the following spiders, we’ll assume you have a projectwith a TestItem declared in a myproject.items module:

import scrapyclass TestItem(scrapy.Item): id = scrapy.Field() name = scrapy.Field() description = scrapy.Field()

import scrapyfrom scrapy.spiders import CrawlSpider, Rulefrom scrapy.linkextractors import LinkExtractorclass MySpider(CrawlSpider): name = "example.com" allowed_domains = ["example.com"] start_urls = ["http://www.example.com"] rules = ( # Extract links matching 'category.php' (but not matching 'subsection.php') # and follow links from them (since no callback means follow=True by default). Rule(LinkExtractor(allow=(r"category\.php",), deny=(r"subsection\.php",))), # Extract links matching 'item.php' and parse them with the spider's method parse_item Rule(LinkExtractor(allow=(r"item\.php",)), callback="parse_item"), ) def parse_item(self, response): self.logger.info("Hi, this is an item page! %s", response.url) item = scrapy.Item() item["id"] = response.xpath('//td[@id="item_id"]/text()').re(r"ID: (\d+)") item["name"] = response.xpath('//td[@id="item_name"]/text()').get() item["description"] = response.xpath( '//td[@id="item_description"]/text()' ).get() item["link_text"] = response.meta["link_text"] url = response.xpath('//td[@id="additional_data"]/@href').get() return response.follow( url, self.parse_additional_page, cb_kwargs=dict(item=item) ) def parse_additional_page(self, response, item): item["additional_data"] = response.xpath( '//p[@id="additional_data"]/text()' ).get() return item

from scrapy.spiders import XMLFeedSpiderfrom myproject.items import TestItemclass MySpider(XMLFeedSpider): name = "example.com" allowed_domains = ["example.com"] start_urls = ["http://www.example.com/feed.xml"] iterator = "iternodes" # This is actually unnecessary, since it's the default value itertag = "item" def parse_node(self, response, node): self.logger.info( "Hi, this is a <%s> node!: %s", self.itertag, "".join(node.getall()) ) item = TestItem() item["id"] = node.xpath("@id").get() item["name"] = node.xpath("name").get() item["description"] = node.xpath("description").get() return item

from scrapy.spiders import CSVFeedSpiderfrom myproject.items import TestItemclass MySpider(CSVFeedSpider): name = "example.com" allowed_domains = ["example.com"] start_urls = ["http://www.example.com/feed.csv"] delimiter = ";" quotechar = "'" headers = ["id", "name", "description"] def parse_row(self, response, row): self.logger.info("Hi, this is a row!: %r", row) item = TestItem() item["id"] = row["id"] item["name"] = row["name"] item["description"] = row["description"] return item

from scrapy.spiders import SitemapSpiderclass MySpider(SitemapSpider): sitemap_urls = ["http://www.example.com/sitemap.xml"] def parse(self, response): pass # ... scrape item here ...

from scrapy.spiders import SitemapSpiderclass MySpider(SitemapSpider): sitemap_urls = ["http://www.example.com/sitemap.xml"] sitemap_rules = [ ("/product/", "parse_product"), ("/category/", "parse_category"), ] def parse_product(self, response): pass # ... scrape product ... def parse_category(self, response): pass # ... scrape category ...

from scrapy.spiders import SitemapSpiderclass MySpider(SitemapSpider): sitemap_urls = ["http://www.example.com/robots.txt"] sitemap_rules = [ ("/shop/", "parse_shop"), ] sitemap_follow = ["/sitemap_shops"] def parse_shop(self, response): pass # ... scrape shop here ...

from scrapy.spiders import SitemapSpiderclass MySpider(SitemapSpider): sitemap_urls = ["http://www.example.com/robots.txt"] sitemap_rules = [ ("/shop/", "parse_shop"), ] other_urls = ["http://www.example.com/about"] def start_requests(self): requests = list(super(MySpider, self).start_requests()) requests += [scrapy.Request(x, self.parse_other) for x in self.other_urls] return requests def parse_shop(self, response): pass # ... scrape shop here ... def parse_other(self, response): pass # ... scrape other here ...