Spider API Reference (2024)

Start crawling a website(s) to collect resources.

POSThttps://api.spider.cloud/crawl

• urlrequiredstring

  The URI resource to crawl. This can be a comma split list for multiple urls.

  Test Url

• requeststring

  The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.

• limitnumber

  The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.

  Crawl Limit

• return_formatstring

  The format to return the data in. Possible values are markdown, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.

• proxy_enabledboolean

  Enable premium proxies to prevent detection. Default is false.

• anti_botboolean

  Enable anti-bot mode using various techniques to increase the chance of success. Default is false.

• tldboolean

  Allow TLD's to be included. Default is false.

• depthnumber

  The crawl limit for maximum depth. If 0, no limit will be applied.

  Crawl Depth

• cacheboolean

  Use HTTP caching for the crawl to speed up repeated runs. Default is true

• budgetobject

  Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.

  Crawl Budget

• localestring

  The locale to use for request, example en-US.

• cookiesstring

  Add HTTP cookies to use for request.

• stealthboolean

  Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.

• headersobject

  Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.

• metadataboolean

  Boolean to store metadata about the pages and content found. This could help improve AI interopt. Defaults to false unless you have the website already stored with the configuration enabled.

• viewportobject

  Configure the viewport for chrome. Defaults to 800x600.

• encodingstring

  The type of encoding to use like UTF-8, SHIFT_JIS, or etc.

• blacklistarray

  Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.

• whitelistarray

  Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.

• subdomainsboolean

  Allow subdomains to be included. Default is false.

• user_agentstring

  Add a custom HTTP user agent to the request. By default this is set to a random agent.

• store_databoolean

  Boolean to determine if storage should be used. If set this takes precedence over storageless. Defaults to false.

• gpt_configobject

  Use AI to generate actions to perform during the crawl. You can pass an array for the "prompt" to chain steps.

• fingerprintboolean

  Use advanced fingerprint for chrome.

• storagelessboolean

  Boolean to prevent storing any type of data for the request including storage and AI vectors embedding. Defaults to false unless you have the website already stored.

• readabilityboolean

  Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.

• chunking_algobject

  Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.

• respect_robotsboolean

  Respect the robots.txt file for crawling. Default is true.

• query_selectorstring

  The CSS query selector to use when extracting content from the markup.

  Test Query Selector

• full_resourcesboolean

  Crawl and download all the resources for a website.

• request_timeoutnumber

  The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.

• run_in_backgroundboolean

  Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.

• skip_config_checksboolean

  Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/json',}json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}response = requests.post('https://api.spider.cloud/crawl', headers=headers, json=json_data)print(response.json())

Response

Perform a search and gather a list of websites to start crawling and collect resources.

POSThttps://api.spider.cloud/search

• searchrequiredstring

  The search query you want to search for.

  Search

• search_limitnumber

  The limit amount of urls to fetch or crawl from the search results. Remove the value or set it to 0 to crawl all URLs from the search results.

• fetch_page_contentboolean

  Fetch all the content of the websites by performing crawls. The default is true; if this is disabled, only the search results are returned instead.

• countrystring

  The country code to use for the search. It's a two-letter country code. (e.g. us for the United States).

• locationstring

  The location from where you want the search to originate.

• languagestring

  The language to use for the search. It's a two-letter language code (e.g., en for English).

• numnumber

  The maximum number of results to return for the search.

• tldboolean

  Allow TLD's to be included. Default is false.

• pagenumber

  The page number for the search results.

• cacheboolean

  Use HTTP caching for the crawl to speed up repeated runs. Default is true

• requeststring

  The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.

• limitnumber

  The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.

  Crawl Limit

• return_formatstring

  The format to return the data in. Possible values are markdown, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.

• proxy_enabledboolean

  Enable premium proxies to prevent detection. Default is false.

• anti_botboolean

  Enable anti-bot mode using various techniques to increase the chance of success. Default is false.

• depthnumber

  The crawl limit for maximum depth. If 0, no limit will be applied.

  Crawl Depth

• budgetobject

  Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.

  Crawl Budget

• localestring

  The locale to use for request, example en-US.

• cookiesstring

  Add HTTP cookies to use for request.

• stealthboolean

  Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.

• headersobject

  Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.

• viewportobject

  Configure the viewport for chrome. Defaults to 800x600.

• encodingstring

  The type of encoding to use like UTF-8, SHIFT_JIS, or etc.

• blacklistarray

  Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.

• whitelistarray

  Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.

• subdomainsboolean

  Allow subdomains to be included. Default is false.

• user_agentstring

  Add a custom HTTP user agent to the request. By default this is set to a random agent.

• store_databoolean

  Boolean to determine if storage should be used. If set this takes precedence over storageless. Defaults to false.

• metadataboolean

  Boolean to store metadata about the pages and content found. This could help improve AI interopt. Defaults to false unless you have the website already stored with the configuration enabled.

• gpt_configobject

  Use AI to generate actions to perform during the crawl. You can pass an array for the "prompt" to chain steps.

• fingerprintboolean

  Use advanced fingerprint for chrome.

• storagelessboolean

  Boolean to prevent storing any type of data for the request including storage and AI vectors embedding. Defaults to false unless you have the website already stored.

• readabilityboolean

  Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.

• chunking_algobject

  Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.

• respect_robotsboolean

  Respect the robots.txt file for crawling. Default is true.

• query_selectorstring

  The CSS query selector to use when extracting content from the markup.

  Test Query Selector

• full_resourcesboolean

  Crawl and download all the resources for a website.

• request_timeoutnumber

  The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.

• run_in_backgroundboolean

  Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.

• skip_config_checksboolean

  Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/json',}json_data = {"search":"a sports website","search_limit":3,"limit":25,"return_format":"markdown"}response = requests.post('https://api.spider.cloud/search', headers=headers, json=json_data)print(response.json())

Response

Start crawling a website(s) to collect links found.

POSThttps://api.spider.cloud/links

• urlrequiredstring

  The URI resource to crawl. This can be a comma split list for multiple urls.

  Test Url

• requeststring

  The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.

• limitnumber

  The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.

  Crawl Limit

• return_formatstring

  The format to return the data in. Possible values are markdown, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.

• proxy_enabledboolean

  Enable premium proxies to prevent detection. Default is false.

• anti_botboolean

  Enable anti-bot mode using various techniques to increase the chance of success. Default is false.

• tldboolean

  Allow TLD's to be included. Default is false.

• depthnumber

  The crawl limit for maximum depth. If 0, no limit will be applied.

  Crawl Depth

• cacheboolean

  Use HTTP caching for the crawl to speed up repeated runs. Default is true

• budgetobject

  Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.

  Crawl Budget

• localestring

  The locale to use for request, example en-US.

• cookiesstring

  Add HTTP cookies to use for request.

• stealthboolean

  Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.

• headersobject

  Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.

• metadataboolean

  Boolean to store metadata about the pages and content found. This could help improve AI interopt. Defaults to false unless you have the website already stored with the configuration enabled.

• viewportobject

  Configure the viewport for chrome. Defaults to 800x600.

• encodingstring

  The type of encoding to use like UTF-8, SHIFT_JIS, or etc.

• blacklistarray

  Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.

• whitelistarray

  Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.

• subdomainsboolean

  Allow subdomains to be included. Default is false.

• user_agentstring

  Add a custom HTTP user agent to the request. By default this is set to a random agent.

• store_databoolean

  Boolean to determine if storage should be used. If set this takes precedence over storageless. Defaults to false.

• gpt_configobject

  Use AI to generate actions to perform during the crawl. You can pass an array for the "prompt" to chain steps.

• fingerprintboolean

  Use advanced fingerprint for chrome.

• storagelessboolean

  Boolean to prevent storing any type of data for the request including storage and AI vectors embedding. Defaults to false unless you have the website already stored.

• readabilityboolean

  Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.

• chunking_algobject

  Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.

• respect_robotsboolean

  Respect the robots.txt file for crawling. Default is true.

• query_selectorstring

  The CSS query selector to use when extracting content from the markup.

  Test Query Selector

• full_resourcesboolean

  Crawl and download all the resources for a website.

• request_timeoutnumber

  The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.

• run_in_backgroundboolean

  Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.

• skip_config_checksboolean

  Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/json',}json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}response = requests.post('https://api.spider.cloud/links', headers=headers, json=json_data)print(response.json())

Response

Start taking screenshots of website(s) to collect images to base64 or binary.

POSThttps://api.spider.cloud/screenshot

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/json',}json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}response = requests.post('https://api.spider.cloud/screenshot', headers=headers, json=json_data)print(response.json())

Response

Transform HTML to Markdown or text fast. Each HTML transformation costs 1 credit. You can send up to 10MB of data at once.

POSThttps://api.spider.cloud/transform

• dataobject

  A list of html data to transform. The object list takes the keys html and url. The url key is optional and only used when the readability is enabled.

  Data

• return_formatstring

  The format to return the data in. Possible values are markdown, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.

• readabilityboolean

  Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/json',}json_data = {"return_format":"markdown","data":[{"html":"<html>\n<head>\n <title>Example Transform</title>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\">\n <style type=\"text/css\">\n html {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-size: 16px;\n }\n </style> \n</head>\n<body>\n<div>\n <h1>Example Website</h1>\n <p>This is some example markup to use to test the transform function.</p>\n <p><a href=\"https://spider.cloud/guides\">Guides</a></p>\n</div>\n</body></html>","url":"https://example.com"}]}response = requests.post('https://api.spider.cloud/transform', headers=headers, json=json_data)print(response.json())

Response

Start crawling a website(s) to collect all contacts utilizing AI. A minimum of $25 in credits is necessary for extraction.

POSThttps://api.spider.cloud/pipeline/extract-contacts

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}response = requests.post('https://api.spider.cloud/pipeline/extract-contacts', headers=headers, json=json_data)print(response.json())

Response

Crawl a website and accurately categorize it using AI.

POSThttps://api.spider.cloud/pipeline/label

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}response = requests.post('https://api.spider.cloud/pipeline/label', headers=headers, json=json_data)print(response.json())

Response

Crawl website(s) found from raw text or markdown.

POSThttps://api.spider.cloud/pipeline/crawl-text

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}json_data = {"text":"Check this link: https://example.com and email to example@email.com","limit":25,"return_format":"markdown"}response = requests.post('https://api.spider.cloud/pipeline/crawl-text', headers=headers, json=json_data)print(response.json())

Response

Get the websites stored.

GEThttps://api.spider.cloud/data/websites

• limitstring

  The limit of records to get.

  Crawl Limit

• pagenumber

  The current page to get.

• domainstring

  Filter a single domain record.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/websites?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get the pages/resources stored.

GEThttps://api.spider.cloud/data/pages

• limitstring

  The limit of records to get.

  Crawl Limit

• pagenumber

  The current page to get.

• domainstring

  Filter a single domain record.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/pages?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get the pages metadata/resources stored.

GEThttps://api.spider.cloud/data/pages_metadata

• limitstring

  The limit of records to get.

  Crawl Limit

• pagenumber

  The current page to get.

• domainstring

  Filter a single domain record.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/pages_metadata?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get the pages contacts stored.

GEThttps://api.spider.cloud/data/contacts

• limitstring

  The limit of records to get.

  Crawl Limit

• pagenumber

  The current page to get.

• domainstring

  Filter a single domain record.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/contacts?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get the state of the crawl for the domain.

GEThttps://api.spider.cloud/data/crawl_state

• limitstring

  The limit of records to get.

  Crawl Limit

• pagenumber

  The current page to get.

• domainstring

  Filter a single domain record.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/crawl_state?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get the last 24 hours of logs.

GEThttps://api.spider.cloud/data/crawl_logs

• limitstring

  The limit of records to get.

  Crawl Limit

• pagenumber

  The current page to get.

• domainstring

  Filter a single domain record.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/crawl_logs?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get the remaining credits available.

GEThttps://api.spider.cloud/data/credits

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/credits?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get the cron jobs that are set to keep data fresh.

GEThttps://api.spider.cloud/data/crons

• limitstring

  The limit of records to get.

  Crawl Limit

• pagenumber

  The current page to get.

• domainstring

  Filter a single domain record.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/crons?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get the profile of the user. This returns data such as approved limits and usage for the month.

GEThttps://api.spider.cloud/data/profiles

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/crawl?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Get a real user agent to use for crawling.

GEThttps://api.spider.cloud/data/user_agents

• limitstring

  The limit of records to get.

  Crawl Limit

• osstring

  Filter a by a device ex: Android, Mac OS, Android, Windows, Linux and more.

• pagenumber

  The current page to get.

• platformstring

  Filter a by a platform ex: Chrome, Edge, Safari, Firefox and more.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}response = requests.get('https://api.spider.cloud/data/user_agents?limit=25&return_format=markdown', headers=headers)print(response.json())

Response

Create or update a website by configuration.

POSThttps://api.spider.cloud/data/websites

• urlrequiredstring

  The URI resource to crawl. This can be a comma split list for multiple urls.

  Test Url

• requeststring

  The request type to perform. Possible values are http, chrome, and smart. Use smart to perform HTTP request by default until JavaScript rendering is needed for the HTML.

• limitnumber

  The maximum amount of pages allowed to crawl per website. Remove the value or set it to 0 to crawl all pages.

  Crawl Limit

• return_formatstring

  The format to return the data in. Possible values are markdown, raw, text, and bytes. Use raw to return the default format of the page like HTML etc.

• proxy_enabledboolean

  Enable premium proxies to prevent detection. Default is false.

• anti_botboolean

  Enable anti-bot mode using various techniques to increase the chance of success. Default is false.

• tldboolean

  Allow TLD's to be included. Default is false.

• cronstring

  Set a cron period to run the website crawls automatically. Possible values are daily, weekly, and monthly.

• depthnumber

  The crawl limit for maximum depth. If 0, no limit will be applied.

  Crawl Depth

• cacheboolean

  Use HTTP caching for the crawl to speed up repeated runs. Default is true

• budgetobject

  Object that has paths with a counter for limiting the amount of pages example {"*":1} for only crawling the root page. The wildcard matches all routes and you can set child paths preventing a depth level, example of limiting { "/docs/colors": 10, "/docs/": 100 } which only allows a max of 100 pages if the route matches /docs/:pathname and only 10 pages if it matches /docs/colors/:pathname.

  Crawl Budget

• localestring

  The locale to use for request, example en-US.

• cookiesstring

  Add HTTP cookies to use for request.

• stealthboolean

  Use stealth mode for headless chrome request to help prevent being blocked. The default is false on chrome.

• headersobject

  Forward HTTP headers to use for all request. The object is expected to be a map of key value pairs.

• metadataboolean

  Boolean to store metadata about the pages and content found. This could help improve AI interopt. Defaults to false unless you have the website already stored with the configuration enabled.

• viewportobject

  Configure the viewport for chrome. Defaults to 800x600.

• encodingstring

  The type of encoding to use like UTF-8, SHIFT_JIS, or etc.

• blacklistarray

  Blacklist a set of paths that you do not want to crawl. You can use Regex patterns to help with the list.

• whitelistarray

  Whitelist a set of paths that you want to crawl, ignoring all other routes that do not match the patterns. You can use regex patterns to help with the list.

• subdomainsboolean

  Allow subdomains to be included. Default is false.

• user_agentstring

  Add a custom HTTP user agent to the request. By default this is set to a random agent.

• store_databoolean

  Boolean to determine if storage should be used. If set this takes precedence over storageless. Defaults to false.

• gpt_configobject

  Use AI to generate actions to perform during the crawl. You can pass an array for the "prompt" to chain steps.

• fingerprintboolean

  Use advanced fingerprint for chrome.

• storagelessboolean

  Boolean to prevent storing any type of data for the request including storage and AI vectors embedding. Defaults to false unless you have the website already stored.

• readabilityboolean

  Use readability to pre-process the content for reading. This may drastically improve the content for LLM usage.

• chunking_algobject

  Use a chunking algorithm to segment your content output. Possible values are ByWords, ByLines, ByCharacterLength, and BySentence. The following is an example of the object shape for "chunking_alg": { "type": "bysentence", "value": 2 } which splits the text into an array by every 2 sentences found. This works well when used with markdown or text formats.

• respect_robotsboolean

  Respect the robots.txt file for crawling. Default is true.

• query_selectorstring

  The CSS query selector to use when extracting content from the markup.

  Test Query Selector

• full_resourcesboolean

  Crawl and download all the resources for a website.

• request_timeoutnumber

  The timeout to use for request. Timeouts can be from 5-60. The default is 30 seconds.

• run_in_backgroundboolean

  Run the request in the background. Useful if storing data and wanting to trigger crawls to the dashboard. This has no effect if storageless is set.

• skip_config_checksboolean

  Skip checking the database for website configuration. This may increase performance for request that are using limit=1. The default is false.

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}response = requests.post('https://api.spider.cloud/crawl', headers=headers, json=json_data)print(response.json())

Response

Delete a website from your collection. Remove the url body to delete all websites.

DELETEhttps://api.spider.cloud/data/websites

• urlrequiredstring

  The URI resource to crawl. This can be a comma split list for multiple urls.

  Test Url

Example request

import requests, osheaders = { 'Authorization': os.environ["SPIDER_API_KEY"], 'Content-Type': 'application/jsonl',}json_data = {"limit":25,"return_format":"markdown","url":"https://spider.cloud"}response = requests.post('https://api.spider.cloud/crawl', headers=headers, json=json_data)print(response.json())

Response