Index

_

__nonzero__() (scrapy.selector.XPathSelector method)

A

adapt_response() (scrapy.contrib.spiders.XMLFeedSpider method) add_value() (scrapy.contrib.loader.ItemLoader method) add_xpath() (scrapy.contrib.loader.XPathItemLoader method)

allowed_domains (scrapy.spider.BaseSpider attribute) AWS_ACCESS_KEY_ID setting AWS_SECRET_ACCESS_KEY setting

B

BaseItemExporter (class in scrapy.contrib.exporter) BaseSgmlLinkExtractor (class in scrapy.contrib.linkextractors.sgml) BaseSpider (class in scrapy.spider) body (scrapy.http.Request attribute) (scrapy.http.Response attribute)

body_as_unicode() (scrapy.http.TextResponse method) BOT_NAME setting BOT_VERSION setting

C

ChunkedTransferMiddleware (class in scrapy.contrib.downloadermiddleware.chunked) clear_stats() (scrapy.statscol.StatsCollector method) close_spider() (scrapy.statscol.StatsCollector method) CloseSpider CLOSESPIDER_ERRORCOUNT setting CLOSESPIDER_ITEMCOUNT setting CLOSESPIDER_PAGECOUNT setting CLOSESPIDER_TIMEOUT setting command crawl deploy edit fetch genspider list parse runspider server settings shell startproject version view COMMANDS_MODULE setting Compose (class in scrapy.contrib.loader.processor) CONCURRENT_ITEMS setting CONCURRENT_REQUESTS setting CONCURRENT_REQUESTS_PER_DOMAIN setting

CONCURRENT_REQUESTS_PER_IP setting context (scrapy.contrib.loader.ItemLoader attribute) COOKIES_DEBUG setting COOKIES_ENABLED setting CookiesMiddleware (class in scrapy.contrib.downloadermiddleware.cookies) copy() (scrapy.http.Request method) (scrapy.http.Response method) CoreStats (class in scrapy.contrib.corestats) crawl command CrawlerResource (class in scrapy.contrib.webservice.crawler) CrawlSpider (class in scrapy.contrib.spiders) CRITICAL (in module scrapy.log) CSVFeedSpider (class in scrapy.contrib.spiders) CsvItemExporter (class in scrapy.contrib.exporter)

D

DEBUG (in module scrapy.log) default_input_processor (scrapy.contrib.loader.ItemLoader attribute) DEFAULT_ITEM_CLASS setting default_item_class (scrapy.contrib.loader.ItemLoader attribute) default_output_processor (scrapy.contrib.loader.ItemLoader attribute) DEFAULT_REQUEST_HEADERS setting DEFAULT_RESPONSE_ENCODING setting default_selector_class (scrapy.contrib.loader.XPathItemLoader attribute) DefaultHeadersMiddleware (class in scrapy.contrib.downloadermiddleware.defaultheaders) delimiter (scrapy.contrib.spiders.CSVFeedSpider attribute) deploy command DEPTH_LIMIT setting DEPTH_PRIORITY setting DEPTH_STATS setting DEPTH_STATS_VERBOSE setting DepthMiddleware (class in scrapy.contrib.spidermiddleware.depth) disabled (scrapy.extension.ExtensionManager attribute)

DNSCACHE_ENABLED setting dont_redirect reqmeta dont_retry reqmeta DOWNLOAD_DELAY setting DOWNLOAD_HANDLERS setting DOWNLOAD_HANDLERS_BASE setting DOWNLOAD_TIMEOUT setting DOWNLOADER_DEBUG setting DOWNLOADER_MIDDLEWARES setting DOWNLOADER_MIDDLEWARES_BASE setting DOWNLOADER_STATS setting DownloaderMiddleware (class in scrapy.contrib.downloadermiddleware) DownloaderStats (class in scrapy.contrib.downloadermiddleware.stats) DownloadTimeoutMiddleware (class in scrapy.contrib.downloadermiddleware.downloadtimeout) DropItem DummyStatsCollector (class in scrapy.statscol) DUPEFILTER_CLASS setting

E

edit command EDITOR setting enabled (scrapy.extension.ExtensionManager attribute) encoding (scrapy.contrib.exporter.BaseItemExporter attribute) (scrapy.http.TextResponse attribute) ENCODING_ALIASES setting ENCODING_ALIASES_BASE setting engine_started signal engine_started() (in module scrapy.signals) engine_stopped signal engine_stopped() (in module scrapy.signals) (scrapy.statscol.StatsCollector method)

EngineStatusResource (class in scrapy.contrib.webservice.enginestatus) ERROR (in module scrapy.log) export_empty_fields (scrapy.contrib.exporter.BaseItemExporter attribute) export_item() (scrapy.contrib.exporter.BaseItemExporter method) ExtensionManager (class in scrapy.extension) EXTENSIONS setting EXTENSIONS_BASE setting extract() (scrapy.selector.XPathSelector method) (scrapy.selector.XPathSelectorList method) extract_unquoted() (scrapy.selector.XPathSelectorList method)

F

FEED_EXPORTERS setting FEED_EXPORTERS_BASE setting FEED_FORMAT setting FEED_STORAGES setting FEED_STORAGES_BASE setting FEED_STORE_EMPTY setting FEED_URI setting fetch command

Field (class in scrapy.item) fields (scrapy.item.Item attribute) fields_to_export (scrapy.contrib.exporter.BaseItemExporter attribute) finish_exporting() (scrapy.contrib.exporter.BaseItemExporter method) flags (scrapy.http.Response attribute) FormRequest (class in scrapy.http) from_response() (scrapy.http.FormRequest class method)

G

genspider command get() (scrapy.conf.Settings method) get_collected_values() (scrapy.contrib.loader.ItemLoader method) get_input_processor() (scrapy.contrib.loader.ItemLoader method) get_media_requests() (scrapy.contrib.pipeline.images.ImagesPipeline method) get_oldest() (in module scrapy.utils.trackref) get_output_processor() (scrapy.contrib.loader.ItemLoader method) get_output_value() (scrapy.contrib.loader.ItemLoader method)

get_stats() (scrapy.statscol.StatsCollector method) get_target() (scrapy.contrib.webservice.enginestatus.scrapy.webservice.JsonRpcResource method) get_value() (scrapy.contrib.loader.ItemLoader method) (scrapy.statscol.StatsCollector method) get_xpath() (scrapy.contrib.loader.XPathItemLoader method) getbool() (scrapy.conf.Settings method) getfloat() (scrapy.conf.Settings method) getint() (scrapy.conf.Settings method) getlist() (scrapy.conf.Settings method)

H

handle_httpstatus_list reqmeta headers (scrapy.contrib.spiders.CSVFeedSpider attribute) (scrapy.http.Request attribute) (scrapy.http.Response attribute) HtmlResponse (class in scrapy.http) HtmlXPathSelector (class in scrapy.selector) HttpAuthMiddleware (class in scrapy.contrib.downloadermiddleware.httpauth) HTTPCACHE_DBM_MODULE setting HTTPCACHE_DIR setting HTTPCACHE_ENABLED setting HTTPCACHE_EXPIRATION_SECS setting

HTTPCACHE_IGNORE_HTTP_CODES setting HTTPCACHE_IGNORE_MISSING setting HTTPCACHE_IGNORE_SCHEMES setting HTTPCACHE_STORAGE setting HttpCacheMiddleware (class in scrapy.contrib.downloadermiddleware.httpcache) HttpCompressionMiddleware (class in scrapy.contrib.downloadermiddleware.httpcompression) HttpErrorMiddleware (class in scrapy.contrib.spidermiddleware.httperror) HttpProxyMiddleware (class in scrapy.contrib.downloadermiddleware.httpproxy)

I

Identity (class in scrapy.contrib.loader.processor) IgnoreRequest IMAGES_EXPIRES setting IMAGES_MIN_HEIGHT setting IMAGES_MIN_WIDTH setting IMAGES_STORE setting IMAGES_THUMBS setting ImagesPipeline (class in scrapy.contrib.pipeline.images) inc_value() (scrapy.statscol.StatsCollector method) INFO (in module scrapy.log) Item (class in scrapy.item) item (scrapy.contrib.loader.ItemLoader attribute)

item_completed() (scrapy.contrib.pipeline.images.ImagesPipeline method) item_dropped signal item_dropped() (in module scrapy.signals) ITEM_PIPELINES setting item_scraped signal item_scraped() (in module scrapy.signals) ItemLoader (class in scrapy.contrib.loader) iter_all() (in module scrapy.utils.trackref) iter_spider_stats() (scrapy.statscol.StatsCollector method) iterator (scrapy.contrib.spiders.XMLFeedSpider attribute) itertag (scrapy.contrib.spiders.XMLFeedSpider attribute)

J

Join (class in scrapy.contrib.loader.processor) JsonItemExporter (class in scrapy.contrib.exporter)

JsonLinesItemExporter (class in scrapy.contrib.exporter)

L

list command load() (scrapy.extension.ExtensionManager method) load_item() (scrapy.contrib.loader.ItemLoader method) loaded (scrapy.extension.ExtensionManager attribute) log() (scrapy.spider.BaseSpider method) LOG_ENABLED setting

LOG_ENCODING setting LOG_FILE setting LOG_LEVEL setting LOG_STDOUT setting LogStats (class in scrapy.contrib.logstats)

M

MAIL_FROM setting MAIL_HOST setting MAIL_PASS setting MAIL_PORT setting mail_sent signal mail_sent() (in module scrapy.mail) MAIL_USER setting make_requests_from_url() (scrapy.spider.BaseSpider method) MapCompose (class in scrapy.contrib.loader.processor) max_value() (scrapy.statscol.StatsCollector method) MEMDEBUG_ENABLED setting

MEMDEBUG_NOTIFY setting MemoryStatsCollector (class in scrapy.statscol) MEMUSAGE_ENABLED setting MEMUSAGE_LIMIT_MB setting MEMUSAGE_NOTIFY_MAIL setting MEMUSAGE_REPORT setting MEMUSAGE_WARNING_MB setting meta (scrapy.http.Request attribute) (scrapy.http.Response attribute) method (scrapy.http.Request attribute) min_value() (scrapy.statscol.StatsCollector method) msg() (in module scrapy.log)

N

name (scrapy.spider.BaseSpider attribute) namespaces (scrapy.contrib.spiders.XMLFeedSpider attribute) NEWSPIDER_MODULE setting

NotConfigured NotSupported

O

object_ref (class in scrapy.utils.trackref) OffsiteMiddleware (class in scrapy.contrib.spidermiddleware.offsite)

open_spider() (scrapy.statscol.StatsCollector method)

P

parse command parse() (scrapy.spider.BaseSpider method) parse_node() (scrapy.contrib.spiders.XMLFeedSpider method) parse_row() (scrapy.contrib.spiders.CSVFeedSpider method) PickleItemExporter (class in scrapy.contrib.exporter) PprintItemExporter (class in scrapy.contrib.exporter) print_live_refs() (in module scrapy.utils.trackref) process_exception() (scrapy.contrib.downloadermiddleware.DownloaderMiddleware method)

process_item() process_request() (scrapy.contrib.downloadermiddleware.DownloaderMiddleware method) process_response() (scrapy.contrib.downloadermiddleware.DownloaderMiddleware method) process_results() (scrapy.contrib.spiders.XMLFeedSpider method) process_spider_exception() (scrapy.contrib.spidermiddleware.SpiderMiddleware method) process_spider_input() (scrapy.contrib.spidermiddleware.SpiderMiddleware method) process_spider_output() (scrapy.contrib.spidermiddleware.SpiderMiddleware method) Python Enhancement Proposals PEP 8

R

RANDOMIZE_DOWNLOAD_DELAY setting re() (scrapy.selector.XPathSelector method) (scrapy.selector.XPathSelectorList method) REDIRECT_ENABLED setting REDIRECT_MAX_METAREFRESH_DELAY setting, [1] REDIRECT_MAX_TIMES setting, [1] REDIRECT_PRIORITY_ADJUST setting redirect_urls reqmeta RedirectMiddleware (class in scrapy.contrib.downloadermiddleware.redirect) RefererMiddleware (class in scrapy.contrib.spidermiddleware.referer) register_namespace() (scrapy.selector.XPathSelector method) reload() (scrapy.extension.ExtensionManager method) replace() (scrapy.http.Request method) (scrapy.http.Response method) replace_value() (scrapy.contrib.loader.ItemLoader method) replace_xpath() (scrapy.contrib.loader.XPathItemLoader method) reqmeta dont_redirect dont_retry handle_httpstatus_list redirect_urls Request (class in scrapy.http) request (scrapy.http.Response attribute)

request_received signal request_received() (in module scrapy.signals) Response (class in scrapy.http) response_downloaded signal response_downloaded() (in module scrapy.signals) response_received signal response_received() (in module scrapy.signals) RETRY_ENABLED setting RETRY_HTTP_CODES setting RETRY_TIMES setting RetryMiddleware (class in scrapy.contrib.downloadermiddleware.retry) ROBOTSTXT_OBEY setting RobotsTxtMiddleware (class in scrapy.contrib.downloadermiddleware.robotstxt) Rule (class in scrapy.contrib.spiders) rules (scrapy.contrib.spiders.CrawlSpider attribute) runspider command

S

SCHEDULER setting scrapy.conf (module) scrapy.contrib.closespider (module) scrapy.contrib.closespider.CloseSpider (class in scrapy.contrib.closespider) scrapy.contrib.corestats (module) scrapy.contrib.debug (module) scrapy.contrib.debug.Debugger (class in scrapy.contrib.debug) scrapy.contrib.debug.StackTraceDump (class in scrapy.contrib.debug) scrapy.contrib.downloadermiddleware (module) scrapy.contrib.downloadermiddleware.chunked (module) scrapy.contrib.downloadermiddleware.cookies (module) scrapy.contrib.downloadermiddleware.defaultheaders (module) scrapy.contrib.downloadermiddleware.downloadtimeout (module) scrapy.contrib.downloadermiddleware.httpauth (module) scrapy.contrib.downloadermiddleware.httpcache (module) scrapy.contrib.downloadermiddleware.httpcompression (module) scrapy.contrib.downloadermiddleware.httpproxy (module) scrapy.contrib.downloadermiddleware.redirect (module) scrapy.contrib.downloadermiddleware.retry (module) scrapy.contrib.downloadermiddleware.robotstxt (module) scrapy.contrib.downloadermiddleware.stats (module) scrapy.contrib.downloadermiddleware.useragent (module) scrapy.contrib.exporter (module) scrapy.contrib.linkextractors (module) scrapy.contrib.linkextractors.sgml (module) scrapy.contrib.loader (module) scrapy.contrib.loader.processor (module) scrapy.contrib.logstats (module) scrapy.contrib.memdebug (module) scrapy.contrib.memdebug.MemoryDebugger (class in scrapy.contrib.memdebug) scrapy.contrib.memusage (module) scrapy.contrib.memusage.MemoryUsage (class in scrapy.contrib.memusage) scrapy.contrib.pipeline.images (module) scrapy.contrib.spidermiddleware (module) scrapy.contrib.spidermiddleware.depth (module) scrapy.contrib.spidermiddleware.httperror (module) scrapy.contrib.spidermiddleware.offsite (module) scrapy.contrib.spidermiddleware.referer (module) scrapy.contrib.spidermiddleware.urllength (module) scrapy.contrib.spiders (module) scrapy.contrib.statsmailer (module) scrapy.contrib.statsmailer.StatsMailer (class in scrapy.contrib.statsmailer) scrapy.contrib.webservice (module) scrapy.contrib.webservice.crawler (module) scrapy.contrib.webservice.enginestatus (module) scrapy.contrib.webservice.stats (module) scrapy.contrib_exp.djangoitem (module) scrapy.exceptions (module) scrapy.extension (module) scrapy.http (module) scrapy.item (module) scrapy.log (module) scrapy.mail (module) scrapy.selector (module) scrapy.signals (module) scrapy.spider (module) scrapy.statscol (module)

scrapy.telnet (module), [1] scrapy.telnet.TelnetConsole (class in scrapy.telnet) scrapy.utils.trackref (module) scrapy.webservice (module) scrapy.webservice.JsonResource (class in scrapy.contrib.webservice.enginestatus) scrapy.webservice.JsonRpcResource (class in scrapy.contrib.webservice.enginestatus) scrapy.webservice.WebService (class in scrapy.webservice) select() (scrapy.selector.XPathSelector method) (scrapy.selector.XPathSelectorList method) selector (scrapy.contrib.loader.XPathItemLoader attribute) send() (in module scrapy.mail) serialize_field() (scrapy.contrib.exporter.BaseItemExporter method) server command set_stats() (scrapy.statscol.StatsCollector method) set_value() (scrapy.statscol.StatsCollector method) setting AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY BOT_NAME BOT_VERSION CLOSESPIDER_ERRORCOUNT CLOSESPIDER_ITEMCOUNT CLOSESPIDER_PAGECOUNT CLOSESPIDER_TIMEOUT COMMANDS_MODULE CONCURRENT_ITEMS CONCURRENT_REQUESTS CONCURRENT_REQUESTS_PER_DOMAIN CONCURRENT_REQUESTS_PER_IP COOKIES_DEBUG COOKIES_ENABLED DEFAULT_ITEM_CLASS DEFAULT_REQUEST_HEADERS DEFAULT_RESPONSE_ENCODING DEPTH_LIMIT DEPTH_PRIORITY DEPTH_STATS DEPTH_STATS_VERBOSE DNSCACHE_ENABLED DOWNLOADER_DEBUG DOWNLOADER_MIDDLEWARES DOWNLOADER_MIDDLEWARES_BASE DOWNLOADER_STATS DOWNLOAD_DELAY DOWNLOAD_HANDLERS DOWNLOAD_HANDLERS_BASE DOWNLOAD_TIMEOUT DUPEFILTER_CLASS EDITOR ENCODING_ALIASES ENCODING_ALIASES_BASE EXTENSIONS EXTENSIONS_BASE FEED_EXPORTERS FEED_EXPORTERS_BASE FEED_FORMAT FEED_STORAGES FEED_STORAGES_BASE FEED_STORE_EMPTY FEED_URI HTTPCACHE_DBM_MODULE HTTPCACHE_DIR HTTPCACHE_ENABLED HTTPCACHE_EXPIRATION_SECS HTTPCACHE_IGNORE_HTTP_CODES HTTPCACHE_IGNORE_MISSING HTTPCACHE_IGNORE_SCHEMES HTTPCACHE_STORAGE IMAGES_EXPIRES IMAGES_MIN_HEIGHT IMAGES_MIN_WIDTH IMAGES_STORE IMAGES_THUMBS ITEM_PIPELINES LOG_ENABLED LOG_ENCODING LOG_FILE LOG_LEVEL LOG_STDOUT MAIL_FROM MAIL_HOST MAIL_PASS MAIL_PORT MAIL_USER MEMDEBUG_ENABLED MEMDEBUG_NOTIFY MEMUSAGE_ENABLED MEMUSAGE_LIMIT_MB MEMUSAGE_NOTIFY_MAIL MEMUSAGE_REPORT MEMUSAGE_WARNING_MB NEWSPIDER_MODULE RANDOMIZE_DOWNLOAD_DELAY REDIRECT_ENABLED REDIRECT_MAX_METAREFRESH_DELAY, [1] REDIRECT_MAX_TIMES, [1] REDIRECT_PRIORITY_ADJUST RETRY_ENABLED RETRY_HTTP_CODES RETRY_TIMES ROBOTSTXT_OBEY SCHEDULER SPIDER_MIDDLEWARES SPIDER_MIDDLEWARES_BASE SPIDER_MODULES STATSMAILER_RCPTS STATS_CLASS STATS_DUMP STATS_ENABLED TELNETCONSOLE_ENABLED TELNETCONSOLE_HOST TELNETCONSOLE_PORT, [1] TEMPLATES_DIR URLLENGTH_LIMIT USER_AGENT WEBSERVICE_ENABLED WEBSERVICE_HOST WEBSERVICE_LOGFILE WEBSERVICE_PORT settings command Settings (class in scrapy.conf) SgmlLinkExtractor (class in scrapy.contrib.linkextractors.sgml) shell command signal engine_started engine_stopped item_dropped item_scraped mail_sent request_received response_downloaded response_received spider_closed spider_error spider_idle spider_opened stats_spider_closed stats_spider_closing stats_spider_opened update_telnet_vars sitemap_follow (scrapy.contrib.spiders.SitemapSpider attribute) sitemap_rules (scrapy.contrib.spiders.SitemapSpider attribute) sitemap_urls (scrapy.contrib.spiders.SitemapSpider attribute) SitemapSpider (class in scrapy.contrib.spiders) spider_closed signal spider_closed() (in module scrapy.signals) spider_error signal spider_error() (in module scrapy.signals) spider_idle signal spider_idle() (in module scrapy.signals) SPIDER_MIDDLEWARES setting SPIDER_MIDDLEWARES_BASE setting SPIDER_MODULES setting spider_opened signal spider_opened() (in module scrapy.signals) spider_stats (scrapy.statscol.MemoryStatsCollector attribute) SpiderMiddleware (class in scrapy.contrib.spidermiddleware) start() (in module scrapy.log) start_exporting() (scrapy.contrib.exporter.BaseItemExporter method) start_requests() (scrapy.spider.BaseSpider method) start_urls (scrapy.spider.BaseSpider attribute) started (in module scrapy.log) startproject command STATS_CLASS setting STATS_DUMP setting STATS_ENABLED setting stats_spider_closed signal stats_spider_closed() (in module scrapy.signals) stats_spider_closing signal stats_spider_closing() (in module scrapy.signals) stats_spider_opened signal stats_spider_opened() (in module scrapy.signals) StatsCollector (class in scrapy.statscol) STATSMAILER_RCPTS setting StatsResource (class in scrapy.contrib.webservice.stats) status (scrapy.http.Response attribute)

T

TakeFirst (class in scrapy.contrib.loader.processor) TELNETCONSOLE_ENABLED setting TELNETCONSOLE_HOST setting

TELNETCONSOLE_PORT setting, [1] TEMPLATES_DIR setting TextResponse (class in scrapy.http)

U

update_telnet_vars signal update_telnet_vars() (in module scrapy.telnet) url (scrapy.http.Request attribute) (scrapy.http.Response attribute) URLLENGTH_LIMIT setting

UrlLengthMiddleware (class in scrapy.contrib.spidermiddleware.urllength) USER_AGENT setting UserAgentMiddleware (class in scrapy.contrib.downloadermiddleware.useragent)

V

version command

view command

W

WARNING (in module scrapy.log) WEBSERVICE_ENABLED setting WEBSERVICE_HOST setting

WEBSERVICE_LOGFILE setting WEBSERVICE_PORT setting ws_name (scrapy.contrib.webservice.enginestatus.scrapy.webservice.JsonResource attribute)

X

XMLFeedSpider (class in scrapy.contrib.spiders) XmlItemExporter (class in scrapy.contrib.exporter) XmlResponse (class in scrapy.http) XmlXPathSelector (class in scrapy.selector)

XPathItemLoader (class in scrapy.contrib.loader) XPathSelector (class in scrapy.selector) XPathSelectorList (class in scrapy.selector)