链接提取LinkExtractor与全站爬取利器CrawlSpider

摘要：关于LinkExtractor的详细使用，可以作为文档，参考。

LinkExtractor

对于提取链接，之前提到过可以通过 Selector 来提取，但 Selector 比较适合于爬去的连接比较简单其模式比较固定的情况。scrapy提供了另一个链接提取的方法 scrapy.linkextractors.LinkExtractor ，这种方法比较适合于爬去整站链接，并且只需声明一次就可使用多次。先来看看 LinkExtractor 构造的参数：

LinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), deny_extensions=None, restrict_xpaths=(), restrict_css=(), tags=('a', 'area'), attrs=('href', ), canonicalize=False, unique=True, process_value=None, strip=True)

下面看看各个参数并用实例讲解：

body = """
<!DOCTYPE html>
<html>
    <head>
    </head>
    <body>
        <div class='scrapyweb'>
            <p>Scrapy主站</p>
            <a href="https://scrapy.org/download/">Download</a>
            <a href="https://scrapy.org/doc/">Doc</a>
            <a href="https://scrapy.org/resources/">Resources</a>
        </div>
        <div class='scrapydoc'>
            <p>Scrapy 开发文档</p>
            <a href="https://docs.scrapy.org/en/latest/intro/overview.html">Scrapy at a glance</a>
            <a href="https://docs.scrapy.org/en/latest/intro/install.html">Installation guide</a>
            <a href="https://docs.scrapy.org/en/latest/intro/tutorial.html">Scrapy Tutorial</a>
            <a href="https://github.com/scrapy/scrapy/blob/1.5/docs/topics/media-pipeline.rst">Docs in github</a>
        </div>
    </body>
</html>
""".encode('utf8')
response = scrapy.http.HtmlResponse(url='', body = body)

allow：一个正则表达式或正则表达式的列表，只有匹配正则表达式的才会被提取出来，如果没有提供，就会爬取所有链接。

>>> pattern = r'/intro/\w+$'
>>> link_extractor = LinkExtractor(allow = pattern)
>>>
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='https://docs.scrapy.org/en/latest/intro/overview.html', text='Scrapy at a glance', fragment='', nofollow=False), Link(url='https://docs.scrapy.org/en/latest/intro/install.html', text='Installation guide', fragment='', nofollow=False), Link(url='https://docs.scrapy.org/en/latest/intro/tutorial.html', text='Scrapy Tutorial', fragment='', nofollow=False)]

deny：一个正则表达式或正则表达式的列表，与allow相反，匹配该正则表达式的链接不会被提取。

>>> pattern = r'/intro/\w+$'
>>> link_extractor = LinkExtractor(deny = pattern)
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='https://scrapy.org/download/', text='Download', fragment='', nofollow=False), Link(url='https://scrapy.org/doc/', text='Doc', fragment='', nofollow=False), Link(url='https://scrapy.org/resources/', text='Resources', fragment='', nofollow=False), Link(url='https://docs.scrapy.org/en/latest/intro/overview.html', text='Scrapy at a glance', fragment='', nofollow=False), Link(url='https://docs.scrapy.org/en/latest/intro/install.html', text='Installation guide', fragment='', nofollow=False), Link(url='https://docs.scrapy.org/en/latest/intro/tutorial.html', text='Scrapy Tutorial', fragment='', nofollow=False)]

allow_domains：域名或域名列表，该域名下的链接会被爬取；

>>> allow_domain = 'docs.scrapy.org'
>>> link_extractor = LinkExtractor(allow_domains = allow_domain)
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='https://docs.scrapy.org/en/latest/intro/overview.html', text='Scrapy at a glance', fragment='', nofollow=False), Link(url='https://docs.scrapy.org/en/latest/intro/install.html', text='Installation guide', fragment='', nofollow=False), Link(url='https://docs.scrapy.org/en/latest/intro/tutorial.html', text='Scrapy Tutorial', fragment='', nofollow=False)]

deny_domains：域名或域名列表，该域名下的链接不会被爬取；

>>> deny_domain = 'docs.scrapy.org'
>>> link_extractor = LinkExtractor(deny_domains = deny_domain)
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='https://scrapy.org/download/', text='Download', fragment='', nofollow=False), Link(url='https://scrapy.org/doc/', text='Doc', fragment='', nofollow=False), Link(url='https://scrapy.org/resources/', text='Resources', fragment='', nofollow=False)]

deny_extensions：字符串或字符串的列表，属于该后缀名的链接不会被爬取，若不提供的话，会使用默认选项；

>>> deny_extensions = 'html'
>>> link_extractor = LinkExtractor(deny_extensions = deny_extensions)
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='https://scrapy.org/download/', text='Download', fragment='', nofollow=False), Link(url='https://scrapy.org/doc/', text='Doc', fragment='', nofollow=False), Link(url='https://scrapy.org/resources/', text='Resources', fragment='', nofollow=False), Link(url='https://github.com/scrapy/scrapy/blob/1.5/docs/topics/media-pipeline.rst', text='Docs in github', fragment='', nofollow=False)]

restrict_xpahs：xpath或xpath的列表，符合该xpath的列表才会被爬取；

>>> xpath = '//div[@class="scrapyweb"]'
>>> link_extractor = LinkExtractor(restrict_xpaths=xpath)
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='https://scrapy.org/download/', text='Download', fragment='', nofollow=False), Link(url='https://scrapy.org/doc/', text='Doc', fragment='', nofollow=False), Link(url='https://scrapy.org/resources/', text='Resources', fragment='', nofollow=False)]

restrict_css：同上；

>>> css = 'div.scrapyweb'
>>> link_extractor = LinkExtractor(restrict_css=css)
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='https://scrapy.org/download/', text='Download', fragment='', nofollow=False), Link(url='https://scrapy.org/doc/', text='Doc', fragment='', nofollow=False), Link(url='https://scrapy.org/resources/', text='Resources', fragment='', nofollow=False)]

tags：tag或tag的list，提取指定标签中的链接，默认为[a,area]；
attrs：属性或属性的列表，提取指定属性内的链接，默认为'href'；

>>> body = b"""<img src="http://p0.so.qhmsg.com/bdr/326__/t010ebf2ec5ab7eed55.jpg"/>"""
>>> response = scrapy.http.HtmlResponse(url='', body = body)
>>> tag = 'img'
>>> attr='src'
>>> link_extractor = LinkExtractor(tags = tag, attrs=attr, deny_extensions='') #默认jpg是不会爬到的
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='http://p0.so.qhmsg.com/bdr/326__/t010ebf2ec5ab7eed55.jpg', text='', fragment='', nofollow=False)]

process_value：回调函数，该函数会对每一个链接进行处理，回调函数要么返回一个处理后的链接，要么返回None表示忽略该链接，默认函数为lambda x:x。

>>> from urllib.parse import urljoin
>>> def process(href):
...     return urljoin('http://example.com', href)
...
>>> body = b"""<a href="example.html"/>"""
>>> response = scrapy.http.HtmlResponse(url='', body = body)
>>> link_extractor = LinkExtractor(process_value = process)
>>> links = link_extractor.extract_links(response)
>>> links
[Link(url='http://example.com/example.html', text='', fragment='', nofollow=False)]

下面我们用LinkExtractor来提取第二篇博客如何编写一个Spider的下一页链接为例，看怎么在scrapy中应用LinkExtractor；

修改后的quotes如下：

# -*- coding: utf-8 -*-
import scrapy
from ..items import QuoteItem
from scrapy.linkextractors import LinkExtractor

class QuotesSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['quotes.toscrape.com']
    start_urls = ['http://quotes.toscrape.com/']
    link_extractor = LinkExtractor(allow=r'/page/\d+/',restrict_css='li.next')  #声明一个LinkExtractor对象
#   def start_requests(self):
#      url = "http://quotes.toscrape.com/"
#      yield scrapy.Request(url, callback = self.parse)

    def parse(self, response):
        quote_selector_list = response.css('body > div > div:nth-child(2) > div.col-md-8 div.quote')

        for quote_selector in quote_selector_list:
            quote = quote_selector.css('span.text::text').extract_first()
            author = quote_selector.css('span small.author::text').extract_first()
            tags = quote_selector.css('div.tags a.tag::text').extract()

            yield QuoteItem({'quote':quote, 'author':author, 'tags':tags})
        links = self.link_extractor.extract_links(response) #爬取链接
        
        if links:
            yield scrapy.Request(links[0].url, callback = self.parse)
LinkExtrator与CrawlSpider结合用的比较多，后面提到CrawlSpider的时候回讲到如何应用。

CrawlSpider

scrapy除了提供基础的spider类，还提供了一个更为强大的类CrawlSpider，CrawlSpider是基于Spider改造的，是为全站爬取而生的，非常适合爬取京东、知乎这张有规律的网站。CrawlSpider基于ExtractorLink制定了跟进url的规则，如果打算从网页中获得url之后继续爬取，非常适合使用 CrawlSpider。

先来看下scrapy.spiders.CrawlSpider，CrawlSpider有2个新的属性：

rules：Rule对象的列表，定义了爬取link的规则及处理方式；
parse_start_url(response)：用来爬取起始相应，默认返回空列表，子类可重写，可返回Item对象或Request对象，或者它们的可迭代对象。

在来看下Rule:

scrapy.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None)

link_extractor：LinkExtractor对象；
callback：爬取后连接的回调函数，该回调函数接收Response对象，并返回Item/Response（）或它们的子类（不要使用parse作为其回调，CrawlSpider使用parse方法实现了自己的逻辑）；
cb_kwargs：字典，用于作为**kwargs参数，传递给callback；
follow：是否跟进，若callback=None，则follow默认为True，否则默认为False；
process_links：可调用对象，针对每一个link_extractor提取的链接会调用该对象，通常作为链接的预处理用；
process_request：可调用对象，针对每一个链接构成的Request对象会调用，返回一个Request对象或None。

下面以爬取http://books.toscrape.com/的网站获取书名和对象的价格来看下怎么使用CrawlSpider；代码如下：

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class BookToscrapeSpider(CrawlSpider):
    name = 'book_toscrape'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com/']

    rules = (
        Rule(LinkExtractor(allow=r'catalogue/[\w\-\d]+/index.html'), callback='parse_item', follow=False),  #爬取详情页，不follow
        Rule(LinkExtractor(allow=r'page-\d+.html')), #爬取下一页，默认follow
    )

    def parse_item(self, response):
        title = response.css('div.product_main h1::text').extract_first()
        price = response.css('div.product_main p.price_color::text').extract_first()
        #简单返回
        yield {
            'title':title,
            'price':price,
        }

运行scrapy crawl book_toscrape -o sell.csv，可以获取到该网站的所有书名与对应价格。

总结

本篇简单介绍了用于爬取固定模板链接的LinkExtractor，然后讲了与之经常使用的爬虫类CrawlSpider，使用CrawlSpider可以用来爬取模式比较固定的网站。下篇我们来看看scrapy中间件的使用。

参考

参考自：链接提取LinkExtractor与全站爬取利器CrawlSpider

keepnight

链接提取LinkExtractor与全站爬取利器CrawlSpider

LinkExtractor

CrawlSpider

总结

参考

添加新评论

轻语

最新文章

分类

标签

归档