默认的去重逻辑

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2017/5/7 22:43
# @Author  : Aries
# @File    : scrapy_filter.py
# @Software: PyCharm

import scrapy
from scrapy.http.request import Request


class FilterSpider(scrapy.Spider):
    name = 'filter'
    headers = {
        'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36", }

    def start_requests(self):
        yield Request(url='https://www.baidu.com/s?wd=22', headers=self.headers)

    def parse_print(self, response):
        self.logger.info(response.url)

    def parse(self, response):
        self.logger.info("--------------------------")
        yield Request(url='https://www.baidu.com/s?wd=1', callback=self.parse_print, headers=self.headers)
        yield Request(url='https://www.baidu.com/s?wd=3', callback=self.parse_print, headers=self.headers)
        yield Request(url='https://www.baidu.com/s?wd=3', callback=self.parse_print, headers=self.headers)

        
# 运行结果如下
2017-05-07 23:33:36 [scrapy.core.engine] INFO: Spider opened
2017-05-07 23:33:36 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-05-07 23:33:36 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-05-07 23:33:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.baidu.com/s?wd=22> (referer: None)
2017-05-07 23:33:37 [filter] INFO: --------------------------
2017-05-07 23:33:37 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.baidu.com/s?wd=3> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2017-05-07 23:33:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.baidu.com/s?wd=3> (referer: https://www.baidu.com/s?wd=22)
2017-05-07 23:33:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.baidu.com/s?wd=1> (referer: https://www.baidu.com/s?wd=22)
2017-05-07 23:33:37 [filter] INFO: https://www.baidu.com/s?wd=3
2017-05-07 23:33:37 [filter] INFO: https://www.baidu.com/s?wd=1
2017-05-07 23:33:37 [scrapy.core.engine] INFO: Closing spider (finished)
2017-05-07 23:33:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
# 从运行结果中看到：两个https://www.baidu.com/s?wd=3 的请求有一个被默认的去重逻辑处理掉了
2017-05-07 23:33:37 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.baidu.com/s?wd=3> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)

默认去重逻辑的瓶颈

默认去重： 'scrapy.dupefilters.RFPDupeFilter'
通过request_fingerprint将request中传递过来的url做类似指纹认证。
如果指纹已经存在，则丢弃返回true
如果指纹不存在，则add加入到fingerprints中（此处self.fingerprints = set()）

class RFPDupeFilter(BaseDupeFilter):
    def request_seen(self, request):
        fp = self.request_fingerprint(request)
        if fp in self.fingerprints:
            return True
        self.fingerprints.add(fp)
        if self.file:
            self.file.write(fp + os.linesep)

按照这个去重处理逻辑，默认情况下法实现如下情况：

• 例子一：不同日期发起的同一个url请求（时间不同页面内容会改变）
  https://www.baidu.com/s?wd=3 （2017.05.05抓取）
  https://www.baidu.com/s?wd=3 （2017.05.07抓取 - 默认情况下这个会被处理掉）

• 例子二：我只关注URL中部分参数（我只关注https://www.baidu.com/s?wd=3）
  https://www.baidu.com/s?wd=3&s=1 （s=1）
  https://www.baidu.com/s?wd=3&s=2 （虽然s=2但是前面wd于前一个请求一样，需要去掉）

解决方法：

• 以例子二为例：通过mate将wd的传值到request_fingerprint的函数中，将URL返回，让filter生成指纹
• 例子一的解决方法类似，将日期传递给request_fingerprint。重新返回对应的URL就可以了（return response + "--" + datetime）

第一步：重写RFPDupeFilter

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2017/5/7 23:17
# @Author  : Aries
# @File    : custom_filter.py
# @Software: PyCharm

from scrapy.dupefilters import RFPDupeFilter


class CustomURLFilter(RFPDupeFilter):
    def request_fingerprint(self, request):

        if "wd" in request.meta:
            return "https://www.baidu.com/s" + "--" + request.meta["wd"]
        else:
            return request.url

第二步：启用custom_settings并设置好对应的meta

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2017/5/7 22:43
# @Author  : Aries
# @File    : scrapy_filter.py
# @Software: PyCharm

import scrapy
from scrapy.http.request import Request


class FilterSpider(scrapy.Spider):
    name = 'filter'
    headers = {
        'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.96 Safari/537.36", }
    custom_settings = {
        'DUPEFILTER_DEBUG': True,
        'DUPEFILTER_CLASS': "lagou.custom_filter.CustomURLFilter"
    }

    def start_requests(self):
        yield Request(url='https://www.baidu.com/s?wd=22', headers=self.headers, meta={"wd": "22"})

    def parse_print(self, response):
        self.logger.info(response.url)

    def parse(self, response):
        self.logger.info("--------------------------")
        yield Request(url='https://www.baidu.com/s?wd=1', callback=self.parse_print, headers=self.headers, meta={"wd": "1"})
        yield Request(url='https://www.baidu.com/s?wd=3&s=1', callback=self.parse_print, headers=self.headers, meta={"wd": "3"})
        yield Request(url='https://www.baidu.com/s?wd=3&s=2', callback=self.parse_print, headers=self.headers, meta={"wd": "3"})

# 运行结果如下
2017-05-07 23:31:14 [scrapy.core.engine] INFO: Spider opened
2017-05-07 23:31:14 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-05-07 23:31:14 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-05-07 23:31:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.baidu.com/s?wd=22> (referer: None)
2017-05-07 23:31:14 [filter] INFO: --------------------------
2017-05-07 23:31:14 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.baidu.com/s?wd=3&s=2>
2017-05-07 23:31:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.baidu.com/s?wd=3&s=1> (referer: https://www.baidu.com/s?wd=22)
2017-05-07 23:31:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.baidu.com/s?wd=1> (referer: https://www.baidu.com/s?wd=22)
2017-05-07 23:31:14 [filter] INFO: https://www.baidu.com/s?wd=3&s=1
2017-05-07 23:31:15 [filter] INFO: https://www.baidu.com/s?wd=1
2017-05-07 23:31:15 [scrapy.core.engine] INFO: Closing spider (finished)
2017-05-07 23:31:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

# 从运行结果中看到：两个https://www.baidu.com/s?wd=3&s=2 的请求被处理掉了
2017-05-07 23:31:14 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.baidu.com/s?wd=3&s=2>