Skip to content

Instantly share code, notes, and snippets.

@ian-kerins

ian-kerins/google_serp_scrapy_p6.py

Created April 13, 2021 11:23
Show Gist options
  • Save ian-kerins/4ceae8257432d47930ee3e17d1c1406b to your computer and use it in GitHub Desktop.
Save ian-kerins/4ceae8257432d47930ee3e17d1c1406b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
google_serp_scrapy_p6.py
custom_settings = {'ROBOTSTXT_OBEY': False, 'LOG_LEVEL': 'INFO',
'CONCURRENT_REQUESTS_PER_DOMAIN': 10,
'RETRY_TIMES': 5}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment