Skip to content

Instantly share code, notes, and snippets.

@ian-kerins

ian-kerins/google_serp_scrapy_p3.py

Created April 12, 2021 13:56
Show Gist options
  • Save ian-kerins/9627a1bca674a8fca56b1b0ae4a370e7 to your computer and use it in GitHub Desktop.
Save ian-kerins/9627a1bca674a8fca56b1b0ae4a370e7 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
google_serp_scrapy_p3.py
def start_requests(self):
queries = ['scrapy’, ‘beautifulsoup’]
for query in queries:
url = create_google_url(query)
yield scrapy.Request(get_url(url), callback=self.parse, meta={'pos': 0})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment