Joseph Turian turian
turian
/ extractors.py
Created
August 5, 2012 02:11
— forked from osiloke/extractors.py
A scrapy link extractor that uses BeautifulSoup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Modified from: https://gist.github.com/1142142 | |
| to include the text of the link. | |
| """ | |
| import re | |
| from scrapy.link import Link | |
| from urlparse import urljoin | |
| from BeautifulSoup import BeautifulSoup |