Skip to content

Instantly share code, notes, and snippets.

@rtsio

rtsio/1_pdfliberation_hackathon_activity.md

Forked from gregelin/1_pdfliberation_hackathon_activity.md
Last active January 3, 2016 17:39
Show Gist options
  • Save rtsio/8496913 to your computer and use it in GitHub Desktop.
Save rtsio/8496913 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
1_pdfliberation_hackathon_activity.md
Raw
2_who.md

Who

Who is working together?

Rostislav Tsiomenko [OpenGov Foundation]

Marjorie Roswell

Travis Korte

Nick Lyell

Raw
3_challenge.md

Challenge

Which challenge are you working on?

  • Amnesty International Annual Reports – Torture Incident Database
  • Comprehensive Annual Financial Reports
  • Federal Communications Commission Daily Releases
  • House of Representatives Financial Disclosures (OpenSecrets.org)
  • IRS Form 990 – Not-for-Profit Organization Reports
  • New York City Council and Community Board Documents
  • New York City Economic Development Commission Monthly Snapshot
  • New York City Environmental Impact Statements
  • US Foreign Aid Reports (USAID)
  • Other: List/Describe here
Raw
4_pdfs.md

PDF Samples

How would you categorize the PDFs?

Sample documents

PDF URL Document Title
http://www.domain.org/docs/docurl.pdf Report of Economic Data 2012

Content category

  • Disclosure (filing, forms, report, ...)
  • Legislative doc (laws, analysis, ...)
  • Financial (statements, reports)
  • Government statistical data
  • Non-Government statistical data
  • Press (press releases, statements, ...)
  • Government reports
  • Non-Government reports
  • Directory
  • Other:

Number of pages

  • 1 page
  • 2 to 9 pages
  • 10+ pages
  • 100+ pages

Other observations

  • Collection includes PDFs made from scanned documents
  • PDFs include hand-written text

PDF Generation

  • Human authored
  • Machine generated
  • God only knows
Raw
5_data.md

Type of data embedded in PDF

  • Simple table of data
  • Complex table of data
  • Multiple tables of data from document
  • Table data greater than one page in length
  • Highly-structured form data
  • Loosely-structured form data
  • Has human-written text
  • Structure text of a report (e.g., headings, subheadings, ...)
  • Other:

Desired output of data

  • CSV
  • JSON
  • text version (e.g., markdown)
Raw
6_tools.md

Tool

What tool(s) are you using to extract the data?

Tool How we used it
ABBYY Cloud SDK Loaded up PDFs and converted to txt, rtf, or XML

Notes

ABBYY is commercial :(

Raw
7_how.md

How

How did you extract the desired data that produced the best results?

https://github.com/rtsio/financial_disclosure_scraping/tree/master/ABBYY-working-example/README.txt

ABBYY provides best results for tabular data. Tesseract and etc. unfortunately do not come close (note that this OCR, not text PDFs).

Improvements

What would have to be changed/added to the tool or process to achieve success?

Account for general structure, different Schedule tables (III, IV, V, etc.), many more improvements to go - only result of 5 hours of research into PDFs.

Raw
8_results.md

Results quality

  • 99%+
  • 90%+
  • 80%+
  • 50% to 75%
  • less than 50%
  • utter crap

Speed

How fast is the data extracted?

  • < 10 seconds
  • < 30 seconds
  • < 1 minute
  • < 5 minutes
  • < 10 minutes
  • < 20 minutes
  • Other:

Notes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment