Created
April 6, 2017 15:28
-
-
Save dustinkirkland/d14df7f490440f85a36373f279e5cf7d to your computer and use it in GitHub Desktop.
Fetch a HackerNews article and all comments into a JSON document
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/python3 | |
# hackernews.py - fetch a hackernews article and comments into a json doc | |
# Copyright (C) 2017 Dustin Kirkland | |
# | |
# Author: Dustin Kirkland | |
# | |
# This program is free software: you can redistribute it and/or modify | |
# it under the terms of the GNU General Public License as published by | |
# the Free Software Foundation, version 3 of the License. | |
# | |
# This program is distributed in the hope that it will be useful, | |
# but WITHOUT ANY WARRANTY; without even the implied warranty of | |
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | |
# GNU General Public License for more details. | |
# | |
# You should have received a copy of the GNU General Public License | |
# along with this program. If not, see <http://www.gnu.org/licenses/>. | |
import json | |
import requests | |
import sys | |
#https://hacker-news.firebaseio.com/v0/item/14002821.json?print=pretty | |
# Example: | |
# ./hackernews.py $ARTICLE_NUMBER | |
def get_json_from_url(item): | |
url = "https://hacker-news.firebaseio.com/v0/item/%s.json" % item | |
data = json.loads(requests.get(url=url).text) | |
# Recursively fetch and insert all nested comments | |
if "kids" in data and len(data["kids"]) > 0: | |
for k in data["kids"]: | |
data[k] = json.loads(get_json_from_url(k)) | |
return json.dumps(data) | |
data = json.loads(get_json_from_url(sys.argv[1])) | |
print(json.dumps(data, indent=4, sort_keys=False)) |
Nope, sorry, never tried it async. Honestly, I only ran it once, ever, and never again. And that was years ago.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
Thanks for the code above, did you ever try to do it async?
i try to reuse your code and move it into async mode, but without a success