Skip to content

Instantly share code, notes, and snippets.

@aborruso
Forked from marklit/roads.md
Created October 7, 2024 10:36
Show Gist options
  • Save aborruso/3b30cb1bdf3dc128d791ef4d0c239a9b to your computer and use it in GitHub Desktop.
Save aborruso/3b30cb1bdf3dc128d791ef4d0c239a9b to your computer and use it in GitHub Desktop.
MS Roads Detection in Parquet
$ wget https://usaminedroads.blob.core.windows.net/road-detections/MiddleEast-Full.zip
$ unzip MiddleEast-Full.zip
$ ~/duckdb roads.duckdb
CREATE OR REPLACE TABLE roads AS
    SELECT column0 as iso2,
           ST_GEOMFROMGeoJSON(column1::JSON->'$.geometry') geom
    FROM READ_CSV('MiddleEast-Full.tsv', header=false)
    ORDER BY HILBERT_ENCODE([ST_Y(ST_CENTROID(geom)),
                             ST_X(ST_CENTROID(geom))]::double[2]);

COPY(
    SELECT iso2,
           ST_AsWKB(geom) AS geom
    FROM   roads
) TO 'MiddleEast-Full.pq' (
    FORMAT 'PARQUET',
    CODEC  'ZSTD',
    ROW_GROUP_SIZE 15000);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment