Skip to content

Instantly share code, notes, and snippets.

@rclark

rclark/backfill.md

Last active February 1, 2016 23:38
Download ZIP

Revisions

  1. Ryan Clark revised this gist Feb 1, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion backfill.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,7 +4,7 @@
    |--------- really no time at all -----------|------------ 7 to 14 days -------------|-------------- really no time at all ----------------|----------------- shouldn't be too long -----------------|
    | | | | |
    | | | | |
    Start stack in backfill mode: Start backfilling dynamodb Backfill is done Feed changed feature ids down Dynamosm pipeline All done!
    Start stack in backfill mode: Start backfilling dynamodb: Backfill is done: Feed changed feature ids down Dynamosm pipeline: All done!
    - changes begin accumulating in dynamodb - chop up a pbf, ingest it - switch out of backfill mode - Dynamosm pipeline will only ever make GeoJSON with - keep on chugging through ongoing changes
    - no changes in geojson cache - convert the pbf to geojson and - new changes start landing new geojson on S3 the current version of data in dynamodb, so this - reduce table capacities to normal levels
    - increase table capacity write it all to S3 - gather change XML since pbf creation into a file won't run the risk of overwriting newer changes
  2. Ryan Clark revised this gist Feb 1, 2016. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions backfill.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,8 +6,8 @@
    | | | | |
    Start stack in backfill mode: Start backfilling dynamodb Backfill is done Feed changed feature ids down Dynamosm pipeline All done!
    - changes begin accumulating in dynamodb - chop up a pbf, ingest it - switch out of backfill mode - Dynamosm pipeline will only ever make GeoJSON with - keep on chugging through ongoing changes
    - no changes in geojson cache - convert the pbf to geojson and - new changes start landing the current version of data in dynamodb, so this - reduce table capacities to normal levels
    - increase table capacity write it all to S3 new geojson on S3 won't run the risk of overwriting newer changes
    - gather change XML since pbf creation into a file - There will not be an inordinate number of things
    - parse and dedupe to get ids of features that changed - Table throughputs can still be high from backfill
    - no changes in geojson cache - convert the pbf to geojson and - new changes start landing new geojson on S3 the current version of data in dynamodb, so this - reduce table capacities to normal levels
    - increase table capacity write it all to S3 - gather change XML since pbf creation into a file won't run the risk of overwriting newer changes
    - parse and dedupe to get ids of features that changed - There will not be an inordinate number of things
    - Table throughputs can still be high from backfill
    ```
  3. Ryan Clark revised this gist Feb 1, 2016. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion backfill.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    ```
    | | | | |
    | | | | |
    |--------- really no time at all -----------|------------ 7 to 14 days -------------|--------- really no time at all ---------------------|----------------- shouldn't be too long -----------------|
    |--------- really no time at all -----------|------------ 7 to 14 days -------------|-------------- really no time at all ----------------|----------------- shouldn't be too long -----------------|
    | | | | |
    | | | | |
    Start stack in backfill mode: Start backfilling dynamodb Backfill is done Feed changed feature ids down Dynamosm pipeline All done!
  4. Ryan Clark created this gist Feb 1, 2016.
    13 changes: 13 additions & 0 deletions backfill.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,13 @@
    ```
    | | | | |
    | | | | |
    |--------- really no time at all -----------|------------ 7 to 14 days -------------|--------- really no time at all ---------------------|----------------- shouldn't be too long -----------------|
    | | | | |
    | | | | |
    Start stack in backfill mode: Start backfilling dynamodb Backfill is done Feed changed feature ids down Dynamosm pipeline All done!
    - changes begin accumulating in dynamodb - chop up a pbf, ingest it - switch out of backfill mode - Dynamosm pipeline will only ever make GeoJSON with - keep on chugging through ongoing changes
    - no changes in geojson cache - convert the pbf to geojson and - new changes start landing the current version of data in dynamodb, so this - reduce table capacities to normal levels
    - increase table capacity write it all to S3 new geojson on S3 won't run the risk of overwriting newer changes
    - gather change XML since pbf creation into a file - There will not be an inordinate number of things
    - parse and dedupe to get ids of features that changed - Table throughputs can still be high from backfill
    ```