Skip to content

Instantly share code, notes, and snippets.

@hilam

hilam/how-to-split-a-file-into-smaller-chunks.md

Forked from nepsilon/how-to-split-a-file-into-smaller-chunks.md
Created March 9, 2017 10:24
Show Gist options
  • Save hilam/5e9290781ac11af91a3a1d8d62ee00b7 to your computer and use it in GitHub Desktop.
Save hilam/5e9290781ac11af91a3a1d8d62ee00b7 to your computer and use it in GitHub Desktop.
Download ZIP
How to split a file into smaller chunks? — First published in fullweb.io issue #90
Raw
how-to-split-a-file-into-smaller-chunks.md

How to split a file into smaller chunks?

You may want to upload a 100GB file over an unstable network, or feed your scripts smaller inputs to load in RAM. In both cases, just splitting your file into smaller chunks is an easy solution.

Great news is Unix/Linux systems have the split utility already installed. And using it is simple as pie:

Cut a binary file into chunks of X bytes:

split -b X bigfile.avi

*Cut a text file into chunks of N lines:

split -l N bigfile.csv

split being a Unix tool, you can actually pipe to it like this:

./gen_lotta_data.py | split -l 1000 -

It will save chunks into files named xaa, xab, xac, etc. Use the -a option to specify a filename suffix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment