Last active
June 15, 2020 14:23
-
-
Save stuarteberg/85074550fc0b39f653b7b789c169f337 to your computer and use it in GitHub Desktop.
Quick test of compressing random data with/without shuffling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In [20]: import numpy as np | |
...: import lz4.frame | |
...: | |
...: a = np.random.randint(2**16, size=1_000_000, dtype=int) | |
...: buf = lz4.frame.compress(a) | |
...: print("Compressed size without shuffling:", len(buf)) | |
...: | |
...: shuffled = a.view(np.uint8).reshape(-1, 8).transpose().copy(order='C') | |
...: buf2 = lz4.frame.compress(shuffled) | |
...: print("Compressed size with shuffling: ", len(buf2)) | |
...: | |
Compressed size without shuffling: 4077762 | |
Compressed size with shuffling: 2025044 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment