How to compress (bzip / gzip) a very large text quickly ?
Compressing files is very important when you are using platforms like Hadoop or FTP. Here is how I do it easily and quickly.
Step 1 :
Install GNU Parallels in your machine. How check here ?
Step 2 : csv to bz2
$ cat largefile.csv | /usr/local/bin/parallel — pipe -k bzip2 — best > largefile.bz2
Step 2 : csv to gz
$ cat largefile.csv | /usr/local/bin/parallel — pipe -k gzip > largefile.gz