jueves, junio 06, 2013

Comparación de programas de compresión vs versión paralela.

Comparación de los programas de compresión, tiempos y tamaños, se usan versión mono y versión paralela.

gzip

pv TABLA_QYDTTK4_F130604.TXT | gzip > TABLA_QYDTTK4_F130604.txt.gz
1.04GiB 0:00:45 [23.5MiB/s] [====================================================>] 100%
pv TABLA_QYDTTK4_F130604.TXT | pigz > TABLA_QYDTTK4_F130604.txt.pig.gz
1.04GiB 0:00:04 [ 236MiB/s] [====================================================>] 100%

bzip2

pv TABLA_QYDTTK4_F130604.TXT | bzip2 > TABLA_QYDTTK4_F130604.txt.bz2
1.04GiB 0:04:55 [3.59MiB/s] [====================================================>] 100%
pv TABLA_QYDTTK4_F130604.TXT | lbzip2 > TABLA_QYDTTK4_F130604.txt.lbzip2.bz2
1.04GiB 0:00:07 [ 134MiB/s] [====================================================>] 100%
pv TABLA_QYDTTK4_F130604.TXT | pbzip2 > TABLA_QYDTTK4_F130604.txt.pbzip2.bz2
1.04GiB 0:00:26 [39.3MiB/s] [====================================================>] 100%

xz

pv TABLA_QYDTTK4_F130604.TXT | xz > TABLA_QYDTTK4_F130604.txt.xz
1.04GiB 0:11:02 [ 1.6MiB/s] [====================================================>] 100%
pv TABLA_QYDTTK4_F130604.TXT | pxz > TABLA_QYDTTK4_F130604.txt.pxz.xz
1.04GiB 0:01:06 [15.9MiB/s] [====================================================>] 100%
pv TABLA_QYDTTK4_F130604.TXT | pixz > TABLA_QYDTTK4_F130604.txt.pixz.xz
1.04GiB 0:00:58 [18.1MiB/s] [====================================================>] 100%

human readable sizes

-rw-r--r-- 1 jobs jobs 1.1G jun  5 06:28 TABLA_QYDTTK4_F130604.TXT
-rw-r--r-- 1 jobs jobs  86M jun  5 13:11 TABLA_QYDTTK4_F130604.rar
-rw-r--r-- 1 jobs jobs 101M jun  6 11:23 TABLA_QYDTTK4_F130604.txt.gz
-rw-r--r-- 1 jobs jobs 101M jun  6 11:31 TABLA_QYDTTK4_F130604.txt.pig.gz
-rw-r--r-- 1 jobs jobs  80M jun  6 11:29 TABLA_QYDTTK4_F130604.txt.bz2
-rw-r--r-- 1 jobs jobs  80M jun  6 12:16 TABLA_QYDTTK4_F130604.txt.lbzip2.bz2
-rw-r--r-- 1 jobs jobs  80M jun  6 12:21 TABLA_QYDTTK4_F130604.txt.pbzip2.bz2
-rw-r--r-- 1 jobs jobs  63M jun  6 12:02 TABLA_QYDTTK4_F130604.txt.pixz.xz
-rw-r--r-- 1 jobs jobs  63M jun  6 11:55 TABLA_QYDTTK4_F130604.txt.pxz.xz
-rw-r--r-- 1 jobs jobs  63M jun  6 11:49 TABLA_QYDTTK4_F130604.txt.xz

sizes, more detail

-rw-r--r-- 1 jobs jobs 1112120271 jun  5 06:28 TABLA_QYDTTK4_F130604.TXT
-rw-r--r-- 1 jobs jobs   90148917 jun  5 13:11 TABLA_QYDTTK4_F130604.rar

-rw-r--r-- 1 jobs jobs  105128200 jun  6 11:23 TABLA_QYDTTK4_F130604.txt.gz
-rw-r--r-- 1 jobs jobs  105153673 jun  6 11:31 TABLA_QYDTTK4_F130604.txt.pig.gz

-rw-r--r-- 1 jobs jobs   82889350 jun  6 11:29 TABLA_QYDTTK4_F130604.txt.bz2
-rw-r--r-- 1 jobs jobs   83134496 jun  6 12:16 TABLA_QYDTTK4_F130604.txt.lbzip2.bz2
-rw-r--r-- 1 jobs jobs   82913562 jun  6 12:21 TABLA_QYDTTK4_F130604.txt.pbzip2.bz2

-rw-r--r-- 1 jobs jobs   65921560 jun  6 12:02 TABLA_QYDTTK4_F130604.txt.pixz.xz
-rw-r--r-- 1 jobs jobs   65884988 jun  6 11:55 TABLA_QYDTTK4_F130604.txt.pxz.xz
-rw-r--r-- 1 jobs jobs   65667608 jun  6 11:49 TABLA_QYDTTK4_F130604.txt.xz

Notas:
Se uso un archivo de texto de 1.04 Gb, en una workstation con 64Gb de RAM usando /dev/shm, sistema operativo Gentoo, y el software se compilo del repositorio.

No hay comentarios.: