LogoTeluq
English
Logo
Répertoire de publications
de recherche en accès libre

Stream VByte: Faster byte-oriented integer compression [r-libre/1197]

Lemire, Daniel; Kurz, Nathan et Rupp, Christoph (2018). Stream VByte: Faster byte-oriented integer compression. Information Processing Letters, 130. https://doi.org/10.1016/j.ipl.2017.09.011

Fichier(s) associé(s) à ce document :
[img]  PDF - streamvbyte.pdf
Contenu du fichier : Manuscrit soumis (avant évaluation)
Licence : Creative Commons CC BY.
 
Catégorie de document : Articles de revues
Évaluation par un comité de lecture : Oui
Étape de publication : Publié
Résumé : Arrays of integers are often compressed in search engines. Though there are many ways to compress integers, we are interested in the popular byte-oriented integer compression techniques (e.g., VByte or Google's Varint-GB). They are appealing due to their simplicity and engineering convenience. Amazon's varint-G8IU is one of the fastest byte-oriented compression technique published so far. It makes judicious use of the powerful single-instruction-multiple-data (SIMD) instructions available in commodity processors. To surpass varint-G8IU, we present Stream VByte, a novel byte-oriented compression technique that separates the control stream from the encoded data. Like varint-G8IU, Stream VByte is well suited for SIMD instructions. We show that Stream VByte decoding can be up to twice as fast as varint-G8IU decoding over real data sets. In this sense, Stream VByte establishes new speed records for byte-oriented integer compression, at times exceeding the speed of the memcpy function. On a 3.4GHz Haswell processor, it decodes more than 4 billion differentially-coded integers per second from RAM to L1 cache.
Déposant: Lemire, Daniel
Responsable : Daniel Lemire
Dépôt : 27 sept. 2017 12:52
Dernière modification : 10 oct. 2017 00:39

Actions (connexion requise)

RÉVISER RÉVISER