I developed a Rust library for compressing vectors of u64 integers using instantaneous codes the from dsi-bitstream library. Offers fast random access via sampling to balance speed and memory.
Features
- Efficient Compression: Leverage various codecs like Gamma, Delta, and Rice coding.
- Fast Random Access: Achieve $O(1)$ access with configurable sampling.
- Memory Analysis: Integrate with
mem-dbgfor memory profiling. - Flexible Codecs: Select codecs based on data distribution for optimal compression.
The sampling parameter determines how often full positions are stored to speed up access. Higher values reduce memory overhead but may increase access time. For example, sampling_param = 32 is usually a good trade-off for large datasets.
More can be found at:
- Crate: compressed-intvec
- Documentation: docs.rs
- Repository: GitHub