Big indexes

By swapping the large indexes to disk, this feature allows supporting them with limited usage of memory.

Technically, the user will define a size beyond which an index is put on disk in order to free up RAM memory space and to improve Indexima’s cluster performance.

Configuration

This feature needs to be activated with the configuration parameter big.index.enable. All configuration parameters to use this feature are described in detail in galactica.conf.

Description

With big index activated (ie big.index.enable=true), the engine will identify big indexes when they reach the big.index.limit.mb size.

In order to reduce the memory footprint, those indexes won't be fully loaded into memory, as they are always ready to be swapped in memory.

When an index is used for a query, the required index partitions are loaded into memory. Only big.index.max.active.partitions partitions of the index will be loaded into memory at the same time on one node.