frozen-indices.asciidoc

Frozen indices

Best practices

Since frozen indices provide a much higher disk to heap ratio at the expense of search latency, it is advisable to allocate frozen indices to dedicated nodes to prevent searches on frozen indices influencing traffic on low latency nodes. There is significant overhead in loading data structures on demand which can cause page faults and garbage collections, which further slow down query execution.

Since indices that are eligible for freezing are unlikely to change in the future, disk space can be optimized as described in [tune-for-disk-usage].

It’s highly recommended to _forcemerge your indices prior to freezing to ensure that each shard has only a single segment on disk. This not only provides much better compression but also simplifies the data structures needed to service aggregation or sorted search requests.

POST /twitter/_forcemerge?max_num_segments=1

Searching a frozen index

Frozen indices are throttled in order to limit memory consumptions per node. The number of concurrently loaded frozen indices per node is limited by the number of threads in the search_throttled threadpool, which is 1 by default. Search requests will not be executed against frozen indices by default, even if a frozen index is named explicitly. This is to prevent accidental slowdowns by targeting a frozen index by mistake. To include frozen indices a search request must be executed with the query parameter ignore_throttled=false.

GET /twitter/_search?q=user:kimchy&ignore_throttled=false

Important

While frozen indices are slow to search, they can be pre-filtered efficiently. The request parameter pre_filter_shard_size specifies a threshold that, when exceeded, will enforce a round-trip to pre-filter search shards that cannot possibly match. This filter phase can limit the number of shards significantly. For instance, if a date range filter is applied, then all indices (frozen or unfrozen) that do not contain documents within the date range can be skipped efficiently. The default value for pre_filter_shard_size is 128 but it’s recommended to set it to 1 when searching frozen indices. There is no significant overhead associated with this pre-filter phase.

Monitoring frozen indices

Frozen indices are ordinary indices that use search throttling and a memory efficient shard implementation. For API’s like the [cat-indices] frozen indices may identified by an index’s search.throttled property (sth).

GET /_cat/indices/twitter?v&h=i,sth

The response looks like:

i         sth
twitter   true