Inverted indexes [experimental]

Inverted indexes are an experimental type of secondary indexes which provide fast text search capabilities for String or FixedString columns. The main idea of an inverted index is to store a mapping from "terms" to the rows which contain these terms. "Terms" are tokenized cells of the string column. For example, the string cell "I will be a little late" is by default tokenized into six terms "I", "will", "be", "a", "little" and "late". Another kind of tokenizer is n-grams. For example, the result of 3-gram tokenization will be 21 terms "I w", " wi", "wil", "ill", "ll ", "l b", " be" etc. The more fine-granular the input strings are tokenized, the bigger but also the more useful the resulting inverted index will be.

:::warning Inverted indexes are experimental and should not be used in production environments yet. They may change in the future in backward-incompatible ways, for example with respect to their DDL/DQL syntax or performance/compression characteristics. :::

Usage

To use inverted indexes, first enable them in the configuration:

SET allow_experimental_inverted_index = true;

An inverted index can be defined on a string column using the following syntax

CREATE TABLE tab
(
    `key` UInt64,
    `str` String,
    INDEX inv_idx str TYPE inverted(0) GRANULARITY 1
)
ENGINE = MergeTree
ORDER BY key

where N specifies the tokenizer:

inverted(0) (or shorter: inverted()) set the tokenizer to "tokens", i.e. split strings along spaces,
inverted(N) with N between 2 and 8 sets the tokenizer to "ngrams(N)"

Being a type of skipping index, inverted indexes can be dropped or added to a column after table creation:

ALTER TABLE tbl DROP INDEX inv_idx;
ALTER TABLE tbl ADD INDEX inv_idx(s) TYPE inverted(2) GRANULARITY 1;

To use the index, no special functions or syntax are required. Typical string search predicates automatically leverage the index. As examples, consider:

SELECT * from tab WHERE s == 'Hello World;
SELECT * from tab WHERE s IN (‘Hello’, ‘World’);
SELECT * from tab WHERE s LIKE ‘%Hello%’;
SELECT * from tab WHERE multiSearchAny(s, ‘Hello’, ‘World’);
SELECT * from tab WHERE hasToken(s, ‘Hello’);
SELECT * from tab WHERE multiSearchAll(s, [‘Hello’, ‘World’]);

The inverted index also works on columns of type Array(String), Array(FixedString), Map(String) and Map(String).

Like for other secondary indices, each column part has its own inverted index. Furthermore, each inverted index is internally divided into "segments". The existence and size of the segments are generally transparent to users but the segment size determines the memory consumption during index construction (e.g. when two parts are merged). Configuration parameter "max_digestion_size_per_segment" (default: 256 MB) controls the amount of data read consumed from the underlying column before a new segment is created. Incrementing the parameter raises the intermediate memory consumption for index construction but also improves lookup performance since fewer segments need to be checked on average to evaluate a query.

Unlike other secondary indices, inverted indexes (for now) map to row numbers (row ids) instead of granule ids. The reason for this design is performance. In practice, users often search for multiple terms at once. For example, filter predicate WHERE s LIKE '%little%' OR s LIKE '%big%' can be evaluated directly using an inverted index by forming the union of the row id lists for terms "little" and "big". This also means that the parameter GRANULARITY supplied to index creation has no meaning (it may be removed from the syntax in the future).

4.0 KiB Raw Blame History Unescape Escape

Inverted indexes [experimental]

Usage

4.0 KiB

Raw Blame History