mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-12-02 04:22:03 +00:00
8623cb232c
* CLICKHOUSE-4063: less manual html @ index.md * CLICKHOUSE-4063: recommend markdown="1" in README.md * CLICKHOUSE-4003: manually purge custom.css for now * CLICKHOUSE-4064: expand <details> before any print (including to pdf) * CLICKHOUSE-3927: rearrange interfaces/formats.md a bit * CLICKHOUSE-3306: add few http headers * Remove copy-paste introduced in #3392 * Hopefully better chinese fonts #3392 * get rid of tabs @ custom.css * Apply comments and patch from #3384 * Add jdbc.md to ToC and some translation, though it still looks badly incomplete * minor punctuation * Add some backlinks to official website from mirrors that just blindly take markdown sources * Do not make fonts extra light * find . -name '*.md' -type f | xargs -I{} perl -pi -e 's//g' {} * find . -name '*.md' -type f | xargs -I{} perl -pi -e 's/ sql/g' {} * Remove outdated stuff from roadmap.md * Not so light font on front page too * Refactor Chinese formats.md to match recent changes in other languages
41 lines
4.0 KiB
Markdown
41 lines
4.0 KiB
Markdown
<a name="table_engine-collapsingmergetree"></a>
|
|
|
|
# CollapsingMergeTree
|
|
|
|
*This engine is used specifically for Yandex.Metrica.*
|
|
|
|
It differs from `MergeTree` in that it allows automatic deletion, or "collapsing" certain pairs of rows when merging.
|
|
|
|
Yandex.Metrica has normal logs (such as hit logs) and change logs. Change logs are used for incrementally calculating statistics on data that is constantly changing. Examples are the log of session changes, or logs of changes to user histories. Sessions are constantly changing in Yandex.Metrica. For example, the number of hits per session increases. We refer to changes in any object as a pair (?old values, ?new values). Old values may be missing if the object was created. New values may be missing if the object was deleted. If the object was changed, but existed previously and was not deleted, both values are present. In the change log, one or two entries are made for each change. Each entry contains all the attributes that the object has, plus a special attribute for differentiating between the old and new values. When objects change, only the new entries are added to the change log, and the existing ones are not touched.
|
|
|
|
The change log makes it possible to incrementally calculate almost any statistics. To do this, we need to consider "new" rows with a plus sign, and "old" rows with a minus sign. In other words, incremental calculation is possible for all statistics whose algebraic structure contains an operation for taking the inverse of an element. This is true of most statistics. We can also calculate "idempotent" statistics, such as the number of unique visitors, since the unique visitors are not deleted when making changes to sessions.
|
|
|
|
This is the main concept that allows Yandex.Metrica to work in real time.
|
|
|
|
CollapsingMergeTree accepts an additional parameter - the name of an Int8-type column that contains the row's "sign". Example:
|
|
|
|
``` sql
|
|
CollapsingMergeTree(EventDate, (CounterID, EventDate, intHash32(UniqID), VisitID), 8192, Sign)
|
|
```
|
|
|
|
Here, `Sign` is a column containing -1 for "old" values and 1 for "new" values.
|
|
|
|
When merging, each group of consecutive identical primary key values (columns for sorting data) is reduced to no more than one row with the column value 'sign_column = -1' (the "negative row") and no more than one row with the column value 'sign_column = 1' (the "positive row"). In other words, entries from the change log are collapsed.
|
|
|
|
If the number of positive and negative rows matches, the first negative row and the last positive row are written.
|
|
If there is one more positive row than negative rows, only the last positive row is written.
|
|
If there is one more negative row than positive rows, only the first negative row is written.
|
|
Otherwise, there will be a logical error and none of the rows will be written. (A logical error can occur if the same section of the log was accidentally inserted more than once. The error is just recorded in the server log, and the merge continues.)
|
|
|
|
Thus, collapsing should not change the results of calculating statistics.
|
|
Changes are gradually collapsed so that in the end only the last value of almost every object is left.
|
|
Compared to MergeTree, the CollapsingMergeTree engine allows a multifold reduction of data volume.
|
|
|
|
There are several ways to get completely "collapsed" data from a `CollapsingMergeTree` table:
|
|
|
|
1. Write a query with GROUP BY and aggregate functions that accounts for the sign. For example, to calculate quantity, write 'sum(Sign)' instead of 'count()'. To calculate the sum of something, write 'sum(Sign * x)' instead of 'sum(x)', and so on, and also add 'HAVING sum(Sign) `>` 0'. Not all amounts can be calculated this way. For example, the aggregate functions 'min' and 'max' can't be rewritten.
|
|
2. If you must extract data without aggregation (for example, to check whether rows are present whose newest values match certain conditions), you can use the FINAL modifier for the FROM clause. This approach is significantly less efficient.
|
|
|
|
|
|
[Original article](https://clickhouse.yandex/docs/en/operations/table_engines/collapsingmergetree/) <!--hide-->
|