ClickHouse/docs/en/engines/table-engines/special/buffer.md

70 lines
5.5 KiB
Markdown
Raw Normal View History

2020-04-03 13:23:32 +00:00
---
toc_priority: 45
toc_title: Buffer
---
2020-06-10 20:07:17 +00:00
# Buffer Table Engine {#buffer}
2017-04-03 19:49:50 +00:00
2017-04-26 17:26:17 +00:00
Buffers the data to write in RAM, periodically flushing it to another table. During the read operation, data is read from the buffer and the other table simultaneously.
2020-03-20 10:10:48 +00:00
``` sql
Buffer(database, table, num_layers, min_time, max_time, min_rows, max_rows, min_bytes, max_bytes)
```
CLICKHOUSE-2720: progress on website (#865) * update presentations * CLICKHOUSE-2936: redirect from clickhouse.yandex.ru and clickhouse.yandex.com * update submodule * lost files * CLICKHOUSE-2981: prefer sphinx docs over original reference * CLICKHOUSE-2981: docs styles more similar to main website + add flags to switch language links * update presentations * Less confusing directory structure (docs -> doc/reference/) * Minify sphinx docs too * Website release script: fail fast + pass docker hash on deploy * Do not underline links in docs * shorter * cleanup docker images * tune nginx config * CLICKHOUSE-3043: get rid of habrastorage links * Lost translation * CLICKHOUSE-2936: temporary client-side redirect * behaves weird in test * put redirect back * CLICKHOUSE-3047: copy docs txts to public too * move to proper file * remove old pages to avoid confusion * Remove reference redirect warning for now * Refresh README.md * Yellow buttons in docs * Use svg flags instead of unicode ones in docs * fix test website instance * Put flags to separate files * wrong flag * Copy Yandex.Metrica introduction from main page to docs * Yet another home page structure change, couple new blocks (CLICKHOUSE-3045) * Update Contacts section * CLICKHOUSE-2849: more detailed legal information * CLICKHOUSE-2978 preparation - split by files * More changes in Contacts block * Tune texts on index page * update presentations * One more benchmark * Add usage sections to index page, adapted from slides * Get the roadmap started, based on slides from last ClickHouse Meetup * CLICKHOUSE-2977: some rendering tuning * Get rid of excessive section in the end of getting started * Make headers linkable * CLICKHOUSE-2981: links to editing reference - https://github.com/yandex/ClickHouse/issues/849 * CLICKHOUSE-2981: fix mobile styles in docs * Ban crawling of duplicating docs * Open some external links in new tab * Ban old docs too * Lots of trivial fixes in english docs * Lots of trivial fixes in russian docs * Remove getting started copies in markdown * Add Yandex.Webmaster * Fix some sphinx warnings * More warnings fixed in english docs * More sphinx warnings fixed * Add code-block:: text * More code-block:: text * These headers look not that well * Better switch between documentation languages * merge use_case.rst into ya_metrika_task.rst * Edit the agg_functions.rst texts * Add lost empty lines
2017-06-13 04:15:47 +00:00
Engine parameters:
- `database` Database name. Instead of the database name, you can use a constant expression that returns a string.
- `table` Table to flush data to.
- `num_layers` Parallelism layer. Physically, the table will be represented as `num_layers` of independent buffers. Recommended value: 16.
- `min_time`, `max_time`, `min_rows`, `max_rows`, `min_bytes`, and `max_bytes` Conditions for flushing data from the buffer.
Data is flushed from the buffer and written to the destination table if all the `min*` conditions or at least one `max*` condition are met.
2017-04-03 19:49:50 +00:00
- `min_time`, `max_time` Condition for the time in seconds from the moment of the first write to the buffer.
- `min_rows`, `max_rows` Condition for the number of rows in the buffer.
- `min_bytes`, `max_bytes` Condition for the number of bytes in the buffer.
2017-04-03 19:49:50 +00:00
During the write operation, data is inserted to a `num_layers` number of random buffers. Or, if the data part to insert is large enough (greater than `max_rows` or `max_bytes`), it is written directly to the destination table, omitting the buffer.
2017-04-03 19:49:50 +00:00
The conditions for flushing the data are calculated separately for each of the `num_layers` buffers. For example, if `num_layers = 16` and `max_bytes = 100000000`, the maximum RAM consumption is 1.6 GB.
2017-04-03 19:49:50 +00:00
2017-04-26 17:26:17 +00:00
Example:
2020-03-20 10:10:48 +00:00
``` sql
CREATE TABLE merge.hits_buffer AS merge.hits ENGINE = Buffer(merge, hits, 16, 10, 100, 10000, 1000000, 10000000, 100000000)
```
2017-04-03 19:49:50 +00:00
2020-06-10 20:07:17 +00:00
Creating a `merge.hits_buffer` table with the same structure as `merge.hits` and using the Buffer engine. When writing to this table, data is buffered in RAM and later written to the merge.hits table. 16 buffers are created. The data in each of them is flushed if either 100 seconds have passed, or one million rows have been written, or 100 MB of data have been written; or if simultaneously 10 seconds have passed and 10,000 rows and 10 MB of data have been written. For example, if just one row has been written, after 100 seconds it will be flushed, no matter what. But if many rows have been written, the data will be flushed sooner.
2017-04-03 19:49:50 +00:00
2020-06-10 20:07:17 +00:00
When the server is stopped, with `DROP TABLE` or `DETACH TABLE`, buffer data is also flushed to the destination table.
2017-04-03 19:49:50 +00:00
2017-04-26 17:26:17 +00:00
You can set empty strings in single quotation marks for the database and table name. This indicates the absence of a destination table. In this case, when the data flush conditions are reached, the buffer is simply cleared. This may be useful for keeping a window of data in memory.
2017-04-03 19:49:50 +00:00
2017-04-26 17:26:17 +00:00
When reading from a Buffer table, data is processed both from the buffer and from the destination table (if there is one).
Note that the Buffer tables does not support an index. In other words, data in the buffer is fully scanned, which might be slow for large buffers. (For data in a subordinate table, the index that it supports will be used.)
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
If the set of columns in the Buffer table doesnt match the set of columns in a subordinate table, a subset of columns that exist in both tables is inserted.
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
If the types dont match for one of the columns in the Buffer table and a subordinate table, an error message is entered in the server log and the buffer is cleared.
The same thing happens if the subordinate table doesnt exist when the buffer is flushed.
2017-04-03 19:49:50 +00:00
2017-04-26 17:26:17 +00:00
If you need to run ALTER for a subordinate table and the Buffer table, we recommend first deleting the Buffer table, running ALTER for the subordinate table, then creating the Buffer table again.
2017-04-03 19:49:50 +00:00
2017-04-26 17:26:17 +00:00
If the server is restarted abnormally, the data in the buffer is lost.
2017-04-03 19:49:50 +00:00
2020-06-10 20:07:17 +00:00
`FINAL` and `SAMPLE` do not work correctly for Buffer tables. These conditions are passed to the destination table, but are not used for processing data in the buffer. If these features are required we recommend only using the Buffer table for writing, while reading from the destination table.
2017-04-03 19:49:50 +00:00
2017-04-26 17:26:17 +00:00
When adding data to a Buffer, one of the buffers is locked. This causes delays if a read operation is simultaneously being performed from the table.
2017-04-03 19:49:50 +00:00
2020-06-10 20:07:17 +00:00
Data that is inserted to a Buffer table may end up in the subordinate table in a different order and in different blocks. Because of this, a Buffer table is difficult to use for writing to a CollapsingMergeTree correctly. To avoid problems, you can set `num_layers` to 1.
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
If the destination table is replicated, some expected characteristics of replicated tables are lost when writing to a Buffer table. The random changes to the order of rows and sizes of data parts cause data deduplication to quit working, which means it is not possible to have a reliable exactly once write to replicated tables.
2017-04-03 19:49:50 +00:00
2017-04-26 17:26:17 +00:00
Due to these disadvantages, we can only recommend using a Buffer table in rare cases.
2017-04-03 19:49:50 +00:00
2020-03-20 10:10:48 +00:00
A Buffer table is used when too many INSERTs are received from a large number of servers over a unit of time and data cant be buffered before insertion, which means the INSERTs cant run fast enough.
2020-03-20 10:10:48 +00:00
Note that it doesnt make sense to insert data one row at a time, even for Buffer tables. This will only produce a speed of a few thousand rows per second, while inserting larger blocks of data can produce over a million rows per second (see the section “Performance”).
2020-01-30 10:34:55 +00:00
[Original article](https://clickhouse.tech/docs/en/operations/table_engines/buffer/) <!--hide-->