Commit Graph

139 Commits

Author SHA1 Message Date
Azat Khuzhin
19e0439629 Add ability to flush buffer only in background for StorageBuffer
Add 3 new engine arguments:
- flush_time
- flush_rows
- flush_bytes

That will be checked only for background flush, this maybe useful if
INSERT latency is "crucial".
2021-04-15 21:22:13 +03:00
alesapin
e27715e55e
Merge pull request #21331 from godliness/master
Fix error configuration for cluster secret
2021-03-10 10:09:29 +03:00
Azat Khuzhin
6965ac26c3 Distributed: Add ability to delay/throttle INSERT until pending data will be reduced
Add two new settings for the Distributed engine:
- bytes_to_delay_insert
- max_delay_to_insert

If at the beginning of INSERT there will be too much pending data, more
then bytes_to_delay_insert, then the INSERT will wait until it will be
shrinked, and not more then max_delay_to_insert seconds.

If after this there will be still too much pending, it will throw an
exception.

Also new profile events were added (by analogy to the MergeTree):
- DistributedDelayedInserts (although you can use system.errors instead
  of this, but still)
- DistributedRejectedInserts
- DistributedDelayedInsertsMilliseconds
2021-03-03 23:30:23 +03:00
Azat Khuzhin
b5a5778589 Distributed: Add ability to limit amount of pending bytes for async INSERT
Right now with distributed_directory_monitor_batch_inserts=1 and
insert_distributed_sync=0 INSERT into Distributed table will store
blocks that should be sent to remote (and in case of
prefer_localhost_replica=0 to the localhost too) on the local
filesystem, and sent it in background.

However there is no limit for this storage, and if the remote is
unavailable (or some other error), these pending blocks may take
significant space, and this is not always desired behaviour.

Add new Distributed setting - bytes_to_throw_insert, that will set the
limit for how much pending bytes is allowed, if the limit will be
reached an exception will be throw.

By default was set to 0, to avoid surprises.
2021-03-03 23:30:00 +03:00
Chao Ma
c2b8612525 Fix error configuration for cluster secret 2021-03-01 16:30:42 +08:00
lehasm
bb2ee9609f Compression mentioned in File engine 2021-02-23 16:43:40 +03:00
Azat Khuzhin
471deab63a Rename fsync_tmp_directory to fsync_directories for Distributed engine 2021-01-09 17:51:30 +03:00
Mikhail Filimonov
4cabfa356e Update documentation for Distributed fsync settings. 2021-01-09 11:31:32 +03:00
Azat Khuzhin
b5ace27014 Add fsync support for Distributed engine.
Two new settings (by analogy with MergeTree family) has been added:

- `fsync_after_insert` - Do fsync for every inserted. Will decreases
  performance of inserts.

- `fsync_tmp_directory` - Do fsync for temporary directory (that is used
  for async INSERT only) after all part operations (writes, renames,
  etc.).

Refs: #17380 (p1)
2021-01-09 11:31:32 +03:00
gyuton
0d1fcd2f96
DOCSUP-3227: Documented the persistent setting (#16579)
* added description of disable_set_and_join_persistency setting.

* Changed setting name to persistent, added references about persistent to join table engine and set table engine

* Resolved conflict.

* Added reviewers suggestions

* Added translation for Russian.

* Fixed typos

* Update docs/en/operations/settings/settings.md

Fixed minor stuff.

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update settings.md

Updated translation.

* Fixed description and translation

* Fixed letter cases

Co-authored-by: George <gyuton@yandex-team.ru>
Co-authored-by: BayoNet <da-daos@yandex.ru>
2020-11-06 14:23:04 +03:00
Alexey Milovidov
db4db42b65 Fix broken links in docs 2020-10-13 20:23:29 +03:00
Azat Khuzhin
0159c74f21 Secure inter-cluster query execution (with initial_user as current query user) [v3]
Add inter-server cluster secret, it is used for Distributed queries
inside cluster, you can configure in the configuration file:

  <remote_servers>
      <logs>
          <shard>
              <secret>foobar</secret> <!-- empty -- works as before -->
              ...
          </shard>
      </logs>
  </remote_servers>

And this will allow clickhouse to make sure that the query was not
faked, and was issued from the node that knows the secret. And since
trust appeared it can use initial_user for query execution, this will
apply correct *_for_user (since with inter-server secret enabled, the
query will be executed from the same user on the shards as on initator,
unlike "default" user w/o it).

v2: Change user to the initial_user for Distributed queries if secret match
v3: Add Protocol::Cluster package
v4: Drop Protocol::Cluster and use plain Protocol::Hello + user marker
v5: Do not use user from Hello for cluster-secure (superfluous)
2020-09-15 01:36:28 +03:00
Ivan Blinkov
ef6a7b2578
fix #13819 2020-08-17 12:05:34 +03:00
BayoNet
52f95e53fb
DOCS-229: insert_distributed_sync (#12579)
* Revolg docsup 727 insert distributed sync setting (#130)

* Add insert_distributed_sync setting

* insert_distributed_sync en doc upd

* Translated to russian.

* Update docs/ru/operations/settings/settings.md

Co-authored-by: BayoNet <da-daos@yandex.ru>

* Added the link to the setting in the russian version.

Co-authored-by: Olga Revyakina <revolg@yandex-team.ru>
Co-authored-by: BayoNet <da-daos@yandex.ru>

* Update settings.md

* Update docs/ru/operations/settings/settings.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
Co-authored-by: Olga Revyakina <revolg@yandex-team.ru>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
2020-07-21 15:40:03 +03:00
Artem Zuikov
ff372ec0f0
Standard join type syntax position (#12520) 2020-07-16 11:56:45 +03:00
Ivan Blinkov
ceac649c01
[docs] split misc statements (#12403) 2020-07-11 14:05:49 +03:00
Ivan Blinkov
258d2fd499
[docs] split various kinds of CREATE queries into separate articles (#12328)
* normalize

* split & adjust links

* re-normalize

* adjust ru links

* adjust ja/tr links

* partially apply e0d19d2aea

* reset contribs
2020-07-09 18:10:35 +03:00
Ivan Blinkov
8d65deaf85
[docs] introduction for special table engines (#12170)
* [docs] introduction for integration table engines

* Update jdbc.md

* Update odbc.md

* Update mysql.md

* Update kafka.md

* Update hdfs.md

* [docs] introduction for special table engines

* Update index.md

* Update index.md
2020-07-07 01:48:11 +03:00
Azat Khuzhin
3395276748 Add replica priority into documentation 2020-06-29 23:03:28 +03:00
Ivan Blinkov
7170f3c534
[docs] split aggregate function and system table references (#11742)
* prefer relative links from root

* wip

* split aggregate function reference

* split system tables
2020-06-18 11:24:31 +03:00
Ivan Blinkov
14cda2d1db
Update file.md 2020-06-10 23:22:56 +03:00
Ivan Blinkov
ba57fbaa20
Update merge.md 2020-06-10 23:21:42 +03:00
Ivan Blinkov
5084496ca0
Update dictionary.md 2020-06-10 23:20:54 +03:00
Ivan Blinkov
139d357892
Update external-data.md 2020-06-10 23:19:58 +03:00
Ivan Blinkov
f5b7665271
Update distributed.md 2020-06-10 23:18:36 +03:00
Ivan Blinkov
7d5a1ea485
Update null.md 2020-06-10 23:16:54 +03:00
Ivan Blinkov
06ba3bc480
Update set.md 2020-06-10 23:14:51 +03:00
Ivan Blinkov
33d4ed893a
Update join.md 2020-06-10 23:14:00 +03:00
Ivan Blinkov
ceb34f352e
Update url.md 2020-06-10 23:11:41 +03:00
Ivan Blinkov
79120387a4
Update view.md 2020-06-10 23:09:39 +03:00
Ivan Blinkov
8be01aae1d
Update materializedview.md 2020-06-10 23:08:56 +03:00
Ivan Blinkov
d4daa006f9
Update memory.md 2020-06-10 23:08:20 +03:00
Ivan Blinkov
38347cde2e
Update buffer.md 2020-06-10 23:07:17 +03:00
Ivan Blinkov
6ad0f240ce
Update generate.md 2020-06-10 23:05:22 +03:00
Mikhail f. Shiryaev
7f09bb8264
Replase back/forward quotes and apostrophes by straight 2020-06-10 12:52:41 +02:00
BayoNet
58786f9929
DOCS-620: Background pools settings (#11358)
* DOCSUP-1036: Describe settings for different pools (#118)

* describe settings for different pools

* minor changes

* add refs to and from distributed, server settings, kafka, replication, mergetree

* changes in links description

* add description and links to RU version

* add descrtiption and links to ru version

* minor changes

* CLICKHOUSEDOCS-620: Returned text lost by the last merge.

Co-authored-by: Evgenia Sudarikova <56156889+otrazhenia@users.noreply.github.com>
Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
2020-06-02 21:11:56 +03:00
Ivan Blinkov
cd14f9ebcb
SQL reference refactoring (#10857)
* split up select.md

* array-join.md basic refactoring

* distinct.md basic refactoring

* format.md basic refactoring

* from.md basic refactoring

* group-by.md basic refactoring

* having.md basic refactoring

* additional index.md refactoring

* into-outfile.md basic refactoring

* join.md basic refactoring

* limit.md basic refactoring

* limit-by.md basic refactoring

* order-by.md basic refactoring

* prewhere.md basic refactoring

* adjust operators/index.md links

* adjust sample.md links

* adjust more links

* adjust operatots links

* fix some links

* adjust aggregate function article titles

* basic refactor of remaining select clauses

* absolute paths in make_links.sh

* run make_links.sh

* remove old select.md locations

* translate docs/es

* translate docs/fr

* translate docs/fa

* remove old operators.md location

* change operators.md links

* adjust links in docs/es

* adjust links in docs/es

* minor texts adjustments

* wip

* update machine translations to use new links

* fix changelog

* es build fixes

* get rid of some select.md links

* temporary adjust ru links

* temporary adjust more ru links

* improve curly brace handling

* adjust ru as well

* fa build fix

* ru link fixes

* zh link fixes

* temporary disable part of anchor checks
2020-05-15 07:34:54 +03:00
Ivan Blinkov
21d8b6288d
[docs] generate AMP versions for docs articles (#10732)
* Refactoring in preparation for AMP support

* infrastructure for AMP validation

* Add Metrika counter on AMP page + adjust layout

* more content fixes

* improve amp layout

* improve navigation

* Move converting admonitions to generation time

* strict amp test check

* Batch AMP validation

* Add date published/modified to docs articles and their ld+json meta

* few more content fixes

* improve ld+json meta

* adjust margins

* skip published/modified for stable release docs

* adapt single page mode

* update po

* skip published/modified for single page docs

* update po

* adjust layout

* adjust layout
2020-05-08 11:04:09 +03:00
Ivan Blinkov
d91c97d15d
[docs] replace underscores with hyphens (#10606)
* Replace underscores with hyphens

* remove temporary code

* fix style check

* fix collapse
2020-04-30 21:19:18 +03:00