Commit Graph

55 Commits

Author SHA1 Message Date
Ilya Golshtein
46be5e5782 system_kafka_consumers: cleanup, test should be more stable 2023-08-10 22:35:05 +00:00
Ilya Golshtein
dd6727be3e system_kafka_consumers: style check fixes 2023-08-10 22:35:05 +00:00
Ilya Golshtein
3f205f4997 system_kafka_consumers: tiny doc changes 2023-08-10 22:35:05 +00:00
rfraposa
b5d9ed5b0c Alphabetize table functions and engines 2023-06-23 07:16:22 -06:00
AVMusorin
418a61a68c
Allow using Alias column type for KafkaEngine
```
create table kafka
(
 a UInt32,
 a_str String Alias toString(a)
) engine = Kafka;

create table data
(
  a UInt32;
  a_str String
) engine = MergeTree
order by tuple();

create materialized view data_mv to data
(
  a UInt32,
  a_str String
) as
select a, a_str from kafka;
```
Alias type works as expected in comparison with MATERIALIZED/EPHEMERAL
or column with default expression.

Ref: https://github.com/ClickHouse/ClickHouse/pull/47138

Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
2023-05-15 15:39:58 +02:00
Robert Schulze
c406663442
Docs: Replace annoying three spaces in enumerations by a single space 2023-04-19 15:56:55 +00:00
filimonov
fbb22348ea
Refactor reading the pool setting & from server config. (#48055)
After #36425 there was a lot of confusions/problems with configuring pools - when the message was confusing, and settings need to be ajusted in several places.
See some examples in #44251, #43351, #47900, #46515.

The commit includes the following changes:
1) Introduced a unified mechanism for reading pool sizes from the configuration file(s). Previously, pool sizes were read from the Context.cpp with fallbacks to profiles, whereas main_config_reloader in Server.cpp read them directly without fallbacks.
2) Corrected the data type for background_merges_mutations_concurrency_ratio. It should be float instead of int.
3) Refactored the default values for settings. Previously, they were defined in multiple places throughout the codebase, but they are now defined in one place (or two, to be exact: Settings.h and ServerSettings.h).
4) Improved documentation, including the correct message in system.settings.

Additionally make the code more conform with #46550.
2023-03-30 16:44:11 +02:00
DanRoscigno
6d8a2bbd48 standardize admonitions 2023-03-27 14:54:05 -04:00
Rich Raposa
de68dade71
Revert "standardize admonitions" 2023-03-09 14:30:40 -07:00
DanRoscigno
95351bc2d3 standardize admonitions 2023-03-09 12:05:26 -05:00
Robert Schulze
ff8fc0634a
Docs: Small cleanups after Kafka fix #47138 2023-03-07 19:50:42 +00:00
AVMusorin
0f2ae72141
prohibit DEFAULT/EPHEMERAL/ALIAS in KafkaEngine 2023-03-03 16:25:42 +01:00
Robert Schulze
cca0bead44
Slighly improved example 2023-02-23 20:07:06 +00:00
Robert Schulze
b3eacf0894
Whitespace for the win 2023-02-23 20:06:24 +00:00
Robert Schulze
6906cafa99
Change to S3 cfg syntax 2023-02-23 20:04:41 +00:00
Robert Schulze
81bf43157f
Allow configuration of Kafka topics with periods
The Kafka table engine allows global configuration and per-Kafka-topic
configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic
"football":

  <kafka_football>
      <retry_backoff_ms>250</retry_backoff_ms>
      <fetch_min_bytes>100000</fetch_min_bytes>
  </kafka_football>

Some users had to find out the hard way that such configuration doesn't
take effect if the topic name contains a period, e.g. "sports.football".
The reason is that ClickHouse configuration framework already uses
periods as level separators to descend the configuration hierarchy.
(Besides that, per-topic configuration at the same level as global
configuration could be considered ugly.)

Note that Kafka topics may contain characters "a-zA-Z0-9._-" (*) and
a tree-like topic organization using periods is quite common in
practice.

This PR deprecates the existing per-topic configuration syntax (but
continues to support it for backward compat) and introduces a new
per-topic configuration syntax below the global Kafka configuration of
the form:

<kafka>
   <topic name="football">
       <retry_backoff_ms>250</retry_backoff_ms>
       <fetch_min_bytes>100000</fetch_min_bytes>
   </topic>
</kafka>

The period restriction doesn't apply to XML attributes, so <topic
name="sports.football"> will work. Also, everything Kafka-related is
below <kafka>.

Considered but rejected alternatives:
- Extending Poco ConfigurationView with custom separators (e.g."/"
  instead of "."). Won't work easily because ConfigurationView only
  builds a path but defers descending the configuration tree to the
  normal configuration classes.
- Reloading the configuration file in StorageKafka (instead of reading
  the loaded file) but with a custom separator. This mode is supported
  by XML configuration. Too ugly and error-prone since the true
  configuration is composed from multiple configuration files.

(*) https://stackoverflow.com/a/37067544
2023-02-22 20:35:09 +00:00
Ivan Blinkov
61c2f23713 Remove leftover empty lines at the end of markdown files 2023-01-09 15:15:18 +01:00
Ivan Blinkov
b7e082d033 Remove "Original article links" 2023-01-09 15:13:36 +01:00
avogar
8e13d1f1ec Improve and refactor Kafka/StorageMQ/NATS and data formats 2022-10-28 16:41:10 +00:00
DanRoscigno
5b5fcc56aa add slugs 2022-08-28 10:53:34 -04:00
Yakov Olkhovskiy
d5f65ece9b
Merge pull request #38105 from arenadata/ADQM-419
Add kerberosInit function as a replacement for kinit executable calls in Kafka and HDFS
2022-06-27 14:19:24 -04:00
Roman Vasin
3cfea6e76f Cleanup code in KerberosInit, HDFSCommon and StorageKafka; update English and Russian documentation. 2022-06-08 17:57:45 +03:00
rfraposa
869967de41 Remove H1 anchor tags from docs 2022-06-02 04:55:18 -06:00
Mark Andreev
ebb9694920
Add Kafka headers virtual columns
Add documentation for https://github.com/ClickHouse/ClickHouse/pull/11283
2022-05-11 18:38:09 +04:00
rfraposa
fb08c5b7e1 Merge branch 'master' into docs-staging 2022-04-15 07:25:27 -06:00
rfraposa
8f01fe9c49 Revised /en folder 2022-04-09 07:34:21 -06:00
rfraposa
5250d9ad11 Removed /ja folder, cleaned up /ru markdown 2022-04-09 07:29:05 -06:00
Alexey Milovidov
9854b55835
Revert "Format changes for new docs" 2022-04-04 02:05:35 +03:00
rfraposa
72a00e2c62 Fixed broken links 2022-04-02 17:38:54 -06:00
rfraposa
00ddb72eea Update /engines docs 2022-03-29 17:43:34 -06:00
Oleg Taizov
2ce6571968
kafka_num_consumers prop upper bound doc update
Add clarification about the upper bound of `kafka_num_consumers` property that was added in [#26640](https://github.com/ClickHouse/ClickHouse/pull/26642)
2022-03-12 11:09:12 +03:00
Xudong Zhang
9c4cc70dae update document: add _timestamp_ms of kafka 2022-02-19 23:49:37 +08:00
alexey-milovidov
b5f44de8f0
Merge pull request #31948 from ramazanpolat/patch-2
fix some broken links
2021-11-30 05:59:45 +03:00
Ramazan Polat
daed39d000
fix some broken links
max_insert_block_size and stream-flush-interval-ms links fixed.
2021-11-29 14:34:46 +03:00
khamadiev
df23893f69
Update kafka.md
Updated See Also section, since number of kafka consumers controlled by background_message_broker_schedule_pool_size setting
2021-11-25 17:20:16 +05:00
Ivan Blinkov
f429db1ee9 find . -type f -name '*.md'| xargs -I{} perl -pi -e 's|https://clickhouse.tech|https://clickhouse.com|g' {} 2021-09-19 23:05:54 +03:00
Romain Neutron
7b515c7235
Avoid short syntax 2021-05-27 21:44:11 +02:00
Olga Revyakina
89f5a64cc0 Merge branch 'master' into revolg-DOCSUP-5920-PostgreSQL_function_dictionary_engine 2021-03-20 12:48:25 +03:00
olgarev
bad099d15e
Merge branch 'master' into revolg-DOCSUP-5920-PostgreSQL_function_dictionary_engine 2021-03-14 20:35:22 +03:00
alexey-milovidov
c1a96e977a Update kafka.md 2021-03-08 03:40:29 +03:00
Slach
ddd5acf251 try to fix "fake" nowhere links according to https://github.com/ClickHouse/ClickHouse/pull/21268#issuecomment-787106299 2021-03-08 03:40:29 +03:00
Olga Revyakina
64322d2212 Table engine and function 2021-02-27 03:01:02 +03:00
George
838dab7564 Edit and translated Kafka 2021-02-11 21:07:38 +03:00
Alexey Milovidov
db4db42b65 Fix broken links in docs 2020-10-13 20:23:29 +03:00
Alexander Tokmakov
a7d3a024c7 Revert "Revert "Test and doc for PR12771 krb5 + cyrus-sasl + kerberized kafka""
This reverts commit c298c633a7.
2020-09-29 11:56:37 +03:00
tavplubix
c298c633a7
Revert "Test and doc for PR12771 krb5 + cyrus-sasl + kerberized kafka" 2020-09-24 16:10:24 +03:00
Ilya Golshtein
2d5c31d588 test_storage_kerberized_kafka cleanup + negative test 2020-09-17 16:51:34 +03:00
Peng Jian
ac25c41f22 Update docs for Kafka engine 2020-09-06 11:17:19 +08:00
Peng Jian
fd82272cfe Fix code style, and update docs for Kafka engine 2020-09-06 11:17:19 +08:00
Ivan Blinkov
a2047de471
[docs] introduction for integration table engines (#12167)
* [docs] introduction for integration table engines

* Update jdbc.md

* Update odbc.md

* Update mysql.md

* Update kafka.md

* Update hdfs.md
2020-07-06 20:18:37 +03:00