thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 10:22:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
134,142 Commits 9,328 Branches 1,783 Tags 2.1 GiB
8655c11280
Commit Graph

16110 Commits

Author SHA1 Message Date
Duc Canh Le
3502245ecb use fs::absolute instead of fs::canonical 
To allow users to work with symlinks in user_files_path

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-03 04:58:26 +00:00
Alexey Milovidov
44432b1ffc Fix build 2024-01-02 21:54:13 +01:00
Alexey Milovidov
deac0b5f3e Merge branch 'master' into insert-quorum-host-name-check 2024-01-02 21:46:12 +01:00
avogar
488e36de46 More fixes 2024-01-02 20:20:39 +00:00
Nikolai Kochetov
9c25cb6692 Cleanup 2024-01-02 18:08:04 +00:00
Nikolai Kochetov
4f99a8bc1f Remove more unused code. 2024-01-02 17:54:20 +00:00
Nikolai Kochetov
1b20ce5162 Cleanup 2024-01-02 17:50:06 +00:00
Nikolai Kochetov
c808b03e55 Remove unneeded code 2024-01-02 17:27:33 +00:00
Nikolai Kochetov
8936c8376a Use predicate in getTaskIteratorExtension. 2024-01-02 17:14:16 +00:00
avogar
c5ba97f0cd Fix tests 2024-01-02 16:38:45 +00:00
Kruglov Pavel
f2dfe8bdda
Fix build 2024-01-02 16:42:17 +01:00
Nikolai Kochetov
3e3fed1cbe Add reading step to URL 2024-01-02 15:18:13 +00:00
Nikolai Kochetov
0f76967f97 Add reading step to Azure. 2024-01-02 13:46:08 +00:00
Alexander Tokmakov
d13abac912
Merge pull request #58333 from ClickHouse/fix_no_such_key_detached_part 
Fix lost blobs after dropping a replica with broken detached parts
 2024-01-02 14:21:55 +01:00
Nikolai Kochetov
b7cc6d4615 Fixing tests. 2024-01-02 13:08:04 +00:00
Duc Canh Le
d623702378 Merge branch 'master' into final_no_copy 
Resolve conflicts

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-02 06:07:49 +00:00
Bharat Nallan Chakravarthy
00b5e28d6a Merge upstream/master into ncb/system-database-engines 2023-12-30 23:10:48 -08:00
Bharat Nallan Chakravarthy
3c4ac0ba08 add system.database_engines table 2023-12-30 20:31:25 -08:00
Alexey Milovidov
7bded0a5e7
Merge pull request #58316 from ClickHouse/reintroduce_is_deleted 
Re-introduce `is_deleted` column for ReplacingMergeTree
 2023-12-31 00:57:19 +01:00
Alexey Milovidov
063463b0a8
Merge pull request #58359 from ClickHouse/all-system-tables-local 
Attach all system tables in `clickhouse-local`
 2023-12-30 17:51:27 +01:00
Alexey Milovidov
c9467d8d47
Update ReplicatedMergeTreeSink.cpp 2023-12-30 15:19:50 +03:00
Alexey Milovidov
2421173574
Merge branch 'master' into insert-quorum-host-name-check 2023-12-30 13:13:24 +01:00
Alexey Milovidov
8fc05e25fe
Merge pull request #58310 from azat/kafka-fix-stat-leak-resubmit 
Create consumers for Kafka tables on fly with TTL (resubmit)
 2023-12-30 13:03:16 +01:00
Alexey Milovidov
e1812f3b58
Merge pull request #58266 from ClickHouse/vdimir/simple_fix_tuple_elimination 
Analyzer: fix tuple comparison when result is always null
 2023-12-30 13:02:38 +01:00
Alexey Milovidov
aa6ecd2d59
Merge pull request #58343 from azat/s3/optional-gcs-compose 
Avoid sending ComposeObject requests after upload to GCS
 2023-12-30 12:40:04 +01:00
Alexey Milovidov
40ca9c202d
Merge pull request #58346 from ClickHouse/check-what-would-be-ifremove-array-joined-columns-from-key-condition 
Check what happen if remove array joined columns from KeyCondition
 2023-12-30 12:38:57 +01:00
Alexey Milovidov
f058394d92
Merge pull request #58351 from ClickHouse/fix_00002 
Keep exception format string in retries ctl
 2023-12-30 12:37:36 +01:00
Alexey Milovidov
39b239683c Attach all system tables in clickhouse-local 2023-12-29 21:25:22 +01:00
Nikolai Kochetov
b95bdef09e Update StorageS3 and StorageS3Cluster 2023-12-29 17:41:11 +00:00
Kruglov Pavel
f57939096c
Merge branch 'master' into ignore-mv-with-dropped-target-table 2023-12-29 17:02:23 +01:00
Nikolai Kochetov
5521e5d9b1 Refactor StorageHDFS and StorageFile virtual columns filtering 2023-12-29 15:58:01 +00:00
robot-ch-test-poll3
07ba672e37
Merge pull request #58142 from canhld94/final_less_compare 
MergeTree FINAL to not compare rows from same non-L0 part
 2023-12-29 16:47:14 +01:00
Azat Khuzhin
a12df35be4 Eliminate possible race between ALTER_METADATA and MERGE_PARTS 
v2: move metadata version check after checking that the part is not covering part
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 16:46:10 +01:00
Azat Khuzhin
c7fa93d704 Add infrastructure for testing replicated MergeTree queue 
- replicated_queue_fail_next_entry - to fail next queue entry
- replicated_queue_unfail_entries - to "unfail" all queue entries (if
  any)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 16:43:01 +01:00
Alexander Tokmakov
1013f6b23f
Merge branch 'master' into reintroduce_is_deleted 2023-12-29 15:46:24 +01:00
Alexander Tokmakov
72a0797b88 keep exception format string in retries ctl 2023-12-29 15:21:46 +01:00
Alexey Milovidov
ea03cc82aa
Merge pull request #58320 from ClickHouse/mv3 
Refreshable materialized views again
 2023-12-29 14:44:50 +01:00
Azat Khuzhin
853fdfe775 Clean cached messages on destroy kafka consumer 
The callchain of the kafka consumer is very tricky, so for the sake of
common sense let's just clean the messages on moving out consumer (and
in dtor, but this is just to keep that two code path in sync).

(Also reported by @filimonov)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 14:30:21 +01:00
Azat Khuzhin
b3d6caf37f Unsubscribe kafka consumer before cleaning it by TTL 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 14:03:53 +01:00
Nikolai Kochetov
0e8232a8c3 Check what happen if remove array joined columns from KeyCondition 2023-12-29 12:24:19 +00:00
Azat Khuzhin
f578541ded Fix destructing kafka consumer via member orders 
We've discussed this with @filimonov and he pointed out that everything
else (except for rdkafka_stat/rdkafka_stat_mutex) is done via members
orders, so let's do it in the same style.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 13:19:11 +01:00
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Duc Canh Le
91a87d6b6c better implementation 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-29 07:27:10 +00:00
Igor Nikonov
208a9193f6 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-28 21:28:36 +00:00
Alexander Tokmakov
852f397a97 fix lost blobs after dropping a replica with broken detached parts 2023-12-28 21:47:19 +01:00
Kruglov Pavel
fbd3f7cd59
Merge pull request #56132 from Avogar/flatten-only-true-nested 
Flatten only true Nested type if flatten_nested=1, not all Array(Tuple)
 2023-12-28 20:58:28 +01:00
Michael Kolupaev
c4f4516a37 Fix WriteBuffer assert if refresh is cancelled at the wrong moment 2023-12-28 18:34:28 +00:00
Nikolai Kochetov
490a8bce9e Remove commented code. 2023-12-28 18:01:08 +00:00
Michael Kolupaev
ea138fe8c9 space 2023-12-28 17:56:06 +00:00
Michael Kolupaev
96c68e5aae Remove pausing, enable multithreading, kick off refresh on table creation unless the query says EMPTY 2023-12-28 17:56:06 +00:00
Michael Kolupaev
4d732cdf1e Add to system.process, improve test slightly 2023-12-28 17:56:05 +00:00
Michael Kolupaev
f0417d0ec3 Things 2023-12-28 17:56:05 +00:00
Michael Kolupaev
0fc7535eba Fixes 2023-12-28 17:56:05 +00:00
Michael Kolupaev
609b2c216c Fix some of the CI 2023-12-28 17:56:05 +00:00
Michael Kolupaev
8b8ef41407 Documentation 2023-12-28 17:56:05 +00:00
Michael Kolupaev
64e6deb197 Slightly more things 2023-12-28 17:56:05 +00:00
Michael Kolupaev
dda0606f67 Things 2023-12-28 17:56:05 +00:00
Michael Kolupaev
98dbd105ad Overhaul timestamp arithmetic 2023-12-28 17:56:04 +00:00
Michael Kolupaev
a524e8c51e Overhaul dependencies 2023-12-28 17:56:04 +00:00
Michael Kolupaev
bd18522cad Overhaul RefreshTask 2023-12-28 17:56:04 +00:00
Michael Kolupaev
29a8edb40e Simple review comments 2023-12-28 17:56:04 +00:00
koloshmet
49367186e3 fix fix fix 2023-12-28 17:56:04 +00:00
koloshmet
fb420a160b proper tmp table cleanup 2023-12-28 17:56:04 +00:00
koloshmet
0999a6d98e proper tmp table cleanup 2023-12-28 17:56:04 +00:00
koloshmet
238741dafe fixed style 2023-12-28 17:56:04 +00:00
koloshmet
c52aa984ee refreshable materialized views 2023-12-28 17:56:04 +00:00
Nikolai Kochetov
4c68716df7 Fix another test. 2023-12-28 17:51:11 +00:00
Nikolai Kochetov
d7a473e386 Fix some test. 2023-12-28 17:34:28 +00:00
avogar
e66701dd10 Add setting ignore_materialized_views_with_dropped_target_table 2023-12-28 15:00:39 +00:00
Nikolai Kochetov
50e9c9bb4e Fixing tests. 2023-12-28 14:59:33 +00:00
Azat Khuzhin
ecf7188d52 Fix use-after-free in KafkaConsumer due to statistics callback 
CI founds [1]:

    Exception: Sanitizer assert found for instance �=================================================================
    ==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x5250006a4100 at pc 0x55d4ed46d2e2 bp 0x7f7e33b40190 sp 0x7f7e33b3f950
    WRITE of size 5390 at 0x5250006a4100 thread T2 (TCPHandler)
       8 0x55d50eba9497 in DB::KafkaConsumer::setRDKafkaStat(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./src/Storages/Kafka/KafkaConsumer.h:117:22
       12 0x55d51e0eebfe in cppkafka::stats_callback_proxy(rd_kafka_s*, char*, unsigned long, void*) build_docker/./contrib/cppkafka/src/configuration.cpp:92:5
       13 0x55d51e151e3d in rd_kafka_poll_cb build_docker/./contrib/librdkafka/src/rdkafka.c:3790:7
       14 0x55d51e15531b in rd_kafka_consumer_close build_docker/./contrib/librdkafka/src/rdkafka.c:3200:31
       15 0x55d51e0f3241 in cppkafka::Consumer::close() build_docker/./contrib/cppkafka/src/consumer.cpp:293:33
       16 0x55d51e0f3241 in cppkafka::Consumer::~Consumer() build_docker/./contrib/cppkafka/src/consumer.cpp:82:9
       20 0x55d50eb8d12e in DB::KafkaConsumer::~KafkaConsumer() build_docker/./src/Storages/Kafka/KafkaConsumer.cpp:179:1

    0x5250006a4100 is located 0 bytes inside of 8736-byte region [0x5250006a4100,0x5250006a6320)
    freed by thread T2 (TCPHandler) here:
       0 0x55d4ed4a26b2 in operator delete(void*, unsigned long) (/usr/bin/clickhouse+0xa94b6b2) (BuildId: 74ec4a14a5109c41de109e82d56d8d863845144d)
       1 0x55d50eb8ca55 in void std::__1::__libcpp_operator_delete[abi:v15000]<void*, unsigned long>(void*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/new:256:3
       2 0x55d50eb8ca55 in void std::__1::__do_deallocate_handle_size[abi:v15000]<>(void*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/new:282:10
       3 0x55d50eb8ca55 in std::__1::__libcpp_deallocate[abi:v15000](void*, unsigned long, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/new:296:14
       4 0x55d50eb8ca55 in std::__1::allocator<char>::deallocate[abi:v15000](char*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator.h:128:13
       5 0x55d50eb8ca55 in std::__1::allocator_traits<std::__1::allocator<char>>::deallocate[abi:v15000](std::__1::allocator<char>&, char*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:282:13
       6 0x55d50eb8ca55 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::~basic_string() build_docker/./contrib/llvm-project/libcxx/include/string:2334:9
       7 0x55d50eb8ca55 in DB::KafkaConsumer::~KafkaConsumer() build_docker/./src/Storages/Kafka/KafkaConsumer.cpp:179:1

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/745d9bb47f3425e28e5660ed7c730038ffece4ee/integration_tests__asan__analyzer__%5B6_6%5D/integration_run_parallel4_0.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-28 15:48:43 +01:00
Azat Khuzhin
4a14112af1 Move StorageKafka::createConsumer() into KafkaConsumer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit ebad1bf4f3)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
87f3f6619a Fix data-race between StorageKafka::startup() and cleanConsumers() 
Actually now we can create consumer object in the ctor, no need to do
this in startup(), since consumer now do not connects to kafka.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 03218202d3)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
3c139d7135 Update comment for statistics.interval.ms librdkafka option 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 1f03a21033)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
6f85306510 Use separate thread for kafka consumers cleanup 
Since pool may exceed threads, while we need to run this thread always
to avoid memory leaking.

And this should not be a problem since librdkafka has multiple threads
for each consumer (5!) anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 06a9e9a9ca)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
7d2b82c37c Add ability to configure TTL for kafka consumers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit b19b70b8fc)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
bea1610219 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 2ff0bfb0a1)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
71fdde76c2 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit db74549940)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
d66be02dc3 Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit e7592c140e)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
a6841c8915 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 51d4f583e6)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
3541d9a05f Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 123d63e824)
 2023-12-28 15:32:39 +01:00
avogar
e1a9baa5b0 Fix 2023-12-28 13:51:37 +00:00
Nikolai Kochetov
737563296b
Merge branch 'master' into filter-virtual-columns-storage-merge 2023-12-28 14:47:41 +01:00
Alexander Tokmakov
bdada351c8 Revert "Merge pull request #58274 from ClickHouse/revert-58267" 
This reverts commit 583b9637c2, reversing
changes made to 224e937620.
 2023-12-28 14:07:59 +01:00
Alexander Tokmakov
5fcbf9cfb0 Revert "Merge pull request #58251 from ClickHouse/reintroduce-compatibility-with-a-misfeature" 
This reverts commit a811d5b761, reversing
changes made to 583b9637c2.
 2023-12-28 14:06:56 +01:00
kssenii
3d2e95dbf5 Fix build 2023-12-28 13:49:49 +01:00
kssenii
234e13ac7e Merge remote-tracking branch 'origin' into broken-projections-better-handling 2023-12-28 13:44:04 +01:00
Alexander Tokmakov
38fe70c68a
Revert "Refreshable materialized views (takeover)" 2023-12-28 13:12:20 +01:00
Duc Canh Le
238c5e66d5 use ChunkInfo to carry part level 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-28 11:01:18 +00:00
Alexey Milovidov
4bb8592434 Update autogenerated version to 23.13.1.1 and contributors 2023-12-28 11:22:16 +01:00
凌涛
4bd6cb9cc9 Inverted Index support function match 2023-12-28 17:49:43 +08:00
Alexey Milovidov
524d53199d
Merge branch 'master' into mv 2023-12-28 04:11:48 +01:00
Alexey Milovidov
a811d5b761
Merge pull request #58251 from ClickHouse/reintroduce-compatibility-with-a-misfeature 
Reintroduce compatibility with `is_deleted` on a syntax level
 2023-12-28 04:11:04 +01:00
Alexey Milovidov
c7efd2afea Revert #58267 2023-12-28 04:09:33 +01:00
Alexey Milovidov
40a5dbdeba
Merge branch 'master' into mv 2023-12-28 03:16:27 +01:00
Alexey Milovidov
c52886eb81
Revert "Create consumers for Kafka tables on fly (but keep them for some period since last used)" 2023-12-28 03:35:57 +03:00
Alexey Milovidov
1d9dbfd18b
Merge pull request #49103 from ClickHouse/check-about-global-sorting 
Fixed a sorting order breakage in TTL GROUP BY
 2023-12-28 01:35:14 +01:00
Alexey Milovidov
d7a35773c1
Merge pull request #58252 from Algunenano/i51543 
Avoid throwing ABORTED on normal situations
 2023-12-28 00:28:51 +01:00
Alexey Milovidov
8d984df135
Merge pull request #58237 from azat/build/fwd-decl-exception 
Some code refactoring (was an attempt to improve build time, but failed)
 2023-12-28 00:21:09 +01:00
Alexey Milovidov
c024dc9c3d
Merge pull request #58265 from ClickHouse/remove-mayBenefitFromIndexForIn 
Remove mayBenefitFromIndexForIn
 2023-12-28 00:15:04 +01:00
Alexey Milovidov
dcbd3b9c26
Merge pull request #58267 from ClickHouse/fix_is_deleted_compatibility 
Re-introduce `is_deleted` column for ReplacingMergeTree
 2023-12-28 00:13:01 +01:00
Michael Kolupaev
4d4d8e0545 space 2023-12-27 20:25:35 +00:00
Michael Kolupaev
b9cbecb0df Remove pausing, enable multithreading, kick off refresh on table creation unless the query says EMPTY 2023-12-27 20:24:56 +00:00
Michael Kolupaev
de8567660c Add to system.process, improve test slightly 2023-12-27 20:24:55 +00:00
Michael Kolupaev
538b23d862 Things 2023-12-27 20:24:55 +00:00
Michael Kolupaev
802961f0a2 Fixes 2023-12-27 20:24:55 +00:00
Michael Kolupaev
673743e2ac Fix some of the CI 2023-12-27 20:24:55 +00:00
Michael Kolupaev
7786b12a89 Documentation 2023-12-27 20:24:55 +00:00
Michael Kolupaev
418423a304 Slightly more things 2023-12-27 20:24:55 +00:00
Michael Kolupaev
ef4cc5ec7f Things 2023-12-27 20:24:55 +00:00
Michael Kolupaev
a7c369e14f Overhaul timestamp arithmetic 2023-12-27 20:24:55 +00:00
Michael Kolupaev
01369a0a8a Overhaul dependencies 2023-12-27 20:24:54 +00:00
Michael Kolupaev
01345981e2 Overhaul RefreshTask 2023-12-27 20:24:54 +00:00
Michael Kolupaev
5dc04a13a7 Simple review comments 2023-12-27 20:24:54 +00:00
koloshmet
808cb0fa05 fix fix fix 2023-12-27 20:24:54 +00:00
koloshmet
f1161566b4 proper tmp table cleanup 2023-12-27 20:24:54 +00:00
koloshmet
f14114dafc proper tmp table cleanup 2023-12-27 20:24:54 +00:00
koloshmet
d1932763f3 fixed style 2023-12-27 20:24:54 +00:00
koloshmet
c762898adb refreshable materialized views 2023-12-27 20:24:54 +00:00
Alexander Tokmakov
a3cba8e06f
Update StorageReplicatedMergeTree.cpp 2023-12-27 20:27:15 +01:00
Alexander Tokmakov
f5bcfaffa5 disable vertical merges with cleanup 2023-12-27 19:28:50 +01:00
vdimir
1137461aaf
Analyzer: fix tuple comparison when result is always null 2023-12-27 18:19:39 +00:00
Nikolai Kochetov
e493789bf3 Remove from indexes as well. 2023-12-27 17:51:23 +00:00
Nikolai Kochetov
bcd34b25b2 Remove mayBenefitFromIndexForIn 2023-12-27 17:42:40 +00:00
Nikolai Kochetov
9f9b080b00
Update StorageMerge.cpp 2023-12-27 18:33:00 +01:00
Alexander Tokmakov
f924848347 partially revert #54368 (f28ad1e136) 2023-12-27 18:17:59 +01:00
Raúl Marín
dfe7b0e973 Keep message 2023-12-27 18:13:22 +01:00
Nikolai Kochetov
2f50d3da50 Filter virtual columns for StorageMerge from plan filter condition. 2023-12-27 17:05:23 +00:00
Raúl Marín
5f183649b2 Avoid throwing ABORTED on normal situations 2023-12-27 17:44:46 +01:00
Alexey Milovidov
64b4e1a66f Reintroduce compatibility with is_deleted on a syntax level 2023-12-27 17:42:51 +01:00
Nikolai Kochetov
3ec1b2a852 Refactor StorageMerge. 2023-12-27 16:32:21 +00:00
avogar
9ef8de21b2 Read column once while reading more that one subcolumn from it in Compact parts 2023-12-27 16:30:04 +00:00
Nikita Mikhaylov
3dbd3b3e61 Better 2023-12-27 15:50:20 +00:00
Nikita Mikhaylov
b60109d43e Better 2023-12-27 15:50:20 +00:00
Alexey Milovidov
f00337e2ba
Merge pull request #57872 from CurtizJ/optimize-aggregation-consecutive-keys 
Better optimization of consecutive keys in aggregation
 2023-12-27 15:44:22 +01:00
Azat Khuzhin
b9233f6d4f Move Allocator code into module part 
This should reduce amount of code that should be recompiled on
Exception.h changes (and everything else that had been included there).

This will actually not help a lot, because it is also included into
PODArray.h and ThreadPool.h at least... Sigh.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 15:42:08 +01:00
Alexander Tokmakov
01d042c490 Revert "Merge pull request #57932 from ClickHouse/remove-shit-cleanup" 
This reverts commit 2d58dc512c, reversing
changes made to 41873dc4a3.
 2023-12-27 13:46:06 +01:00
Alexander Tokmakov
eeadeaa89d Revert "Merge pull request #58104 from ClickHouse/cleanup-replication-compatibility" 
This reverts commit 34fd555ee6, reversing
changes made to cb53ee63be.
 2023-12-27 13:03:38 +01:00
Azat Khuzhin
ebad1bf4f3 Move StorageKafka::createConsumer() into KafkaConsumer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
03218202d3 Fix data-race between StorageKafka::startup() and cleanConsumers() 
Actually now we can create consumer object in the ctor, no need to do
this in startup(), since consumer now do not connects to kafka.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
1f03a21033 Update comment for statistics.interval.ms librdkafka option 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
06a9e9a9ca Use separate thread for kafka consumers cleanup 
Since pool may exceed threads, while we need to run this thread always
to avoid memory leaking.

And this should not be a problem since librdkafka has multiple threads
for each consumer (5!) anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
b19b70b8fc Add ability to configure TTL for kafka consumers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
2ff0bfb0a1 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
db74549940 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
e7592c140e Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
51d4f583e6 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
123d63e824 Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Duc Canh Le
476ca4246d Merge branch 'master' into final_no_copy 
Resolve conflicts + add some comments

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-27 07:00:58 +00:00
Igor Nikonov
bee15325fc Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-26 21:56:46 +00:00
Alexey Milovidov
a0fccb0498
Merge pull request #58224 from amosbird/part_offset_pk 
Primary key analysis for _part_offset
 2023-12-26 14:51:57 +01:00
Alexey Milovidov
31a081bd83
Merge pull request #58226 from Algunenano/cleanup_known_short 
Cleanup some known short messages
 2023-12-26 14:40:58 +01:00
Raúl Marín
e87b9751bd Cleanup some known short messages 2023-12-26 12:58:50 +01:00
Amos Bird
66660ee4e2
Add comment 2023-12-26 17:04:00 +08:00
Amos Bird
bfcccf9fa3
Primary key analysis for _part_offset 2023-12-26 17:03:59 +08:00
santrancisco
a59d874bf9
fix syntax 2023-12-26 16:56:58 +11:00
凌涛
a09bdd4367 Merge branch 'master' into optimization/BF_support_rg 2023-12-26 10:09:58 +08:00
Azat Khuzhin
837f4ea676 Add ability to throttle merges/mutations 
Main motivation was to has an ability to throttle background tasks, to
avoid affecting queries.

To new server settings had been added for this:
- max_mutations_bandwidth_for_server
- max_merges_bandwidth_for_server

Note, that they limit only reading, since usually you will not write
more data then you read, but sometimes it is possible in case of ALTER
UPDATE.

But for now, to keep things simple, I decided to limit this with only
2 settings instead of 4.

Note, that if the write throttling will be needed, then they can use the
same settings, and just create new throttler for write.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:31:49 +01:00
Azat Khuzhin
79de5c16c9 Apply all reader settings for merges/mutations 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:29:43 +01:00
Azat Khuzhin
e71f6893cc Add brief comment for MergeTreeSequentialSource 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:29:42 +01:00
Azat Khuzhin
3be3b0a280 Fix incorrect Exceptions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 21:26:32 +01:00
Alexey Milovidov
ae51334ba5 Merge branch 'master' into fix-error-in-archive-reader 2023-12-24 05:53:22 +01:00
Alexey Milovidov
e98c49a58f Fix a benign error in archive reader 2023-12-24 05:44:24 +01:00
Alexey Milovidov
3f4c8e4ae8
Merge pull request #58167 from jrdi/part-log-uncompressed-bytes 
Add bytes_uncompressed to system.part_log
 2023-12-24 04:11:35 +01:00
Alexey Milovidov
b4bf1d1c4c
Merge pull request #58136 from azat/system.stack_trace-rt_tgsigqueueinfo-v2 
Fix system.stack_trace for threads with blocked SIGRTMIN (resubmit)
 2023-12-24 03:51:13 +01:00
Alexey Milovidov
4f3f69521d
Merge pull request #58173 from ClickHouse/parallel-replicas-used-count 
Profile event 'ParallelReplicasUsedCount'
 2023-12-24 03:46:09 +01:00
Alexey Milovidov
00fa9085b1
Merge pull request #58178 from chhetripradeep/add-base-backup-name-to-system-tables 
Add base backup name to system.backups and system.backup_log tables
 2023-12-24 03:38:20 +01:00
Azat Khuzhin
2f6c0487ad Ignore ENOENT for SigBlk check for system.stack_trace 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-23 14:35:38 +01:00
Azat Khuzhin
ac542199c5 Add some comments about racy code for system.stack_trace 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-23 13:42:26 +01:00
Igor Nikonov
d644a208bf Merge remote-tracking branch 'origin/master' into parallel-replicas-used-count 2023-12-23 11:02:28 +00:00
Igor Nikonov
3a485a8bbf Fix:moved request object was used 2023-12-23 11:02:24 +00:00
Alexey Milovidov
dc4b9a1013 Obfuscator: keep settings and timezones 2023-12-23 04:55:55 +01:00
Yakov Olkhovskiy
d7fe86279f StoragesInfoStreamBase refactoring, additional test, style fix 2023-12-23 03:47:43 +00:00
Pradeep Chhetri
b5c8c4050b Add base backup name to system.backups and system.backup_log tables 2023-12-23 11:08:50 +08:00
Jordi Villar
bff0b9c790 Fix mutations new part uncompressed bytes 2023-12-22 22:33:58 +01:00
Igor Nikonov
1deafa1a00 Profile event 'ParallelReplicasUsedCount' 2023-12-22 20:54:52 +00:00
jinjunzh
aaf98f0ed7 refine setting with enable_zstd_qat_codec 2023-12-22 14:24:15 -05:00
Jordi Villar
b4c3969d3a Add bytes_uncompressed to system.part_log 2023-12-22 18:35:33 +01:00
Alexey Milovidov
08ff37f64e
Merge pull request #57682 from azat/system.stack_trace/analyzer 
Add support for system.stack_trace filtering optimizations for analyzer
 2023-12-22 16:28:28 +01:00
Azat Khuzhin
d29762f19f Do not send signals to threads that blocks SIGRTMIN for system.stack_trace 
That way we can avoid superior timeouts during reading from
system.stack_trace.

Two known cases of such threads are:
- rdk: -- librdkafka threads
- iou-wrk -- io_uring threads

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-22 12:41:20 +01:00
Duc Canh Le
17ed277191 FINAL to not compare rows from same non-L0 part 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-22 03:57:06 +00:00
Azat Khuzhin
aa5a6449f0 Fix system.stack_trace for threads with blocked SIGRTMIN 
Some third-party libraries (i.e. librdkafka) could block it, and in this
case system.stack_trace will return stacktrace for the main process
(usually, basically it could be any thread with non blocked signal).

By replacing sigqueue() with more precise rt_tgsigqueueinfo(), other
threads will not respond to the signal.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 106042cf41)
 2023-12-21 19:41:56 +01:00
Raúl Marín
2e522b9405 Remove requestUnconditionalRetry 
It's confusing, only me used it and I used it wrong
 2023-12-21 19:19:50 +01:00
Igor Nikonov
5982a3f80e Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-21 18:07:49 +00:00
Yakov Olkhovskiy
2ad9c7d96d fix for Ordinary 2023-12-21 16:56:02 +00:00
Kseniia Sumarokova
189d0ec147
Merge branch 'master' into broken-projections-better-handling 2023-12-21 17:17:50 +01:00
Raúl Marín
ceed935b30 Remove debug comment and fix unconditional_retry logic 2023-12-21 17:11:34 +01:00
Raúl Marín
6d9da8edd5 Merge remote-tracking branch 'blessed/master' into zk_retries_quorum 2023-12-21 17:03:29 +01:00
skyoct
342a4f7b96 add doc and opt some code 2023-12-21 14:25:18 +00:00
Alexey Milovidov
ab6a712ec6 Add compatibility in the replication protocol for a removed feature 2023-12-21 09:31:59 +01:00
Alexey Milovidov
7be39a27cc
Revert "Consider lightweight deleted rows when selecting parts to merge" 2023-12-21 01:46:38 +03:00
Igor Nikonov
db87ea0b86 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-20 20:48:51 +00:00
Nikita Mikhaylov
cb6c9557ff
Merge branch 'master' into remove-old-projections-code 2023-12-20 17:45:38 +01:00
Kruglov Pavel
ca7001d70f
Merge pull request #57850 from shanielh/feature/s3-session-token 
Feature/s3 session token
 2023-12-20 17:22:09 +01:00
Anton Popov
87d0cec055
Merge pull request #57433 from CurtizJ/materialize-lightweight-delete 
Add mutation command `APPLY DELETED MASK`
 2023-12-20 14:44:11 +01:00
skyoct
296a0bf981
Merge branch 'master' into feat/server_settings 2023-12-20 21:30:36 +08:00
Alexey Milovidov
af32b33e93
Merge pull request #57648 from jewelzqiu/refine-lwd-merge 
Consider lightweight deleted rows when selecting parts to merge
 2023-12-20 01:10:32 +01:00
Yakov Olkhovskiy
43fcea64b6
Merge pull request #57186 from chenlx0/bytes_uncompressed 
Show uncompressed data size in system.tables
 2023-12-19 19:04:47 -05:00
Jordi Villar
7c281d9a6f
Allow max_size_to_drop settings in query time (#57452) 2023-12-19 23:16:03 +01:00
jsc0218
0a9475abc7
Merge pull request #57874 from joelynch/joelynch/host-psql 
bugfix: addresses_expr ignored for psql named collections
 2023-12-19 15:24:23 -05:00
Julia Kartseva
c7eff0e573
Merge pull request #58015 from jkartseva/fix-fuzzjson-segfault 
Fix segfault in FuzzJSON engine
 2023-12-19 11:19:20 -08:00
Julia Kartseva
58f75c9679
Update src/Storages/StorageFuzzJSON.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-12-19 10:17:15 -08:00
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint 
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
 2023-12-19 19:16:01 +01:00
Yakov Olkhovskiy
3d8933f241
Revert "Revert "Add system.dropped_tables_parts table"" 2023-12-19 10:44:44 -05:00
Nikita Taranov
a375b1eab8
Fix handling of unavailable replicas before first request happened (#57933) 2023-12-19 16:41:51 +01:00
skyoct
00576d2092 add onlyDecrease to enums 2023-12-19 15:23:20 +00:00
Raúl Marín
b458c77340 Fix integration test 2023-12-19 14:36:03 +01:00
skyoct
62b8486964
Merge branch 'master' into feat/server_settings 2023-12-19 20:37:15 +08:00
skyoct
267b35ff67 feat: add server setting config 2023-12-19 12:24:32 +00:00
Robert Schulze
6df2548417
Some minor adjustments 2023-12-19 10:39:32 +00:00
Antonio Andelic
52b3caec90
Revert "Add system.dropped_tables_parts table" 2023-12-19 10:36:22 +01:00
Shani Elharrar
679a0e1300 StorageS3 / TableFunctionS3: Allow passing session_token to AuthSettings 
This can help users that want to pass temporary credentials that
issued by AWS in order to load data from S3 without changing
configuration or creating an IAM User.

Fixes #57848
 2023-12-19 08:06:36 +02:00
Julia Kartseva
6014dca114 Allow multiple columns in StorageFuzzJSON 2023-12-19 02:34:23 +00:00
Julia Kartseva
0fc402c106 Fix segfault in FuzzJSON engine 
Allow only String type for FuzzJSON engine table columns.

Fixes: https://github.com/ClickHouse/ClickHouse/issues/57858
 2023-12-19 02:20:37 +00:00
Anton Popov
bfd403cc8c
Merge pull request #57996 from CurtizJ/better-trivial-count-merge 
Better trivial count optimization for storage `Merge`
 2023-12-19 02:25:58 +01:00
Yakov Olkhovskiy
fd460566f6
fix setting description 2023-12-18 19:34:08 -05:00
Han Fei
032fd08559
Merge pull request #57991 from hanfei1991/hanfei/refine-err-msg 
refine error message
 2023-12-19 01:14:18 +01:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Alexey Milovidov
1a7de9158f
Merge pull request #57979 from ClickHouse/add-test-49708 
Add a test for #49708
 2023-12-18 21:29:11 +01:00
Alexey Milovidov
7178bd5c88
Merge pull request #57910 from Algunenano/log_message_string 
Improvements for 00002_log_and_exception_messages_formatting
 2023-12-18 21:02:23 +01:00
Alexander Tokmakov
de7a349f0d
Update MergeTreeData.cpp 2023-12-18 20:44:50 +01:00
Raúl Marín
46067adce8 Improve ZooKeeperRetriesControl so it's easier to reuse 2023-12-18 17:43:10 +01:00
Yakov Olkhovskiy
8e39c61401 Merge remote-tracking branch 'origin/master' into ft-dropped_tables_parts 2023-12-18 15:40:07 +00:00
Anton Popov
7ebb7d2955 better trivial count optimization for storage Merge 2023-12-18 15:13:43 +00:00
Anton Popov
10bc2db56d add setting 2023-12-18 13:57:07 +00:00
Raúl Marín
88dcaefde8 Style 2023-12-18 14:51:46 +01:00
Han Fei
028d1c9478 refine error message 2023-12-18 14:47:54 +01:00
skyoct
9b628aa9d9 feat: add server setting runtime reload 2023-12-18 13:25:43 +00:00
Igor Nikonov
52db759c92 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-18 11:59:13 +00:00
Raúl Marín
0346850897 Support keeper failures in quorum check 2023-12-18 12:54:15 +01:00
Duc Canh Le
c3ffce1491 Merge branch 'master' into column_level_compress_block 
Resolve conflicts

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-18 11:03:44 +00:00
Raúl Marín
2639d0715f Merge remote-tracking branch 'blessed/master' into log_message_string 2023-12-18 10:40:18 +01:00
vdimir
f024e39e3b
Merge pull request #57888 from ucasfl/fix-transfer-external-query 
Fix transform query for external database
 2023-12-18 10:28:32 +01:00
Raúl Marín
f10dae4761
Merge pull request #56917 from Algunenano/backup_1 
Prepare the introduction of more keeper faults
 2023-12-18 10:27:48 +01:00
凌涛
5c2a5759a8
Merge branch 'master' into optimization/BF_support_rg 2023-12-18 16:03:44 +08:00
凌涛
93b18a32c3 add tests 2023-12-18 16:02:19 +08:00
Alexey Milovidov
69e6e59e96 Add a test for #49708 2023-12-18 08:34:53 +01:00
Alexey Milovidov
b0eb670776
Merge pull request #57974 from ClickHouse/revert-stack-trace-changes 
Revert "Merge pull request #57907 from azat/system.stack_trace-rt_tgsigqueueinfo"
 2023-12-18 04:50:12 +01:00
Alexey Milovidov
58396c5546
Merge pull request #57218 from tntnatbry/issue-43666 
Issue 43666: Add skip_unavailable_shards as a setting for Distributed table.
 2023-12-18 04:48:57 +01:00
Alexey Milovidov
2859ac08c9 Revert "Merge pull request #57907 from azat/system.stack_trace-rt_tgsigqueueinfo" 
This reverts commit a33d6b0c96, reversing
changes made to cf031a1b8e.
 2023-12-18 02:01:14 +01:00
Igor Nikonov
af6401761e Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-17 13:32:53 +00:00
Robert Schulze
7f675ddf80
Merge pull request #57878 from Jpnock/master 
Apply full-text skipping index when using `hasAny()`
 2023-12-17 11:06:29 +01:00
Robert Schulze
fc67d2c0e9
Merge pull request #57867 from skyoct/read_code 
Support trivial count optimization for `Merge`-engine tables
 2023-12-17 10:45:21 +01:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Sergei Trifonov
3634b949a8
Merge pull request #57793 from ClickHouse/rename-system-table 
Rename `system.async_loader` into `system.asynchronous_loader`
 2023-12-16 17:33:04 +01:00
Nikolai Kochetov
288888f950 Another try. 2023-12-16 15:41:02 +00:00
Nikolai Kochetov
e7e5fc2891 Another attempt 2023-12-16 12:01:36 +00:00
Sergei Trifonov
37a01bc109
Merge branch 'master' into ft-dropped_tables_parts 2023-12-16 11:53:37 +01:00
Kseniia Sumarokova
4279dd2bf1
Merge pull request #57687 from ClickHouse/make-system-filesystem-cache-query-no-memory-intensive 
Make querying system.filesystem_cache not memory intensive
 2023-12-16 11:28:28 +01:00
Alexey Milovidov
ee502ef6b9 Remove trash (OPTIMIZE CLEANUP) 2023-12-15 22:41:58 +01:00
Igor Nikonov
a735820400 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-15 21:14:43 +00:00
Raúl Marín
6cf8c9b607 Review improvements 2023-12-15 19:44:13 +01:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
James Nock
9aa1ec488c Apply full-text skipping index when using hasAll() 2023-12-15 17:28:33 +00:00
Azat Khuzhin
106042cf41 Fix system.stack_trace for threads with blocked SIGRTMIN 
Some third-party libraries (i.e. librdkafka) could block it, and in this
case system.stack_trace will return stacktrace for the main process
(usually, basically it could be any thread with non blocked signal).

By replacing sigqueue() with more precise rt_tgsigqueueinfo(), other
threads will not respond to the signal.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-15 17:47:01 +01:00
flynn
82fc51ffa6 add test 2023-12-15 15:29:15 +00:00
Robert Schulze
066579f7d7
Improve code aesthetics 2023-12-15 15:07:40 +00:00
Nikita Mikhaylov
1780671443
ClickHouse Cloud promotion (#57638) 
Co-authored-by: Alexander Sapin <alesapin@gmail.com>
 2023-12-15 15:23:23 +01:00
Raúl Marín
946defa8a6
Merge pull request #57860 from zvonand/zvonand-fix-57819 
Try to fix memory leak in StorageHDFS
 2023-12-15 11:50:17 +01:00
kssenii
7bbb456ae9 Fix style check 2023-12-15 11:32:26 +01:00
kssenii
5de9c143a5 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-15 11:32:14 +01:00
Kseniia Sumarokova
f285a01b17
Merge pull request #57863 from ClickHouse/fix-flaky-pg-test 
Follow up to #57568
 2023-12-15 11:29:32 +01:00
Kseniia Sumarokova
06438cc17b
Merge pull request #57578 from ClickHouse/allow-to-change-some-cache-settings-without-restart 
Allow to apply some fs cache config settings changes without server restart
 2023-12-15 11:18:19 +01:00
flynn
2d45213f70 Fix transform query for external database 2023-12-15 08:34:58 +00:00
skyoct
3319934055 reafactor function 2023-12-15 04:49:38 +00:00
凌涛
57a5bef09e optimize code 2023-12-15 11:49:56 +08:00
凌涛
ebc570aef5 optimize code 2023-12-15 11:31:07 +08:00
凌涛
e36cd6a06e BloomFilter support match function 2023-12-15 10:56:52 +08:00
凌涛
18e29bc6a2 BloomFilter support match function 2023-12-15 10:46:30 +08:00
Alexey Milovidov
23cee7ef37
Merge pull request #57851 from azat/revert-55271-local-lazy-database 
Revert "Replace --no-system-tables with loading virtual tables of system database lazily"
 2023-12-15 03:25:59 +01:00
Andrey Zvonov
0174b94469
Merge branch 'ClickHouse:master' into zvonand-fix-57819 2023-12-14 20:14:54 +01:00
Igor Nikonov
1370a8c6c4 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-14 18:29:38 +00:00
Raúl Marín
546484d46b Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 19:03:59 +01:00
Duc Canh Le
93dd6b83e7
Do not allow creating replicated table with inconsistent merge params (#56833) 
* save all merge params to zookeeper

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* calculate hash for graphite merge params

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* add graphite params hash to zookeeper + fix tests

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* install new graphite for testing

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* fix backward incompatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* minor fix test

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* Update src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* remove peekString and add more comments

- peekString doesn't always work even for ReadBufferFromString
- more comment re. backward compatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

---------

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-12-14 17:26:35 +01:00
joelynch
b4bef78e63
bugfix: addresses_expr ignored for psql named collections 2023-12-14 17:17:14 +01:00
Andrey Zvonov
f9579580e9 Retrigger CI + make docs a bit better 2023-12-14 16:34:10 +01:00
kssenii
3605caa8d9 Fxi 2023-12-14 16:05:40 +01:00
kssenii
ea7e55b929 Simplify 2023-12-14 15:58:06 +01:00
skyoct
0a4882d781 feat: update style 2023-12-14 14:26:47 +00:00
skyoct
d700eb2501 feat: support trivial count optimization for Merge 2023-12-14 14:08:24 +00:00
kssenii
a4a264c785 Fix 2023-12-14 14:04:17 +01:00
Andrey Zvonov
7d7aea9826 try fix memory leak in StorageHDFS 2023-12-14 13:01:47 +01:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
Igor Nikonov
42d0afef1f
Merge pull request #57749 from ClickHouse/pr-perf-improve-hope 
Parallel replicas: announcement response handling improvement
 2023-12-14 11:48:51 +01:00
Raúl Marín
efcacd3c34 Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 11:08:50 +01:00
Azat Khuzhin
0e86e16a01
Revert "Replace --no-system-tables with loading virtual tables of system database lazily" 2023-12-14 09:07:47 +01:00
Alexey Milovidov
7a2edb4804
Merge pull request #57822 from azat/kafka-disable-stat 
Disable system.kafka_consumers by default (due to possible live memory leak)
 2023-12-14 04:37:50 +01:00
Yakov Olkhovskiy
9e1d583f54 fix style 2023-12-13 23:36:54 +00:00
Kseniia Sumarokova
79db3c66df
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-13 23:33:59 +01:00
Yakov Olkhovskiy
441050adce fix style, cleanup 2023-12-13 22:26:38 +00:00
Julia Kartseva
b8d274d070
Add malformed output generation to JSON fuzzer (#57646) 
Randomly modify structural characters of a valid JSON ('{', '}', '[', ']',
':', '"', ',') to generate output that cannot be parsed as JSON.

Follow-up to https://github.com/ClickHouse/ClickHouse/pull/56490
 2023-12-13 19:59:31 +01:00
Alexey Milovidov
bf4964b177
Merge pull request #57712 from ClickHouse/clickbench-ci 
Add ClickBench to CI
 2023-12-13 19:52:44 +01:00
Nikita Taranov
560e66f3ca
More respect to min_number_of_marks in ParallelReplicasReadingCoordinator (#57763) 2023-12-13 19:51:49 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Yakov Olkhovskiy
f4e2933f5d use storage uuid instead of table name as a key in storage info because there are can be multiple dropped table with the same name 2023-12-13 18:08:41 +00:00
Azat Khuzhin
055c231438 Disable system.kafka_consumers by default (due to possible live memory leak) 
It is not safe to use statistics because of how KafkaEngine works - it
pre-creates consumers, and this leads to the situation when this
statistics entries generated (RD_KAFKA_OP_STATS), but never consumed.

Which creates a live memory leak for a server with Kafka tables, but
without materialized view attached to it (and no SELECT).

Another problem is that this makes shutdown very slow, because of how
pending queue entries are handled in librdkafka, it uses
TAILQ_INSERT_SORTED, which is sorted insert into linked list, which
works incredibly slow (likely you will never wait till it ends and kill
the server)

For instance in my production setup the server was running for ~67 days
with such table, and it got 1'942'233 `TAILQ_INSERT_SORTED` entries
(which perfectly matches by the way - `67*86400/3` = 1'929'600), and it
moved only 289'806 entries for a few hours, though I'm not sure how much
time the process was in the running state, since most of the time it was
with debugger attached.

So for now let's disable it, to make this patch easy for backporting,
and I will think about long term fix - do not pre-create consumers in
Kafka engine.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-13 18:13:13 +01:00
Anton Popov
7598403f78 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-13 16:55:17 +00:00
Igor Nikonov
72f1c415f1 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-13 15:03:05 +00:00
Nikolai Kochetov
2085b5dd6f
Merge branch 'master' into analyzer-execution-names 2023-12-13 15:22:48 +01:00
Yakov Olkhovskiy
0ea4b16530
Merge branch 'master' into ft-dropped_tables_parts 2023-12-13 09:01:59 -05:00
Igor Nikonov
c165be76ab
Parallel replicas: friendly settings (#57542) 2023-12-13 14:42:06 +01:00
Raúl Marín
dd405a655e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-13 13:39:58 +01:00
Kseniia Sumarokova
b1325450e9
Merge pull request #57754 from ClickHouse/fix-loading-dependent-table-materialized-postgresql 
Table engine MaterializedPostgreSQL fix dependency loading
 2023-12-13 12:17:01 +01:00
Yakov Olkhovskiy
5121bfcd78 major refactoring 2023-12-13 07:16:30 +00:00
Alexey Milovidov
62b6d1ef5e iMerge branch 'master' of github.com:ClickHouse/ClickHouse into clickbench-ci 2023-12-13 01:41:27 +01:00
Alexey Milovidov
f11b90e7bb Allow buckets without List access 2023-12-13 00:10:58 +01:00
Igor Nikonov
616d054dee Merge remote-tracking branch 'origin/master' into pr-perf-improve-hope 2023-12-12 21:27:31 +00:00
Igor Nikonov
7f4a028196 Test and fix 2023-12-12 21:26:58 +00:00
Nikolai Kochetov
3e1dc6bcf9
Merge branch 'master' into analyzer-execution-names 2023-12-12 19:32:50 +01:00
Nikolai Kochetov
63533ad5bf Merge branch 'master' into analyzer-execution-names 2023-12-12 18:30:59 +00:00
serxa
57d7a58dbe Rename system.async_loader into system.asynchronous_loader 2023-12-12 16:55:27 +00:00
kssenii
cc71b40002 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 16:40:04 +01:00
Alexander Gololobov
09d6e6b945
Merge pull request #57743 from ClickHouse/other_timeout_for_wait 
Increase async block cache deduplication timeout
 2023-12-12 15:40:04 +01:00
Raúl Marín
049fb60c73 Fix error on retries due to TABLE_IS_READ_ONLY 2023-12-12 15:37:28 +01:00
kssenii
fa0b2b9e50 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 15:31:01 +01:00
avogar
096c55aece Fix tests 2023-12-12 14:13:15 +00:00
avogar
cbad0add9c Flatten only true Nested type if flatten_nested=1, not all Array(Tuple) 2023-12-12 14:13:15 +00:00
Chen Lixiang
68af81196d
Merge branch 'ClickHouse:master' into bytes_uncompressed 2023-12-12 22:06:45 +08:00
Kruglov Pavel
8a447bf57c
Merge pull request #55892 from Avogar/schema-inference-union 
Add 'union' mode for schema inference
 2023-12-12 15:02:06 +01:00
Nikolai Kochetov
696a2d9f4f
Merge pull request #57614 from ClickHouse/vdimir/join_array_lowcard_totals_crash 
Fix type correction in HashJoin for nested low cardinality
 2023-12-12 14:56:41 +01:00
Raúl Marín
923c3b7b32 Implement retries when ZK connection fails without committing the transaction 2023-12-12 12:00:46 +01:00
Raúl Marín
9d8d5dfdf6 Partially revert "make stages commit" 
This reverts commit e4becc01ba.
 2023-12-12 11:26:48 +01:00
Chen Lixiang
0b89fbbdd9 fix style issue 2023-12-12 16:49:58 +08:00
Zhuo Qiu
e34c13b9d9 Consider lightweight deleted rows when selecting parts to merge 2023-12-12 11:40:19 +08:00
alesapin
7cdc3d29fb Get rid of tiny function 2023-12-11 22:29:51 +01:00
alesapin
bfc2a7bb75 Update cache only after conflicts 2023-12-11 21:27:48 +01:00
alesapin
4b3e845dab Merge branch 'master' into other_timeout_for_wait 2023-12-11 20:43:21 +01:00
Igor Nikonov
30dabd42d3 One lookup for 2 checks 
(1) if it's the same part
(2) if annouced part has intersections with parts in working set
 2023-12-11 19:37:42 +00:00
kssenii
05e56f76ed Merge remote-tracking branch 'origin/master' into fix-loading-dependent-table-materialized-postgresql 2023-12-11 20:17:26 +01:00
Kseniia Sumarokova
907d0a79eb
Merge pull request #57568 from ClickHouse/fix-materialized-pg-issue-with-generated-columns 
MaterializedPostgreSQL: Support "generated columns" and default values
 2023-12-11 19:57:41 +01:00
Kruglov Pavel
3b1ffbd77c
Merge branch 'master' into schema-inference-union 2023-12-11 18:59:15 +01:00
kssenii
48ce040621 Fix 2023-12-11 17:30:23 +01:00
Raúl Marín
e1965bb6b5 WIP: Remove UNCERTAIN_COMMIT in INSERT 2023-12-11 17:27:56 +01:00
vdimir
ea09080c73
fix 2023-12-11 15:50:27 +00:00
Raúl Marín
e6be38be4d Adapt from HEAD 2023-12-11 16:28:00 +01:00
Raúl Marín
a55a0c068e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-11 15:27:21 +01:00
Igor Nikonov
4ccf4e11a8 Parallel replicas (perf): announcement response handling improvement 2023-12-11 13:00:08 +00:00
kssenii
1e4d61d55a Change error code 2023-12-11 13:44:12 +01:00
kssenii
b8caa00d24 Commit forgotten file 2023-12-11 13:28:36 +01:00
kssenii
c8c4db5984 Fxi test 2023-12-11 13:24:31 +01:00
alesapin
a62a0b92de Increase async block cache deduplication timeout 2023-12-11 12:46:22 +01:00
kssenii
966a09fb70 Review fixes 2023-12-11 12:42:23 +01:00
kssenii
a06179da00 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-11 11:15:00 +01:00
Vitaly Baranov
f1ccf4e29f Show owner query ids in the message for the DEADLOCK_AVOIDED error. 2023-12-11 00:56:17 +01:00
Alexey Milovidov
644ef7b63f
Revert "Fix RWLock inconsistency after write lock timeout" 2023-12-10 23:51:59 +03:00
Igor Nikonov
f7e6a17052 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-10 20:21:47 +00:00
Alexey Milovidov
0a20ce5d32
Merge pull request #55271 from azat/local-lazy-database 
Replace --no-system-tables with loading virtual tables of system database lazily
 2023-12-10 19:51:17 +01:00
Alexey Milovidov
d119052895
Merge pull request #51858 from ClickHouse/CurtizJ-fix-ttl-non-deterministic 
Merging #37286
 2023-12-10 17:50:36 +01:00
Vitaly Baranov
a058a26b7e
Merge pull request #57454 from vitlibar/fix-rwlock 
Fix RWLock inconsistency after write lock timeout
 2023-12-10 14:09:29 +01:00