thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 10:22:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
132,559 Commits 9,328 Branches 1,783 Tags 2.1 GiB
211c285a0b
Commit Graph

15847 Commits

Author SHA1 Message Date
Nikolai Kochetov
9f9b080b00
Update StorageMerge.cpp 2023-12-27 18:33:00 +01:00
Alexander Tokmakov
f924848347 partially revert #54368 (f28ad1e136) 2023-12-27 18:17:59 +01:00
Raúl Marín
dfe7b0e973 Keep message 2023-12-27 18:13:22 +01:00
Nikolai Kochetov
2f50d3da50 Filter virtual columns for StorageMerge from plan filter condition. 2023-12-27 17:05:23 +00:00
Raúl Marín
5f183649b2 Avoid throwing ABORTED on normal situations 2023-12-27 17:44:46 +01:00
Alexey Milovidov
64b4e1a66f Reintroduce compatibility with is_deleted on a syntax level 2023-12-27 17:42:51 +01:00
Nikolai Kochetov
3ec1b2a852 Refactor StorageMerge. 2023-12-27 16:32:21 +00:00
avogar
9ef8de21b2 Read column once while reading more that one subcolumn from it in Compact parts 2023-12-27 16:30:04 +00:00
Nikita Mikhaylov
3dbd3b3e61 Better 2023-12-27 15:50:20 +00:00
Nikita Mikhaylov
b60109d43e Better 2023-12-27 15:50:20 +00:00
Alexey Milovidov
f00337e2ba
Merge pull request #57872 from CurtizJ/optimize-aggregation-consecutive-keys 
Better optimization of consecutive keys in aggregation
 2023-12-27 15:44:22 +01:00
Azat Khuzhin
b9233f6d4f Move Allocator code into module part 
This should reduce amount of code that should be recompiled on
Exception.h changes (and everything else that had been included there).

This will actually not help a lot, because it is also included into
PODArray.h and ThreadPool.h at least... Sigh.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 15:42:08 +01:00
Alexander Tokmakov
01d042c490 Revert "Merge pull request #57932 from ClickHouse/remove-shit-cleanup" 
This reverts commit 2d58dc512c, reversing
changes made to 41873dc4a3.
 2023-12-27 13:46:06 +01:00
Alexander Tokmakov
eeadeaa89d Revert "Merge pull request #58104 from ClickHouse/cleanup-replication-compatibility" 
This reverts commit 34fd555ee6, reversing
changes made to cb53ee63be.
 2023-12-27 13:03:38 +01:00
Azat Khuzhin
ebad1bf4f3 Move StorageKafka::createConsumer() into KafkaConsumer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
03218202d3 Fix data-race between StorageKafka::startup() and cleanConsumers() 
Actually now we can create consumer object in the ctor, no need to do
this in startup(), since consumer now do not connects to kafka.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
1f03a21033 Update comment for statistics.interval.ms librdkafka option 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
06a9e9a9ca Use separate thread for kafka consumers cleanup 
Since pool may exceed threads, while we need to run this thread always
to avoid memory leaking.

And this should not be a problem since librdkafka has multiple threads
for each consumer (5!) anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
b19b70b8fc Add ability to configure TTL for kafka consumers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
2ff0bfb0a1 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
db74549940 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
e7592c140e Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
51d4f583e6 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
123d63e824 Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Igor Nikonov
bee15325fc Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-26 21:56:46 +00:00
Alexey Milovidov
a0fccb0498
Merge pull request #58224 from amosbird/part_offset_pk 
Primary key analysis for _part_offset
 2023-12-26 14:51:57 +01:00
Alexey Milovidov
31a081bd83
Merge pull request #58226 from Algunenano/cleanup_known_short 
Cleanup some known short messages
 2023-12-26 14:40:58 +01:00
Raúl Marín
e87b9751bd Cleanup some known short messages 2023-12-26 12:58:50 +01:00
Amos Bird
66660ee4e2
Add comment 2023-12-26 17:04:00 +08:00
Amos Bird
bfcccf9fa3
Primary key analysis for _part_offset 2023-12-26 17:03:59 +08:00
santrancisco
a59d874bf9
fix syntax 2023-12-26 16:56:58 +11:00
凌涛
a09bdd4367 Merge branch 'master' into optimization/BF_support_rg 2023-12-26 10:09:58 +08:00
Azat Khuzhin
837f4ea676 Add ability to throttle merges/mutations 
Main motivation was to has an ability to throttle background tasks, to
avoid affecting queries.

To new server settings had been added for this:
- max_mutations_bandwidth_for_server
- max_merges_bandwidth_for_server

Note, that they limit only reading, since usually you will not write
more data then you read, but sometimes it is possible in case of ALTER
UPDATE.

But for now, to keep things simple, I decided to limit this with only
2 settings instead of 4.

Note, that if the write throttling will be needed, then they can use the
same settings, and just create new throttler for write.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:31:49 +01:00
Azat Khuzhin
79de5c16c9 Apply all reader settings for merges/mutations 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:29:43 +01:00
Azat Khuzhin
e71f6893cc Add brief comment for MergeTreeSequentialSource 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:29:42 +01:00
Azat Khuzhin
3be3b0a280 Fix incorrect Exceptions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 21:26:32 +01:00
Alexey Milovidov
ae51334ba5 Merge branch 'master' into fix-error-in-archive-reader 2023-12-24 05:53:22 +01:00
Alexey Milovidov
e98c49a58f Fix a benign error in archive reader 2023-12-24 05:44:24 +01:00
Alexey Milovidov
3f4c8e4ae8
Merge pull request #58167 from jrdi/part-log-uncompressed-bytes 
Add bytes_uncompressed to system.part_log
 2023-12-24 04:11:35 +01:00
Alexey Milovidov
b4bf1d1c4c
Merge pull request #58136 from azat/system.stack_trace-rt_tgsigqueueinfo-v2 
Fix system.stack_trace for threads with blocked SIGRTMIN (resubmit)
 2023-12-24 03:51:13 +01:00
Alexey Milovidov
4f3f69521d
Merge pull request #58173 from ClickHouse/parallel-replicas-used-count 
Profile event 'ParallelReplicasUsedCount'
 2023-12-24 03:46:09 +01:00
Alexey Milovidov
00fa9085b1
Merge pull request #58178 from chhetripradeep/add-base-backup-name-to-system-tables 
Add base backup name to system.backups and system.backup_log tables
 2023-12-24 03:38:20 +01:00
Azat Khuzhin
2f6c0487ad Ignore ENOENT for SigBlk check for system.stack_trace 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-23 14:35:38 +01:00
Azat Khuzhin
ac542199c5 Add some comments about racy code for system.stack_trace 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-23 13:42:26 +01:00
Igor Nikonov
d644a208bf Merge remote-tracking branch 'origin/master' into parallel-replicas-used-count 2023-12-23 11:02:28 +00:00
Igor Nikonov
3a485a8bbf Fix:moved request object was used 2023-12-23 11:02:24 +00:00
Alexey Milovidov
dc4b9a1013 Obfuscator: keep settings and timezones 2023-12-23 04:55:55 +01:00
Yakov Olkhovskiy
d7fe86279f StoragesInfoStreamBase refactoring, additional test, style fix 2023-12-23 03:47:43 +00:00
Pradeep Chhetri
b5c8c4050b Add base backup name to system.backups and system.backup_log tables 2023-12-23 11:08:50 +08:00
Jordi Villar
bff0b9c790 Fix mutations new part uncompressed bytes 2023-12-22 22:33:58 +01:00
Igor Nikonov
1deafa1a00 Profile event 'ParallelReplicasUsedCount' 2023-12-22 20:54:52 +00:00
jinjunzh
aaf98f0ed7 refine setting with enable_zstd_qat_codec 2023-12-22 14:24:15 -05:00
Jordi Villar
b4c3969d3a Add bytes_uncompressed to system.part_log 2023-12-22 18:35:33 +01:00
Alexey Milovidov
08ff37f64e
Merge pull request #57682 from azat/system.stack_trace/analyzer 
Add support for system.stack_trace filtering optimizations for analyzer
 2023-12-22 16:28:28 +01:00
Azat Khuzhin
d29762f19f Do not send signals to threads that blocks SIGRTMIN for system.stack_trace 
That way we can avoid superior timeouts during reading from
system.stack_trace.

Two known cases of such threads are:
- rdk: -- librdkafka threads
- iou-wrk -- io_uring threads

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-22 12:41:20 +01:00
Duc Canh Le
17ed277191 FINAL to not compare rows from same non-L0 part 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-22 03:57:06 +00:00
Azat Khuzhin
aa5a6449f0 Fix system.stack_trace for threads with blocked SIGRTMIN 
Some third-party libraries (i.e. librdkafka) could block it, and in this
case system.stack_trace will return stacktrace for the main process
(usually, basically it could be any thread with non blocked signal).

By replacing sigqueue() with more precise rt_tgsigqueueinfo(), other
threads will not respond to the signal.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 106042cf41)
 2023-12-21 19:41:56 +01:00
Raúl Marín
2e522b9405 Remove requestUnconditionalRetry 
It's confusing, only me used it and I used it wrong
 2023-12-21 19:19:50 +01:00
Igor Nikonov
5982a3f80e Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-21 18:07:49 +00:00
Yakov Olkhovskiy
2ad9c7d96d fix for Ordinary 2023-12-21 16:56:02 +00:00
Raúl Marín
ceed935b30 Remove debug comment and fix unconditional_retry logic 2023-12-21 17:11:34 +01:00
Raúl Marín
6d9da8edd5 Merge remote-tracking branch 'blessed/master' into zk_retries_quorum 2023-12-21 17:03:29 +01:00
skyoct
342a4f7b96 add doc and opt some code 2023-12-21 14:25:18 +00:00
Alexey Milovidov
ab6a712ec6 Add compatibility in the replication protocol for a removed feature 2023-12-21 09:31:59 +01:00
Alexey Milovidov
7be39a27cc
Revert "Consider lightweight deleted rows when selecting parts to merge" 2023-12-21 01:46:38 +03:00
Igor Nikonov
db87ea0b86 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-20 20:48:51 +00:00
Nikita Mikhaylov
cb6c9557ff
Merge branch 'master' into remove-old-projections-code 2023-12-20 17:45:38 +01:00
Kruglov Pavel
ca7001d70f
Merge pull request #57850 from shanielh/feature/s3-session-token 
Feature/s3 session token
 2023-12-20 17:22:09 +01:00
Anton Popov
87d0cec055
Merge pull request #57433 from CurtizJ/materialize-lightweight-delete 
Add mutation command `APPLY DELETED MASK`
 2023-12-20 14:44:11 +01:00
skyoct
296a0bf981
Merge branch 'master' into feat/server_settings 2023-12-20 21:30:36 +08:00
Alexey Milovidov
af32b33e93
Merge pull request #57648 from jewelzqiu/refine-lwd-merge 
Consider lightweight deleted rows when selecting parts to merge
 2023-12-20 01:10:32 +01:00
Yakov Olkhovskiy
43fcea64b6
Merge pull request #57186 from chenlx0/bytes_uncompressed 
Show uncompressed data size in system.tables
 2023-12-19 19:04:47 -05:00
Jordi Villar
7c281d9a6f
Allow max_size_to_drop settings in query time (#57452) 2023-12-19 23:16:03 +01:00
jsc0218
0a9475abc7
Merge pull request #57874 from joelynch/joelynch/host-psql 
bugfix: addresses_expr ignored for psql named collections
 2023-12-19 15:24:23 -05:00
Julia Kartseva
c7eff0e573
Merge pull request #58015 from jkartseva/fix-fuzzjson-segfault 
Fix segfault in FuzzJSON engine
 2023-12-19 11:19:20 -08:00
Julia Kartseva
58f75c9679
Update src/Storages/StorageFuzzJSON.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-12-19 10:17:15 -08:00
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint 
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
 2023-12-19 19:16:01 +01:00
Yakov Olkhovskiy
3d8933f241
Revert "Revert "Add system.dropped_tables_parts table"" 2023-12-19 10:44:44 -05:00
Nikita Taranov
a375b1eab8
Fix handling of unavailable replicas before first request happened (#57933) 2023-12-19 16:41:51 +01:00
skyoct
00576d2092 add onlyDecrease to enums 2023-12-19 15:23:20 +00:00
Raúl Marín
b458c77340 Fix integration test 2023-12-19 14:36:03 +01:00
skyoct
62b8486964
Merge branch 'master' into feat/server_settings 2023-12-19 20:37:15 +08:00
skyoct
267b35ff67 feat: add server setting config 2023-12-19 12:24:32 +00:00
Robert Schulze
6df2548417
Some minor adjustments 2023-12-19 10:39:32 +00:00
Antonio Andelic
52b3caec90
Revert "Add system.dropped_tables_parts table" 2023-12-19 10:36:22 +01:00
Shani Elharrar
679a0e1300 StorageS3 / TableFunctionS3: Allow passing session_token to AuthSettings 
This can help users that want to pass temporary credentials that
issued by AWS in order to load data from S3 without changing
configuration or creating an IAM User.

Fixes #57848
 2023-12-19 08:06:36 +02:00
Julia Kartseva
6014dca114 Allow multiple columns in StorageFuzzJSON 2023-12-19 02:34:23 +00:00
Julia Kartseva
0fc402c106 Fix segfault in FuzzJSON engine 
Allow only String type for FuzzJSON engine table columns.

Fixes: https://github.com/ClickHouse/ClickHouse/issues/57858
 2023-12-19 02:20:37 +00:00
Anton Popov
bfd403cc8c
Merge pull request #57996 from CurtizJ/better-trivial-count-merge 
Better trivial count optimization for storage `Merge`
 2023-12-19 02:25:58 +01:00
Yakov Olkhovskiy
fd460566f6
fix setting description 2023-12-18 19:34:08 -05:00
Han Fei
032fd08559
Merge pull request #57991 from hanfei1991/hanfei/refine-err-msg 
refine error message
 2023-12-19 01:14:18 +01:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Alexey Milovidov
1a7de9158f
Merge pull request #57979 from ClickHouse/add-test-49708 
Add a test for #49708
 2023-12-18 21:29:11 +01:00
Alexey Milovidov
7178bd5c88
Merge pull request #57910 from Algunenano/log_message_string 
Improvements for 00002_log_and_exception_messages_formatting
 2023-12-18 21:02:23 +01:00
Alexander Tokmakov
de7a349f0d
Update MergeTreeData.cpp 2023-12-18 20:44:50 +01:00
Raúl Marín
46067adce8 Improve ZooKeeperRetriesControl so it's easier to reuse 2023-12-18 17:43:10 +01:00
Yakov Olkhovskiy
8e39c61401 Merge remote-tracking branch 'origin/master' into ft-dropped_tables_parts 2023-12-18 15:40:07 +00:00
Anton Popov
7ebb7d2955 better trivial count optimization for storage Merge 2023-12-18 15:13:43 +00:00
Anton Popov
10bc2db56d add setting 2023-12-18 13:57:07 +00:00
Raúl Marín
88dcaefde8 Style 2023-12-18 14:51:46 +01:00
Han Fei
028d1c9478 refine error message 2023-12-18 14:47:54 +01:00
skyoct
9b628aa9d9 feat: add server setting runtime reload 2023-12-18 13:25:43 +00:00
Igor Nikonov
52db759c92 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-18 11:59:13 +00:00
Raúl Marín
0346850897 Support keeper failures in quorum check 2023-12-18 12:54:15 +01:00
Raúl Marín
2639d0715f Merge remote-tracking branch 'blessed/master' into log_message_string 2023-12-18 10:40:18 +01:00
vdimir
f024e39e3b
Merge pull request #57888 from ucasfl/fix-transfer-external-query 
Fix transform query for external database
 2023-12-18 10:28:32 +01:00
Raúl Marín
f10dae4761
Merge pull request #56917 from Algunenano/backup_1 
Prepare the introduction of more keeper faults
 2023-12-18 10:27:48 +01:00
凌涛
5c2a5759a8
Merge branch 'master' into optimization/BF_support_rg 2023-12-18 16:03:44 +08:00
凌涛
93b18a32c3 add tests 2023-12-18 16:02:19 +08:00
Alexey Milovidov
69e6e59e96 Add a test for #49708 2023-12-18 08:34:53 +01:00
Alexey Milovidov
b0eb670776
Merge pull request #57974 from ClickHouse/revert-stack-trace-changes 
Revert "Merge pull request #57907 from azat/system.stack_trace-rt_tgsigqueueinfo"
 2023-12-18 04:50:12 +01:00
Alexey Milovidov
58396c5546
Merge pull request #57218 from tntnatbry/issue-43666 
Issue 43666: Add skip_unavailable_shards as a setting for Distributed table.
 2023-12-18 04:48:57 +01:00
Alexey Milovidov
2859ac08c9 Revert "Merge pull request #57907 from azat/system.stack_trace-rt_tgsigqueueinfo" 
This reverts commit a33d6b0c96, reversing
changes made to cf031a1b8e.
 2023-12-18 02:01:14 +01:00
Igor Nikonov
af6401761e Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-17 13:32:53 +00:00
Robert Schulze
7f675ddf80
Merge pull request #57878 from Jpnock/master 
Apply full-text skipping index when using `hasAny()`
 2023-12-17 11:06:29 +01:00
Robert Schulze
fc67d2c0e9
Merge pull request #57867 from skyoct/read_code 
Support trivial count optimization for `Merge`-engine tables
 2023-12-17 10:45:21 +01:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Sergei Trifonov
3634b949a8
Merge pull request #57793 from ClickHouse/rename-system-table 
Rename `system.async_loader` into `system.asynchronous_loader`
 2023-12-16 17:33:04 +01:00
Nikolai Kochetov
288888f950 Another try. 2023-12-16 15:41:02 +00:00
Nikolai Kochetov
e7e5fc2891 Another attempt 2023-12-16 12:01:36 +00:00
Sergei Trifonov
37a01bc109
Merge branch 'master' into ft-dropped_tables_parts 2023-12-16 11:53:37 +01:00
Kseniia Sumarokova
4279dd2bf1
Merge pull request #57687 from ClickHouse/make-system-filesystem-cache-query-no-memory-intensive 
Make querying system.filesystem_cache not memory intensive
 2023-12-16 11:28:28 +01:00
Alexey Milovidov
ee502ef6b9 Remove trash (OPTIMIZE CLEANUP) 2023-12-15 22:41:58 +01:00
Igor Nikonov
a735820400 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-15 21:14:43 +00:00
Raúl Marín
6cf8c9b607 Review improvements 2023-12-15 19:44:13 +01:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
James Nock
9aa1ec488c Apply full-text skipping index when using hasAll() 2023-12-15 17:28:33 +00:00
Azat Khuzhin
106042cf41 Fix system.stack_trace for threads with blocked SIGRTMIN 
Some third-party libraries (i.e. librdkafka) could block it, and in this
case system.stack_trace will return stacktrace for the main process
(usually, basically it could be any thread with non blocked signal).

By replacing sigqueue() with more precise rt_tgsigqueueinfo(), other
threads will not respond to the signal.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-15 17:47:01 +01:00
flynn
82fc51ffa6 add test 2023-12-15 15:29:15 +00:00
Robert Schulze
066579f7d7
Improve code aesthetics 2023-12-15 15:07:40 +00:00
Nikita Mikhaylov
1780671443
ClickHouse Cloud promotion (#57638) 
Co-authored-by: Alexander Sapin <alesapin@gmail.com>
 2023-12-15 15:23:23 +01:00
Raúl Marín
946defa8a6
Merge pull request #57860 from zvonand/zvonand-fix-57819 
Try to fix memory leak in StorageHDFS
 2023-12-15 11:50:17 +01:00
kssenii
7bbb456ae9 Fix style check 2023-12-15 11:32:26 +01:00
kssenii
5de9c143a5 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-15 11:32:14 +01:00
Kseniia Sumarokova
f285a01b17
Merge pull request #57863 from ClickHouse/fix-flaky-pg-test 
Follow up to #57568
 2023-12-15 11:29:32 +01:00
Kseniia Sumarokova
06438cc17b
Merge pull request #57578 from ClickHouse/allow-to-change-some-cache-settings-without-restart 
Allow to apply some fs cache config settings changes without server restart
 2023-12-15 11:18:19 +01:00
flynn
2d45213f70 Fix transform query for external database 2023-12-15 08:34:58 +00:00
skyoct
3319934055 reafactor function 2023-12-15 04:49:38 +00:00
凌涛
57a5bef09e optimize code 2023-12-15 11:49:56 +08:00
凌涛
ebc570aef5 optimize code 2023-12-15 11:31:07 +08:00
凌涛
e36cd6a06e BloomFilter support match function 2023-12-15 10:56:52 +08:00
凌涛
18e29bc6a2 BloomFilter support match function 2023-12-15 10:46:30 +08:00
Alexey Milovidov
23cee7ef37
Merge pull request #57851 from azat/revert-55271-local-lazy-database 
Revert "Replace --no-system-tables with loading virtual tables of system database lazily"
 2023-12-15 03:25:59 +01:00
Andrey Zvonov
0174b94469
Merge branch 'ClickHouse:master' into zvonand-fix-57819 2023-12-14 20:14:54 +01:00
Igor Nikonov
1370a8c6c4 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-14 18:29:38 +00:00
Raúl Marín
546484d46b Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 19:03:59 +01:00
Duc Canh Le
93dd6b83e7
Do not allow creating replicated table with inconsistent merge params (#56833) 
* save all merge params to zookeeper

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* calculate hash for graphite merge params

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* add graphite params hash to zookeeper + fix tests

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* install new graphite for testing

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* fix backward incompatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* minor fix test

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* Update src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* remove peekString and add more comments

- peekString doesn't always work even for ReadBufferFromString
- more comment re. backward compatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

---------

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-12-14 17:26:35 +01:00
joelynch
b4bef78e63
bugfix: addresses_expr ignored for psql named collections 2023-12-14 17:17:14 +01:00
Andrey Zvonov
f9579580e9 Retrigger CI + make docs a bit better 2023-12-14 16:34:10 +01:00
kssenii
3605caa8d9 Fxi 2023-12-14 16:05:40 +01:00
kssenii
ea7e55b929 Simplify 2023-12-14 15:58:06 +01:00
skyoct
0a4882d781 feat: update style 2023-12-14 14:26:47 +00:00
skyoct
d700eb2501 feat: support trivial count optimization for Merge 2023-12-14 14:08:24 +00:00
kssenii
a4a264c785 Fix 2023-12-14 14:04:17 +01:00
Andrey Zvonov
7d7aea9826 try fix memory leak in StorageHDFS 2023-12-14 13:01:47 +01:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
Igor Nikonov
42d0afef1f
Merge pull request #57749 from ClickHouse/pr-perf-improve-hope 
Parallel replicas: announcement response handling improvement
 2023-12-14 11:48:51 +01:00
Raúl Marín
efcacd3c34 Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 11:08:50 +01:00
Azat Khuzhin
0e86e16a01
Revert "Replace --no-system-tables with loading virtual tables of system database lazily" 2023-12-14 09:07:47 +01:00
Alexey Milovidov
7a2edb4804
Merge pull request #57822 from azat/kafka-disable-stat 
Disable system.kafka_consumers by default (due to possible live memory leak)
 2023-12-14 04:37:50 +01:00
Yakov Olkhovskiy
9e1d583f54 fix style 2023-12-13 23:36:54 +00:00
Kseniia Sumarokova
79db3c66df
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-13 23:33:59 +01:00
Yakov Olkhovskiy
441050adce fix style, cleanup 2023-12-13 22:26:38 +00:00
Julia Kartseva
b8d274d070
Add malformed output generation to JSON fuzzer (#57646) 
Randomly modify structural characters of a valid JSON ('{', '}', '[', ']',
':', '"', ',') to generate output that cannot be parsed as JSON.

Follow-up to https://github.com/ClickHouse/ClickHouse/pull/56490
 2023-12-13 19:59:31 +01:00
Alexey Milovidov
bf4964b177
Merge pull request #57712 from ClickHouse/clickbench-ci 
Add ClickBench to CI
 2023-12-13 19:52:44 +01:00
Nikita Taranov
560e66f3ca
More respect to min_number_of_marks in ParallelReplicasReadingCoordinator (#57763) 2023-12-13 19:51:49 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Yakov Olkhovskiy
f4e2933f5d use storage uuid instead of table name as a key in storage info because there are can be multiple dropped table with the same name 2023-12-13 18:08:41 +00:00
Azat Khuzhin
055c231438 Disable system.kafka_consumers by default (due to possible live memory leak) 
It is not safe to use statistics because of how KafkaEngine works - it
pre-creates consumers, and this leads to the situation when this
statistics entries generated (RD_KAFKA_OP_STATS), but never consumed.

Which creates a live memory leak for a server with Kafka tables, but
without materialized view attached to it (and no SELECT).

Another problem is that this makes shutdown very slow, because of how
pending queue entries are handled in librdkafka, it uses
TAILQ_INSERT_SORTED, which is sorted insert into linked list, which
works incredibly slow (likely you will never wait till it ends and kill
the server)

For instance in my production setup the server was running for ~67 days
with such table, and it got 1'942'233 `TAILQ_INSERT_SORTED` entries
(which perfectly matches by the way - `67*86400/3` = 1'929'600), and it
moved only 289'806 entries for a few hours, though I'm not sure how much
time the process was in the running state, since most of the time it was
with debugger attached.

So for now let's disable it, to make this patch easy for backporting,
and I will think about long term fix - do not pre-create consumers in
Kafka engine.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-13 18:13:13 +01:00
Anton Popov
7598403f78 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-13 16:55:17 +00:00
Igor Nikonov
72f1c415f1 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-13 15:03:05 +00:00
Nikolai Kochetov
2085b5dd6f
Merge branch 'master' into analyzer-execution-names 2023-12-13 15:22:48 +01:00
Yakov Olkhovskiy
0ea4b16530
Merge branch 'master' into ft-dropped_tables_parts 2023-12-13 09:01:59 -05:00
Igor Nikonov
c165be76ab
Parallel replicas: friendly settings (#57542) 2023-12-13 14:42:06 +01:00
Raúl Marín
dd405a655e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-13 13:39:58 +01:00
Kseniia Sumarokova
b1325450e9
Merge pull request #57754 from ClickHouse/fix-loading-dependent-table-materialized-postgresql 
Table engine MaterializedPostgreSQL fix dependency loading
 2023-12-13 12:17:01 +01:00
Yakov Olkhovskiy
5121bfcd78 major refactoring 2023-12-13 07:16:30 +00:00
Alexey Milovidov
62b6d1ef5e iMerge branch 'master' of github.com:ClickHouse/ClickHouse into clickbench-ci 2023-12-13 01:41:27 +01:00
Alexey Milovidov
f11b90e7bb Allow buckets without List access 2023-12-13 00:10:58 +01:00
Igor Nikonov
616d054dee Merge remote-tracking branch 'origin/master' into pr-perf-improve-hope 2023-12-12 21:27:31 +00:00
Igor Nikonov
7f4a028196 Test and fix 2023-12-12 21:26:58 +00:00
Nikolai Kochetov
3e1dc6bcf9
Merge branch 'master' into analyzer-execution-names 2023-12-12 19:32:50 +01:00
Nikolai Kochetov
63533ad5bf Merge branch 'master' into analyzer-execution-names 2023-12-12 18:30:59 +00:00
serxa
57d7a58dbe Rename system.async_loader into system.asynchronous_loader 2023-12-12 16:55:27 +00:00
kssenii
cc71b40002 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 16:40:04 +01:00
Alexander Gololobov
09d6e6b945
Merge pull request #57743 from ClickHouse/other_timeout_for_wait 
Increase async block cache deduplication timeout
 2023-12-12 15:40:04 +01:00
Raúl Marín
049fb60c73 Fix error on retries due to TABLE_IS_READ_ONLY 2023-12-12 15:37:28 +01:00
kssenii
fa0b2b9e50 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 15:31:01 +01:00
avogar
096c55aece Fix tests 2023-12-12 14:13:15 +00:00
avogar
cbad0add9c Flatten only true Nested type if flatten_nested=1, not all Array(Tuple) 2023-12-12 14:13:15 +00:00
Chen Lixiang
68af81196d
Merge branch 'ClickHouse:master' into bytes_uncompressed 2023-12-12 22:06:45 +08:00
Kruglov Pavel
8a447bf57c
Merge pull request #55892 from Avogar/schema-inference-union 
Add 'union' mode for schema inference
 2023-12-12 15:02:06 +01:00
Nikolai Kochetov
696a2d9f4f
Merge pull request #57614 from ClickHouse/vdimir/join_array_lowcard_totals_crash 
Fix type correction in HashJoin for nested low cardinality
 2023-12-12 14:56:41 +01:00
Raúl Marín
923c3b7b32 Implement retries when ZK connection fails without committing the transaction 2023-12-12 12:00:46 +01:00
Raúl Marín
9d8d5dfdf6 Partially revert "make stages commit" 
This reverts commit e4becc01ba.
 2023-12-12 11:26:48 +01:00
Chen Lixiang
0b89fbbdd9 fix style issue 2023-12-12 16:49:58 +08:00
Zhuo Qiu
e34c13b9d9 Consider lightweight deleted rows when selecting parts to merge 2023-12-12 11:40:19 +08:00
alesapin
7cdc3d29fb Get rid of tiny function 2023-12-11 22:29:51 +01:00
alesapin
bfc2a7bb75 Update cache only after conflicts 2023-12-11 21:27:48 +01:00
alesapin
4b3e845dab Merge branch 'master' into other_timeout_for_wait 2023-12-11 20:43:21 +01:00
Igor Nikonov
30dabd42d3 One lookup for 2 checks 
(1) if it's the same part
(2) if annouced part has intersections with parts in working set
 2023-12-11 19:37:42 +00:00
kssenii
05e56f76ed Merge remote-tracking branch 'origin/master' into fix-loading-dependent-table-materialized-postgresql 2023-12-11 20:17:26 +01:00
Kseniia Sumarokova
907d0a79eb
Merge pull request #57568 from ClickHouse/fix-materialized-pg-issue-with-generated-columns 
MaterializedPostgreSQL: Support "generated columns" and default values
 2023-12-11 19:57:41 +01:00
Kruglov Pavel
3b1ffbd77c
Merge branch 'master' into schema-inference-union 2023-12-11 18:59:15 +01:00
kssenii
48ce040621 Fix 2023-12-11 17:30:23 +01:00
Raúl Marín
e1965bb6b5 WIP: Remove UNCERTAIN_COMMIT in INSERT 2023-12-11 17:27:56 +01:00
vdimir
ea09080c73
fix 2023-12-11 15:50:27 +00:00
Raúl Marín
e6be38be4d Adapt from HEAD 2023-12-11 16:28:00 +01:00
Raúl Marín
a55a0c068e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-11 15:27:21 +01:00
Igor Nikonov
4ccf4e11a8 Parallel replicas (perf): announcement response handling improvement 2023-12-11 13:00:08 +00:00
kssenii
1e4d61d55a Change error code 2023-12-11 13:44:12 +01:00
kssenii
b8caa00d24 Commit forgotten file 2023-12-11 13:28:36 +01:00
alesapin
a62a0b92de Increase async block cache deduplication timeout 2023-12-11 12:46:22 +01:00
kssenii
966a09fb70 Review fixes 2023-12-11 12:42:23 +01:00
kssenii
a06179da00 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-11 11:15:00 +01:00
Vitaly Baranov
f1ccf4e29f Show owner query ids in the message for the DEADLOCK_AVOIDED error. 2023-12-11 00:56:17 +01:00
Alexey Milovidov
644ef7b63f
Revert "Fix RWLock inconsistency after write lock timeout" 2023-12-10 23:51:59 +03:00
Igor Nikonov
f7e6a17052 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-10 20:21:47 +00:00
Alexey Milovidov
0a20ce5d32
Merge pull request #55271 from azat/local-lazy-database 
Replace --no-system-tables with loading virtual tables of system database lazily
 2023-12-10 19:51:17 +01:00
Alexey Milovidov
d119052895
Merge pull request #51858 from ClickHouse/CurtizJ-fix-ttl-non-deterministic 
Merging #37286
 2023-12-10 17:50:36 +01:00
Vitaly Baranov
a058a26b7e
Merge pull request #57454 from vitlibar/fix-rwlock 
Fix RWLock inconsistency after write lock timeout
 2023-12-10 14:09:29 +01:00
Alexey Milovidov
100b36e8a9 Do not check in replication 2023-12-10 01:32:42 +01:00
Alexey Milovidov
33b0ec7d55 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-10 01:21:43 +01:00
Azat Khuzhin
4f9d90761b Attach tables lazily only for clickhouse-local 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
c14721bb03 Move lazy table initialization interface into IDatabase 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
cc9afe6f6e Replace --no-system-tables with loading virtual tables of system database lazily 
First of all the problem is that that simple 'SELECT 1' cannot be runned
without system.one, which makes --no-system-tables almost useless:

    $ ./clickhouse-debug local --no-system-tables -q "select 1"
    Code: 81. DB::Exception: Database system does not exist. (UNKNOWN_DATABASE)

Secondly there are just too much flags, and this one
(--no-system-tables) is too damn specific.

This patch should improve startup time of the clickhouse-local almost
3x in debug builds.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Alexey Milovidov
c77183a597
Merge pull request #57480 from azat/dist/async-INSERT-fixes 
Fix possible distributed sends stuck due to "No such file or directory" (during recovering batch from disk)
 2023-12-09 17:11:35 +01:00
Alexey Milovidov
56cd9196b0
Merge pull request #57641 from azat/system.stack_trace-fix 
Avoid hangs of system.stack_trace by correctly prohibit parallel read from it
 2023-12-09 17:07:48 +01:00
Chen Lixiang
7cd770e9a7 fix doc and comments 2023-12-09 20:09:41 +08:00
Nikolay Degterinsky
c5e52d260f
Merge pull request #57553 from evillique/fix-local-crash 
Fix crash in clickhouse-local
 2023-12-08 22:57:52 +01:00
Gagan Goel
e547db0a8c Issue 43666: Add skip_unavailable_shards as a setting for Distributed table. 
This setting, when enabled (disabled by default), allows ClickHouse to
silently skip unavailable shards of a Distributed table during a query
execution, instead of throwing an exception to the client.
 2023-12-08 15:43:59 -05:00
avogar
ee7af95bc0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-12-08 20:29:28 +00:00
Azat Khuzhin
f876bea050 Add support for system.stack_trace filtering optimizations for analyzer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 18:13:36 +01:00
Nikita Mikhaylov
c979124cfe Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-08 16:25:02 +00:00
Azat Khuzhin
b1043b7ad3 Increase log verbosity of some messages in system.stack_trace 
This can help to debug 01051_system_stack_trace test failures.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 16:18:49 +01:00
Kseniia Sumarokova
928993f2bd
Merge branch 'master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 15:37:29 +01:00
Kseniia Sumarokova
bd3665863c
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-08 15:36:40 +01:00
zhongyuankai
7b0f8d44e8
Make DirectoryMonitor handle cluster node list change (#42826) 2023-12-08 14:41:51 +01:00
kssenii
d77938c3ff Add validation of columns 2023-12-08 12:42:38 +01:00
kssenii
ffd0d16f79 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 12:42:21 +01:00
Azat Khuzhin
16cbcf700e Prohibit parallel read from system.stack_trace 
Before rewriting system.stack_trace to handle max_block_size (in #54946)
parallel reading from system.stack_trace was prohibited, because this
could lead to hang of system.stack_trace table.

But that rewrite broke this guarantee, so let's fix it to avoid possible
hung.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 11:05:25 +01:00
Alexey Milovidov
ce9e134043 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-08 03:10:09 +01:00
alesapin
46f04d6d0e Throw more clear exception 2023-12-07 17:15:39 +01:00
Igor Nikonov
94fb23af0b Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-07 11:52:40 +00:00
kssenii
8be3c9d218 Merge remote-tracking branch 'origin/master' into allow-to-change-some-cache-settings-without-restart 2023-12-07 12:14:24 +01:00
kssenii
614da21144 Better 2023-12-07 12:12:10 +01:00
Chen Lixiang
9e95fe5d29 conflict fixed. 2023-12-07 18:10:45 +08:00
Kseniia Sumarokova
b3f22449f6
Merge pull request #57515 from ClickHouse/fix-materialized-pg-issue-with-incorrect-connection-options 
MaterializedPostgreSQL: fix issue #41922, add test for #41923
 2023-12-07 10:42:14 +01:00
Alexey Milovidov
b85214ca1a
Merge pull request #57546 from rschu1ze/revert-revert-support_alias_column_in_indices 
Revert "Revert "Support use alias column in indices""
 2023-12-07 01:22:03 +01:00
Alexey Milovidov
5fe1b8926c
Merge pull request #57595 from ClickHouse/23.12-prepare 
Update version after release
 2023-12-06 23:53:53 +01:00