thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,599 Commits 9,328 Branches 1,783 Tags 2.1 GiB
852f397a97
Commit Graph

15616 Commits

Author SHA1 Message Date
Azat Khuzhin
2ff0bfb0a1 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
db74549940 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
e7592c140e Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
51d4f583e6 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Azat Khuzhin
123d63e824 Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-27 09:49:07 +01:00
Alexey Milovidov
a0fccb0498
Merge pull request #58224 from amosbird/part_offset_pk 
Primary key analysis for _part_offset
 2023-12-26 14:51:57 +01:00
Alexey Milovidov
31a081bd83
Merge pull request #58226 from Algunenano/cleanup_known_short 
Cleanup some known short messages
 2023-12-26 14:40:58 +01:00
Raúl Marín
e87b9751bd Cleanup some known short messages 2023-12-26 12:58:50 +01:00
Amos Bird
66660ee4e2
Add comment 2023-12-26 17:04:00 +08:00
Amos Bird
bfcccf9fa3
Primary key analysis for _part_offset 2023-12-26 17:03:59 +08:00
santrancisco
a59d874bf9
fix syntax 2023-12-26 16:56:58 +11:00
Azat Khuzhin
3be3b0a280 Fix incorrect Exceptions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 21:26:32 +01:00
Alexey Milovidov
ae51334ba5 Merge branch 'master' into fix-error-in-archive-reader 2023-12-24 05:53:22 +01:00
Alexey Milovidov
e98c49a58f Fix a benign error in archive reader 2023-12-24 05:44:24 +01:00
Alexey Milovidov
3f4c8e4ae8
Merge pull request #58167 from jrdi/part-log-uncompressed-bytes 
Add bytes_uncompressed to system.part_log
 2023-12-24 04:11:35 +01:00
Alexey Milovidov
b4bf1d1c4c
Merge pull request #58136 from azat/system.stack_trace-rt_tgsigqueueinfo-v2 
Fix system.stack_trace for threads with blocked SIGRTMIN (resubmit)
 2023-12-24 03:51:13 +01:00
Alexey Milovidov
4f3f69521d
Merge pull request #58173 from ClickHouse/parallel-replicas-used-count 
Profile event 'ParallelReplicasUsedCount'
 2023-12-24 03:46:09 +01:00
Alexey Milovidov
00fa9085b1
Merge pull request #58178 from chhetripradeep/add-base-backup-name-to-system-tables 
Add base backup name to system.backups and system.backup_log tables
 2023-12-24 03:38:20 +01:00
Azat Khuzhin
2f6c0487ad Ignore ENOENT for SigBlk check for system.stack_trace 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-23 14:35:38 +01:00
Azat Khuzhin
ac542199c5 Add some comments about racy code for system.stack_trace 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-23 13:42:26 +01:00
Igor Nikonov
d644a208bf Merge remote-tracking branch 'origin/master' into parallel-replicas-used-count 2023-12-23 11:02:28 +00:00
Igor Nikonov
3a485a8bbf Fix:moved request object was used 2023-12-23 11:02:24 +00:00
Alexey Milovidov
dc4b9a1013 Obfuscator: keep settings and timezones 2023-12-23 04:55:55 +01:00
Pradeep Chhetri
b5c8c4050b Add base backup name to system.backups and system.backup_log tables 2023-12-23 11:08:50 +08:00
Jordi Villar
bff0b9c790 Fix mutations new part uncompressed bytes 2023-12-22 22:33:58 +01:00
Igor Nikonov
1deafa1a00 Profile event 'ParallelReplicasUsedCount' 2023-12-22 20:54:52 +00:00
Jordi Villar
b4c3969d3a Add bytes_uncompressed to system.part_log 2023-12-22 18:35:33 +01:00
Alexey Milovidov
08ff37f64e
Merge pull request #57682 from azat/system.stack_trace/analyzer 
Add support for system.stack_trace filtering optimizations for analyzer
 2023-12-22 16:28:28 +01:00
Azat Khuzhin
d29762f19f Do not send signals to threads that blocks SIGRTMIN for system.stack_trace 
That way we can avoid superior timeouts during reading from
system.stack_trace.

Two known cases of such threads are:
- rdk: -- librdkafka threads
- iou-wrk -- io_uring threads

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-22 12:41:20 +01:00
Azat Khuzhin
aa5a6449f0 Fix system.stack_trace for threads with blocked SIGRTMIN 
Some third-party libraries (i.e. librdkafka) could block it, and in this
case system.stack_trace will return stacktrace for the main process
(usually, basically it could be any thread with non blocked signal).

By replacing sigqueue() with more precise rt_tgsigqueueinfo(), other
threads will not respond to the signal.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 106042cf41)
 2023-12-21 19:41:56 +01:00
Raúl Marín
2e522b9405 Remove requestUnconditionalRetry 
It's confusing, only me used it and I used it wrong
 2023-12-21 19:19:50 +01:00
Raúl Marín
ceed935b30 Remove debug comment and fix unconditional_retry logic 2023-12-21 17:11:34 +01:00
Raúl Marín
6d9da8edd5 Merge remote-tracking branch 'blessed/master' into zk_retries_quorum 2023-12-21 17:03:29 +01:00
Alexey Milovidov
ab6a712ec6 Add compatibility in the replication protocol for a removed feature 2023-12-21 09:31:59 +01:00
Alexey Milovidov
7be39a27cc
Revert "Consider lightweight deleted rows when selecting parts to merge" 2023-12-21 01:46:38 +03:00
Kruglov Pavel
ca7001d70f
Merge pull request #57850 from shanielh/feature/s3-session-token 
Feature/s3 session token
 2023-12-20 17:22:09 +01:00
Anton Popov
87d0cec055
Merge pull request #57433 from CurtizJ/materialize-lightweight-delete 
Add mutation command `APPLY DELETED MASK`
 2023-12-20 14:44:11 +01:00
Alexey Milovidov
af32b33e93
Merge pull request #57648 from jewelzqiu/refine-lwd-merge 
Consider lightweight deleted rows when selecting parts to merge
 2023-12-20 01:10:32 +01:00
Yakov Olkhovskiy
43fcea64b6
Merge pull request #57186 from chenlx0/bytes_uncompressed 
Show uncompressed data size in system.tables
 2023-12-19 19:04:47 -05:00
Jordi Villar
7c281d9a6f
Allow max_size_to_drop settings in query time (#57452) 2023-12-19 23:16:03 +01:00
jsc0218
0a9475abc7
Merge pull request #57874 from joelynch/joelynch/host-psql 
bugfix: addresses_expr ignored for psql named collections
 2023-12-19 15:24:23 -05:00
Julia Kartseva
c7eff0e573
Merge pull request #58015 from jkartseva/fix-fuzzjson-segfault 
Fix segfault in FuzzJSON engine
 2023-12-19 11:19:20 -08:00
Julia Kartseva
58f75c9679
Update src/Storages/StorageFuzzJSON.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2023-12-19 10:17:15 -08:00
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint 
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
 2023-12-19 19:16:01 +01:00
Nikita Taranov
a375b1eab8
Fix handling of unavailable replicas before first request happened (#57933) 2023-12-19 16:41:51 +01:00
Raúl Marín
b458c77340 Fix integration test 2023-12-19 14:36:03 +01:00
Antonio Andelic
52b3caec90
Revert "Add system.dropped_tables_parts table" 2023-12-19 10:36:22 +01:00
Shani Elharrar
679a0e1300 StorageS3 / TableFunctionS3: Allow passing session_token to AuthSettings 
This can help users that want to pass temporary credentials that
issued by AWS in order to load data from S3 without changing
configuration or creating an IAM User.

Fixes #57848
 2023-12-19 08:06:36 +02:00
Julia Kartseva
6014dca114 Allow multiple columns in StorageFuzzJSON 2023-12-19 02:34:23 +00:00
Julia Kartseva
0fc402c106 Fix segfault in FuzzJSON engine 
Allow only String type for FuzzJSON engine table columns.

Fixes: https://github.com/ClickHouse/ClickHouse/issues/57858
 2023-12-19 02:20:37 +00:00
Anton Popov
bfd403cc8c
Merge pull request #57996 from CurtizJ/better-trivial-count-merge 
Better trivial count optimization for storage `Merge`
 2023-12-19 02:25:58 +01:00
Yakov Olkhovskiy
fd460566f6
fix setting description 2023-12-18 19:34:08 -05:00
Han Fei
032fd08559
Merge pull request #57991 from hanfei1991/hanfei/refine-err-msg 
refine error message
 2023-12-19 01:14:18 +01:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Alexey Milovidov
1a7de9158f
Merge pull request #57979 from ClickHouse/add-test-49708 
Add a test for #49708
 2023-12-18 21:29:11 +01:00
Alexey Milovidov
7178bd5c88
Merge pull request #57910 from Algunenano/log_message_string 
Improvements for 00002_log_and_exception_messages_formatting
 2023-12-18 21:02:23 +01:00
Alexander Tokmakov
de7a349f0d
Update MergeTreeData.cpp 2023-12-18 20:44:50 +01:00
Raúl Marín
46067adce8 Improve ZooKeeperRetriesControl so it's easier to reuse 2023-12-18 17:43:10 +01:00
Yakov Olkhovskiy
8e39c61401 Merge remote-tracking branch 'origin/master' into ft-dropped_tables_parts 2023-12-18 15:40:07 +00:00
Anton Popov
7ebb7d2955 better trivial count optimization for storage Merge 2023-12-18 15:13:43 +00:00
Anton Popov
10bc2db56d add setting 2023-12-18 13:57:07 +00:00
Raúl Marín
88dcaefde8 Style 2023-12-18 14:51:46 +01:00
Han Fei
028d1c9478 refine error message 2023-12-18 14:47:54 +01:00
Raúl Marín
0346850897 Support keeper failures in quorum check 2023-12-18 12:54:15 +01:00
Raúl Marín
2639d0715f Merge remote-tracking branch 'blessed/master' into log_message_string 2023-12-18 10:40:18 +01:00
vdimir
f024e39e3b
Merge pull request #57888 from ucasfl/fix-transfer-external-query 
Fix transform query for external database
 2023-12-18 10:28:32 +01:00
Raúl Marín
f10dae4761
Merge pull request #56917 from Algunenano/backup_1 
Prepare the introduction of more keeper faults
 2023-12-18 10:27:48 +01:00
Alexey Milovidov
69e6e59e96 Add a test for #49708 2023-12-18 08:34:53 +01:00
Alexey Milovidov
b0eb670776
Merge pull request #57974 from ClickHouse/revert-stack-trace-changes 
Revert "Merge pull request #57907 from azat/system.stack_trace-rt_tgsigqueueinfo"
 2023-12-18 04:50:12 +01:00
Alexey Milovidov
58396c5546
Merge pull request #57218 from tntnatbry/issue-43666 
Issue 43666: Add skip_unavailable_shards as a setting for Distributed table.
 2023-12-18 04:48:57 +01:00
Alexey Milovidov
2859ac08c9 Revert "Merge pull request #57907 from azat/system.stack_trace-rt_tgsigqueueinfo" 
This reverts commit a33d6b0c96, reversing
changes made to cf031a1b8e.
 2023-12-18 02:01:14 +01:00
Robert Schulze
7f675ddf80
Merge pull request #57878 from Jpnock/master 
Apply full-text skipping index when using `hasAny()`
 2023-12-17 11:06:29 +01:00
Robert Schulze
fc67d2c0e9
Merge pull request #57867 from skyoct/read_code 
Support trivial count optimization for `Merge`-engine tables
 2023-12-17 10:45:21 +01:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Sergei Trifonov
3634b949a8
Merge pull request #57793 from ClickHouse/rename-system-table 
Rename `system.async_loader` into `system.asynchronous_loader`
 2023-12-16 17:33:04 +01:00
Sergei Trifonov
37a01bc109
Merge branch 'master' into ft-dropped_tables_parts 2023-12-16 11:53:37 +01:00
Kseniia Sumarokova
4279dd2bf1
Merge pull request #57687 from ClickHouse/make-system-filesystem-cache-query-no-memory-intensive 
Make querying system.filesystem_cache not memory intensive
 2023-12-16 11:28:28 +01:00
Alexey Milovidov
ee502ef6b9 Remove trash (OPTIMIZE CLEANUP) 2023-12-15 22:41:58 +01:00
Raúl Marín
6cf8c9b607 Review improvements 2023-12-15 19:44:13 +01:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
James Nock
9aa1ec488c Apply full-text skipping index when using hasAll() 2023-12-15 17:28:33 +00:00
Azat Khuzhin
106042cf41 Fix system.stack_trace for threads with blocked SIGRTMIN 
Some third-party libraries (i.e. librdkafka) could block it, and in this
case system.stack_trace will return stacktrace for the main process
(usually, basically it could be any thread with non blocked signal).

By replacing sigqueue() with more precise rt_tgsigqueueinfo(), other
threads will not respond to the signal.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-15 17:47:01 +01:00
flynn
82fc51ffa6 add test 2023-12-15 15:29:15 +00:00
Robert Schulze
066579f7d7
Improve code aesthetics 2023-12-15 15:07:40 +00:00
Nikita Mikhaylov
1780671443
ClickHouse Cloud promotion (#57638) 
Co-authored-by: Alexander Sapin <alesapin@gmail.com>
 2023-12-15 15:23:23 +01:00
Raúl Marín
946defa8a6
Merge pull request #57860 from zvonand/zvonand-fix-57819 
Try to fix memory leak in StorageHDFS
 2023-12-15 11:50:17 +01:00
kssenii
7bbb456ae9 Fix style check 2023-12-15 11:32:26 +01:00
kssenii
5de9c143a5 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-15 11:32:14 +01:00
Kseniia Sumarokova
f285a01b17
Merge pull request #57863 from ClickHouse/fix-flaky-pg-test 
Follow up to #57568
 2023-12-15 11:29:32 +01:00
Kseniia Sumarokova
06438cc17b
Merge pull request #57578 from ClickHouse/allow-to-change-some-cache-settings-without-restart 
Allow to apply some fs cache config settings changes without server restart
 2023-12-15 11:18:19 +01:00
flynn
2d45213f70 Fix transform query for external database 2023-12-15 08:34:58 +00:00
skyoct
3319934055 reafactor function 2023-12-15 04:49:38 +00:00
Alexey Milovidov
23cee7ef37
Merge pull request #57851 from azat/revert-55271-local-lazy-database 
Revert "Replace --no-system-tables with loading virtual tables of system database lazily"
 2023-12-15 03:25:59 +01:00
Andrey Zvonov
0174b94469
Merge branch 'ClickHouse:master' into zvonand-fix-57819 2023-12-14 20:14:54 +01:00
Raúl Marín
546484d46b Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 19:03:59 +01:00
Duc Canh Le
93dd6b83e7
Do not allow creating replicated table with inconsistent merge params (#56833) 
* save all merge params to zookeeper

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* calculate hash for graphite merge params

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* add graphite params hash to zookeeper + fix tests

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* install new graphite for testing

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* fix backward incompatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* minor fix test

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* Update src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* remove peekString and add more comments

- peekString doesn't always work even for ReadBufferFromString
- more comment re. backward compatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

---------

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-12-14 17:26:35 +01:00
joelynch
b4bef78e63
bugfix: addresses_expr ignored for psql named collections 2023-12-14 17:17:14 +01:00
Andrey Zvonov
f9579580e9 Retrigger CI + make docs a bit better 2023-12-14 16:34:10 +01:00
kssenii
3605caa8d9 Fxi 2023-12-14 16:05:40 +01:00
kssenii
ea7e55b929 Simplify 2023-12-14 15:58:06 +01:00