thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
132,656 Commits 9,328 Branches 1,783 Tags 2.1 GiB
9cfdff2ddb
Commit Graph

15874 Commits

Author SHA1 Message Date
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint 
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
 2023-12-19 19:16:01 +01:00
Yakov Olkhovskiy
3d8933f241
Revert "Revert "Add system.dropped_tables_parts table"" 2023-12-19 10:44:44 -05:00
Nikita Taranov
a375b1eab8
Fix handling of unavailable replicas before first request happened (#57933) 2023-12-19 16:41:51 +01:00
skyoct
00576d2092 add onlyDecrease to enums 2023-12-19 15:23:20 +00:00
Raúl Marín
b458c77340 Fix integration test 2023-12-19 14:36:03 +01:00
skyoct
62b8486964
Merge branch 'master' into feat/server_settings 2023-12-19 20:37:15 +08:00
skyoct
267b35ff67 feat: add server setting config 2023-12-19 12:24:32 +00:00
Robert Schulze
6df2548417
Some minor adjustments 2023-12-19 10:39:32 +00:00
Antonio Andelic
52b3caec90
Revert "Add system.dropped_tables_parts table" 2023-12-19 10:36:22 +01:00
Shani Elharrar
679a0e1300 StorageS3 / TableFunctionS3: Allow passing session_token to AuthSettings 
This can help users that want to pass temporary credentials that
issued by AWS in order to load data from S3 without changing
configuration or creating an IAM User.

Fixes #57848
 2023-12-19 08:06:36 +02:00
Julia Kartseva
6014dca114 Allow multiple columns in StorageFuzzJSON 2023-12-19 02:34:23 +00:00
Julia Kartseva
0fc402c106 Fix segfault in FuzzJSON engine 
Allow only String type for FuzzJSON engine table columns.

Fixes: https://github.com/ClickHouse/ClickHouse/issues/57858
 2023-12-19 02:20:37 +00:00
Anton Popov
bfd403cc8c
Merge pull request #57996 from CurtizJ/better-trivial-count-merge 
Better trivial count optimization for storage `Merge`
 2023-12-19 02:25:58 +01:00
Yakov Olkhovskiy
fd460566f6
fix setting description 2023-12-18 19:34:08 -05:00
Han Fei
032fd08559
Merge pull request #57991 from hanfei1991/hanfei/refine-err-msg 
refine error message
 2023-12-19 01:14:18 +01:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Alexey Milovidov
1a7de9158f
Merge pull request #57979 from ClickHouse/add-test-49708 
Add a test for #49708
 2023-12-18 21:29:11 +01:00
Alexey Milovidov
7178bd5c88
Merge pull request #57910 from Algunenano/log_message_string 
Improvements for 00002_log_and_exception_messages_formatting
 2023-12-18 21:02:23 +01:00
Alexander Tokmakov
de7a349f0d
Update MergeTreeData.cpp 2023-12-18 20:44:50 +01:00
Raúl Marín
46067adce8 Improve ZooKeeperRetriesControl so it's easier to reuse 2023-12-18 17:43:10 +01:00
Yakov Olkhovskiy
8e39c61401 Merge remote-tracking branch 'origin/master' into ft-dropped_tables_parts 2023-12-18 15:40:07 +00:00
Anton Popov
7ebb7d2955 better trivial count optimization for storage Merge 2023-12-18 15:13:43 +00:00
Anton Popov
10bc2db56d add setting 2023-12-18 13:57:07 +00:00
Raúl Marín
88dcaefde8 Style 2023-12-18 14:51:46 +01:00
Han Fei
028d1c9478 refine error message 2023-12-18 14:47:54 +01:00
skyoct
9b628aa9d9 feat: add server setting runtime reload 2023-12-18 13:25:43 +00:00
Igor Nikonov
52db759c92 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-18 11:59:13 +00:00
Raúl Marín
0346850897 Support keeper failures in quorum check 2023-12-18 12:54:15 +01:00
Raúl Marín
2639d0715f Merge remote-tracking branch 'blessed/master' into log_message_string 2023-12-18 10:40:18 +01:00
vdimir
f024e39e3b
Merge pull request #57888 from ucasfl/fix-transfer-external-query 
Fix transform query for external database
 2023-12-18 10:28:32 +01:00
Raúl Marín
f10dae4761
Merge pull request #56917 from Algunenano/backup_1 
Prepare the introduction of more keeper faults
 2023-12-18 10:27:48 +01:00
凌涛
5c2a5759a8
Merge branch 'master' into optimization/BF_support_rg 2023-12-18 16:03:44 +08:00
凌涛
93b18a32c3 add tests 2023-12-18 16:02:19 +08:00
Alexey Milovidov
69e6e59e96 Add a test for #49708 2023-12-18 08:34:53 +01:00
Alexey Milovidov
b0eb670776
Merge pull request #57974 from ClickHouse/revert-stack-trace-changes 
Revert "Merge pull request #57907 from azat/system.stack_trace-rt_tgsigqueueinfo"
 2023-12-18 04:50:12 +01:00
Alexey Milovidov
58396c5546
Merge pull request #57218 from tntnatbry/issue-43666 
Issue 43666: Add skip_unavailable_shards as a setting for Distributed table.
 2023-12-18 04:48:57 +01:00
Alexey Milovidov
2859ac08c9 Revert "Merge pull request #57907 from azat/system.stack_trace-rt_tgsigqueueinfo" 
This reverts commit a33d6b0c96, reversing
changes made to cf031a1b8e.
 2023-12-18 02:01:14 +01:00
Igor Nikonov
af6401761e Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-17 13:32:53 +00:00
Robert Schulze
7f675ddf80
Merge pull request #57878 from Jpnock/master 
Apply full-text skipping index when using `hasAny()`
 2023-12-17 11:06:29 +01:00
Robert Schulze
fc67d2c0e9
Merge pull request #57867 from skyoct/read_code 
Support trivial count optimization for `Merge`-engine tables
 2023-12-17 10:45:21 +01:00
Shani Elharrar
c696c0bfe7 S3Common.AuthSettings: Allow passing SESSION_TOKEN to AWSCredentials 
This sets the infrastructure of loading session_token and passing it directly
to all AWSCredentials instances that are created using the AuthSettings.

The default SESSION_TOKEN is set to an empty string as documented in AWS SDK
reference: https://sdk.amazonaws.com/cpp/api/0.12.9/d4/d27/class_aws_1_1_auth_1_1_a_w_s_credentials.html
 2023-12-17 10:29:15 +02:00
Sergei Trifonov
3634b949a8
Merge pull request #57793 from ClickHouse/rename-system-table 
Rename `system.async_loader` into `system.asynchronous_loader`
 2023-12-16 17:33:04 +01:00
Nikolai Kochetov
288888f950 Another try. 2023-12-16 15:41:02 +00:00
Nikolai Kochetov
e7e5fc2891 Another attempt 2023-12-16 12:01:36 +00:00
Sergei Trifonov
37a01bc109
Merge branch 'master' into ft-dropped_tables_parts 2023-12-16 11:53:37 +01:00
Kseniia Sumarokova
4279dd2bf1
Merge pull request #57687 from ClickHouse/make-system-filesystem-cache-query-no-memory-intensive 
Make querying system.filesystem_cache not memory intensive
 2023-12-16 11:28:28 +01:00
Alexey Milovidov
ee502ef6b9 Remove trash (OPTIMIZE CLEANUP) 2023-12-15 22:41:58 +01:00
Igor Nikonov
a735820400 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-15 21:14:43 +00:00
Raúl Marín
6cf8c9b607 Review improvements 2023-12-15 19:44:13 +01:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
James Nock
9aa1ec488c Apply full-text skipping index when using hasAll() 2023-12-15 17:28:33 +00:00
Azat Khuzhin
106042cf41 Fix system.stack_trace for threads with blocked SIGRTMIN 
Some third-party libraries (i.e. librdkafka) could block it, and in this
case system.stack_trace will return stacktrace for the main process
(usually, basically it could be any thread with non blocked signal).

By replacing sigqueue() with more precise rt_tgsigqueueinfo(), other
threads will not respond to the signal.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-15 17:47:01 +01:00
flynn
82fc51ffa6 add test 2023-12-15 15:29:15 +00:00
Robert Schulze
066579f7d7
Improve code aesthetics 2023-12-15 15:07:40 +00:00
Nikita Mikhaylov
1780671443
ClickHouse Cloud promotion (#57638) 
Co-authored-by: Alexander Sapin <alesapin@gmail.com>
 2023-12-15 15:23:23 +01:00
Raúl Marín
946defa8a6
Merge pull request #57860 from zvonand/zvonand-fix-57819 
Try to fix memory leak in StorageHDFS
 2023-12-15 11:50:17 +01:00
kssenii
7bbb456ae9 Fix style check 2023-12-15 11:32:26 +01:00
kssenii
5de9c143a5 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-15 11:32:14 +01:00
Kseniia Sumarokova
f285a01b17
Merge pull request #57863 from ClickHouse/fix-flaky-pg-test 
Follow up to #57568
 2023-12-15 11:29:32 +01:00
Kseniia Sumarokova
06438cc17b
Merge pull request #57578 from ClickHouse/allow-to-change-some-cache-settings-without-restart 
Allow to apply some fs cache config settings changes without server restart
 2023-12-15 11:18:19 +01:00
flynn
2d45213f70 Fix transform query for external database 2023-12-15 08:34:58 +00:00
skyoct
3319934055 reafactor function 2023-12-15 04:49:38 +00:00
凌涛
57a5bef09e optimize code 2023-12-15 11:49:56 +08:00
凌涛
ebc570aef5 optimize code 2023-12-15 11:31:07 +08:00
凌涛
e36cd6a06e BloomFilter support match function 2023-12-15 10:56:52 +08:00
凌涛
18e29bc6a2 BloomFilter support match function 2023-12-15 10:46:30 +08:00
Alexey Milovidov
23cee7ef37
Merge pull request #57851 from azat/revert-55271-local-lazy-database 
Revert "Replace --no-system-tables with loading virtual tables of system database lazily"
 2023-12-15 03:25:59 +01:00
Andrey Zvonov
0174b94469
Merge branch 'ClickHouse:master' into zvonand-fix-57819 2023-12-14 20:14:54 +01:00
Igor Nikonov
1370a8c6c4 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-14 18:29:38 +00:00
Raúl Marín
546484d46b Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 19:03:59 +01:00
Duc Canh Le
93dd6b83e7
Do not allow creating replicated table with inconsistent merge params (#56833) 
* save all merge params to zookeeper

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* calculate hash for graphite merge params

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* add graphite params hash to zookeeper + fix tests

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* install new graphite for testing

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* fix backward incompatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* minor fix test

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

* Update src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* remove peekString and add more comments

- peekString doesn't always work even for ReadBufferFromString
- more comment re. backward compatibility

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>

---------

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-12-14 17:26:35 +01:00
joelynch
b4bef78e63
bugfix: addresses_expr ignored for psql named collections 2023-12-14 17:17:14 +01:00
Andrey Zvonov
f9579580e9 Retrigger CI + make docs a bit better 2023-12-14 16:34:10 +01:00
kssenii
3605caa8d9 Fxi 2023-12-14 16:05:40 +01:00
kssenii
ea7e55b929 Simplify 2023-12-14 15:58:06 +01:00
skyoct
0a4882d781 feat: update style 2023-12-14 14:26:47 +00:00
skyoct
d700eb2501 feat: support trivial count optimization for Merge 2023-12-14 14:08:24 +00:00
kssenii
a4a264c785 Fix 2023-12-14 14:04:17 +01:00
Andrey Zvonov
7d7aea9826 try fix memory leak in StorageHDFS 2023-12-14 13:01:47 +01:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
Igor Nikonov
42d0afef1f
Merge pull request #57749 from ClickHouse/pr-perf-improve-hope 
Parallel replicas: announcement response handling improvement
 2023-12-14 11:48:51 +01:00
Raúl Marín
efcacd3c34 Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 11:08:50 +01:00
Azat Khuzhin
0e86e16a01
Revert "Replace --no-system-tables with loading virtual tables of system database lazily" 2023-12-14 09:07:47 +01:00
Alexey Milovidov
7a2edb4804
Merge pull request #57822 from azat/kafka-disable-stat 
Disable system.kafka_consumers by default (due to possible live memory leak)
 2023-12-14 04:37:50 +01:00
Yakov Olkhovskiy
9e1d583f54 fix style 2023-12-13 23:36:54 +00:00
Kseniia Sumarokova
79db3c66df
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-13 23:33:59 +01:00
Yakov Olkhovskiy
441050adce fix style, cleanup 2023-12-13 22:26:38 +00:00
Julia Kartseva
b8d274d070
Add malformed output generation to JSON fuzzer (#57646) 
Randomly modify structural characters of a valid JSON ('{', '}', '[', ']',
':', '"', ',') to generate output that cannot be parsed as JSON.

Follow-up to https://github.com/ClickHouse/ClickHouse/pull/56490
 2023-12-13 19:59:31 +01:00
Alexey Milovidov
bf4964b177
Merge pull request #57712 from ClickHouse/clickbench-ci 
Add ClickBench to CI
 2023-12-13 19:52:44 +01:00
Nikita Taranov
560e66f3ca
More respect to min_number_of_marks in ParallelReplicasReadingCoordinator (#57763) 2023-12-13 19:51:49 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
Yakov Olkhovskiy
f4e2933f5d use storage uuid instead of table name as a key in storage info because there are can be multiple dropped table with the same name 2023-12-13 18:08:41 +00:00
Azat Khuzhin
055c231438 Disable system.kafka_consumers by default (due to possible live memory leak) 
It is not safe to use statistics because of how KafkaEngine works - it
pre-creates consumers, and this leads to the situation when this
statistics entries generated (RD_KAFKA_OP_STATS), but never consumed.

Which creates a live memory leak for a server with Kafka tables, but
without materialized view attached to it (and no SELECT).

Another problem is that this makes shutdown very slow, because of how
pending queue entries are handled in librdkafka, it uses
TAILQ_INSERT_SORTED, which is sorted insert into linked list, which
works incredibly slow (likely you will never wait till it ends and kill
the server)

For instance in my production setup the server was running for ~67 days
with such table, and it got 1'942'233 `TAILQ_INSERT_SORTED` entries
(which perfectly matches by the way - `67*86400/3` = 1'929'600), and it
moved only 289'806 entries for a few hours, though I'm not sure how much
time the process was in the running state, since most of the time it was
with debugger attached.

So for now let's disable it, to make this patch easy for backporting,
and I will think about long term fix - do not pre-create consumers in
Kafka engine.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-13 18:13:13 +01:00
Anton Popov
7598403f78 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-13 16:55:17 +00:00
Igor Nikonov
72f1c415f1 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-13 15:03:05 +00:00
Nikolai Kochetov
2085b5dd6f
Merge branch 'master' into analyzer-execution-names 2023-12-13 15:22:48 +01:00
Yakov Olkhovskiy
0ea4b16530
Merge branch 'master' into ft-dropped_tables_parts 2023-12-13 09:01:59 -05:00
Igor Nikonov
c165be76ab
Parallel replicas: friendly settings (#57542) 2023-12-13 14:42:06 +01:00
Raúl Marín
dd405a655e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-13 13:39:58 +01:00
Kseniia Sumarokova
b1325450e9
Merge pull request #57754 from ClickHouse/fix-loading-dependent-table-materialized-postgresql 
Table engine MaterializedPostgreSQL fix dependency loading
 2023-12-13 12:17:01 +01:00
Yakov Olkhovskiy
5121bfcd78 major refactoring 2023-12-13 07:16:30 +00:00
Alexey Milovidov
62b6d1ef5e iMerge branch 'master' of github.com:ClickHouse/ClickHouse into clickbench-ci 2023-12-13 01:41:27 +01:00
Alexey Milovidov
f11b90e7bb Allow buckets without List access 2023-12-13 00:10:58 +01:00
Igor Nikonov
616d054dee Merge remote-tracking branch 'origin/master' into pr-perf-improve-hope 2023-12-12 21:27:31 +00:00
Igor Nikonov
7f4a028196 Test and fix 2023-12-12 21:26:58 +00:00
Nikolai Kochetov
3e1dc6bcf9
Merge branch 'master' into analyzer-execution-names 2023-12-12 19:32:50 +01:00
Nikolai Kochetov
63533ad5bf Merge branch 'master' into analyzer-execution-names 2023-12-12 18:30:59 +00:00
serxa
57d7a58dbe Rename system.async_loader into system.asynchronous_loader 2023-12-12 16:55:27 +00:00
kssenii
cc71b40002 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 16:40:04 +01:00
Alexander Gololobov
09d6e6b945
Merge pull request #57743 from ClickHouse/other_timeout_for_wait 
Increase async block cache deduplication timeout
 2023-12-12 15:40:04 +01:00
Raúl Marín
049fb60c73 Fix error on retries due to TABLE_IS_READ_ONLY 2023-12-12 15:37:28 +01:00
kssenii
fa0b2b9e50 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 15:31:01 +01:00
avogar
096c55aece Fix tests 2023-12-12 14:13:15 +00:00
avogar
cbad0add9c Flatten only true Nested type if flatten_nested=1, not all Array(Tuple) 2023-12-12 14:13:15 +00:00
Chen Lixiang
68af81196d
Merge branch 'ClickHouse:master' into bytes_uncompressed 2023-12-12 22:06:45 +08:00
Kruglov Pavel
8a447bf57c
Merge pull request #55892 from Avogar/schema-inference-union 
Add 'union' mode for schema inference
 2023-12-12 15:02:06 +01:00
Nikolai Kochetov
696a2d9f4f
Merge pull request #57614 from ClickHouse/vdimir/join_array_lowcard_totals_crash 
Fix type correction in HashJoin for nested low cardinality
 2023-12-12 14:56:41 +01:00
Raúl Marín
923c3b7b32 Implement retries when ZK connection fails without committing the transaction 2023-12-12 12:00:46 +01:00
Raúl Marín
9d8d5dfdf6 Partially revert "make stages commit" 
This reverts commit e4becc01ba.
 2023-12-12 11:26:48 +01:00
Chen Lixiang
0b89fbbdd9 fix style issue 2023-12-12 16:49:58 +08:00
Zhuo Qiu
e34c13b9d9 Consider lightweight deleted rows when selecting parts to merge 2023-12-12 11:40:19 +08:00
alesapin
7cdc3d29fb Get rid of tiny function 2023-12-11 22:29:51 +01:00
alesapin
bfc2a7bb75 Update cache only after conflicts 2023-12-11 21:27:48 +01:00
alesapin
4b3e845dab Merge branch 'master' into other_timeout_for_wait 2023-12-11 20:43:21 +01:00
Igor Nikonov
30dabd42d3 One lookup for 2 checks 
(1) if it's the same part
(2) if annouced part has intersections with parts in working set
 2023-12-11 19:37:42 +00:00
kssenii
05e56f76ed Merge remote-tracking branch 'origin/master' into fix-loading-dependent-table-materialized-postgresql 2023-12-11 20:17:26 +01:00
Kseniia Sumarokova
907d0a79eb
Merge pull request #57568 from ClickHouse/fix-materialized-pg-issue-with-generated-columns 
MaterializedPostgreSQL: Support "generated columns" and default values
 2023-12-11 19:57:41 +01:00
Kruglov Pavel
3b1ffbd77c
Merge branch 'master' into schema-inference-union 2023-12-11 18:59:15 +01:00
kssenii
48ce040621 Fix 2023-12-11 17:30:23 +01:00
Raúl Marín
e1965bb6b5 WIP: Remove UNCERTAIN_COMMIT in INSERT 2023-12-11 17:27:56 +01:00
vdimir
ea09080c73
fix 2023-12-11 15:50:27 +00:00
Raúl Marín
e6be38be4d Adapt from HEAD 2023-12-11 16:28:00 +01:00
Raúl Marín
a55a0c068e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-11 15:27:21 +01:00
Igor Nikonov
4ccf4e11a8 Parallel replicas (perf): announcement response handling improvement 2023-12-11 13:00:08 +00:00
kssenii
1e4d61d55a Change error code 2023-12-11 13:44:12 +01:00
kssenii
b8caa00d24 Commit forgotten file 2023-12-11 13:28:36 +01:00
alesapin
a62a0b92de Increase async block cache deduplication timeout 2023-12-11 12:46:22 +01:00
kssenii
966a09fb70 Review fixes 2023-12-11 12:42:23 +01:00
kssenii
a06179da00 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-11 11:15:00 +01:00
Vitaly Baranov
f1ccf4e29f Show owner query ids in the message for the DEADLOCK_AVOIDED error. 2023-12-11 00:56:17 +01:00
Alexey Milovidov
644ef7b63f
Revert "Fix RWLock inconsistency after write lock timeout" 2023-12-10 23:51:59 +03:00
Igor Nikonov
f7e6a17052 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-10 20:21:47 +00:00
Alexey Milovidov
0a20ce5d32
Merge pull request #55271 from azat/local-lazy-database 
Replace --no-system-tables with loading virtual tables of system database lazily
 2023-12-10 19:51:17 +01:00
Alexey Milovidov
d119052895
Merge pull request #51858 from ClickHouse/CurtizJ-fix-ttl-non-deterministic 
Merging #37286
 2023-12-10 17:50:36 +01:00
Vitaly Baranov
a058a26b7e
Merge pull request #57454 from vitlibar/fix-rwlock 
Fix RWLock inconsistency after write lock timeout
 2023-12-10 14:09:29 +01:00
Alexey Milovidov
100b36e8a9 Do not check in replication 2023-12-10 01:32:42 +01:00
Alexey Milovidov
33b0ec7d55 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-10 01:21:43 +01:00
Azat Khuzhin
4f9d90761b Attach tables lazily only for clickhouse-local 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
c14721bb03 Move lazy table initialization interface into IDatabase 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Azat Khuzhin
cc9afe6f6e Replace --no-system-tables with loading virtual tables of system database lazily 
First of all the problem is that that simple 'SELECT 1' cannot be runned
without system.one, which makes --no-system-tables almost useless:

    $ ./clickhouse-debug local --no-system-tables -q "select 1"
    Code: 81. DB::Exception: Database system does not exist. (UNKNOWN_DATABASE)

Secondly there are just too much flags, and this one
(--no-system-tables) is too damn specific.

This patch should improve startup time of the clickhouse-local almost
3x in debug builds.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 23:08:29 +01:00
Alexey Milovidov
c77183a597
Merge pull request #57480 from azat/dist/async-INSERT-fixes 
Fix possible distributed sends stuck due to "No such file or directory" (during recovering batch from disk)
 2023-12-09 17:11:35 +01:00
Alexey Milovidov
56cd9196b0
Merge pull request #57641 from azat/system.stack_trace-fix 
Avoid hangs of system.stack_trace by correctly prohibit parallel read from it
 2023-12-09 17:07:48 +01:00
Chen Lixiang
7cd770e9a7 fix doc and comments 2023-12-09 20:09:41 +08:00
Nikolay Degterinsky
c5e52d260f
Merge pull request #57553 from evillique/fix-local-crash 
Fix crash in clickhouse-local
 2023-12-08 22:57:52 +01:00
Gagan Goel
e547db0a8c Issue 43666: Add skip_unavailable_shards as a setting for Distributed table. 
This setting, when enabled (disabled by default), allows ClickHouse to
silently skip unavailable shards of a Distributed table during a query
execution, instead of throwing an exception to the client.
 2023-12-08 15:43:59 -05:00
avogar
ee7af95bc0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-12-08 20:29:28 +00:00
Azat Khuzhin
f876bea050 Add support for system.stack_trace filtering optimizations for analyzer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 18:13:36 +01:00
Nikita Mikhaylov
c979124cfe Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-08 16:25:02 +00:00
Azat Khuzhin
b1043b7ad3 Increase log verbosity of some messages in system.stack_trace 
This can help to debug 01051_system_stack_trace test failures.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 16:18:49 +01:00
Kseniia Sumarokova
928993f2bd
Merge branch 'master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 15:37:29 +01:00
Kseniia Sumarokova
bd3665863c
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-08 15:36:40 +01:00
zhongyuankai
7b0f8d44e8
Make DirectoryMonitor handle cluster node list change (#42826) 2023-12-08 14:41:51 +01:00
kssenii
d77938c3ff Add validation of columns 2023-12-08 12:42:38 +01:00
kssenii
ffd0d16f79 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 12:42:21 +01:00
Azat Khuzhin
16cbcf700e Prohibit parallel read from system.stack_trace 
Before rewriting system.stack_trace to handle max_block_size (in #54946)
parallel reading from system.stack_trace was prohibited, because this
could lead to hang of system.stack_trace table.

But that rewrite broke this guarantee, so let's fix it to avoid possible
hung.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-08 11:05:25 +01:00
Alexey Milovidov
ce9e134043 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-08 03:10:09 +01:00
alesapin
46f04d6d0e Throw more clear exception 2023-12-07 17:15:39 +01:00
Igor Nikonov
94fb23af0b Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-07 11:52:40 +00:00
kssenii
8be3c9d218 Merge remote-tracking branch 'origin/master' into allow-to-change-some-cache-settings-without-restart 2023-12-07 12:14:24 +01:00
kssenii
614da21144 Better 2023-12-07 12:12:10 +01:00
Chen Lixiang
9e95fe5d29 conflict fixed. 2023-12-07 18:10:45 +08:00
Kseniia Sumarokova
b3f22449f6
Merge pull request #57515 from ClickHouse/fix-materialized-pg-issue-with-incorrect-connection-options 
MaterializedPostgreSQL: fix issue #41922, add test for #41923
 2023-12-07 10:42:14 +01:00
Alexey Milovidov
b85214ca1a
Merge pull request #57546 from rschu1ze/revert-revert-support_alias_column_in_indices 
Revert "Revert "Support use alias column in indices""
 2023-12-07 01:22:03 +01:00
Alexey Milovidov
5fe1b8926c
Merge pull request #57595 from ClickHouse/23.12-prepare 
Update version after release
 2023-12-06 23:53:53 +01:00
Alexey Milovidov
c8fb574fc8 Update autogenerated version to 23.12.1.1 and contributors 2023-12-06 23:53:06 +01:00
Yakov Olkhovskiy
ac7210b9b3
Merge pull request #57461 from ClickHouse/fix-ephemeral-matview 
Fix: don't exclude ephemeral column when building pushing to view chain
 2023-12-06 16:46:54 -05:00
Alexey Milovidov
05bc8ef1e0
Merge pull request #57548 from ClickHouse/s3queue_production_ready 
`S3Queue` is production ready
 2023-12-06 22:21:08 +01:00
Alexey Milovidov
3723573f81 Revert "Merge pull request #56724 from canhld94/ch_replicated_column_mismatch" 
This reverts commit a9bb76378f, reversing
changes made to 588fd16518.
 2023-12-06 19:09:07 +01:00
kssenii
806061642a Support GENERATED and DEFAULT columns 2023-12-06 15:01:43 +01:00
vdimir
8f32293104
Merge pull request #57416 from ClickHouse/vdimir/fix_test_encrypted_disk 
Change type of s3_cache in test_encrypted_disk
 2023-12-06 14:37:40 +01:00
Yakov Olkhovskiy
8f00678abe fix style 2023-12-06 13:24:25 +00:00
Alexey Milovidov
f8de7f6bc2 Merge branch 'master' into s3queue_production_ready 2023-12-06 12:25:29 +01:00
Kseniia Sumarokova
e02883cd4a
Merge branch 'master' into fix-materialized-pg-issue-with-incorrect-connection-options 2023-12-06 12:16:04 +01:00
Igor Nikonov
b4922eece4 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-06 10:44:30 +00:00
alesapin
e8b017d323
Merge pull request #57513 from ClickHouse/enable_async_block_ids_cache_by_default 
Enable async block ids cache by default
 2023-12-06 11:35:25 +01:00
Nikolai Kochetov
718b542f40
Merge pull request #57518 from ClickHouse/cleanup-around-distributed 
Simple cleanup in distributed (while dealing with parallel replicas)
 2023-12-06 10:22:17 +01:00
Yakov Olkhovskiy
f5fb169c65 add system.dropped_tables_parts table 2023-12-06 06:00:00 +00:00
Nikolay Degterinsky
ed618d3be1 Fix crash in clickhouse-local 2023-12-06 04:24:43 +00:00
Nikita Taranov
7d8badc4c1
Small change in log message in MergeTreeDataMergerMutator 2023-12-06 00:08:24 +01:00
Alexey Milovidov
76bbeaa676 S3Queue is production ready 2023-12-05 21:10:19 +01:00
Robert Schulze
8e4e37d8ed
Revert "Revert "Support use alias column in indices"" 
This reverts commit d0b591be2a.
 2023-12-05 18:22:32 +00:00
Alexey Milovidov
d0b591be2a
Revert "Support use alias column in indices" 2023-12-05 18:21:45 +03:00
Anton Popov
eb990d863d fix tests 2023-12-05 15:15:46 +00:00
Anton Popov
913ae1d267 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-05 14:47:32 +00:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Igor Nikonov
87a4c0bde4 Simple cleanup in distributed (while dealing with parallel replicas) 2023-12-05 12:25:02 +00:00
kssenii
0d313acd03 Fix 2023-12-05 13:22:35 +01:00
Igor Nikonov
71204c2a82 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-05 12:00:45 +00:00
Alexander Sapin
93a16b7cd6 Fix comments 2023-12-05 12:39:00 +01:00
Alexander Sapin
2587f368a6 Enable async block ids cache by default 2023-12-05 12:37:22 +01:00
Anton Popov
5770def9af
Merge pull request #57491 from jkartseva/limit-table-projections 
limit merge tree projections
 2023-12-05 12:13:41 +01:00
Julia Kartseva
2ac104240d limit merge tree projections 
Fixes: https://github.com/ClickHouse/ClickHouse/issues/56427
 2023-12-05 06:44:39 +00:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
Anton Popov
69896d94be fix tests 2023-12-04 21:40:19 +00:00
Anton Popov
40a1744b7c Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-04 18:10:43 +00:00
kssenii
4a28f10c3d Minor cache changes 2023-12-04 19:02:37 +01:00
Alexey Milovidov
02439eee58
Merge pull request #49351 from ClickHouse/async-loader-integration 
Asynchronous loading of tables
 2023-12-04 17:16:12 +01:00
Azat Khuzhin
7986fe619a Introduce DistributedAsyncInsertionFailures - event for async INSERT failures 
Useful for alerts

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
Azat Khuzhin
604cec475a Fix possible distributed sends stuck due to "No such file or directory" 
In case of restoring from current_batch.txt it is possible that the some
file from the batch will not be exist, and the fix submitted in #49884
was not complete, since it will fail later in markAsSend() (due to it
tries to obtain file size there):

    2023.12.04 05:43:12.676658 [ 5006 ] {} <Error> dist.DirectoryMonitor.work4: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in file_size: No such file or directory ["/work4/clickhouse/data/dist/shard8_all_replicas//150426396.bin"], Stack trace (when copying this message, always include the lines below):

    0. ./.build/./contrib/llvm-project/libcxx/include/exception:134: std::runtime_error::runtime_error(String const&) @ 0x00000000177e83f4 in /usr/lib/debug/usr/bin/clickhouse.debug
    1. ./.build/./contrib/llvm-project/libcxx/include/string:1499: std::system_error::system_error(std::error_code, String const&) @ 0x00000000177f0fd5 in /usr/lib/debug/usr/bin/clickhouse.debug
    2. ./.build/./contrib/llvm-project/libcxx/include/__filesystem/filesystem_error.h:42: std::__fs::filesystem::filesystem_error::filesystem_error[abi:v15000](String const&, std::__fs::filesystem::path const&, std::error_code) @ 0x000000000b844ca1 in /usr/lib/debug/usr/bin/clickhouse.debug
    3. ./.build/./contrib/llvm-project/libcxx/include/__filesystem/filesystem_error.h:90: void std::__fs::filesystem::__throw_filesystem_error[abi:v15000]<String&, std::__fs::filesystem::path const&, std::error_code const&>(String&, std::__fs::filesystem::path const&, std::error_code const&) @ 0x000000001778f953 in /usr/lib/debug/usr/bin/clickhouse.debug
    4. ./.build/./contrib/llvm-project/libcxx/src/filesystem/filesystem_common.h:0: std::__fs::filesystem::detail::(anonymous namespace)::ErrorHandler<unsigned long>::report(std::error_code const&) const @ 0x0000000017793ef7 in /usr/lib/debug/usr/bin/clickhouse.debug
    5. ./.build/./contrib/llvm-project/libcxx/src/filesystem/operations.cpp:0: std::__fs::filesystem::__file_size(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000017793e26 in /usr/lib/debug/usr/bin/clickhouse.debug
    6. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:707: DB::DistributedAsyncInsertDirectoryQueue::markAsSend(String const&) @ 0x0000000011cd92c5 in /usr/lib/debug/usr/bin/clickhouse.debug
    7. ./.build/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h💯 DB::DistributedAsyncInsertBatch::send() @ 0x0000000011cdd81c in /usr/lib/debug/usr/bin/clickhouse.debug
    8. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:0: DB::DistributedAsyncInsertDirectoryQueue::processFilesWithBatching() @ 0x0000000011cd5054 in /usr/lib/debug/usr/bin/clickhouse.debug
    9. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:417: DB::DistributedAsyncInsertDirectoryQueue::processFiles() @ 0x0000000011cd3440 in /usr/lib/debug/usr/bin/clickhouse.debug
    10. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:0: DB::DistributedAsyncInsertDirectoryQueue::run() @ 0x0000000011cd3878 in /usr/lib/debug/usr/bin/clickhouse.debug
    11. ./.build/./contrib/llvm-project/libcxx/include/__functional/function.h:0: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x00000000103dbc34 in /usr/lib/debug/usr/bin/clickhouse.debug
    12. ./.build/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x00000000103de1b6 in /usr/lib/debug/usr/bin/clickhouse.debug
    13. ./.build/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000103de7d1 in /usr/lib/debug/usr/bin/clickhouse.debug
    14. ./.build/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000b8c5502 in /usr/lib/debug/usr/bin/clickhouse.debug
    15. ./.build/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000b8c936e in /usr/lib/debug/usr/bin/clickhouse.debug
    16. ? @ 0x00007f1be8b30fd4 in ?
    17. ? @ 0x00007f1be8bb15bc in ?

And instead of ignoring errors, DistributedAsyncInsertBatch::valid() had
been added, that should be called when the files had been read from the
current_batch.txt, if it is not valid (some files from the batch did not
exist), then there is no sense in trying to send the same batch, so just
this file will be ignored, and files will be processed in a regular
order.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
Azat Khuzhin
638d0102f8 Fix error_count in case of distributed_directory_monitor_max_sleep_time_ms>5min 
In this case the error counter will be decremented everytime.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
vdimir
7e624ae911
Review fixes 2023-12-04 13:56:51 +00:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data 
Background merges correctly use temporary data storage in the cache
 2023-12-04 14:52:20 +01:00
Anton Popov
8d30e22a09 fix lighweight delete with heavy delete 2023-12-04 13:00:50 +00:00
vdimir
b2e29d1416
Change type of s3_cache in test_encrypted_disk 2023-12-04 10:34:13 +00:00
robot-ch-test-poll1
d63b652dae
Merge pull request #57385 from vitlibar/fix-inconsistent-metadata-for-backup-2 
Stop using INCONSISTENT_METADATA_FOR_BACKUP that much
 2023-12-04 10:29:57 +01:00
Yakov Olkhovskiy
b601b66146 don't exclude ephemeral column when pushing to view 2023-12-04 06:56:33 +00:00
Vitaly Baranov
0457aba4a9 Show owner query ids in the message for the DEADLOCK_AVOIDED error. 2023-12-04 00:15:31 +01:00
Nikolay Degterinsky
2150308c23
Merge pull request #56948 from amosbird/fix-56932 
Prevent incompatible ALTER of projection columns
 2023-12-03 04:24:51 +01:00
Sema Checherinda
c61cd6aab2
Merge pull request #57335 from sichenzhao/increase_default_replicated_deduplication_window 
increase default replicated_deduplication_window from 100 to 1k
 2023-12-02 21:23:39 +01:00
Nikolay Degterinsky
a53e604429
Merge branch 'master' into fix-56932 2023-12-02 21:19:45 +01:00
Sema Checherinda
32e3b2657b
Merge pull request #57205 from CheSema/zero-locks-leak 
fix zero-copy locks leaking
 2023-12-02 15:54:12 +01:00
Sema Checherinda
a1c9501c07 fix 01158_zookeeper_log_long and 02442_auxiliary_zookeeper_endpoint 2023-12-02 04:57:33 +01:00
Anton Popov
7fbe7462b6 add mutation command to apply deleted mask 2023-12-01 19:12:05 +00:00
JackyWoo
8a68d51261
Add index to table system.numbers (#50909) 2023-12-01 19:59:25 +01:00
robot-ch-test-poll
1b49463bd2
Merge pull request #55841 from nickitat/optimize_reading3 
Optimize reading from cache
 2023-12-01 17:36:57 +01:00
Nikolay Degterinsky
ceed0f73d3
Merge pull request #57391 from evillique/better-partitioned-write-to-file 
Allow wildcards in directories for partitioned write with File storage
 2023-12-01 17:23:54 +01:00
Nikolai Kochetov
5430ac2c02
Merge pull request #57099 from ClickHouse/analyzer-fuzz-3 
Analyzer fuzzer 3 (aggregate_functions_null_for_empty for projections)
 2023-12-01 16:57:34 +01:00
Anton Popov
61c517bc02
Merge pull request #57352 from kitaisreal/mutations-reuse-source-part-granularity 
MergeTree mutations reuse source part index granularity
 2023-12-01 16:45:41 +01:00
Nikolai Kochetov
0b4131546a
Merge pull request #56813 from jsc0218/SystemTablesFilterEngine 
Able to Filter Engine When Scanning System Tables
 2023-12-01 16:02:27 +01:00
alesapin
fb98b212c5
Merge pull request #57311 from ClickHouse/make_alter_mat_view_non_experimental 
Make alter materialized view non experimental
 2023-12-01 12:18:43 +01:00
vdimir
fd373024f5
Update buffer re-reading in MergeTask.cpp 2023-12-01 10:11:08 +00:00
Sema Checherinda
82270627ed fix the relock 2023-12-01 02:12:35 +01:00
Igor Nikonov
a763c55c2e Fix 2023-11-30 22:42:57 +00:00
Igor Nikonov
45ebe78d32 Cleanup 2023-11-30 21:40:45 +00:00
Vitaly Baranov
1bc1563e0e Stop using INCONSISTENT_METADATA_FOR_BACKUP that much. If possible prefer to continue scanning instead of stopping and starting the scanning for backup again. 2023-11-30 21:37:42 +01:00
Sema Checherinda
47daa4f235 remarks 2023-11-30 21:13:56 +01:00
Sema Checherinda
42f648f0d2 relock block when sessin expired 2023-11-30 21:13:56 +01:00
Sema Checherinda
e4becc01ba make stages commit 2023-11-30 21:13:56 +01:00
Sema Checherinda
7efa76f08d remove rename_part_to_temporary 2023-11-30 21:13:56 +01:00
Sema Checherinda
2cc31752b1 unlockSharedData in 1 more place, and do it in a best effort way 2023-11-30 21:13:56 +01:00
Sema Checherinda
ad6897ddbd make shellcheck happy 2023-11-30 21:13:55 +01:00
Sema Checherinda
ad7b51c3c8 fix zero locks leak 2023-11-30 21:13:55 +01:00
Igor Nikonov
8ba5c32f1c Remove commented code 2023-11-30 18:42:45 +00:00
Sergey Kviatkevich
376d297da2 fix alter table tests 2023-11-30 21:42:13 +03:00
Sergey Kviatkevich
afae432e99 allow ALTER for TEMPORARY table 2023-11-30 21:42:12 +03:00
Nikolay Degterinsky
95b5b9eb35 Fix partitioned write for File storage 2023-11-30 16:18:12 +00:00
Igor Nikonov
b4bf611d5d Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-11-30 14:09:26 +00:00
Igor Nikonov
59daabe95a Some refactoring and cleanup 
+ change load balancing for custom key w/o hedged connections
  (need the test though)
 2023-11-30 14:08:01 +00:00
Kruglov Pavel
575ee1e665
Merge pull request #56868 from zvonand/zvonand-fileCluster 
Introduce `fileCluster` table function
 2023-11-30 13:49:53 +01:00
Antonio Andelic
9955f571ef
Merge pull request #57318 from mkmkme/mkmkme/remove-unused-vars 
Remove unused Strings from MergeTreeData
 2023-11-30 13:48:54 +01:00
vdimir
54fe2e1410
Fix clang tidy 2023-11-30 12:40:34 +00:00
alesapin
d8bdbf29c4 Merge branch 'master' into make_alter_mat_view_non_experimental 2023-11-30 11:14:25 +01:00
robot-ch-test-poll
07ca6236f6
Merge pull request #57372 from jkartseva/fuzz-json-verbose-exception 
verbose exception messages for StorageFuzzJSON
 2023-11-30 08:20:54 +01:00
zvonand
6730a29a0b upd 2023-11-30 01:28:25 +01:00
Julia Kartseva
ed65c26faf verbose exception messages for StorageFuzzJSON 
Follow-up to https://github.com/ClickHouse/ClickHouse/pull/56490#discussion_r1409556111
 2023-11-29 20:42:21 +00:00
Kruglov Pavel
c6fecfb1af
Merge pull request #56901 from KevinyhZou/Fix_allow_cr_end_of_csv_line 
Fix allow cr end of line for csv
 2023-11-29 20:57:58 +01:00
Nikolai Kochetov
a36c11b21e Disable transform_null_in as well just in case. 2023-11-29 17:30:22 +00:00
Nikolai Kochetov
fc02b88596 Merge branch 'master' into analyzer-fuzz-3 2023-11-29 17:19:56 +00:00
Nikolai Kochetov
dc43a36ee2 Fixing style 2023-11-29 16:40:18 +00:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
Nikolai Kochetov
390c27f7d8 Ignore aggregate_functions_null_for_empty for projections at insert. 2023-11-29 16:16:22 +00:00
zvonand
c329e382ab resolve conflicts 2023-11-29 16:03:07 +01:00
zvonand
a4d9fbdfc2 Fix indents, fix style, fix test 2023-11-29 15:55:24 +01:00
Kruglov Pavel
30148972ed
Merge pull request #57328 from Avogar/size-virtual-column-table-functions 
Add information about new _size virtual column in file/s3/url/hdfs/azure table functions
 2023-11-29 15:05:28 +01:00
Maksim Kita
6578fe5baa Updated implementation 2023-11-29 16:21:07 +03:00
Maksim Kita
f0e53fde64 MergeTree mutations reuse source part index granularity 2023-11-29 14:36:57 +03:00
Han Fei
4c0efb0262
Merge pull request #53240 from hanfei1991/hanfei/statistic 
use statistic to order prewhere conditions better
 2023-11-29 12:26:44 +01:00
Nikolai Kochetov
731a1bfcf8 Cleanup 2023-11-29 11:11:58 +00:00
vdimir
470a092e3d
Merge pull request #51745 from ClickHouse/vdimir/setting_join_algorithm_order 
Setting JoinAlgorithm respect specified order
 2023-11-29 11:24:39 +01:00
Raúl Marín
b102145a4f
Merge pull request #57301 from Algunenano/fix_reports 
Fix code reports
 2023-11-29 11:11:33 +01:00
Chen Lixiang
88920f08f1 add tests and fix whitespace issue 2023-11-29 17:26:11 +08:00
Robert Schulze
bc3da5032e
Merge pull request #57330 from rschu1ze/fix-no-lib-build 
Follow-up to #56490: Fix build with `cmake -DENABLE_LIBRARIES=0`
 2023-11-29 09:42:12 +01:00
Antonio Andelic
f8811351fa
Merge pull request #57267 from ClickHouse/fix-readonly-replica 
Fix `ReadonlyReplica` metric for all cases
 2023-11-29 08:32:25 +01:00
Sichen Zhao
1b8b1c0a63 increase default replicated_deduplication_window from 100 to 1k 2023-11-28 14:24:06 -08:00
Robert Schulze
5f5e8633c7
Merge pull request #57220 from ucasfl/alias_column_in_indices 
Support use alias column in indices
 2023-11-28 21:31:34 +01:00
Robert Schulze
7d26990752
Follow-up to #56490: Fix build with -DENABLE_LIBRARIES=0 2023-11-28 19:59:43 +00:00
vdimir
b748405976
Intorduce *List definition for muli enum settings 2023-11-28 19:09:02 +00:00
vdimir
a6eb2ad930
Fix JoinStep::describeJoinActions 2023-11-28 19:09:01 +00:00
Nikolai Kochetov
4d906d6e3c Cleanup 2023-11-28 18:35:19 +00:00
avogar
4d9a1b50f9 Add information about new _size virtual column in file/s3/url/hdfs/azure table functions 2023-11-28 18:15:07 +00:00
Igor Nikonov
a06a71834a Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-11-28 18:07:04 +00:00
Nikolai Kochetov
84194f10f2 Fixing tests. 2023-11-28 18:04:30 +00:00
Nikolai Kochetov
08a7575984 Re-implement filtering a bit. 2023-11-28 16:17:35 +00:00
Han Fei
e4421e2808 remove wrong code 2023-11-28 16:46:55 +01:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Han Fei
6733438adc Merge branch 'master' into hanfei/statistic 2023-11-28 16:35:00 +01:00
Han Fei
8e84c734bf rename some code 2023-11-28 16:32:47 +01:00
zvonand
c306d21b54 merge master + resolve conflicts 2023-11-28 15:51:21 +01:00
Mikhail Koviazin
daaa575106
Remove unused Strings from MergeTreeData 
Compiler does not emit a warning when there is an unused variable of
a type with a non-trivial constructor. std::string is one of them.
Hence, these two variables were overlooked. This commit removes them
 2023-11-28 16:33:44 +02:00
Alexander Tokmakov
e40c71a74c
Merge pull request #56533 from ClickHouse/rmt_check_shutdown_flags_in_retry_loops 
ReplicatedMergeTree: check shutdown flags in retry loops
 2023-11-28 15:24:40 +01:00
Alexander Sapin
2cd9f5f3cc Fix known issue 2023-11-28 15:13:12 +01:00
Alexander Sapin
e6855cef4d Followup 2023-11-28 14:01:31 +01:00
Alexander Sapin
b5404f4a9f Followup 2023-11-28 13:34:05 +01:00
Alexander Sapin
e1aa62d9cd Make alter materialized view non expiremental and deprecate allow_experimental_alter_materialized_view_structure obsolete 2023-11-28 13:17:55 +01:00
Antonio Andelic
636ac3f349 Fix assert 2023-11-28 12:07:05 +00:00
vdimir
af81cb8252
Setting JoinAlgorithm respect specified order 2023-11-28 11:01:35 +00:00
Antonio Andelic
9dd4783675 Merge branch 'master' into fix-readonly-replica 2023-11-28 10:43:29 +00:00
Robert Schulze
d459fa1fc6
Merge remote-tracking branch 'ClickHouse/master' into alias_column_in_indices 2023-11-28 10:15:25 +00:00
Raúl Marín
ff0df49e2b Fix comment in MergeTreeDataPartWide::checkConsistency 2023-11-28 10:09:03 +00:00
Nikolai Kochetov
7e1f625179 Revert system tables to master. 2023-11-28 10:07:36 +00:00
Julia Kartseva
57a4492275
Merge pull request #56490 from jkartseva/fuzz_json 
Fuzz JSON table function
 2023-11-27 21:47:53 -08:00
Ubuntu
a85cf758b0 Fuzz JSON table function 
Create a table function `fuzzJSON`

An example query:
```
CREATE NAMED COLLECTION json_fuzzer AS json_str={};

SELECT *
FROM fuzzJSON(json_fuzzer, json_str = '{"students" : ["Alice", "Bob"]}', random_seed = 666, max_output_length = 128, probability = 0.9)
LIMIT 3

Query id: 7f802052-efb0-41b4-87fa-03b7dd290e9d

┌─json──────────────────────────────────────────────────────────────────────────────────┐
│ {"ade0yX":[9200406572736542991, true, "sm"]}                                          │
│ {"students":["Alice", "eSN3WO#a6NYTBe0$#OWwyIQ"], "cVoP2BuQugQ":17384271928263249403} │
│ {"students":["Alice", "Bob", "T+-k4+PJGkL%XRRaF2BoeN@&A"]}                            │
└───────────────────────────────────────────────────────────────────────────────────────┘
```

Next step:
* Generate invalid string

Fixes #35962

add Object('json')

use named collection
 2023-11-28 01:58:24 +00:00
Han Fei
7bd2ea3add Merge branch 'master' into hanfei/statistic 2023-11-27 23:31:23 +01:00
Antonio Andelic
544f6081eb
Merge pull request #57259 from ClickHouse/ignore-column-comments 
Ignore comments when comparing column descriptions
 2023-11-27 20:43:39 +01:00
Nikolai Kochetov
e1b00b3ee0 Merge branch 'master' into SystemTablesFilterEngine 2023-11-27 18:54:21 +00:00
Igor Nikonov
e823046646 Fixes 2023-11-27 15:27:04 +00:00
Antonio Andelic
9077550bb1 Better Readonly metric 2023-11-27 14:25:45 +00:00
Antonio Andelic
9707796869 Ignore comments when comparing column descriptions 2023-11-27 12:25:41 +00:00
vdimir
bbc202a891
Merge pull request #56816 from AVMusorin/improve-settings-rocksdb 
Show total_bytes and total_rows in system tables for RocksDB storage
 2023-11-27 11:42:15 +01:00
Kruglov Pavel
b10e46b2bc
Merge pull request #57126 from Avogar/size-virtual-column 
Add _size virtual column to s3/file/hdfs/url/azureBlobStorage engines
 2023-11-27 10:12:18 +01:00
flynn
aa15a4e46d Merge branch 'master' of github.com:ClickHouse/ClickHouse into alias_column_in_indices 2023-11-27 03:52:00 +00:00
flynn
f995ac87c5 review fix 2023-11-27 03:50:34 +00:00
Igor Nikonov
63adf4d20c Remove commented code 2023-11-26 22:07:57 +00:00
Igor Nikonov
774347d231 Parallel replicas custom key: skip unavailable replicas 2023-11-26 21:38:49 +00:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read 
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
 2023-11-26 19:20:19 +01:00
Alexey Milovidov
a9bb76378f
Merge pull request #56724 from canhld94/ch_replicated_column_mismatch 
Do not allow tables on different replicas have different aggregate functions in SimpleAggregateFunction column
 2023-11-26 17:20:10 +01:00
Alexey Milovidov
1cc33f3430
Merge pull request #56164 from azat/not-byte-identical-message 
Add more details to "Data after merge is not byte-identical to data on another replicas"
 2023-11-26 04:14:15 +01:00
Alexey Milovidov
304d6375be
Merge pull request #56225 from azat/rocksdb-compact 
Allow manual compaction of rocksdb via OPTIMIZE query
 2023-11-26 03:59:54 +01:00
Robert Schulze
5b7d2a903d
Merge pull request #57152 from rschu1ze/fine-granular-plan-opt-settings 
Fine-granular enablement/disabling of plan-level optimizations
 2023-11-25 20:44:23 +01:00
flynn
9f174b1c5e Support use alias column in indices 2023-11-25 15:40:22 +00:00
Vitaly Baranov
2e7f314599
Merge pull request #50209 from ilejn/merge_row_policy 
Engine Merge obeys row policy
 2023-11-25 10:34:22 +01:00
Sema Checherinda
bd36dce81b
Merge pull request #57129 from ClickHouse/fix_something 
Fix something in ReplicatedMergeTree
 2023-11-24 20:19:25 +01:00
Alexander Tokmakov
deda8483be
Merge pull request #56536 from ClickHouse/move-partition-inactive-replica 
Fix race between REPLACE_RANGE and GET_PART (set actual part name when fetching)
 2023-11-24 20:09:24 +01:00
Aleksandr Musorin
c3b617bd7b Move class functions upper 2023-11-24 19:00:00 +01:00
Raúl Marín
63fe821f7b Merge remote-tracking branch 'blessed/master' into backup_1 2023-11-24 16:21:59 +01:00
Raúl Marín
2539100afb Review improvements 2023-11-24 16:21:56 +01:00
Kseniia Sumarokova
13187bf4f1
Merge pull request #56959 from melvynator/master 
Add experimental flag for MaterializedPostgreSQL table engine.
 2023-11-24 16:16:42 +01:00
Kseniia Sumarokova
c25c6ec403
Merge pull request #57031 from aiven-sal/aiven-sal/fix_namedcoll2 
NamedCollections: make exception message more informative.
 2023-11-24 16:15:16 +01:00
alesapin
a3c2896e9d
Merge pull request #56846 from aalexfvk/add_cancellation_point_for_moving_background_operation 
Add cancellation hook for moving background operation
 2023-11-24 12:12:47 +01:00
Chen Lixiang
2b35e98ef8 add comments and doc 2023-11-24 16:40:19 +08:00
Chen Lixiang
89fa729235 show uncompressed size in system.tables 2023-11-24 15:54:27 +08:00
Alexey Milovidov
81a50703c3 Fix tests 2023-11-23 23:39:40 +01:00
Alexey Milovidov
ade67cc57e Merge branch 'master' of github.com:ClickHouse/ClickHouse into CurtizJ-fix-ttl-non-deterministic 2023-11-23 23:04:11 +01:00
Alexey Milovidov
d72bc854d1
Merge pull request #57074 from ClickHouse/less-allocation-in-arenas 
Avoid excessive allocation in Arena
 2023-11-23 23:02:26 +01:00
Igor Nikonov
0434044a7e
Merge pull request #57054 from ClickHouse/pr-cleanup-narrow-dependency 
Parallel replicas: cleanup, narrow dependency
 2023-11-23 20:12:37 +01:00
Robert Schulze
77d48e4d55
"optimise" --> "optimize". Sorry, friends from the UK. 2023-11-23 18:41:35 +00:00
Alexey Milovidov
c5fabcc472 Fix tests 2023-11-23 17:50:50 +01:00
Alexey Milovidov
dc0d6e432c Fix build 2023-11-23 16:11:54 +01:00
Alexey Milovidov
6bbf449808 Fix build 2023-11-23 16:06:35 +01:00
Alexey Milovidov
39ea0c426a Fix build 2023-11-23 16:05:15 +01:00
Alexey Milovidov
89656723fb Fix tests 2023-11-23 16:02:33 +01:00
Alexey Milovidov
93b789169b Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-11-23 15:46:01 +01:00
Aleksei Filatov
1a03f5f7f4 Merge remote-tracking branch 'upstream/master' into add_cancellation_point_for_moving_background_operation 2023-11-23 16:43:33 +03:00
serxa
663c8cd944 Merge branch 'master' into dashboards-table 2023-11-23 10:29:48 +00:00
serxa
0fc25fbda8 Merge branch 'master' into async-loader-integration 2023-11-23 09:32:43 +00:00
Alexander Tokmakov
a0ce9f46cf fix 2023-11-22 23:15:20 +01:00
Alexey Milovidov
2ba2f48051 Merge branch 'master' of github.com:ClickHouse/ClickHouse into less-allocation-in-arenas 2023-11-22 22:11:59 +01:00
avogar
70f45c11bd Fix 2023-11-22 19:56:42 +00:00
Alexander Tokmakov
ef3e558a65 fix no such key for empty parts 2023-11-22 20:51:10 +01:00
Alexander Tokmakov
d9eb68ba35 fix unexpected outdated parts 2023-11-22 20:29:57 +01:00