Commit Graph

13198 Commits

Author SHA1 Message Date
Robert Schulze
7608e08eed
Cosmetics: more constness 2023-06-05 09:28:58 +00:00
Robert Schulze
423f692282
Cosmetics: Remove unnecessary toString() 2023-06-05 09:28:55 +00:00
Robert Schulze
3b77e40902
Cosmetics: Remove dots from exception messages 2023-06-05 09:28:53 +00:00
Robert Schulze
c5ededdc5b
Cosmetics: Switch arguments in MTIConditionAnnoy ctor 2023-06-05 09:28:50 +00:00
Robert Schulze
594572b0de
Cosmetics: AnnoyIndex --> AnnoyIndexWithSerialization 2023-06-05 09:28:47 +00:00
Robert Schulze
dc88d4e742
Cosmetics: Factorize repeated typedefs into a single typedef 2023-06-05 09:28:44 +00:00
Robert Schulze
e373cf682a
Cosmetics: Unwrap Annoy index from nested namespace 2023-06-05 09:28:41 +00:00
Robert Schulze
f8c1f2bd33
Cosmetics: Remove absolute namespace qualification of Annoy library internals 2023-06-05 09:28:38 +00:00
Robert Schulze
a8bf7af291
Cosmetics: Move ErrorCodes to top 2023-06-05 09:28:35 +00:00
Robert Schulze
4631595cf6
Cosmetics: number_of_trees --> trees 2023-06-05 09:28:32 +00:00
Robert Schulze
1018677f46
Cosmetics: Move ctors into cpp file 2023-06-05 09:28:29 +00:00
Robert Schulze
edad92a7f2
Cosmetics: Minor aesthetic fixes 2023-06-05 09:28:26 +00:00
Robert Schulze
3275629230
Cosmetics: Remove parentheses in single statement if/for/while 2023-06-05 09:28:24 +00:00
Robert Schulze
3657ef05ff
Cosmetics: Fix indentation 2023-06-05 09:28:19 +00:00
Alexey Milovidov
3c5bd78856 Fix typo 2023-06-05 06:13:39 +02:00
Alexey Milovidov
881bdbf839
Merge pull request #50153 from amosbird/fix_44504
Fix invalid index analysis for date related keys
2023-06-05 05:00:20 +03:00
Alexey Milovidov
5fc8838b04
Update KeyCondition.cpp 2023-06-05 04:58:29 +03:00
Alexey Milovidov
f118cba17e
Merge pull request #50238 from azat/dist-pending-bytes
Add new metrics BrokenDistributedBytesToInsert/DistributedBytesToInsert
2023-06-05 04:50:42 +03:00
Alexey Milovidov
3b8c781b2c
Merge pull request #50448 from ClickHouse/rm_parts_from_zk_paranoid
Paranoid fix for removing parts from ZooKeeper
2023-06-05 04:42:15 +03:00
Alexey Milovidov
18817517ed
Merge pull request #50538 from ClickHouse/fix-generaterandom
Control memory usage in generateRandom
2023-06-05 02:50:24 +03:00
alesapin
8d8d06285f Add integration test 2023-06-04 18:59:22 +02:00
alesapin
02923138a3 Merge branch 'master' into azure 2023-06-04 16:43:28 +02:00
Amos Bird
e24c9267bc
fix 2023-06-04 20:06:27 +08:00
Amos Bird
97bd3f0483
Proper mutation of skip indices and projections 2023-06-04 18:32:08 +08:00
Alexey Milovidov
c62558f982 Control memory usage in generateRandom 2023-06-04 04:44:51 +02:00
alesapin
a0df856605 Able to insert 2023-06-03 23:11:03 +02:00
Azat Khuzhin
69aec7af9b Add new metrics BrokenDistributedBytesToInsert/DistributedBytesToInsert
Useful to see at the server status overall.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-03 20:49:19 +02:00
Azat Khuzhin
13a1226971 Fix nested distributed SELECT in WITH clause
For the CSE (common scalar expression, form of CTE) form of WITH it will
set max_result_rows to 1, since there should not be more rows, but this
will be applied for the DESC TABLE as well (service distributed query
that required to obtain the structure).

Note, that it is a problem only for nested distributed queries because
getSubqueryContext() from InterpreterSelectQuery.cpp resets those limits
as well, but this does not helps, for the nested DESC since it is
executed on the remote node that has max_result_rows=1.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-03 20:22:51 +02:00
Alexey Milovidov
8018e58b74
Merge pull request #49894 from evillique/postgresql-uuid
Add UUID data type to PostgreSQL
2023-06-03 07:13:41 +03:00
Alexey Milovidov
ccba3500dd
Update StoragePostgreSQL.cpp 2023-06-03 07:13:26 +03:00
Dmitry Novik
530f743ed0 Fix Object data type for StorageDistributed 2023-06-02 23:41:25 +02:00
Alexey Milovidov
9e938d7eec
Merge pull request #49287 from nickitat/optimize_reading2
Optimise hot reading from remote tables
2023-06-02 23:51:57 +03:00
Kruglov Pavel
f87f98fb80
Merge pull request #47409 from Avogar/random-structure
Add new function generateRandomStructure
2023-06-02 18:50:35 +02:00
Alexander Tokmakov
179b6aca6a fix 2023-06-02 17:52:57 +02:00
Alexander Tokmakov
4209ccfc08 fix 2023-06-02 17:42:04 +02:00
tpanetti
05dfe584b9 Merge branch 'master' into mysql_compatible_types 2023-06-02 08:33:06 -07:00
alesapin
8e4a2a42fb Some code 2023-06-02 16:51:09 +02:00
János Benjamin Antal
5e17adc9c0 Add system.user_processes table 2023-06-02 14:08:14 +00:00
Alexander Tokmakov
e148c60d5a
Fixes for MergeTree with readonly disks (#50244)
* fixes for MergeTree with readonly disks

* Automatic style fix

* Update test.py

* Automatic style fix

* Update test.py

* Update test.py

* Automatic style fix

* Update test.py

---------

Co-authored-by: robot-clickhouse <robot-clickhouse@users.noreply.github.com>
Co-authored-by: alesapin <alesapin@clickhouse.com>
2023-06-02 16:18:32 +03:00
Raúl Marín
2f08b6738f Support parallel replicas with the analyzer 2023-06-02 11:24:21 +02:00
JackyWoo
ef6fde8264 fix build error for dwrwin 2023-06-02 10:05:54 +08:00
JackyWoo
dc61023927 add sync mode to redis storage truncate 2023-06-02 10:05:54 +08:00
JackyWoo
e6d1b3c351 little fix 2023-06-02 10:05:54 +08:00
JackyWoo
119c2200a9 fix merged error 2023-06-02 10:05:54 +08:00
JackyWoo
bcf22c1ec7 fix code style 2023-06-02 10:05:54 +08:00
JackyWoo
0106704573 add truncate to redis storage 2023-06-02 10:05:54 +08:00
JackyWoo
1df1dfc3e5 add update/delete/insert to redis storage 2023-06-02 10:05:54 +08:00
JackyWoo
f4f939162d new redis engine schema design 2023-06-02 10:05:54 +08:00
JackyWoo
357df40c8f fix tests 2023-06-02 10:05:54 +08:00
JackyWoo
8c822a7edf add tests to redis engine 2023-06-02 10:05:54 +08:00
JackyWoo
3c2b447472 move get configuration to RedisCommon 2023-06-02 10:05:54 +08:00
JackyWoo
3281aec335 make some redis engine args optional 2023-06-02 10:05:54 +08:00
JackyWoo
412d9ba259 add tests for redis storage 2023-06-02 10:05:54 +08:00
JackyWoo
b35867d907 unify storage type 2023-06-02 10:05:54 +08:00
JackyWoo
70cfd7a222 fix typos 2023-06-02 10:05:54 +08:00
JackyWoo
40cc8d2107 fix code style 2023-06-02 10:05:54 +08:00
JackyWoo
ce203b5ce6 Check redis table structure 2023-06-02 10:05:54 +08:00
JackyWoo
23d6c835d8 fix poco redis array NPE 2023-06-02 10:05:54 +08:00
JackyWoo
9a495cbf99 Push down filter into Redis 2023-06-02 10:05:54 +08:00
JackyWoo
e91867373c Add table function Redis 2023-06-02 10:05:54 +08:00
JackyWoo
7398b22fa5 Add redis storage 2023-06-02 10:05:54 +08:00
Michael Kolupaev
b51064a508 Get rid of SeekableReadBufferFactory, add SeekableReadBuffer::readBigAt() instead 2023-06-01 18:48:30 -07:00
Alexey Milovidov
31773d7eee
Merge pull request #50432 from azat/fix-min-for-seek
Fix merge_tree_min_rows_for_seek/merge_tree_min_bytes_for_seek for data skipping indexes
2023-06-02 04:09:35 +03:00
Alexey Milovidov
60dccff77c
Merge pull request #50177 from amosbird/fix_50094
Fix broken index analysis when binary operator contains a null constant argument
2023-06-02 01:29:03 +03:00
Alexey Milovidov
52ffaa4630
Merge pull request #50203 from Avogar/head-requests-on-shcema-inference
Don't send head request for all keys in Iceberg schema inference
2023-06-02 01:28:00 +03:00
Andrey Zvonov
2e427ee028
Merge branch 'master' into zvonand-implicit-tz 2023-06-02 01:15:56 +03:00
Alexander Tokmakov
f62faaedc3 paranoid fix for removing parts from zk 2023-06-01 21:20:39 +02:00
Kruglov Pavel
d9113a3b75
Style 2023-06-01 16:57:43 +02:00
Kruglov Pavel
f99a7366da
Fix tests 2023-06-01 16:56:18 +02:00
Dmitry Novik
eb7ae91d01 Do not add alias to a temporary table 2023-06-01 14:34:30 +00:00
Dmitry Novik
c6dcb69b85 Fix GLOBAL IN 2023-06-01 14:34:30 +00:00
Dmitry Novik
85e5ed79e5 Fix distributed JOINs 2023-06-01 14:34:30 +00:00
Dmitry Novik
b86516131b Attempt to fix global JOINs and INs 2023-06-01 14:34:30 +00:00
Dmitry Novik
a4cb82127d Analyzer: WIP on distributed queries 2023-06-01 14:34:29 +00:00
Azat Khuzhin
53eb360ac2 Fix merge_tree_min_rows_for_seek/merge_tree_min_bytes_for_seek for data skipping indexes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-01 15:04:16 +02:00
SmitaRKulkarni
06432929a7
Merge pull request #50339 from ClickHouse/43983_Updated_to_hold_lock_fetch_part
Improved fetch part by holding directory lock longer
2023-06-01 13:42:34 +02:00
Alexey Milovidov
fb86fe8f9d Remove useless code 2023-06-01 03:08:05 +02:00
Alexey Milovidov
956c399b2a Remove useless code 2023-06-01 03:04:29 +02:00
Nikita Mikhaylov
3543d95980
Parallel replicas always skip unavailable ones (#50293) 2023-05-31 22:10:33 +02:00
SmitaRKulkarni
123cd505ee
Merge branch 'master' into 43983_Updated_to_hold_lock_fetch_part 2023-05-31 21:59:55 +02:00
avogar
0b62be649f Add docs, fix style 2023-05-31 17:52:29 +00:00
Alexander Tokmakov
a18b998076
Merge branch 'master' into rmt_better_background_tasks_scheduling 2023-05-31 19:47:29 +03:00
Kruglov Pavel
0beca0336d
Merge pull request #49112 from ClickHouse/Avogar-patch-3
Fix possible terminate called for uncaught exception in some places
2023-05-31 16:55:43 +02:00
Kruglov Pavel
4d1b5cc89f
Merge branch 'master' into renamefile 2023-05-31 13:25:11 +02:00
Sema Checherinda
ab027ca1ce
Merge pull request #50275 from CheSema/d-tor-free
buffers d-tor finalize free
2023-05-31 12:26:19 +02:00
Andrey Zvonov
c2b30f8903
Merge branch 'master' into zvonand-implicit-tz 2023-05-31 00:44:32 +03:00
tpanetti
38634cc5c5 Convert Clickhouse Types to MySQL types in Compatibility mode
This changes MySQL compatibility mode to display MySQL compatible types
2023-05-30 13:29:17 -07:00
avogar
d4efbbfbd3 Allow to skip empty files in file/s3/url/hdfs table functions 2023-05-30 19:32:24 +00:00
Kseniia Sumarokova
55b2e42150
Merge pull request #50232 from kssenii/fix-iceberg
Fix iceberg metadata parsing
2023-05-30 20:11:11 +02:00
Alexander Tokmakov
c8bb1f64ad fix 2023-05-30 18:46:49 +02:00
Alexander Tokmakov
82e1d5ae67 Merge branch 'master' into rmt_better_background_tasks_scheduling 2023-05-30 17:40:13 +02:00
Smita Kulkarni
fe944a55dc Updated to hold temporary_directory_lock till part is fetched, renamed and committed 2023-05-30 13:30:47 +02:00
Kruglov Pavel
f863dee8e7
Merge branch 'master' into renamefile 2023-05-30 12:26:40 +02:00
Sema Checherinda
2d2b411c26 fix build 2023-05-30 12:26:20 +02:00
Sema Checherinda
0b4ea3e2e1 remove reminder comments 2023-05-30 12:26:20 +02:00
Sema Checherinda
13dcb62ffb fix logs engins 2023-05-30 12:26:20 +02:00
Sema Checherinda
a5dcd8dabb do not call finalize in d-tor MergedBlockOutputStream::Finalizer 2023-05-30 12:26:20 +02:00
Sema Checherinda
d95e5b51af mark all finalize calls in buffers d-tors with issue id 2023-05-30 12:26:20 +02:00
Sema Checherinda
fe3939287b add test, fix .gin_sid files 2023-05-30 12:26:20 +02:00
Sema Checherinda
0e019c8e83 turn off some d-tor finalize 2023-05-30 12:26:20 +02:00
Andrey Zvonov
87fc780c14
Merge branch 'master' into zvonand-implicit-tz 2023-05-30 13:13:46 +03:00
SmitaRKulkarni
f2172ab8dd
Merge pull request #46661 from ClickHouse/42192_Lower_parallel_parsing_threads_with_globs
When reading from multiple files displace parallel parsing
2023-05-30 08:53:08 +02:00
alekseygolub
63225b37a2
Merge branch 'ClickHouse:master' into renamefile 2023-05-29 16:52:41 +03:00
Nikita Taranov
be62c8faba fix build 2023-05-29 15:27:39 +02:00
Nikita Taranov
a3652f1727 Merge branch 'master' into optimize_reading2 2023-05-29 15:26:18 +02:00
alesapin
6b7e2a0041
Merge pull request #49961 from CheSema/mem-consumption-s3-write
Introduce a limitation of async calls inside s3 write buffer, make a test that memory is bounded even when s3 is slow
2023-05-29 13:29:01 +02:00
alesapin
1bf32f70a0
Merge branch 'master' into fix-iceberg 2023-05-28 21:59:13 +02:00
alesapin
3a3cee586a
Merge pull request #50257 from CurtizJ/useless-refactoring
Yet another refactoring
2023-05-28 21:42:16 +02:00
Nikita Taranov
42a9072888 Merge branch 'master' into optimize_reading2 2023-05-28 18:04:44 +02:00
Sema Checherinda
206efee9b7 add test with merge tree table 2023-05-26 18:59:21 +02:00
Sema Checherinda
4249bda449 s3_max_inflight_parts_for_one_file with tests 2023-05-26 18:59:11 +02:00
Sergei Trifonov
0d1f2e297b
Unify priorities: lower value means higher priority (#50205) 2023-05-26 15:55:30 +02:00
Anton Popov
612173e734 refactoring near alter conversions 2023-05-25 22:54:54 +00:00
kssenii
3fefacbf20 Fix 2023-05-25 15:48:56 +02:00
Kseniia Sumarokova
f1a3c9cfd5
Merge pull request #50109 from kssenii/abstract-async-prefetched-buffer
Make async prefetched buffer work with arbitrary impl
2023-05-25 15:06:44 +02:00
Nikita Mikhaylov
1c3b6738f4
Fixes for parallel replicas (#50195) 2023-05-25 14:41:04 +02:00
Sema Checherinda
3329a8428d
Merge pull request #49779 from helifu/master3
Add 'partitions' field for system.query_log
2023-05-25 12:51:40 +02:00
Alexander Tokmakov
d8f936987e Merge branch 'master' into rmt_better_background_tasks_scheduling 2023-05-25 12:36:41 +02:00
何李夫
e4c8c4cecf
Add zookeeper name in endpoint id (#49780)
* Add zookeeper name in endpoint id

When we migrate a replicated table from one zookeeper cluster to
another (the reason why we migration is that zookeeper's load is
too high), we will create a new table with the same zpath, but it
will fail and the old table will be in trouble.

Here is some infomation:
1.old table:
  CREATE TABLE a1 (`id` UInt64)
  ENGINE = ReplicatedMergeTree('/clickhouse/tables/default/a1/{shard}', '{replica}')
  ORDER BY (id);
2.new table:
  CREATE TABLE a2 (`id` UInt64)
  ENGINE = ReplicatedMergeTree('aux1:/clickhouse/tables/default/a1/{shard}', '{replica}')
  ORDER BY (id);
3.error info:
  <Error> executeQuery: Code: 220. DB::Exception: Duplicate interserver IO endpoint:
          DataPartsExchange:/clickhouse/tables/default/a1/01/replicas/02.
          (DUPLICATE_INTERSERVER_IO_ENDPOINT)
  <Error> InterserverIOHTTPHandler: Code: 221. DB::Exception: No interserver IO endpoint
          named DataPartsExchange:/clickhouse/tables/default/a1/01/replicas/02.
          (NO_SUCH_INTERSERVER_IO_ENDPOINT)

* Revert "Add zookeeper name in endpoint id"

This reverts commit 9deb75b249619b7abdd38e3949ca8b3a76c9df8e.

* Add zookeeper name in endpoint id

When we migrate a replicated table from one zookeeper cluster to
another (the reason why we migration is that zookeeper's load is
too high), we will create a new table with the same zpath, but it
will fail and the old table will be in trouble.

* Fix incompatible with a new setting

* add a test, fix other issues

* Update 02442_auxiliary_zookeeper_endpoint_id.sql

* Update 02735_system_zookeeper_connection.reference

* Update 02735_system_zookeeper_connection.sql

* Update run.sh

* Remove the 'no-fasttest' tag

* Update 02442_auxiliary_zookeeper_endpoint_id.sql

---------

Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-05-25 12:50:14 +03:00
avogar
bc527c7588 Don't send head request for all keys in Iceberg schema inference 2023-05-24 17:07:31 +00:00
alesapin
7c0c49c9d2
Merge pull request #50154 from hanfei1991/hanfei/fix-modify-order-by
do not allow modify order by when there are no order by cols
2023-05-24 15:01:38 +02:00
helifu
2255b0287a Add 'partitions' field for system.query_log 2023-05-24 20:42:31 +08:00
Alexander Tokmakov
ffdd916694
Merge pull request #50180 from ClickHouse/tavplubix-patch-6
Update an exception message
2023-05-24 15:01:50 +03:00
Kruglov Pavel
9545100c9e
Merge pull request #45427 from attack204/urlCluster
Add urlCluster table function and refactor all *Cluster table functions
2023-05-24 13:32:56 +02:00
Alexander Tokmakov
486153d581
Update MergeTreeData.cpp 2023-05-24 13:33:28 +03:00
Amos Bird
8bbfdcc56c
Fix index analysis with binary operator null 2023-05-24 15:47:38 +08:00
alekseygolub
c37b420045
Merge branch 'ClickHouse:master' into renamefile 2023-05-24 09:51:07 +03:00
Amos Bird
b11aa42db9
Fix tests 2023-05-24 14:27:49 +08:00
Alexey Milovidov
3e1267c839
Merge pull request #50152 from ClickHouse/tavplubix-patch-6
Follow-up to #49889
2023-05-24 01:05:24 +03:00
Han Fei
037c5f8a06
Merge branch 'master' into hanfei/fix-modify-order-by 2023-05-23 20:48:27 +02:00
Alexander Tokmakov
f3b4959e05 fix 2023-05-23 20:45:16 +02:00
Igor Nikonov
2c01104c3f Clarification comment on retries controller behavior 2023-05-23 17:30:22 +00:00
Han Fei
584c05d8b8 fix modify order by when there was no order by cols 2023-05-23 18:54:36 +02:00
Raúl Marín
db4b3d19ae
Clearer coordinator log (#50101) 2023-05-23 17:30:27 +02:00
Alexander Tokmakov
64ee8ebb12
Update MutateTask.cpp 2023-05-23 18:11:08 +03:00
Amos Bird
b82ff979d0
Fix invalid index analysis for date related keys 2023-05-23 23:10:34 +08:00
SmitaRKulkarni
55af60ea3f
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-23 13:59:15 +02:00
Alexander Tokmakov
60a696b88b Merge branch 'master' into rmt_better_background_tasks_scheduling 2023-05-23 12:18:07 +02:00
Alexander Tokmakov
141a72d694
Merge pull request #49637 from ClickHouse/less_zookeeper_requests
Provide better partitions hint for merge selecting task
2023-05-23 12:40:39 +03:00
Alexander Tokmakov
dbf08b25fb better scheduling of merge selecting task 2023-05-23 01:25:17 +02:00
Alexander Tokmakov
ec6f07aa78 Merge branch 'less_zookeeper_requests' into rmt_better_background_tasks_scheduling 2023-05-22 23:51:36 +02:00
avogar
4f85d6a1bb Merge branch 'master' of github.com:ClickHouse/ClickHouse into random-structure 2023-05-22 19:43:24 +00:00
avogar
88e4c93abc Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-05-22 19:19:57 +00:00
kssenii
295fe3b228 Merge remote-tracking branch 'upstream/master' into abstract-async-prefetched-buffer 2023-05-22 20:01:38 +02:00
kssenii
c4d862a16f Make async reader work with any impl 2023-05-22 19:54:04 +02:00
Alexander Tokmakov
b8305503d8 more flexible cleanup thread scheduling 2023-05-22 19:07:18 +02:00
Nikolay Degterinsky
d4b89cb643
Merge pull request #49356 from Ziy1-Tan/vcol
Support for `_path` and `_file` virtual columns for table function `url`.
2023-05-22 18:10:32 +02:00
Nikolay Degterinsky
7bed59e1d2
Merge pull request #50000 from evillique/add-schema-inference
Add schema inference to more table engines
2023-05-22 17:24:30 +02:00
Alexander Tokmakov
821b64b420 apply review suggestions 2023-05-22 15:18:29 +02:00
Kruglov Pavel
b5cad024e0
Merge branch 'master' into urlCluster 2023-05-22 14:59:34 +02:00
alesapin
cc3897a84a
Merge pull request #50033 from kssenii/disk-object-storage-minor-changes
Get rid of indirect write buffer in object storages
2023-05-22 14:08:17 +02:00
Alexander Tokmakov
487e510103 Merge branch 'master' into less_zookeeper_requests 2023-05-22 13:59:26 +02:00
Alexander Tokmakov
c89f92e1f6
Merge pull request #50052 from amosbird/fix_49913
Fix reporting broken projection parts
2023-05-22 14:20:06 +03:00
Nikolay Degterinsky
183f90e45a Update MongoDB protocol 2023-05-22 09:05:23 +00:00
Antonio Andelic
93cd47fd5f
Merge pull request #50026 from ClickHouse/fix-deadlock-attach-thread
Avoid deadlock when starting table in attach thread of `ReplicatedMergeTree`
2023-05-22 09:22:33 +02:00
Amos Bird
0a3d986e42
Fix reporting projection broken part 2023-05-21 20:58:58 +08:00
Nikita Taranov
c93836b962 fix 2023-05-19 22:27:53 +00:00
zvonand
7dccb51bc0 fix conflict and update 2023-05-19 23:40:51 +02:00
alekseygolub
2b68a6a22a Fix style 2023-05-19 16:03:22 +00:00
alekseygolub
c85c3afa1f Added option to rename files, loaded via TableFunctionFile, after success processing 2023-05-19 16:03:22 +00:00
kssenii
3e42ee7f2b Get rid of finalize callback in object storages 2023-05-19 17:29:37 +02:00
Antonio Andelic
4af8187464 Activate restarting thread in both cases 2023-05-19 15:06:02 +00:00
mateng915
5237dd0245
New system table zookeeper connection (#45245)
* Feature: Support new system table to show which zookeeper node be connected

Description:
============
Currently we have no place to check which zk node be connected otherwise using
lsof command. It not convenient

Solution:
=========
Implemented a new system table, system.zookeeper_host when CK Server has zk
this table will show the zk node dir which connected by current CK server

Noted: This table can support multi-zookeeper cluster scenario.

* fixed review comments

* added test case

* update test cases

* remove unused code

* fixed review comments and removed unused code

* updated test cases for print host, port and is_expired

* modify the code comments

* fixed CI Failed

* fixed code style check failure

* updated test cases by added Tags

* update test reference

* update test cases

* added system.zookeeper_connection doc

* Update docs/en/operations/system-tables/zookeeper_connection.md

* Update docs/en/operations/system-tables/zookeeper_connection.md

* Update docs/en/operations/system-tables/zookeeper_connection.md

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-05-19 17:06:43 +03:00
Antonio Andelic
acf71c5b9a
Fix typo 2023-05-19 15:48:31 +02:00
Antonio Andelic
3107070e76 Avoid deadlock when starting table in attach thread 2023-05-19 12:48:19 +00:00
Nikolay Degterinsky
a09a8d60c4
Merge branch 'master' into postgresql-uuid 2023-05-19 13:44:17 +02:00
alesapin
e741450b88
Merge branch 'master' into fix_another_zero_copy_bug 2023-05-19 12:40:48 +02:00
alesapin
e5b001abda
Merge branch 'master' into fix_some_tests4 2023-05-19 12:34:03 +02:00
Michael Kolupaev
e84f0895e7 Support hardlinking parts transactionally 2023-05-18 21:05:56 -07:00
Nikolay Degterinsky
ef45956713 Fix style 2023-05-19 01:31:45 +00:00
Nikolay Degterinsky
b8be714830 Add schema inference to more table engines 2023-05-19 00:44:27 +00:00
Nikita Taranov
971cc092d4
Update src/Storages/MergeTree/MergeTreePrefetchedReadPool.cpp 2023-05-18 15:16:47 +02:00
SmitaRKulkarni
a91c793684
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-18 09:24:25 +02:00
Kseniia Sumarokova
1c04085e8f
Update MergeTreeWriteAheadLog.h 2023-05-17 18:15:51 +02:00
Han Fei
ed1d036151
Merge pull request #49884 from azat/dist-fix-async-block-processing
Fix processing pending batch for Distributed async INSERT after restart
2023-05-17 15:19:42 +02:00
Nikolay Degterinsky
194ce2d881 Better 2023-05-17 13:13:57 +00:00
Nikita Taranov
0dd67bacf2
Merge branch 'master' into optimize_reading2 2023-05-17 15:06:41 +02:00
Alexander Tokmakov
36c31e1d79
Improve concurrent parts removal with zero copy replication (#49630)
* improve concurrent parts removal

* fix

* fix
2023-05-17 14:07:34 +03:00
Vitaly Baranov
6c8a923c9d
Merge branch 'master' into write-encrypted-to-backup 2023-05-17 12:37:05 +02:00
Vitaly Baranov
f4ac4c3f9d Corrections after review. 2023-05-17 03:23:16 +02:00
Alexander Tokmakov
242a3fc520 Merge branch 'master' into less_zookeeper_requests 2023-05-16 18:24:11 +02:00
Alexander Tokmakov
0da82945ac fix 2023-05-16 18:18:48 +02:00
Kruglov Pavel
4530f38fdf
Merge branch 'master' into urlCluster 2023-05-16 16:21:23 +02:00
alesapin
50a536bba8 Remove unused code 2023-05-16 15:26:24 +02:00
Alexander Tokmakov
b6716a8f0f Merge branch 'master' into fix_some_tests4 2023-05-16 14:46:27 +02:00
Vitaly Baranov
943707963f Add backup setting "decrypt_files_from_encrypted_disks" 2023-05-16 14:27:27 +02:00
Vitaly Baranov
5198997fd8 Remove ReadSettings from backup entries. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
517e119e03 Move checksum calculation to IBackupEntry. 2023-05-16 14:27:27 +02:00
Vitaly Baranov
c92219f01b BACKUP now writes encrypted data for tables on encrypted disks. 2023-05-16 14:26:33 +02:00
Vitaly Baranov
cc50fcc60a Remove the 'temporary_file_' argument from BackupEntryFromImmutableFile's constructor. 2023-05-16 14:25:37 +02:00
alesapin
93bd09ddd6
Merge branch 'master' into fix_another_zero_copy_bug 2023-05-16 12:24:52 +02:00
alesapin
0b4ab70dd9
Merge pull request #49891 from hanfei1991/hanfei/chassert-1
use chassert in MergeTreeDeduplicationLog to have better log info
2023-05-16 11:50:11 +02:00
Sema Checherinda
03c51208d1
Merge pull request #44869 from CheSema/multi_part_upload
rework WriteBufferFromS3, add tests, add abortion
2023-05-16 10:52:01 +02:00
Nikolay Degterinsky
d9be88a36a Add UUID data type to PostgreSQL 2023-05-15 21:40:10 +00:00
Robert Schulze
59bc3e25be
Merge pull request #49824 from AVMusorin/allow-alias-column-kafka
KafkaEngine: Allow usage of Alias column type
2023-05-15 23:40:03 +02:00
Vitaly Baranov
801cacc13f
Merge pull request #49831 from vitlibar/fix-setting-null-in-profile-def
Fix setting NULL in profile definition
2023-05-15 22:24:49 +02:00
Alexander Tokmakov
65bc702b0b fix 2023-05-15 20:02:30 +02:00
Han Fei
4137a5e058 use chassert in MergeTreeDeduplicationLog to have better log info 2023-05-15 18:51:16 +02:00
avogar
334f062fa0 fix style 2023-05-15 16:39:26 +00:00
avogar
f9e0eb47d7 Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-05-15 16:35:24 +00:00
Alexander Tokmakov
05ae7b2c2d fix some tests 2023-05-15 18:28:12 +02:00
Azat Khuzhin
f2a023140e Fix processing pending batch for Distributed async INSERT after restart
After abnormal server restart current_batch.txt (that contains list of
files to send to the remote shard), may not have all files, if it was
terminated between unlink .bin files and truncation of current_batch.txt

But it should be fixed in a more reliable way, though to backport the
patch I kept it simple.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-15 15:57:30 +02:00
AVMusorin
418a61a68c
Allow using Alias column type for KafkaEngine
```
create table kafka
(
 a UInt32,
 a_str String Alias toString(a)
) engine = Kafka;

create table data
(
  a UInt32;
  a_str String
) engine = MergeTree
order by tuple();

create materialized view data_mv to data
(
  a UInt32,
  a_str String
) as
select a, a_str from kafka;
```
Alias type works as expected in comparison with MATERIALIZED/EPHEMERAL
or column with default expression.

Ref: https://github.com/ClickHouse/ClickHouse/pull/47138

Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
2023-05-15 15:39:58 +02:00
Alexander Tokmakov
25912a2673
Merge pull request #49876 from JackyWoo/fix_typo
fix typo
2023-05-15 13:32:58 +03:00
Kruglov Pavel
558eda4146
Merge pull request #49412 from azat/block-use-dense-hash-map
Switch Block::NameMap to google::dense_hash_map over HashMap
2023-05-15 12:22:55 +02:00
JackyWoo
8d1bcb5c2f fix typo 2023-05-15 16:51:20 +08:00
Alexey Milovidov
b4b5b4dcc5 Amend 2023-05-14 02:52:23 +02:00
Alexey Milovidov
0ca36d4f89 Merge branch 'master' into clang-17 2023-05-14 01:57:40 +02:00
Smita Kulkarni
fd58eac75a Fixed max_threads datatype issue for builds 2023-05-13 10:22:37 +02:00
Alexey Milovidov
a2d1cc1333
Merge pull request #49844 from MikhailBurdukov/in_with_tuple
TYPE_MISMATCH exception for in operator with  single column tuples.
2023-05-13 04:21:12 +03:00
Alexey Milovidov
5a44dc26e7 Fixes for clang-17 2023-05-13 02:57:31 +02:00
Anton Popov
401030e2be
Merge pull request #49757 from CurtizJ/add-forgotten-lock
Add forgotten lock (addition to #49117)
2023-05-12 23:55:09 +02:00
MikhailBurdukov
8c7b63f1e7 Fix 2023-05-12 16:42:01 +00:00
Smita Kulkarni
792565d858 Updated to ULL 2023-05-12 17:23:37 +02:00
avogar
3d00f4cdf6 Remove unneded files 2023-05-12 14:03:04 +00:00
avogar
70a8fd2c50 Fix schema inference with named collection, refactor Cluster table functions 2023-05-12 13:58:45 +00:00
Robert Schulze
b9c185af44
Merge pull request #49678 from azat/build/llvm-16
Switch to LLVM/clang 16 (16.0.3)
2023-05-12 13:47:36 +02:00
Sema Checherinda
a1ee7d52e1
Merge pull request #49432 from CheSema/lost-blobs
all s3-blobs removed when merge aborted, remove part from failed fetch without unlock keper
2023-05-12 13:19:27 +02:00
Sema Checherinda
5706e05757 adding more comments 2023-05-12 13:00:56 +02:00
Nikita Taranov
536c5ac261
Merge branch 'master' into optimize_reading2 2023-05-12 12:59:32 +02:00
Alexander Gololobov
7c7565f094
Merge pull request #49737 from ClickHouse/fix_lwd_and_object
Ignore LWD column in checkPartDynamicColumns
2023-05-12 12:20:07 +02:00
Robert Schulze
922420420c
Merge pull request #49300 from ClickHouse/rs/functdocs
Introduce more fields for in-source function documentation
2023-05-12 11:36:04 +02:00
Vitaly Baranov
eb62030fa4 Fix assigning a setting to NULL in settings profile's definition. 2023-05-12 11:04:30 +02:00
Azat Khuzhin
2c40dd6a4c Switch Block::NameMap to google::dense_hash_map over HashMap
Since HashMap creates 2^8 elements by default, while dense_hash_map
should be good here.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-12 05:52:57 +02:00
Alexey Milovidov
ea979b40a9
Merge pull request #49717 from ClickHouse/disable-mmap-for-server
Disable mmap for StorageFile in clickhouse-server
2023-05-11 23:53:01 +03:00
Anton Popov
d21dc829d9
Merge branch 'master' into add-forgotten-lock 2023-05-11 16:56:13 +02:00
Azat Khuzhin
e5c4eb35e8 Merge remote-tracking branch 'u/master' into build/llvm-16
* u/master:
  enable used flags's reinit only when the hash talbe rehash
  Fix build of libfiu on clang-16
  fix flaky test 02504_regexp_dictionary_ua_parser
  fix convertation
  Fix test that expected CH to apply a wrong optimization
  ActionsDAG: remove wrong optimization
2023-05-11 16:32:25 +02:00
avogar
3ee8de792c Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-05-11 12:46:20 +00:00
Smita Kulkarni
ef1100bb90 Added include to fix build issue 2023-05-11 14:44:36 +02:00
Sema Checherinda
8470aa8fbb
Merge branch 'master' into convertations_to_timespan 2023-05-11 08:24:40 +02:00
Sema Checherinda
66e3712dca
Merge branch 'master' into lost-blobs 2023-05-11 08:23:25 +02:00
Sema Checherinda
9724688608 rename field, do not fetch PreActive 2023-05-11 08:22:37 +02:00
Alexey Milovidov
cfd3f76217
Merge branch 'master' into disable-mmap-for-server 2023-05-11 06:17:32 +03:00
Anton Popov
ff287c7bba add lock 2023-05-10 23:00:49 +00:00
zvonand
ecd0b4cc06 update settings from master 2023-05-11 00:37:58 +02:00
Azat Khuzhin
d56e1423ef
Merge branch 'master' into build/llvm-16 2023-05-10 21:23:26 +02:00
avogar
9096f62efc Merge branch 'master' of github.com:ClickHouse/ClickHouse into random-structure 2023-05-10 18:46:19 +00:00
Sema Checherinda
7fbf87be17 rework WriteBufferFromS3, squashed 2023-05-10 18:31:47 +00:00
Alexander Gololobov
049ef9dda5
Merge branch 'master' into fix_lwd_and_object 2023-05-10 19:54:26 +02:00
Sema Checherinda
f2ad1122a1 fix convertation 2023-05-10 17:50:42 +00:00
Nikita Taranov
ec63a6ee7c
Merge branch 'master' into optimize_reading2 2023-05-10 17:24:37 +02:00
Alexey Milovidov
4a4eb5b171
Merge pull request #49660 from CurtizJ/fix-sparse-columns-reload
Fix reading from sparse columns after restart
2023-05-10 18:08:22 +03:00
Alexey Milovidov
33a58aacea
Merge pull request #49719 from ClickHouse/make-projections-non-experimental
Make projections production-ready
2023-05-10 18:07:02 +03:00
Azat Khuzhin
a0cc5b7e47 Merge remote-tracking branch 'u/master' into build/llvm-16 2023-05-10 16:28:36 +02:00
Alexander Gololobov
e6f4e5cd10 Ignore LWD column in checkPartDynamicColumns 2023-05-10 16:10:04 +02:00
Han Fei
d17aa828b3
Merge pull request #49361 from hanfei1991/hanfei/fiu
support named fault injection and add a simple test
2023-05-10 15:51:51 +02:00
zvonand
7762dde325 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-implicit-tz 2023-05-10 14:49:17 +02:00
Roman Vasin
632c0eeefc
Add support of temporary tables in FROM part of ATTACH|REPLACE PARTITION FROM (#49436)
* Add temp tables after FROM

* Resolve works fine with temp and non-temp tables

* Add usage of temporary table in  00626_replace_partition_from_table test

* Update documentation

* Add tests for ReplicatedMergeTree

* Move tests into 02731_replace_partition_from_temporary_table
2023-05-10 15:25:12 +03:00
Azat Khuzhin
2f9a7cba8a Fix stack-use-after-scope in KafkaProducer::produce()
ASan report:

    ==1==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f7540186b20 at pc 0x55656ef458ad bp 0x7f754114acf0 sp 0x7f754114a4b8
    READ of size 2 at 0x7f7540186b20 thread T2 (TCPHandler)
        0 0x55656ef458ac in __asan_memcpy (/usr/bin/clickhouse+0x11b028ac) (BuildId: 6ba63057df561f6dc16fb8fd7623fd858bf327d1)
        1 0x55659f62daf8 in rd_kafka_msg_new00 build_docker/./contrib/librdkafka/src/rdkafka_msg.c:192:3
        2 0x55659f62daf8 in rd_kafka_msg_new0 build_docker/./contrib/librdkafka/src/rdkafka_msg.c:257:8
        3 0x55659f630635 in rd_kafka_producev build_docker/./contrib/librdkafka/src/rdkafka_msg.c:642:23
        4 0x55659f4a865c in cppkafka::Producer::do_produce() build_docker/./contrib/cppkafka/src/producer.cpp:128:19
        5 0x55659f4a8306 in cppkafka::Producer::produce() build_docker/./contrib/cppkafka/src/producer.cpp:69:5
        6 0x5565961e748c in DB::KafkaProducer::produce() build_docker/./src/Storages/Kafka/KafkaProducer.cpp:70:23
        7 0x5565962155b2 in DB::MessageQueueSink::consume(DB::Chunk) build_docker/./src/Storages/MessageQueueSink.cpp:64:23
        8 0x5565980b9e8e in DB::SinkToStorage::onConsume(DB::Chunk) build_docker/./src/Processors/Sinks/SinkToStorage.cpp:18:5

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-10 12:15:17 +02:00
SmitaRKulkarni
5c030c428c
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs 2023-05-10 09:51:38 +02:00
Alexey Milovidov
8a6e07f0ea Make projections production-ready 2023-05-10 03:35:13 +02:00
Alexey Milovidov
a2c4b8e23d Disable mmap for server 2023-05-10 03:16:52 +02:00
Andrey Zvonov
9237b904be
Merge branch 'master' into zvonand-implicit-tz 2023-05-10 01:25:33 +02:00
Anton Popov
996fcfe120 fix column ttl with sparse columns 2023-05-09 21:37:43 +00:00
Alexander Tokmakov
6f53ee289d fix another issue, add a test 2023-05-09 23:26:56 +02:00
Azat Khuzhin
d03ae2abfa Fix modernize-loop-convert clang-tidy warning
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-09 21:19:37 +02:00
Alexey Milovidov
3556c26075
Merge pull request #49434 from ClickHouse/ins
Make INSERT do more things in parallel to avoid getting bottlenecked on one thread
2023-05-09 08:10:15 +03:00
Alexey Milovidov
a8bdb20fc4
Merge pull request #48787 from ClickHouse/background-memory-tracker
Add MemoryTracker for the background tasks [Resubmit]
2023-05-09 07:58:36 +03:00
Alexey Milovidov
9a73a04fe3
Merge pull request #42587 from ClickHouse/compress-marks-primary-key-by-default
Compress marks and primary key by default
2023-05-09 06:43:10 +03:00
Han Fei
64b67b9a37 Merge branch 'master' into hanfei/fiu 2023-05-09 01:14:30 +02:00
Alexander Tokmakov
aa829c4ddc better partitions hint in mutations finalization 2023-05-09 00:23:21 +02:00
Han Fei
82ce47ce56 address comments 2023-05-08 23:02:36 +02:00
Anton Popov
0612f5d782 fix sparse columns after reload 2023-05-08 11:50:21 +00:00
Kseniia Sumarokova
336bb41c5d
Merge branch 'master' into remove-dependency-from-context 2023-05-08 12:46:10 +02:00
Alexey Milovidov
3cf1da7bf3
Merge branch 'master' into deprecate-in-memory-parts 2023-05-08 01:38:00 +03:00
Alexander Tokmakov
91489243c9 provide better partitions hint for merge selecting task 2023-05-08 00:22:07 +02:00
Alexander Tokmakov
29bbde85a6 split selectPartsToMerge into smaller functions 2023-05-07 22:06:25 +02:00
robot-ch-test-poll
5078231c7e
Merge pull request #49619 from ClickHouse/remove_concurrent
Better concurrent parts removal with zero copy
2023-05-07 21:03:44 +02:00
alesapin
33e691ad51 Better name 2023-05-07 14:26:21 +02:00
alesapin
a2ed457219 Better concurrent parts removal with zero copy 2023-05-07 14:02:24 +02:00
Alexey Milovidov
6b742bb1d1
Merge branch 'master' into system-stack-trace-fix 2023-05-07 06:05:47 +03:00
Alexey Milovidov
3b6edbacab Update tests 2023-05-07 03:45:17 +02:00
Alexey Milovidov
de3e66385c Merge branch 'master' into compress-marks-primary-key-by-default 2023-05-07 02:40:25 +02:00
Sema Checherinda
08bfc8f58d inserted parts has exlusive blobs 2023-05-07 01:51:43 +02:00
Sema Checherinda
56f2917718
Update StorageReplicatedMergeTree.cpp 2023-05-06 17:59:07 +02:00
Alexey Milovidov
9efbe86110
Merge pull request #49563 from evillique/object-column-alter
Forbid the change of data type of Object('json') column
2023-05-06 18:17:16 +03:00
Robert Schulze
983e37048c
Merge pull request #48853 from ClickHouse/rs/qc-totals-extremes2
QueryCache: Support queries with totals and extremes
2023-05-06 15:41:10 +02:00
Igor Nikonov
c23600fb56
Merge pull request #49536 from ClickHouse/propa
Make input_format_parquet_preserve_order imply !parallelize_output_from_storages
2023-05-06 14:53:36 +02:00
Sema Checherinda
8f5afffaa0 Outdated parts has to be unlocked in general case 2023-05-06 14:23:57 +02:00
Sema Checherinda
f302c1986a use flag has_exclusive_blobs to track the part's originate 2023-05-06 03:07:29 +02:00
Alexey Milovidov
75a2589a42
Merge branch 'master' into deprecate-in-memory-parts 2023-05-06 00:49:59 +03:00
Robert Schulze
3dfc0bd265
Merge pull request #49413 from azat/build/headers
Slightly reduce inter-header dependencies
2023-05-05 23:37:58 +02:00
Robert Schulze
45c28e1221
Introduce more fields for in-source function documentation 2023-05-05 21:30:21 +00:00
Nikolay Degterinsky
d29193da51 Also prohibit complex types with Object('json') 2023-05-05 18:45:00 +00:00
Nikolay Degterinsky
f361d96f21 Forbid the change of data type of Object('json') column 2023-05-05 15:51:17 +00:00
Sema Checherinda
efcf7fbe4f
Update src/Storages/StorageReplicatedMergeTree.cpp
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-05-05 17:49:12 +02:00
Sema Checherinda
36b028bc6b
Update src/Storages/StorageReplicatedMergeTree.cpp
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-05-05 17:48:57 +02:00
Robert Schulze
aba5db40a1
Merge remote-tracking branch 'origin/master' into rs/qc-totals-extremes2 2023-05-05 14:11:10 +00:00
zvonand
8338d54c34 Merge branch 'master' of github.com:ClickHouse/ClickHouse into zvonand-implicit-tz 2023-05-05 15:50:26 +02:00
Nikolay Degterinsky
7a727bbbe3
Merge pull request #49520 from IvanTakarlikov-st/it-grammar-fixes
Fix some grammar mistakes in documentation, code and tests
2023-05-05 15:45:58 +02:00
wangxiaobo
f704c0dfec
Implement SYSTEM DROP REPLICA from auxillary ZooKeeper clusters (#48932)
* multiple zookeeper drop replica bug fix.

* add an integration test.

* format code.

* set stay_alive=True for test.

* style check bug fix.

* add check for is_active

* format code

* remove table_settings and has_metadata_out param.
2023-05-05 14:11:11 +03:00
Kseniia Sumarokova
7c203dbcd2
Merge pull request #49481 from snikch/master
Fix postgres database setting
2023-05-05 12:53:51 +02:00
Anton Popov
c87a6e8e7d
Merge pull request #49117 from CurtizJ/limit-mutations
Add settings to delay or throw in case of too many mutations
2023-05-05 12:18:57 +02:00
Alexey Milovidov
09f171822e Merge branch 'master' into deprecate-in-memory-parts 2023-05-05 06:43:23 +02:00
Michael Kolupaev
3bd1489f18 Propagate input_format_parquet_preserve_order to parallelizeOutputAfterReading() 2023-05-05 04:20:27 +00:00
Alexey Milovidov
ab1a1ea5b7
Merge pull request #49490 from ClickHouse/fix-s3cluster-with-headers
Correctly handle s3Cluster arguments
2023-05-05 05:31:23 +03:00
Michael Kolupaev
0fa311ee38 Make INSERT do more things in parallel to avoid getting bottlenecked on one thread 2023-05-05 00:49:27 +00:00
Michael Kolupaev
eb3b774ad0 Better control over Parquet row group size 2023-05-04 14:59:55 -07:00
Sema Checherinda
9a0bc891e4 fix can_remove_callback for Temporary parts 2023-05-04 19:39:16 +02:00
Ivan Takarlikov
8873856ce5 Fix some grammar mistakes in documentation, code and tests 2023-05-04 13:35:18 -03:00