thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 03:42:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
97,814 Commits 9,315 Branches 1,783 Tags 2.1 GiB
ab4db2d0c4
Commit Graph

9499 Commits

Author SHA1 Message Date
FArthur-cmd
0e89fbff53 refactoring 2022-07-12 13:57:16 +03:00
Alexander Tokmakov
b35ad0dbdc
Merge pull request #39091 from ClickHouse/tavplubix-patch-1 
Follow-up to #39036
 2022-07-12 13:47:37 +03:00
Nikolai Kochetov
945b5d4b3f
Merge pull request #38993 from azat/fix-fsync_part_directory-on-fetch 
Fix fsync_part_directory for fetches
 2022-07-12 11:48:42 +02:00
Nikolai Kochetov
cd4211298e Fix typo. 2022-07-12 09:37:00 +00:00
Nikolai Kochetov
03d9301425 Fix log. 2022-07-12 08:31:09 +00:00
Nikolai Kochetov
631be9219b Properly remove projection from part in case it was removed from table metadata. 2022-07-12 08:28:22 +00:00
FArthur-cmd
783e6e398c merge with other working branch 2022-07-11 23:02:58 +03:00
aaapetrenko
aabf0ed67f refactor Storage 2022-07-11 22:00:57 +02:00
FArthur-cmd
6ae227a6bb add test 2022-07-11 22:50:10 +03:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_" 
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
 2022-07-11 19:36:18 +02:00
Sergei Trifonov
6ed777d2a8 add azure support 2022-07-11 19:19:57 +02:00
Sergei Trifonov
add0d1698e respect new limits in HDFS 2022-07-11 17:30:23 +02:00
FArthur-cmd
868628d3fd make stable prototype with arrays support 2022-07-11 18:02:50 +03:00
Sergei Trifonov
43779ec280 add max_remote_{read,write}_network_bandwidth_for_server settings 2022-07-11 14:59:39 +02:00
Alexander Tokmakov
e9ddbc247f
Merge pull request #39016 from Algunenano/clearOldLogs_zkerror 
clearOldLogs: Don't report KEEPER_EXCEPTION on concurrent deletes
 2022-07-11 15:16:18 +03:00
Alexander Tokmakov
048d27189e
Update MergeTreeData.cpp 2022-07-11 15:07:31 +03:00
FArthur-cmd
7b7a9521c7 change submodule path 2022-07-10 18:10:47 +03:00
FArthur-cmd
2f9f5f172a Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy_3 2022-07-10 17:25:21 +03:00
kssenii
4c88527c8e Review fixes 2022-07-09 19:35:07 +02:00
kssenii
bf1c99daab Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
Frank Chen
da57a993e4 Fix CI 2022-07-09 13:43:10 +08:00
Alexander Tokmakov
51130d723c
Update MergeTreeData.cpp 2022-07-08 20:17:29 +03:00
Raúl Marín
a965438eea clearOldLogs: Don't report errors on concurrent deletes 2022-07-08 13:57:53 +02:00
Azat Khuzhin
b2f7f9c0f0 Fix fsync_part_directory for fetches 
Before it produces the following error:

    2022.07.07 23:40:03.131044 [ 44869 ] {} <Error> default.rep_data: auto DB::StorageReplicatedMergeTree::processQueueEntry(ReplicatedMergeTreeQueue::SelectedEntryPtr)::(anonymous class)::operator()(DB::StorageReplicatedMergeTree::LogEntryPtr &) const: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/tmp-fetch_36_25458_25458_0, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

    0. Poco::Exception::Exception() @ 0x1764634c in /usr/bin/clickhouse
    1. DB::Exception::Exception() @ 0xb42caba in /usr/bin/clickhouse
    2. DB::throwFromErrnoWithPath() @ 0xb42d79b in /usr/bin/clickhouse
    3. DB::LocalDirectorySyncGuard::LocalDirectorySyncGuard() @ 0x1404bbd4 in /usr/bin/clickhouse
    4. DB::DiskLocal::getDirectorySyncGuard() const @ 0x140466a4 in /usr/bin/clickhouse
    5. DB::DataPartsExchange::Fetcher::downloadPartToDisk() @ 0x15122af1 in /usr/bin/clickhouse
    6. DB::DataPartsExchange::Fetcher::fetchPart() @ 0x1511de39 in /usr/bin/clickhouse

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-08 13:48:51 +03:00
lgbo-ustc
e01b6cf8a5 update codes 2022-07-08 09:25:45 +08:00
Alexander Tokmakov
b0f8970d07
Merge pull request #38961 from Algunenano/reduce_zk_eexists 
Stop reporting Zookeeper "Node exists" exceptions in system.errors when they are expected
 2022-07-07 23:16:46 +03:00
Dmitry Novik
d1df66687b
Merge branch 'master' into group-by-use-nulls 2022-07-07 20:54:38 +02:00
Raúl Marín
9b0c7714e5 Style 2022-07-07 17:15:23 +02:00
Kseniia Sumarokova
48220f64d6
Merge pull request #38214 from bigo-sg/storagehive_alter 
support alter on `StorageHive` table
 2022-07-07 17:06:43 +02:00
Raúl Marín
c3935000a8 Avoid reporting ZK EEXIST errors on checkPartChecksumsAndCommit 2022-07-07 16:10:05 +02:00
Raúl Marín
33ffb71453 Avoid 'Node exists' errors when creating Ephemeral Locks 2022-07-07 16:10:04 +02:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
alesapin
1bd97fa6fa
Merge pull request #38851 from Algunenano/rmarin_read_only_on_drop 
Deactivate mutations_finalizing_task during shutdown
 2022-07-07 12:55:50 +02:00
Frank Chen
93dc109e36 Fix code 2022-07-07 17:44:19 +08:00
Frank Chen
d3d89f59ca Add tracing support to distributed insert 2022-07-07 17:43:09 +08:00
Frank Chen
57a7e4a7c9 Remove old API reference 2022-07-07 17:42:35 +08:00
FArthur-cmd
8968ff8995 small inmprovements 2022-07-07 11:33:06 +03:00
Amos Bird
19bd3d8d9f
Nullable partition pruning fix 2022-07-07 16:02:38 +08:00
lgbo-ustc
337f379bc3 update codes 2022-07-07 09:45:49 +08:00
lgbo-ustc
cda081dc57 update codes 2022-07-07 09:41:52 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9 
Backup Improvements 9
 2022-07-07 02:24:47 +02:00
FArthur-cmd
a3407ba6db Merge branch 'master' of github.com:FArthur-cmd/ClickHouse into annoy-2 2022-07-06 19:42:55 +03:00
Vitaly Baranov
7f84cf3968 Fix style. 2022-07-06 16:36:59 +02:00
Maksim Kita
b94489d52c
Merge pull request #38859 from kitaisreal/merge-tree-merge-disable-batch-optimization 
MergeTree merge disable batch optimization
 2022-07-06 15:59:40 +02:00
Vladimir C
3c7fdb604d
Merge pull request #35363 from vdimir/kv-join 2022-07-06 15:57:30 +02:00
FArthur-cmd
c94d6a57ec add arrays 2022-07-06 14:39:49 +03:00
Raúl Marín
86d1c319f4 Remove unnecessary log 2022-07-06 12:23:45 +02:00
lgbo-ustc
aeb4f9c6be fixed: crash caused by IHiveFile be shared among threads 2022-07-06 16:56:17 +08:00
lgbo-ustc
946a9b8c63 update codes 2022-07-06 09:26:02 +08:00
Kseniia Sumarokova
a26643a7a6
Merge pull request #38363 from kssenii/pg-add-auto-close 
Support `auto_close` option for postgres engine connection
 2022-07-06 01:10:05 +02:00
kssenii
e55169ebad Fix 2022-07-05 20:15:35 +02:00
Vitaly Baranov
f9204315b5 Store columns.txt in backups for the Memory table engine too. 2022-07-05 19:03:20 +02:00
Kseniia Sumarokova
3e803006a2
Merge pull request #37171 from tchepavel/nats-integration 
Nats Integration
 2022-07-05 17:47:02 +02:00
Maksim Kita
bdc21737d5 MergeTree merge disable batch optimization 2022-07-05 16:15:00 +02:00
Vitaly Baranov
43d35eec1b Write unfinished mutations to backup. 2022-07-05 14:51:09 +02:00
Raúl Marín
9e17ae365b Deactivate mutations_finalizing_task during shutdown 2022-07-05 14:28:20 +02:00
vdimir
17903117d2
better 2022-07-05 10:46:08 +00:00
kssenii
7a4be3b748 Fix build, disable for freebsd 2022-07-05 12:10:49 +02:00
Kruglov Pavel
a6a9824261
Merge pull request #38362 from Avogar/better-exception-messages 
Better exception messages on wrong table engines/functions argument types
 2022-07-05 11:21:44 +02:00
lgbo-ustc
76813ee3c0 update codes 2022-07-05 11:49:55 +08:00
kssenii
b52084265c Merge master 2022-07-04 21:37:43 +02:00
Kseniia Sumarokova
6c3d648f49
Merge branch 'master' into nats-integration 2022-07-04 21:06:07 +02:00
Kseniia Sumarokova
0cd228cab3
Merge pull request #38365 from kssenii/better-rabbit-adddress-setting 
Fix RabbitMQ configuration via connection string setting
 2022-07-04 20:53:43 +02:00
vdimir
c0cb588e45
Fix build 2022-07-04 17:28:34 +00:00
vdimir
43afda408b
do not call setStorageJoin when setting isn't set 2022-07-04 17:28:15 +00:00
vdimir
65ac97ce89
direct join: handle nullable, add tests 2022-07-04 17:28:14 +00:00
vdimir
8359753a85
Move DirectJoin to separate file 2022-07-04 17:28:11 +00:00
vdimir
5ed0911ff4
Fix rocksdb::getSlicedKeys 2022-07-04 17:28:10 +00:00
vdimir
085e70c7a3
wip left kvjoin 2022-07-04 17:28:10 +00:00
vdimir
980000d04c
Change getByKeys, add StorageEmbeddedRocksDB::getByKeysImpl 2022-07-04 17:28:09 +00:00
vdimir
744c692be3
wip key value join 2022-07-04 17:28:09 +00:00
vdimir
8bce6451c6
add method StorageEmbeddedRocksDB::getByKeys 2022-07-04 17:27:37 +00:00
Anton Popov
8b356f0e1b better tests and comments 2022-07-04 15:06:43 +00:00
Dmitry Novik
c66e2c8095
Merge pull request #37155 from amosbird/projection-fix-two 
Fix intermediate header in aggregate projection
 2022-07-04 16:32:34 +02:00
FArthur-cmd
21df8ed12f refactoring 2022-07-04 17:21:50 +03:00
Kruglov Pavel
6c4b6fa834
Merge branch 'master' into better-exception-messages 2022-07-04 15:07:31 +02:00
kssenii
cfff7c4c28 Merge master 2022-07-04 14:13:26 +02:00
FArthur-cmd
21b99e7304 Merge branch 'master' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-07-04 14:54:21 +03:00
Vitaly Baranov
1dba3d9f9c Make temporary files in better places. 2022-07-04 11:02:58 +02:00
lgbo-ustc
932e596689 Merge remote-tracking branch 'ck/master' into storagehive_alter 2022-07-04 09:56:00 +08:00
Raúl Marín
838fd1e41e Fix replication after improper merge process 2022-07-03 15:11:25 +02:00
Vitaly Baranov
92e0ee0b6f More detailed error messages. 2022-07-03 14:20:19 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder 2022-07-02 15:39:44 +02:00
kssenii
831d2253ce Merge master 2022-07-01 18:53:20 +02:00
Anton Popov
ef87e1207c better support of read_in_order in case of fixed prefix of sorting key 2022-07-01 16:45:01 +00:00
Dmitry Novik
81dd90893e Merge remote-tracking branch 'origin/master' into group-by-use-nulls 2022-07-01 16:24:05 +00:00
alesapin
0d7298f3a9 Fix typos 2022-07-01 15:26:27 +02:00
alesapin
57284e6a9d Fix possible deadlocks with MergeTreeData::Transaction 2022-07-01 15:16:32 +02:00
Vitaly Baranov
cadb496894
Merge pull request #38537 from vitlibar/backup-improvements-8 
Backup Improvements 8
 2022-07-01 11:20:46 +02:00
Amos Bird
d73f975849
Fix intermediate header in aggregate projection 2022-07-01 16:47:32 +08:00
alesapin
09faa1641e Some comments 2022-06-30 23:09:32 +02:00
alesapin
979565edf0 Merge branch 'master' into better_data_part_storage_builder 2022-06-30 22:54:42 +02:00
alesapin
eb5046ab26 Simplify everything 2022-06-30 22:51:27 +02:00
Alexander Tokmakov
60dcae9af0
Merge pull request #38627 from Algunenano/rmarin_fix_replicated_part_2 
Adapt some more nodes to avoid issues with pre-22.4 replicas
 2022-06-30 21:11:35 +03:00
Alexander Tokmakov
d2cbdc7c53
Update ReplicatedMergeTreeQueue.cpp 2022-06-30 17:23:53 +03:00
Maksim Kita
a0e5768956
Merge pull request #38623 from loyispa/patch 
Fix typo
 2022-06-30 16:02:36 +02:00
Vitaly Baranov
e367d96964 Fix style. 2022-06-30 15:10:33 +02:00
Raúl Marín
cbcd740dc1 Adapt some more nodes to avoid issues with pre-22.4 replicas 2022-06-30 15:10:09 +02:00
avogar
ee54c4f9b7 Add some fixes and add settings in docs 2022-06-30 12:41:56 +00:00
alesapin
1363b478d4 Merge branch 'fix_disk_tx_write_op' into better_data_part_storage_builder 2022-06-30 14:38:32 +02:00
Anton Popov
7c721578c7
Merge pull request #38320 from CurtizJ/dynamic-columns-16 
Improve performace of insertion to columns of type JSON
 2022-06-30 14:18:20 +02:00
alesapin
cb90aca2ef Some changes in merge tree 2022-06-30 14:12:45 +02:00
loyispa
109c9bcbd5 Fix typo 2022-06-30 18:37:19 +08:00
Vitaly Baranov
5456bde4a2 Improve gathering metadata for storing ACL in backups. 2022-06-30 09:46:37 +02:00
Antonio Andelic
de264117fd
Merge pull request #38118 from bigo-sg/storagehive_struct_type 
Add struct type support in `StorageHive`
 2022-06-30 09:11:13 +02:00
Vitaly Baranov
031ca28fdc Add test for partition clause. More checks for data compatibility on restore. 2022-06-30 08:37:18 +02:00
Vitaly Baranov
11b51d2878 Implement storing UDF in backups. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
efbee5e723 Improve gathering metadata for backup - part 7. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
01921ce9a3 Move most code from setTableStructure() to a separate function. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
7689e0c36f Improve gathering metadata for backup - part 6. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
461a31f237 Improve gathering metadata for backup - part 2. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
64b51a3772 Improve gathering metadata for backup. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
47ac47350b Store projections in backups. 2022-06-30 08:37:17 +02:00
alesapin
67f2fa0ef0 Fix style 2022-06-29 15:18:48 +02:00
alesapin
2f03e821bd Fix stupid bug 2022-06-29 15:08:16 +02:00
alesapin
ac963a7663 Remove redundant check 2022-06-29 13:55:20 +02:00
Alexander Tokmakov
8246e55002
Merge pull request #38486 from azat/fix-parts-removal 
Fix parts removal after incorrect server shutdown
 2022-06-29 14:17:59 +03:00
Alexander Tokmakov
32c6a944ee
Merge pull request #38541 from Algunenano/rmarin_fix_replicated_create_table_compat 
Fix table creation to avoid replication issues with pre-22.4 replicas
 2022-06-29 14:15:50 +03:00
mergify[bot]
1280fc39a0
Merge branch 'master' into better-exception-messages 2022-06-29 11:09:06 +00:00
Kruglov Pavel
b9813bcdea
Merge pull request #38356 from kssenii/fix-rabbitmq-with-peekable 
Fix RabbitMQ with formats based on PeekableReadBuffer
 2022-06-29 13:06:10 +02:00
Alexander Tokmakov
ceb66ade4b
Merge pull request #38335 from ClickHouse/deprecate_ordinary_database 
Deprecate Ordinary database and old *MergeTree syntax
 2022-06-29 13:42:59 +03:00
alesapin
c80a4c27be Merge branch 'master' into better_data_part_storage_builder 2022-06-29 12:33:42 +02:00
Alexey Milovidov
4cda5491f6
Merge pull request #38542 from ClickHouse/unique_lock-to-lock_guard 
Don't use std::unique_lock unless we need to
 2022-06-29 09:40:56 +03:00
Nikita Taranov
f5d26572df
Quick fix for aggregation pipeline (#38295) 2022-06-29 01:16:30 +02:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to 
Replace where possible by std::lock_guard which is more light-weight.
 2022-06-28 19:19:06 +00:00
Raúl Marín
9b4da86e2e Adapt table creation with replication to avoid issues with pre-22.4 replicas 2022-06-28 20:12:07 +02:00
avogar
106f92dcdb Fix tests 2022-06-28 16:13:42 +00:00
Alexander Tokmakov
428628238f Merge branch 'master' into deprecate_ordinary_database 2022-06-28 15:40:33 +02:00
alesapin
6429b72371 Fixes 2022-06-28 14:41:22 +02:00
alesapin
c7620b8217 Use builder instead of disk 2022-06-28 13:19:30 +02:00
kssenii
4c99f608b3 Fix 2022-06-28 13:18:21 +02:00
mergify[bot]
ea416c877f
Merge branch 'master' into better-exception-messages 2022-06-28 11:13:26 +00:00
alesapin
a68bf8fff2 Missed changes 2022-06-28 12:59:54 +02:00
alesapin
0a3fab1cb6 Some sad changes 2022-06-28 12:51:49 +02:00
alesapin
49add19842 Merge branch 'better_data_part_storage_builder' of github.com:ClickHouse/ClickHouse into better_data_part_storage_builder 2022-06-28 12:02:27 +02:00
alesapin
1a4aa8e52e Merge branch 'master' into better_data_part_storage_builder 2022-06-28 11:51:06 +02:00
Alexander Tokmakov
851534c9f7 Merge branch 'master' into deprecate_ordinary_database 2022-06-27 23:47:49 +02:00
Azat Khuzhin
e70cff2cea Fix parts removal after incorrect server shutdown 
Before this patch, and after transaction support had been added #24258,
if the server had not removed old parts on shutdown, then they will left
forever.

Fixes: #24258 (cc @tavplubix)
v2: use Tx::PrehistoricTID over removePartsFromWorkingSet() from ctor
v3: apply black to new test
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-27 23:56:25 +03:00
Azat Khuzhin
f0271dc486 Refactor deactive_part() a little 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-27 23:42:39 +03:00
alesapin
d88cdcd5c1 AAA 2022-06-27 21:41:29 +02:00
Yakov Olkhovskiy
d5f65ece9b
Merge pull request #38105 from arenadata/ADQM-419 
Add kerberosInit function as a replacement for kinit executable calls in Kafka and HDFS
 2022-06-27 14:19:24 -04:00
alesapin
180b4cd64d Merge branch 'master' into better_data_part_storage_builder 2022-06-27 19:46:17 +02:00
Alexander Tokmakov
88dbbcb15d fix tests 2022-06-27 16:10:00 +02:00
avogar
12448f9400 Fix style 2022-06-27 13:56:43 +00:00
Alexander Tokmakov
bec921c9d9
Merge pull request #38265 from ClickHouse/cleanup_garbage_in_store_dir 
Add background cleanup of store/ subdirs
 2022-06-27 16:53:13 +03:00
Kseniia Sumarokova
be283d1c12
Update RabbitMQSource.cpp 2022-06-27 15:39:22 +02:00
Kseniia Sumarokova
cadc202b35
Update RabbitMQSource.cpp 2022-06-27 15:37:58 +02:00
Alexander Tokmakov
7c37079a96 Merge branch 'master' into deprecate_ordinary_database 2022-06-27 14:46:59 +02:00
avogar
5155262a16 Add some additional information to cache keys 2022-06-27 12:43:24 +00:00
kssenii
2c5aeaaa1a Add auto close for postgres connection 2022-06-27 13:46:52 +02:00
kssenii
5c26f99bc5 Merge master 2022-06-27 13:36:41 +02:00
Nikita Taranov
2487ba7f00
Move updateInputStream to ITransformingStep (#37393) 2022-06-27 13:16:52 +02:00
alesapin
50eb364a56
Merge pull request #38436 from kssenii/remote-path-separation-in-object-storage 
more consistent work with paths in object storages
 2022-06-27 10:58:01 +02:00
alesapin
83b584b2dd Something 2022-06-26 18:43:28 +02:00
mergify[bot]
f63c959679
Merge branch 'master' into cleanup_garbage_in_store_dir 2022-06-26 13:35:10 +00:00
Alexey Milovidov
b3098822e0
Merge pull request #38171 from ClickHouse/hyper-to-vectorscan 
Replace hyperscan by vectorscan
 2022-06-26 10:01:45 +03:00
kssenii
725d80d470 get rid of path separation 2022-06-26 01:17:19 +02:00
kssenii
03f8b974ce get rid of path separation 2022-06-25 20:30:36 +02:00
Kseniia Sumarokova
a6e868dddc
Update RabbitMQSource.cpp 2022-06-25 11:26:08 +02:00
alesapin
e2a779b315 Merge with new changes 2022-06-25 00:29:45 +02:00
alesapin
80d27f6381
Merge pull request #38380 from ClickHouse/simplify_parts_commit_merge_tree 
Simplify parts commit methods
 2022-06-25 00:12:52 +02:00
kssenii
708b93a03b Merge master 2022-06-25 00:08:24 +02:00
alesapin
8ccc47c202 Revert some changes 2022-06-24 23:54:10 +02:00
Alexander Gololobov
189d0fffb8
Merge pull request #37165 from ClickHouse/merge_tree_reader 
Merge tree reader support for multiple read/filter steps: row level filter, prewhere, ...
 2022-06-24 21:32:31 +02:00
alesapin
a22c77e118 Merge branch 'master' into better_data_part_storage_builder 2022-06-24 19:52:41 +02:00
Vadim Volodin
3296ba2532 Fix SYSTEM UNFREEZE for ordinary database 2022-06-24 20:46:16 +03:00
Kseniia Sumarokova
5e86d6c55b
Merge pull request #38360 from kssenii/add-check-materialized-pg 
Fix possible `Invalid number of rows in Chunk` in materialized pg
 2022-06-24 19:33:15 +02:00
alesapin
2bc4dc4a1d Fix accident change 2022-06-24 18:38:43 +02:00
Alexander Tokmakov
31dcc7634e Merge branch 'master' into deprecate_ordinary_database 2022-06-24 18:16:07 +02:00
alesapin
d963f262f8 Fix style 2022-06-24 17:43:18 +02:00
alesapin
f685cf2268 Fix comment 2022-06-24 17:33:43 +02:00
alesapin
011d58d7a0 Simplify more 2022-06-24 17:19:59 +02:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache 2022-06-24 16:10:25 +02:00
avogar
23022441a6 Fix style 2022-06-24 14:01:12 +00:00
avogar
ba69193db1 Fix build 2022-06-24 14:00:22 +00:00
alesapin
612c4571d5 Split method into smaller 2022-06-24 15:41:09 +02:00
alesapin
7517e1f4d5 Remove some complexity 2022-06-24 15:24:02 +02:00
mergify[bot]
36b93772b9
Merge branch 'master' into merge_tree_reader 2022-06-24 13:22:53 +00:00
alesapin
c57e07cb49 Style 2022-06-24 14:51:47 +02:00
alesapin
37310dc9df Simpler 2022-06-24 14:10:15 +02:00
avogar
8b88df8b81 Fix build 2022-06-24 11:58:12 +00:00
alesapin
36b810b076 Remove unused parameter 2022-06-24 13:42:36 +02:00
alesapin
af1a9d18ab Remove transaction argument 2022-06-24 13:34:00 +02:00
alesapin
9910395823 Simplify method signature 2022-06-24 13:19:29 +02:00
Robert Schulze
2c828338f4
Replace hyperscan by vectorscan 
This commit migrates ClickHouse to Vectorscan. The first 10 min of
[0] explain the reasons for it.

(*) Addresses (but does not resolve) #38046

(*) Config parameter names (e.g. "max_hyperscan_regexp_length") are
    preserved for compatibility. Likewise, error codes (e.g.
    "ErrorCodes::HYPERSCAN_CANNOT_SCAN_TEXT") and function/class names (e.g.
    "HyperscanDeleter") are preserved as vectorscan aims to be a drop-in
    replacement.

[0] https://www.youtube.com/watch?v=KlZWmmflW6M
 2022-06-24 10:47:52 +02:00
Kruglov Pavel
982fcfc68a
Fix build 2022-06-24 00:54:05 +02:00
avogar
59c1c472cb Better exception messages on wrong table engines/functions argument types 2022-06-23 20:04:06 +00:00
kssenii
bc9b56096f Fix 2022-06-23 21:52:57 +02:00
Alexander Tokmakov
74f38710a8 Merge branch 'master' into cleanup_garbage_in_store_dir 2022-06-23 21:43:28 +02:00
kssenii
6fbd49f554 Merge master 2022-06-23 21:40:01 +02:00
Alexander Tokmakov
dbf2763788 automatically convert system database to Atomic 2022-06-23 21:38:43 +02:00
kssenii
e362e4fff1 Fix 2022-06-23 20:48:32 +02:00
alesapin
2bdedf5e0c Some changes 2022-06-23 18:21:46 +02:00
kssenii
468c98ff42 Better 2022-06-23 17:46:27 +02:00
mergify[bot]
174d8577fa
Merge branch 'master' into merge_tree_reader 2022-06-23 15:16:51 +00:00
kssenii
f7b329ee57 Merge master 2022-06-23 14:56:48 +02:00
mergify[bot]
234f0c6399
Merge branch 'master' into revert-35914-FIPS_compliance 2022-06-23 12:06:17 +00:00
alesapin
5661280ef6 Remove some read methods 2022-06-23 14:01:26 +02:00
Anton Popov
7efbae7728
Merge pull request #38069 from CurtizJ/better-support-gcp 
Better support of GCP storage
 2022-06-23 13:19:14 +02:00
alesapin
ae2feacbd1
Merge pull request #38322 from ClickHouse/ban_projections_in_a_right_way 
Ban projections for zero-copy replication in a right way
 2022-06-23 11:52:30 +02:00
lgbo-ustc
cd8e5c7c49 update headers 2022-06-23 17:43:54 +08:00
Alexander Tokmakov
f00e6b5a7a deprecate old MergeTree syntax 2022-06-23 11:24:54 +02:00
Alexander Tokmakov
2c5a88faba deprecate Ordinary database 2022-06-23 10:20:14 +02:00
Kseniia Sumarokova
af2150a68d
Merge pull request #38176 from kssenii/url-headers-in-ast 
Add ability to pass headers to url table function / storage via sql
 2022-06-23 10:15:07 +02:00
lgbo-ustc
c1770c22b9 Merge remote-tracking branch 'ck/master' into storagehive_struct_type 2022-06-23 15:54:20 +08:00
lgbo-ustc
0fba75b21d Merge remote-tracking branch 'ck/master' into storagehive_alter 2022-06-23 09:38:06 +08:00
alesapin
1b495ec8ad Fix style 2022-06-23 00:17:37 +02:00
kssenii
e40d9bcf55 Merge master 2022-06-22 23:28:52 +02:00
Anton Popov
52db1b35a1 improve performace of insertion to columns of type JSON 2022-06-22 17:45:51 +00:00
Alexander Gololobov
0ee47363d4 Fixed includes 2022-06-22 19:08:18 +02:00
Alexander Gololobov
8138a3a36e Cleanups based on review comments 2022-06-22 19:08:01 +02:00
alesapin
ca0299e927 Ban projections for zero-copy replication in a right way 2022-06-22 19:01:46 +02:00
Alexander Tokmakov
ed8341025b make code less bad 2022-06-22 18:31:42 +02:00
Alexander Gololobov
e5b55b965b Removed incorrect check 2022-06-22 17:23:09 +02:00
Alexander Gololobov
dbc6d1a159 Cleanups 2022-06-22 17:23:09 +02:00
Alexander Gololobov
aea5bc7527 Removed unused should_reorder flag 2022-06-22 17:23:09 +02:00
Alexander Gololobov
5427d5c51b Apply filter at row level security step 2022-06-22 17:23:09 +02:00
Alexander Gololobov
b3922461b3 Properly handle empty actions 2022-06-22 17:23:09 +02:00
Alexander Gololobov
ba89c3954c Do not add the same vitrual if it has been added by prev_reader 2022-06-22 17:23:09 +02:00
Alexander Gololobov
e5c0ebc7b5 Handle read from wide part when no columns are requested 2022-06-22 17:23:09 +02:00
Alexander Gololobov
a9e3b8d29e Don't read the same columns again 2022-06-22 17:23:09 +02:00
Alexander Gololobov
4e426c63cc Debuging test failures 2022-06-22 17:23:09 +02:00
Alexander Gololobov
6a26325fab Test dirty hacks for multiple PREWHERE steps 2022-06-22 17:23:05 +02:00
Alexander Gololobov
4360fd9798 Vector of structs instead of multiple vectors 2022-06-22 17:17:42 +02:00
Alexander Gololobov
87b669f439 Intermediate changes 2022-06-22 17:17:42 +02:00
Alexander Gololobov
159ab765fb Moved estimateNumRows from lambda to function 2022-06-22 17:17:42 +02:00
Alexander Gololobov
538c8c96fc Dump row_level_filter as a part of prewhere_info 2022-06-22 17:17:42 +02:00
Alexander Gololobov
64a2f3734b Protect ReadResult internals from MergeTreeRangeReader clients 2022-06-22 17:17:42 +02:00
kssenii
0acc95fe22 Review fixes 2022-06-22 16:55:47 +02:00
alesapin
0f8eed98c4
Merge branch 'master' into better-support-gcp 2022-06-22 12:46:01 +02:00
Kruglov Pavel
e5a7f53775
Fix misleading error message while s3 schema inference 2022-06-22 12:36:09 +02:00
alesapin
d3bc7c0190
Merge pull request #37971 from Avogar/fix-alter-columns-with-dots 
Fix ALTER column with column names with dots
 2022-06-22 12:20:30 +02:00
Nikolai Kochetov
cc6fdfe0eb
Merge pull request #36555 from ClickHouse/refactor-something-in-part-volumes 
Separate data storage abstraction for MergeTree
 2022-06-22 11:13:36 +02:00
Alexey Milovidov
0f8bc027fd Merge branch 'revert-35914-FIPS_compliance' of github.com:ClickHouse/ClickHouse into revert-35914-FIPS_compliance 2022-06-22 06:17:01 +02:00
Anton Popov
d5d568e235
Merge pull request #37913 from azat/parts-refcnt 
Fix refcnt for unused MergeTree parts in SELECT queries
 2022-06-21 22:56:26 +02:00
kssenii
4178abc0a5 Merge master 2022-06-21 22:38:51 +02:00
kssenii
90ce0b44f0 Fixes 2022-06-21 21:03:18 +02:00
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa 
Support for Clang Thread Safety Analysis (TSA)
 2022-06-21 20:19:33 +02:00
avogar
13a05adcb9 Some fixes 2022-06-21 17:34:08 +00:00
avogar
c14364e3d9 Check last modification time for URL function too 2022-06-21 17:18:14 +00:00
Nikolai Kochetov
dcf6e0d060 Roll back some changes from IPartMetadataManager 2022-06-21 13:05:49 +00:00
avogar
d37ad2e6de Implement cache for schema inference for file/s3/hdfs/url 2022-06-21 13:02:48 +00:00
Kruglov Pavel
8bf78dbc95
Add comment 2022-06-21 14:32:12 +02:00
Nikolai Kochetov
854d148d73 Add some comments. Remove some commented code. 2022-06-21 12:31:02 +00:00
kssenii
4656dd3de3 Headers for url in ast 2022-06-21 14:25:05 +02:00
Nikolai Kochetov
543782436d Process additional filters. 2022-06-21 11:24:46 +00:00
Nikolai Kochetov
46c4fd6cf7 Fixing build. 2022-06-21 11:15:06 +00:00
Kruglov Pavel
3c5a627bd7
Merge pull request #38239 from CurtizJ/fix-reading-from-s3 
Fix reading from s3 in some corner cases
 2022-06-21 12:57:55 +02:00
mergify[bot]
f304fc9179
Merge branch 'master' into fix-alter-columns-with-dots 2022-06-21 10:40:53 +00:00
Nikolai Kochetov
1e8c9ecd4c Merge branch 'master' into refactor-something-in-part-volumes 2022-06-21 12:37:21 +02:00
alesapin
fafcb876be
Merge pull request #38182 from ClickHouse/disk_transaction 
Disk transaction
 2022-06-21 12:28:49 +02:00
Nikolai Kochetov
b8d27aa8dd
Merge pull request #37469 from azat/projections-optimize_aggregation_in_order 
Implement in order aggregation (optimize_aggregation_in_order) for projections for tables with fully materialized projections
 2022-06-21 12:17:35 +02:00
Kseniia Sumarokova
71ee5dbbb2
Merge pull request #38227 from kssenii/diff-reduce 
buffer's getFileSize small changes
 2022-06-21 12:12:22 +02:00
Nikolai Kochetov
e45d552f4d Do not create empty folder for in-memory parts. 2022-06-21 09:59:46 +00:00
Nikolai Kochetov
5232c17b89 Fix style 2022-06-21 07:30:45 +00:00
Nikolai Kochetov
3429a5e07a Fix 2022-06-21 07:26:43 +00:00
lgbo-ustc
fc641d9ce4 some changes 2022-06-21 11:29:55 +08:00
Larry Luo
bbd73ba727 use utility methods to access x509 struct fields. 2022-06-20 21:27:33 -04:00
Nikolai Kochetov
dccf90b1ea Cleanup. 2022-06-20 18:18:17 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Anton Popov
59a9ce6def fix reading from s3 in some corner cases 2022-06-20 15:24:38 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
Nikolai Kochetov
39b02d4ab8 Continure resolving conflicts. 2022-06-20 15:39:05 +02:00
Nikolai Kochetov
7452d04e3a Merge branch 'master' into refactor-something-in-part-volumes 2022-06-20 15:31:02 +02:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
Nikolai Kochetov
ef8b967c3f Fix more tests. 2022-06-20 13:07:14 +00:00
kssenii
90be49faa4 Merge master 2022-06-20 14:26:56 +02:00
Dmitry Novik
77177917bd
Merge pull request #37848 from ClickHouse/window-function-expression 
Support expressions with window functions
 2022-06-20 14:21:13 +02:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6 
Backup Improvements 6
 2022-06-20 14:13:30 +02:00
alesapin
e3184d264b Merge branch 'master' into disk_transaction 2022-06-20 13:47:30 +02:00
mergify[bot]
4ed606a275
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 10:17:51 +00:00
Kseniia Sumarokova
c83594284d
Merge pull request #38205 from azat/fix-window-view 
(Window View is an experimental feature) Fix LOGICAL_ERROR for WINDOW VIEW with incorrect structure
 2022-06-20 11:47:45 +02:00
lgbo-ustc
efec6f9ac5 support alter on storagehive 2022-06-20 14:57:33 +08:00
Vitaly Baranov
638ea23399 Fix build. 2022-06-20 03:44:59 +02:00
mergify[bot]
670a63865e
Merge branch 'master' into window-function-expression 2022-06-19 22:14:54 +00:00
Alexey Milovidov
7700c26076
Merge pull request #38196 from ClickHouse/revert-38194-revert-37015-zstd_window_log_max 
Revert "Revert "Add a setting to use more memory for zstd decompression""
 2022-06-19 19:58:03 +03:00
Azat Khuzhin
bc3e73d776 Fix LOGICAL_ERROR for WINDOW VIEW with incorrect structure 
Caching header of the source table in the WINDOW VIEW should not be
done, since there is no ability to get notification when it had been
changed (ALTER or CREATE/DROP).

And this fires on [CI], when the following tests had been executed in
order in stress tests:
- 01050_window_view_parser_tumble (leaves wm for mt)
- 01748_partition_id_pruning (cache input_header)
- 01188_attach_table_from_path (insert into mt with wm attached and
  incorrect structure)

  [CI]: https://s3.amazonaws.com/clickhouse-test-reports/38056/109980eb275c064d08bc031bfdc14d95b9a7272b/stress_test__undefined__actions_.html

Follow-up for: #37965 (@Vxider)
Fixes: #37815
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-19 19:16:39 +03:00
alesapin
119ce90e5b Fix strange diff 2022-06-19 18:03:43 +02:00
alesapin
08b5dd091b Fix double negative 2022-06-19 17:12:04 +02:00
Vitaly Baranov
8a7c970ce0 Fix style. 2022-06-19 15:58:26 +02:00
alesapin
50801e41c5 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-19 14:05:46 +02:00
Alexey Milovidov
04b7a34870 Maybe fix error 2022-06-19 05:21:59 +02:00
Alexey Milovidov
73709b0488
Revert "Revert "Add a setting to use more memory for zstd decompression"" 2022-06-18 15:55:35 +03:00
alesapin
16e8b85fbf
Revert "Add a setting to use more memory for zstd decompression" 2022-06-18 14:08:14 +02:00
Alexey Milovidov
e20259e9ca
Merge pull request #37015 from wuxiaobai24/zstd_window_log_max 
Add a setting to use more memory for zstd decompression
 2022-06-18 04:19:27 +03:00
Vitaly Baranov
a0c558a17e Implement backup/restore for ACL system tables (system.users, system.roles, etc.) 2022-06-17 18:14:31 +02:00
Alexander Tokmakov
83adf56383 fix race 2022-06-17 18:13:57 +02:00
Alexander Tokmakov
e9c2157f01 Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-17 15:57:30 +02:00
Alexander Tokmakov
5ffc38d9b9 fix style check, add comments 2022-06-17 13:47:14 +02:00
alesapin
e5b405701a Merge branch 'master' into disk_transaction 2022-06-17 12:16:12 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache 
Extract some diff from pr #36171
 2022-06-17 12:07:31 +02:00
mergify[bot]
296905771f
Merge branch 'master' into better-support-gcp 2022-06-17 01:19:11 +00:00
Anton Popov
1523c9c9e5 fix filling of empty Nested + small refactoring 2022-06-17 01:10:52 +00:00
Alexander Tokmakov
39c0219c11 fixes 2022-06-16 19:41:32 +02:00
Dmitry Novik
376412e417 Small refactoring 2022-06-16 15:41:04 +00:00
Mikhail f. Shiryaev
2bef5257b7
Merge pull request #38147 from ClickHouse/22.7-prepare 
Update version after release
 2022-06-16 17:19:40 +02:00
Mikhail f. Shiryaev
06dd85f921
Update version to 22.7.1.1 2022-06-16 17:15:22 +02:00
Anton Popov
13ec7e3092
Merge pull request #37978 from CurtizJ/fix-sparse-s3 
Fix reading of sparse columns from s3
 2022-06-16 15:32:54 +02:00
alesapin
379fff7318 Merge branch 'master' into disk_transaction 2022-06-16 14:36:19 +02:00
Roman Vasin
ed3fe84b63 Fix runKinit() is called only for USE_KRB5 2022-06-16 14:45:27 +03:00
Roman Vasin
0ab6bfd5d8 Add warning about using krb5 parameter in StorageKafka.cpp 2022-06-16 14:25:37 +03:00
Roman Vasin
6e28275569 Add warnings about using krb5 parameters 2022-06-16 14:21:04 +03:00
Roman Vasin
d93fd3bd2d Add complilation support for case when krb5 is not used 2022-06-16 09:30:40 +00:00
lgbo-ustc
a8b17fec84 fixed a bug 2022-06-16 16:53:11 +08:00
lgbo-ustc
35d534c213 nested struct in struct 2022-06-16 16:45:05 +08:00
Azat Khuzhin
4ff82eb9d5 Implement optimize_aggregation_in_order for projections 
v2: use real column name instead of aliases from GROUP BY
    Fixes the following error in 01710_projection_aggregation_in_order:

        Not found column a in block. There are only columns: toStartOfHour(ts), sum(value). (NOT_FOUND_COLUMN_IN_BLOCK)
v2.1: Get back support for projected and non-projected parts
v2.2: merge tests and rename
v3: Reduce copy-paste for optimize_aggregation_in_order for projections
v4: rebase on top of QueryPlanResourceHolder
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
14f9491619 MergeTreeData: preserve order for group_by_elements_order_descr 
This is required for proper optimize_aggregation_in_order for
projections.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
4694929623 Implement merging only for AggregatingStep 
v2: fill AggregateColumnsConstData only for only_merge
    (fixes 01291_aggregation_in_order and some other tests)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
682c93e2e1 Use proper setting compile_aggregate_expressions in MergeTreeDataSelectExecutor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:35 +03:00
lgbo-ustc
e115e3f731 remove unused header 2022-06-16 09:53:04 +08:00
lgbo-ustc
655e42c9bc remove trace logs 2022-06-16 09:44:41 +08:00
lgbo-ustc
4f13521aa6 struct type support for storage hive 2022-06-16 09:35:34 +08:00
mergify[bot]
7f24574609
Merge branch 'master' into better-support-gcp 2022-06-15 23:18:20 +00:00
Anton Popov
5b01f0f341 Merge remote-tracking branch 'upstream/master' into HEAD 2022-06-15 23:01:22 +00:00
Alexander Tokmakov
6ddfec47d2 maybe fix livelock on queue processing 2022-06-15 22:22:27 +02:00
Alexander Tokmakov
5e2b42c85a Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-15 20:47:02 +02:00
Vitaly Baranov
c2c35fad82 Refactoring of the code getting create table queries for backup. 2022-06-15 20:32:35 +02:00
Vitaly Baranov
1198e86295 Fix storing temporary tables and skipping system tables while making a backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
21f3bed435 Simplify path calculations in backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
592f568f83 Move backup/restore code to storages and databases - part 2. 2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57 Move backup/restore code to storages and databases - part 1. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
ce1836f0d2 Lock tables for share before backup and restore. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21 Rework collecting replicated parts. 2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Daniel Kutenin
a769dea8ef
Merge branch 'ClickHouse:master' into master 2022-06-15 19:12:10 +01:00
Roman Vasin
344fbe8de4 Fix code style 2022-06-15 20:26:42 +03:00
Roman Vasin
1c26424371 Change message in StorageKafka; Code style correction 2022-06-15 19:35:21 +03:00
Nikolai Kochetov
25b9990737 Fixing build 2022-06-15 16:32:22 +00:00
Alexander Tokmakov
2ac72319bd
Merge pull request #37185 from amosbird/projection-fix-three 
Fix possible heap-use-after-free error when reading system.projection_parts and system.projection_parts_columns
 2022-06-15 19:00:10 +03:00
kssenii
416c6008ee Merge master 2022-06-15 16:50:25 +02:00
kssenii
500f49972b Extract diff from PR 36171 2022-06-15 16:40:18 +02:00
Alexander Tokmakov
e4d4d0a854 fix race on queue processing 2022-06-15 16:21:26 +02:00
Roman Vasin
dd5b0ee065 Add kerberosInit() function to call KeberosInit 2022-06-15 17:02:53 +03:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Danila Kutenin
08e3f77a9c Optimize most important parts with NEON SIMD 
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.

That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
 2022-06-15 13:19:29 +00:00
alesapin
1b66166501 Merge branch 'master' into disk_transaction 2022-06-15 13:54:10 +02:00
alesapin
b557c1dd80 Interm stage 2022-06-15 13:51:21 +02:00
Nikolai Kochetov
2ee349f29c Fix test_encrypted_disk 2022-06-15 11:42:59 +00:00
Nikolai Kochetov
a274bec0fa Fix test_encrypted_disk 2022-06-15 11:41:08 +00:00
Alexander Tokmakov
56f2121c1a
Revert "Add backoff to merges in replicated queue if always_fetch_merged_part is enabled" 2022-06-15 13:04:01 +03:00
Roman Vasin
9bf6b9d491 Add kinit presence handling in StorageKafka; Cleanup code in HDFSCommon 2022-06-15 11:37:02 +03:00
Anton Popov
a59be0fd5d better support of GCP storage 2022-06-15 00:23:45 +00:00
alesapin
af1cd745e1
Merge pull request #37975 from kssenii/clean-up-broken-detached 
Clean up broken detached parts after timeout
 2022-06-14 20:53:31 +02:00
Amos Bird
d5a7a5be8e
Fix use-after-free in system.projection_parts 2022-06-14 23:41:42 +08:00
Maksim Kita
da8b1b1eba
Merge pull request #38025 from kitaisreal/use-base-sort-instead-of-standard 
Use pdqsort instead of standard sort
 2022-06-14 12:13:35 +02:00
alesapin
3b832a25f6
Merge pull request #37995 from ClickHouse/fix_queue 
Add backoff to merges in replicated queue if `always_fetch_merged_part` is enabled
 2022-06-14 11:38:44 +02:00
kssenii
0d2dae0046 Add const, rename settings 2022-06-14 00:47:03 +02:00
kssenii
74cffaf186 Merge master 2022-06-14 00:45:19 +02:00
kssenii
4813b90281 Merge master 2022-06-13 21:32:00 +02:00
kssenii
34e63b20fb Review fixes 2022-06-13 21:27:54 +02:00
Anton Popov
e6517164f2 add test and comment 2022-06-13 17:30:58 +00:00
Antonio Andelic
b9cf6fe367 Merge branch 'master' into pp-system-unfreeze 2022-06-13 14:46:30 +00:00
Maksim Kita
98a89b50ff Use pdqsort instead of standard sort 2022-06-13 15:31:08 +02:00
alesapin
d984763b95
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-06-13 15:10:20 +02:00
kssenii
debe467ee7 Review fixes 2022-06-13 14:25:42 +02:00
Antonio Andelic
e67b6c3125 Fix trailing whitespace 2022-06-13 09:53:16 +00:00
kssenii
4859e11c27 Review fixes 2022-06-12 01:15:10 +02:00
alesapin
2a81904fbc Revert "Revert "Fix bug with removal of moving parts"" 
This reverts commit c0581bf741.
 2022-06-12 01:08:24 +02:00
mergify[bot]
3b5d1b5efe
Merge branch 'master' into fix_queue 2022-06-11 22:05:11 +00:00
alesapin
8c088bd9ee
Merge pull request #37994 from ClickHouse/fix_removal_moving_parts 
Fix bug with removal of failed to move parts
 2022-06-11 23:54:43 +02:00
kssenii
aaf9a46f07 Merge master 2022-06-11 15:56:05 +02:00
Kseniia Sumarokova
a4e080d144
Merge pull request #37965 from Vxider/wv-add-missing-default 
Add missing default columns when pushing to target table in WindowView
 2022-06-11 13:38:08 +02:00
Vxider
ea2375ffc5 remove duplicate lines 2022-06-10 18:58:53 +00:00
alesapin
c0581bf741 Revert "Fix bug with removal of moving parts" 
This reverts commit e32133ff8f.
 2022-06-10 19:35:57 +02:00
alesapin
301eac7d59 Comment 2022-06-10 19:34:03 +02:00
alesapin
ffff1c22a6 Add backoff for merges when always_fetch_merged_part turned on 2022-06-10 19:33:28 +02:00
alesapin
e32133ff8f Fix bug with removal of moving parts 2022-06-10 18:47:06 +02:00
Kseniia Sumarokova
0ae2168fb6
Merge pull request #36328 from bigo-sg/async_hdfs_read_buffer 
Apply read_method 'threadpool' for StorageHive
 2022-06-10 15:04:21 +02:00
kssenii
fd6c84b831 Review fixes 2022-06-10 14:23:13 +02:00
Anton Popov
6a0b305a14 fix reading of sparse columns from s3 2022-06-10 12:09:37 +00:00
kssenii
498f389c21 Merge master 2022-06-10 12:49:10 +02:00
kssenii
0fd2c4b2c0 Continuation 2022-06-10 12:33:29 +02:00
kssenii
7a2676c7ab Clean up broken detached parts with timeout 2022-06-10 12:27:57 +02:00
Kseniia Sumarokova
508676ab4e
Merge pull request #37804 from ClickHouse/metadata_storage 
Metadata storage abstraction for Disks
 2022-06-10 11:43:28 +02:00
avogar
a4f378f8c3 Fix ALTER column with column names with dots 2022-06-10 08:58:00 +00:00
Vxider
b09dbeed77 support default columns 2022-06-10 07:24:26 +00:00
Yakov Olkhovskiy
349f1b140a
Merge pull request #37850 from bharatnc/ncb/build-options-ssl-version 
expose boring ssl version info in the system.build_options table
 2022-06-09 18:26:02 -04:00
Alexander Tokmakov
7c30d9b5d3
Merge pull request #37839 from ClickHouse/try_fix_37664 
Do not count covered unexpected parts on sanity check
 2022-06-09 18:22:37 +03:00
Alexander Tokmakov
780f7c87c7
Merge pull request #36113 from ClickHouse/remove-useless-code-2 
Remove useless code in ReplicatedMergeTreeRestartingThread
 2022-06-09 17:56:28 +03:00
Roman Vasin
d1d6d87432 Cleanup code in KerberosInit 2022-06-09 11:51:15 +03:00
Roman Vasin
3cfea6e76f Cleanup code in KerberosInit, HDFSCommon and StorageKafka; update English and Russian documentation. 2022-06-08 17:57:45 +03:00
alesapin
378f9246ca Rename one method 2022-06-08 14:57:10 +02:00
alesapin
56a57e649f First try new interface 2022-06-08 14:56:51 +02:00
Vadim Volodin
637d293fbd Add SYSTEM UNFREEZE query 2022-06-08 15:21:14 +03:00
alesapin
81dca513de Review fixes 2022-06-08 13:28:33 +02:00
Nikolai Kochetov
5bc9b32025 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-08 11:10:06 +00:00
alesapin
b88e1a2bef Merge branch 'master' into metadata_storage 2022-06-08 13:02:21 +02:00
alesapin
9fdc783eac
Merge pull request #37906 from ClickHouse/remove_tmp_directories_on_replicated 
Remove temporary `delete_tmp_` on clickhouse server start
 2022-06-08 12:35:51 +02:00
Roman Vasin
2b76d0c6a9 Add new integration test for kerberized Kafka; remove old kinit code from HDFSCommon 2022-06-08 12:26:35 +03:00
Azat Khuzhin
61a2613127 Fix refcnt for unused MergeTree parts in SELECT queries 
Before this patch SELECT queries hold parts even if they were not
required by select (had been eliminated by partition pruning).

This defers removing parts if you have long running queries.

This had been introduced in #23932, with introduction of
StorageSnapshotPtr.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-08 07:04:53 +03:00
taiyang-li
57b6cf6c09 fix build error 2022-06-08 09:58:09 +08:00
alesapin
e32d36d790 Proper fix 2022-06-07 17:58:32 +02:00
alesapin
5741ac2b9e Remove redundant line 2022-06-07 17:41:14 +02:00
Nikolai Kochetov
678d978acf Merge branch 'master' into refactor-something-in-part-volumes 2022-06-07 15:23:00 +00:00
alesapin
e7b76f5dee Remove temporary directories on replicated table start 2022-06-07 16:32:29 +02:00
Bharat Nallan Chakravarthy
4855794246 Merge remote-tracking branch 'upstream/master' into ncb/build-options-ssl-version 2022-06-07 19:02:50 +05:30
mergify[bot]
2d01abf871
Merge branch 'master' into revert-37647-Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-07 13:32:30 +00:00
Roman Vasin
a156a77890 Add KerberosInit into StorageKafka 2022-06-07 14:59:46 +03:00
Nikita Taranov
d09ad5fcd5
Fix build 
https://s3.amazonaws.com/clickhouse-test-reports/36593/a4b9614385e0ecd1654d15f507159678758ec557/fast_test__actions_/build_log.txt
 2022-06-07 13:56:03 +02:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type" 2022-06-07 13:53:10 +02:00
Anton Popov
ef6f5a6500
Merge pull request #37570 from azat/column-ttl-expired-fix 
Do not write expired columns by TTL after subsequent merges
 2022-06-07 13:05:03 +02:00
Kseniia Sumarokova
c4b1f537cd
Merge pull request #37879 from Vxider/wv-inner-target-drop 
Do not drop the inner target table when alter WindowView
 2022-06-07 13:00:15 +02:00
Kseniia Sumarokova
edc6b68801
Merge pull request #37853 from Vxider/fire-when-inserted 
Fire and clean windows in WindowView only when data is inserted
 2022-06-07 11:24:50 +02:00
Roman Vasin
cb53aa15ec Fix HDFSCommon and test_storage_kerberized_hdfs to make running integration tests 2022-06-07 12:06:22 +03:00
Vitaly Baranov
d199478169
Merge pull request #37303 from ClickHouse/fix_trash 
Try to fix some trash
 2022-06-07 10:17:39 +02:00
taiyang-li
73a484256e Merge branch 'master' into async_hdfs_read_buffer 2022-06-07 12:16:46 +08:00
taiyang-li
c65c56fd48 fix typo 2022-06-07 09:58:29 +08:00
mergify[bot]
e237e44b69
Merge branch 'master' into fire-when-inserted 2022-06-06 19:22:55 +00:00
mergify[bot]
dd0603a595
Merge branch 'master' into metadata_storage 2022-06-06 18:59:45 +00:00
Anton Popov
d40b23272e
Merge pull request #37755 from CurtizJ/fix-mutations-again 
Return back #37266
 2022-06-06 20:22:03 +02:00
Vxider
acda04a3d4 do not drop inner target table when alter windowview 2022-06-06 05:06:02 +00:00
taiyang-li
b36d9f8143 refactor readinto 2022-06-06 12:58:22 +08:00
Robert Schulze
2d87af2a15
Merge pull request #37647 from DevTeamBK/Fix-all-CheckTriviallyCopyableMove-Errors 
Fix errors of CheckTriviallyCopyableMove type
 2022-06-05 19:58:47 +02:00
alesapin
a25a01af16 Fix WAL 2022-06-05 16:15:11 +02:00
alesapin
38df94666b Fix merge tree wal 2022-06-05 15:09:39 +02:00
alesapin
a2704bc541 Trying to fix WAL log 2022-06-05 01:05:15 +02:00
Vxider
d4f0e8d4c2 update code style 2022-06-04 17:14:42 +00:00
Vxider
1f10276686 fire and clean only when data inserted 2022-06-04 16:55:46 +00:00
alesapin
083b15fea0 bug impl 2022-06-04 17:33:07 +02:00
Alexey Milovidov
4576b29e57
Update StorageReplicatedMergeTree.cpp 2022-06-04 14:33:29 +03:00
Bharat Nallan Chakravarthy
5fb9336159 add ssl version to build options 2022-06-04 15:31:08 +05:30
Alexey Milovidov
f860407af5
Merge pull request #37777 from ClickHouse/avx512_tail_zero 
Use multitarget framework for numZerosInTail implementation
 2022-06-04 01:46:56 +03:00
Nikolai Kochetov
3f67abc3b0 Fix some tests. 2022-06-03 18:49:12 +00:00
Kseniia Sumarokova
7632de911d
Merge pull request #37807 from Vxider/deley-cleanup 
Delay schedule of the cleaning task in WindowView
 2022-06-03 19:48:57 +02:00
Roman Vasin
8b5bf02927 Add support of cache commands in KerberosInit 2022-06-03 18:07:18 +03:00
Alexander Tokmakov
2358878c38 do not count covered unexpected parts 2022-06-03 16:32:02 +02:00
mergify[bot]
ddf7210ecc
Merge branch 'master' into remove-useless-code-2 2022-06-03 13:58:45 +00:00
Maksim Kita
8469e0d10d
ExecutableLambdaAdapter forward job and callback (#37830) 2022-06-03 15:54:47 +02:00
Nikita Mikhaylov
fe477355df
Added support for schema inference for hdfsCluster [2] (#35812) 2022-06-03 15:54:29 +02:00
alesapin
ac03e3af3e
Merge branch 'master' into metadata_storage 2022-06-03 14:42:21 +02:00
Anton Popov
5cf2c67410
Merge branch 'master' into fix-mutations-again 2022-06-03 14:17:05 +02:00
taiyang-li
f202c35311 Merge branch 'master' into async_hdfs_read_buffer 2022-06-03 17:52:09 +08:00
Kseniia Sumarokova
9c0e838404
Merge pull request #37770 from Vxider/fix-empty-target-id 
Fix empty target table id in WindowView
 2022-06-03 11:23:35 +02:00
Roman Vasin
82ce2d76c3 Add KerberosInit class; add kerberos_init console example; modify HDFSCommon.cpp 2022-06-03 12:06:31 +03:00
Vxider
d3c907cf12 log exception when fire 2022-06-02 18:22:48 +00:00
Vxider
1282d038cc delay schedule of windowview cleanup 2022-06-02 18:21:56 +00:00
Nikolai Kochetov
8991f39412 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 17:00:08 +00:00
alesapin
46bccae078 Trying to create metadata layer 2022-06-02 18:09:40 +02:00
Nikolai Kochetov
25c8a437ea Fixing build. 2022-06-02 15:00:07 +00:00
kssenii
92a799a0c4 Add ObjectStorageProxy, CachedObjectStorage 2022-06-02 16:45:21 +02:00
HeenaBansal2009
e3080f2a97 Merge remote-tracking branch 'origin' into Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-02 07:30:08 -07:00
Alexander Tokmakov
a893595025
Merge pull request #37596 from hanfei1991/hanfei/zk-write 
Support insert into system.zookeeper
 2022-06-02 15:09:07 +03:00
Kruglov Pavel
37a4d11c64
Merge pull request #37645 from azat/fix-write 
Fix possible "Cannot write to finalized buffer"
 2022-06-02 14:01:30 +02:00
mergify[bot]
fbfaaa9c74
Merge branch 'master' into fix-mutations-again 2022-06-02 11:57:19 +00:00
Alexander Tokmakov
45657d203a
Merge pull request #37763 from ClickHouse/fix_replace_range_again 
Fix flaky 01154_move_partition_long
 2022-06-02 14:15:16 +03:00
Alexander Gololobov
0b004b907a Added AVX2 implementation of numZerosInTail 2022-06-02 13:01:13 +02:00
Nikolai Kochetov
fc726c236f
Merge pull request #37661 from ClickHouse/no_alias_actions 
Trying to remove prewhere_info.alias_actions as they seem to always be included in prewhere_actions
 2022-06-02 12:31:04 +02:00
Nikolai Kochetov
89c5855d20 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-02 12:19:07 +02:00
Nikita Mikhaylov
d34e051c69
Support for simultaneous read from local and remote parallel replica (#37204) 2022-06-02 11:46:33 +02:00
Nikolai Kochetov
edac3d6714 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 09:36:20 +00:00
Alexander Gololobov
d8e4dcf9e2 Use multitarget framework for numZerosInTail implementation 2022-06-02 11:23:44 +02:00
Igor Nikonov
8c40f05d4c
Merge pull request #37674 from xlwh/cleanup_unused 
Cleanup unused file
 2022-06-02 10:18:29 +02:00
mergify[bot]
88bed9de20
Merge branch 'master' into fix_replace_range_again 2022-06-02 07:05:44 +00:00
Alexander Gololobov
a0cf902d49
Merge pull request #37588 from yaqi-zhao/avx512_tail_zero 
add avx512 support for mergetree reader
 2022-06-02 08:51:52 +03:00
Han Fei
1424c420fa try to fill in right metadata columns 2022-06-02 13:41:37 +08:00
Vxider
8221fcd5f1 update var name 2022-06-02 04:38:45 +00:00
Vxider
df4db70bb8 fix empty target table id 2022-06-02 04:28:18 +00:00
lthaooo
6632616733
Fix TTL merge scheduling bug (#36387) 2022-06-01 21:09:53 +02:00
Azat Khuzhin
545a56ce45 Fix sinks with onException() handler 
It is possible to call onException() even after onFinish(), in case of
onFinish() throws, and in this case onException() should be no-op for
such sinks.

Also there can be caveats with PartitionedSync.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-01 21:50:30 +03:00
Azat Khuzhin
02af58f41d Fix possible "Cannot write to finalized buffer" 
It is still possible to get this error since onException does not
finalize format correctly.

Here is an example of such error, that was found by CI [1]:

<details>

    [ 2686 ] {fa01bf02-73f6-4f7f-b14f-e725de6d7f9b} <Fatal> : Logical error: 'Cannot write to finalized buffer'.
    [ 34577 ] {} <Fatal> BaseDaemon: ########################################
    [ 34577 ] {} <Fatal> BaseDaemon: (version 22.6.1.1, build id: AB8040A6769E01A0) (from thread 2686) (query_id: fa01bf02-73f6-4f7f-b14f-e725de6d7f9b) (query: insert into test_02302 select number from numbers(10) settings s3_truncate_on_insert=1;) Received signal Aborted (6)
    [ 34577 ] {} <Fatal> BaseDaemon:
    [ 34577 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fcbaa5a703b 0x7fcbaa586859 0xfad9bab 0xfad9e05 0xfaf6a3b 0x24a48c7f 0x258fb9b9 0x258f2004 0x258b88f4 0x258b863b 0x2581773d 0x258177ce 0x24bb5e98 0xfad01d6 0xfad0105 0x2419b11d 0xfad01d6 0xfad0105 0x2215afbb 0x2215aa48 0xfad01d6 0xfad0105 0xfcc265d 0x225cc546 0x249a1c40 0x249bc1b6 0x2685902c 0x26859505 0x269d7767 0x269d504c 0x7fcbaa75e609 0x7fcbaa683163
    [ 34577 ] {} <Fatal> BaseDaemon: 3. raise @ 0x7fcbaa5a703b in ?
    [ 34577 ] {} <Fatal> BaseDaemon: 4. abort @ 0x7fcbaa586859 in ?
    [ 34577 ] {} <Fatal> BaseDaemon: 5. ./build_docker/../src/Common/Exception.cpp:47: DB::abortOnFailedAssertion(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfad9bab in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 6. ./build_docker/../src/Common/Exception.cpp:70: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xfad9e05 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 7. ./build_docker/../src/IO/WriteBuffer.h:0: DB::WriteBuffer::write(char const*, unsigned long) @ 0xfaf6a3b in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 8. ./build_docker/../src/Processors/Formats/Impl/ArrowBufferedStreams.cpp:47: DB::ArrowBufferedOutputStream::Write(void const*, long) @ 0x24a48c7f in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 9. long parquet::ThriftSerializer::Serialize<parquet::format::FileMetaData>(parquet::format::FileMetaData const*, arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) @ 0x258fb9b9 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 10. parquet::FileMetaData::FileMetaDataImpl::WriteTo(arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) const @ 0x258f2004 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 11. parquet::WriteFileMetaData(parquet::FileMetaData const&, arrow::io::OutputStream*) @ 0x258b88f4 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 12. parquet::ParquetFileWriter::~ParquetFileWriter() @ 0x258b863b in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 13. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x2581773d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 14. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x258177ce in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 15. ./build_docker/../src/Processors/Formats/Impl/ParquetBlockOutputFormat.h:27: DB::ParquetBlockOutputFormat::~ParquetBlockOutputFormat() @ 0x24bb5e98 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 16. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 17. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:312: std::__1::unique_ptr<DB::WriteBuffer, std::__1::default_delete<DB::WriteBuffer> >::reset(DB::WriteBuffer*)
    [ 34577 ] {} <Fatal> BaseDaemon: 18.2. inlined from ../contrib/libcxx/include/__memory/unique_ptr.h:269: ~unique_ptr
    [ 34577 ] {} <Fatal> BaseDaemon: 18. ../src/Storages/StorageS3.cpp:566: DB::StorageS3Sink::~StorageS3Sink() @ 0x2419b11d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 19. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 20. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 21. ./build_docker/../contrib/abseil-cpp/absl/container/internal/raw_hash_set.h:1662: absl::lts_20211102::container_internal::raw_hash_set<absl::lts_20211102::container_internal::FlatHashMapPolicy<StringRef, std::__1::shared_ptr<DB::SinkToStorage> >, absl::lts_20211102::hash_internal::Hash<StringRef>, std::__1::equal_to<StringRef>, std::__1::allocator<std::__1::pair<StringRef const, std::__1::shared_ptr<DB::SinkToStorage> > > >::destroy_slots() @ 0x2215afbb in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./build_docker/../contrib/libcxx/include/string:1445: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long() const
    [ 34577 ] {} <Fatal> BaseDaemon: 22.2. inlined from ../contrib/libcxx/include/string:2231: ~basic_string
    [ 34577 ] {} <Fatal> BaseDaemon: 22. ../src/Storages/PartitionedSink.h:14: DB::PartitionedSink::~PartitionedSink() @ 0x2215aa48 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 23. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 24. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 25. ./build_docker/../contrib/libcxx/include/vector:802: std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >::__base_destruct_at_end(std::__1::shared_ptr<DB::IProcessor>*) @ 0xfcc265d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 26.1. inlined from ./build_docker/../contrib/libcxx/include/vector:402: ~vector
    [ 34577 ] {} <Fatal> BaseDaemon: 26.2. inlined from ../src/QueryPipeline/QueryPipeline.cpp:29: ~QueryPipeline
    [ 34577 ] {} <Fatal> BaseDaemon: 26. ../src/QueryPipeline/QueryPipeline.cpp:535: DB::QueryPipeline::reset() @ 0x225cc546 in /usr/bin/clickhouse
    [ 614 ] {} <Fatal> Application: Child process was terminated by signal 6.

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37542/8a224239c1d922158b4dc9f5d6609dca836dfd06/stress_test__undefined__actions_.html

Follow-up for: #36979

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-01 21:50:30 +03:00
tchepavel
75db37cb57 Small engine overhaul 2022-06-01 21:30:08 +03:00
Alexander Tokmakov
06f80770b8 fix stuck REPALCE_RANGE 2022-06-01 20:11:53 +02:00
Yakov Olkhovskiy
e23cec01d5
Merge pull request #37581 from ClickHouse/http-named-collection 
Support for HTTP source for Data Dictionaries in Named Collections
 2022-06-01 11:55:04 -04:00
Nikolai Kochetov
6e924cdc77 Fix some more tests. 2022-06-01 15:21:47 +00:00
Sema Checherinda
2626a49616 FR: Expose what triggered the merge in system.part_log #26255 2022-06-01 16:58:06 +02:00
Alexander Tokmakov
3d346c766a better code 2022-06-01 16:49:26 +02:00
kssenii
aa60de8f63 Merge master 2022-06-01 15:32:01 +02:00
Kseniia Sumarokova
7afcfcbaaf
Merge pull request #37691 from kssenii/fix-rabbitmq-restart-with-no-settings 
Fix rabbitmq restart with empty settings
 2022-06-01 14:59:34 +02:00
Alexander Tokmakov
75f49a48e1 Merge branch 'master' into fix_trash 2022-06-01 14:20:46 +02:00
Anton Popov
011131198c Merge remote-tracking branch 'upstream/master' into fix-mutations-again 2022-06-01 12:04:18 +00:00
Anton Popov
cae2478b3f Revert "Merge pull request #37355 from ClickHouse/revert-37266-fix-mutations-with-object" 
This reverts commit a53cfa9fca, reversing
changes made to 9acb42fcdb.
 2022-06-01 10:57:20 +00:00
Han Fei
ea693dd0c2 add config and change test logic 2022-06-01 14:57:07 +08:00
Anton Popov
6cf9405f09 fix optimize_monotonous_functions_in_order_by in distributed queries 2022-06-01 00:50:28 +00:00
Alexander Gololobov
26609a1875 Style fixes 2022-05-31 21:41:10 +02:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00
Han Fei
5693e6212d add config and fix style check 2022-05-31 23:18:05 +08:00
Nikolai Kochetov
147a819221 Refactor a little bit more. 2022-05-31 14:43:38 +00:00
kssenii
c2087b3145 Fix 2022-05-31 14:38:11 +02:00
mergify[bot]
d85c3ec69e
Merge branch 'master' into turn_on_s3_tests 2022-05-31 11:58:16 +00:00
mergify[bot]
1e08046c47
Merge branch 'master' into cleanup_unused 2022-05-31 10:28:19 +00:00
taiyang-li
047387bf1c fix 2 bugs: 1. select count(1) from hive_table; 2. select _file, _path from hive_table 2022-05-31 17:39:02 +08:00
Kseniia Sumarokova
73ed9c3977
Merge pull request #37619 from Vxider/wv-fix-table-identifier 
Fix bugs in WindowView when using table identifier
 2022-05-31 11:07:11 +02:00
xlwh
ba4cdd43bd Cleanup unused file 2022-05-31 14:37:30 +08:00
yaqi-zhao
a2857491c4 add avx512 support for mergetreereader 2022-05-30 20:53:00 -04:00
mergify[bot]
55913cf8e1
Merge branch 'master' into turn_on_s3_tests 2022-05-30 20:52:40 +00:00
kssenii
bdfd166eb1 Merge master 2022-05-30 21:31:47 +02:00
Nikolai Kochetov
77b07dd0a8
Merge pull request #37163 from ClickHouse/grouping-function 
Add GROUPING function
 2022-05-30 20:45:04 +02:00
HeenaBansal2009
b7eb6bbd38 Fixed clang-tidy-CheckTriviallyCopyableMove-errors 2022-05-30 11:09:03 -07:00
Alexander Gololobov
e2dd6f6249 Removed prewhere_info.alias_actions 2022-05-30 19:58:23 +02:00
Han Fei
e15cdec39c address comments 2022-05-31 01:46:31 +08:00
mergify[bot]
d4e722bbfa
Merge branch 'master' into http-named-collection 2022-05-30 16:40:18 +00:00
Han Fei
af86900c52 Merge branch 'hanfei/zk-write' of github.com:hanfei1991/ClickHouse into hanfei/zk-write 2022-05-31 00:17:38 +08:00
Han Fei
42fca8d5f0 address comments 2022-05-31 00:17:32 +08:00
Han Fei
a464b10afe
Update src/Storages/System/StorageSystemZooKeeper.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2022-05-30 23:52:37 +08:00
Han Fei
194445646a
Update src/Storages/System/StorageSystemZooKeeper.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2022-05-30 23:47:43 +08:00
Nikolai Kochetov
c71256ea38 Remove some commented code. 2022-05-30 13:18:20 +00:00
Kruglov Pavel
0615866aea
Merge pull request #37450 from Avogar/check-format-on-storage-creation 
Check format name on storage creation
 2022-05-30 14:23:20 +02:00
Nikolai Kochetov
5b4658aa5e Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-30 09:47:35 +00:00
taiyang-li
dbb8a09825 merge master and solve conflict 2022-05-30 10:47:04 +08:00
taiyang-li
51a893c8be add some metrics 2022-05-30 10:05:20 +08:00
alesapin
c32b6076fb Remove stranges from code 2022-05-30 00:12:33 +02:00
alesapin
0e8ab36913 Merge branch 'master' into turn_on_s3_tests 2022-05-29 14:37:10 +02:00
alesapin
d2cdbf3956 Fix refactoring issue 2022-05-29 14:09:49 +02:00
alesapin
9dc81e5cc8
Merge pull request #37598 from ClickHouse/revert-37545-revert-37424-fix_fetching_part_deadlock 
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part""
 2022-05-29 13:54:34 +02:00
Alexander Tokmakov
562eec591e
Revert "Implemented changing comment to a ReplicatedMergeTree table" 2022-05-29 09:28:47 +03:00
Alexey Milovidov
97606c324c
Merge pull request #37574 from azat/mt-tiny-refactor 
Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm()
 2022-05-29 07:59:57 +03:00
Alexey Milovidov
73e2e63414
Merge pull request #37612 from ClickHouse/clang-tidy-14 
Fix clang-tidy-14, part 1
 2022-05-29 03:16:32 +03:00
Alexander Tokmakov
4e52f45695 Merge branch 'master' into fix_trash 2022-05-28 19:43:19 +02:00
Anton Popov
1d9b3be7da
Merge pull request #37536 from CurtizJ/profile-events-for-part-types 
Add profile events for introspection of part types
 2022-05-28 14:25:21 +02:00
kssenii
2d3714f6c4 Merge master 2022-05-28 14:07:04 +02:00
Han Fei
340a264a62 fix style 2022-05-28 18:26:14 +08:00
Han Fei
0a0d77bdef fix build 2022-05-28 17:57:59 +08:00
Han Fei
0f71231574 try fix flaky tests and refine code style 2022-05-28 17:25:33 +08:00
Vxider
b24346328d fix parser when using table identifer 2022-05-28 08:22:34 +00:00
Alexey Milovidov
c50791dd3b Fix clang-tidy-14, part 1 2022-05-27 22:52:14 +02:00
Nikolai Kochetov
b80b1940ce Fix some tests. 2022-05-27 20:47:35 +00:00
Yakov Olkhovskiy
41ef0044f0 endpoint is added 2022-05-27 13:43:34 -04:00
mergify[bot]
f5ee337bab
Merge branch 'master' into revert-37545-revert-37424-fix_fetching_part_deadlock 2022-05-27 16:52:00 +00:00
mergify[bot]
923ad2e905
Merge branch 'master' into turn_on_s3_tests 2022-05-27 16:31:43 +00:00
alesapin
f63fa9bcc6
Merge pull request #37416 from Enmk/fix_ReplicatedMergeTree_comments 
Implemented changing comment to a ReplicatedMergeTree table
 2022-05-27 18:29:34 +02:00
kssenii
25b1ed526d Merge master 2022-05-27 16:50:45 +02:00
alesapin
5a296aec01 Fix build 2022-05-27 16:34:16 +02:00
alesapin
be1c3c132b Fix some trash 2022-05-27 16:08:49 +02:00
tchepavel
42cd854688
Merge branch 'master' into nats-integration 2022-05-27 16:05:08 +03:00
alesapin
6d6779f17a
Merge pull request #37139 from ClickHouse/i_object_storage 
Separate object storage operations from disks
 2022-05-27 13:59:50 +02:00
alesapin
c79600c4c8 Fix build 2022-05-27 13:44:29 +02:00
taiyang-li
73d2c889c6 fix log level 2022-05-27 19:23:58 +08:00
alesapin
841858ec30
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part"" 2022-05-27 13:13:36 +02:00
Han Fei
2ea027ffcb Support insert into system.zookeeper 2022-05-27 18:53:12 +08:00
taiyang-li
ea450b86cb add some prefetch metric codes 2022-05-27 18:06:40 +08:00
Kseniia Sumarokova
2943d44bf1
Merge pull request #37554 from msaf1980/cleanup_hdfs 
Cleanup StorageHDFS (unused variables prevent build with clang 12)
 2022-05-27 11:57:18 +02:00
Yakov Olkhovskiy
25884c68f1 http named collection source implemented for dictionary 2022-05-26 20:46:26 -04:00
Azat Khuzhin
c6c60364ae Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm() 
In favor of
MergeTask::ExecuteAndFinalizeHorizontalPart::chooseMergeAlgorithm()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 21:20:50 +03:00
Alexander Tokmakov
eb71dd4c78
Merge pull request #37547 from ClickHouse/followup_37398 
Follow-up to #37398
 2022-05-26 20:29:41 +03:00
Azat Khuzhin
73a99d4eee Improve error message for skipped/expired columns 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
f0cac5417a More optional skipping of fully expired columns (by TTL) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
46a94f395d Move the check into IMergedBlockOutputStream::removeEmptyColumnsFromPart() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
4288d09a85 Do not write expired columns by TTL after merge w/o TTL 
Usually second merge do not perform TTL, since everything is up to date,
however in this case TTLTransform is not used, and hence expired_columns
will not be filled for new part, and so those columns will be written
with default values.

Avoid this, by manually filling expired_columns.

Here is a simpler reproducer:

Simple reproducer:

    ```sql
    create table ttl_02262 (date Date, key Int, value String TTL date + interval 1 month) engine=MergeTree order by key settings min_bytes_for_wide_part=0, min_rows_for_wide_part=0;
    insert into ttl_02262 values ('2010-01-01', 2010, 'foo');
    ```

    ```sh
    # ls -l .server/data/default/ttl_02262/all_*
    .server/data/default/ttl_02262/all_1_1_0:
    total 48
    -rw-r----- 1 root root 335 May 26 14:19 checksums.txt
    -rw-r----- 1 root root  76 May 26 14:19 columns.txt
    -rw-r----- 1 root root   1 May 26 14:19 count.txt
    -rw-r----- 1 root root  28 May 26 14:19 date.bin
    -rw-r----- 1 root root  48 May 26 14:19 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:19 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:19 key.bin
    -rw-r----- 1 root root  48 May 26 14:19 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:19 primary.idx
    -rw-r----- 1 root root  99 May 26 14:19 ttl.txt
    -rw-r----- 1 root root  30 May 26 14:19 value.bin
    -rw-r----- 1 root root  48 May 26 14:19 value.mrk2
    ```

    ```sql
    optimize table ttl_02262 final;
    ```

    ```sh
    .server/data/default/ttl_02262/all_1_1_1:
    total 40
    -rw-r----- 1 root root 279 May 26 14:19 checksums.txt
    -rw-r----- 1 root root  61 May 26 14:19 columns.txt
    -rw-r----- 1 root root   1 May 26 14:19 count.txt
    -rw-r----- 1 root root  28 May 26 14:19 date.bin
    -rw-r----- 1 root root  48 May 26 14:19 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:19 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:19 key.bin
    -rw-r----- 1 root root  48 May 26 14:19 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:19 primary.idx
    -rw-r----- 1 root root  81 May 26 14:19 ttl.txt
    ```

    ```sql
    optimize table ttl_02262 final;
    ```

    ```sh
    .server/data/default/ttl_02262/all_1_1_2:
    total 48
    -rw-r----- 1 root root 349 May 26 14:20 checksums.txt
    -rw-r----- 1 root root  76 May 26 14:20 columns.txt
    -rw-r----- 1 root root   1 May 26 14:20 count.txt
    -rw-r----- 1 root root  28 May 26 14:20 date.bin
    -rw-r----- 1 root root  48 May 26 14:20 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:20 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:20 key.bin
    -rw-r----- 1 root root  48 May 26 14:20 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:20 primary.idx
    -rw-r----- 1 root root  81 May 26 14:20 ttl.txt
    -rw-r----- 1 root root  27 May 26 14:20 value.bin
    -rw-r----- 1 root root  48 May 26 14:20 value.mrk2
    ```

And now we have `value.*` for all_1_1_2, this should not happen.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
8328d7068b Fix updating of MergeTreeDataPartTTLInfo::finished 
Previously you cannot distinguish non-initialized finished with
initialized to false, so update() cannot do the correct thing.

Rename the field to avoid hidden usage.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
0de1a64436 Log empty parts in IMergedBlockOutputStream::removeEmptyColumnsFromPart() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:09 +03:00
Nikolai Kochetov
ec4e8d71b2 Fixing build 2022-05-26 15:33:21 +00:00
Alexander Tokmakov
e8f33fb0d9 fix flaky tests 2022-05-26 14:17:05 +02:00
Azat Khuzhin
dc9ca3d70c
Fix LOGICAL_ERROR in getMaxSourcePartsSizeForMerge during merges (#37413) 2022-05-26 14:14:58 +02:00
Nikolai Kochetov
84f97b53de Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-26 11:07:45 +00:00
Maksim Kita
3a92e61827
Merge pull request #37148 from kitaisreal/dictionary-get-descendants-performance-improvement 
Dictionary getDescendants performance improvement
 2022-05-26 12:29:17 +02:00
Vasily Nemkov
abe6b5d013
Reverted unnecessary modification 2022-05-26 10:09:27 +03:00
taiyang-li
561c87222d add prefetch for hive text 2022-05-26 11:04:35 +08:00
Anton Popov
f488efd27e fix tests 2022-05-26 00:03:31 +00:00
Dmitry Novik
16c6b60703 Introduce AggregationKeysInfo 2022-05-25 23:22:29 +00:00
alesapin
8f1aac0ce4 Fix merge with master 2022-05-26 00:44:45 +02:00
alesapin
1db9cf480b Merge remote-tracking branch 'origin/master' into i_object_storage 2022-05-25 22:50:22 +02:00
kssenii
50cd358c0e Merge master 2022-05-25 22:21:28 +02:00
Alexander Tokmakov
47820c216d
Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part" 2022-05-25 23:10:33 +03:00
alesapin
c7b16065e1 Merge with master 2022-05-25 21:47:05 +02:00
Nikolai Kochetov
6d4a26afac Update ReadProgressCallback. 2022-05-25 19:45:48 +00:00
tchepavel
531bda5a65 Spelling and more black python 2022-05-25 22:13:26 +03:00
tchepavel
d4afc36f8e Style+build fixes; make long tests time limited 2022-05-25 22:02:47 +03:00
kssenii
b89a4edc04 Merge master 2022-05-25 20:55:04 +02:00
alesapin
6f5c86e55e Merge branch 'master' into i_object_storage 2022-05-25 20:49:01 +02:00
Alexander Tokmakov
779e6ea0b9 make it better, fix on cluster queries 2022-05-25 20:17:49 +02:00
Nikolai Kochetov
54d7e4139f Fix build. 2022-05-25 18:16:48 +00:00
alesapin
51868a9a4f
Merge pull request #37424 from metahys/fix_fetching_part_deadlock 
(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part
 2022-05-25 20:15:41 +02:00
alesapin
0a3597da72
Merge pull request #34915 from ianton-ru/MDB-16962 
Fix collision of S3 operation log revision
 2022-05-25 18:15:31 +02:00
Anton Popov
16e839ac71 add profile events for introspection of part types 2022-05-25 14:54:49 +00:00
kssenii
0556237b68 Continuation 2022-05-25 16:49:40 +02:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
msaf1980
fda6ddeffa cleanup StorageHDFS (unused variables) 2022-05-25 19:23:05 +05:00
mergify[bot]
73662b4436
Merge branch 'master' into fix_fetching_part_deadlock 2022-05-25 14:22:35 +00:00
taiyang-li
a7a816dcb6 fix build error 2022-05-25 19:55:11 +08:00
taiyang-li
1d9f65a7d4 Merge branch 'master' into async_hdfs_read_buffer 2022-05-25 17:10:22 +08:00
Kseniia Sumarokova
b50d4549c9
Merge pull request #37356 from amosbird/partition-prune-for-s3 
"Partition pruning" for s3
 2022-05-25 11:03:07 +02:00
avogar
f782fa31c6 Merge branch 'master' of github.com:ClickHouse/ClickHouse into check-format-on-storage-creation 2022-05-25 08:42:54 +00:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
Nikolai Kochetov
333fd09dbf Fixing build. 2022-05-24 19:29:00 +00:00
Vasily Nemkov
59b4d4a643 ALTER COMMENT is now local-only operation and immediately observable 2022-05-24 21:08:30 +03:00
tchepavel
43226a57fc
Merge branch 'master' into nats-integration 2022-05-24 18:15:11 +03:00
alesapin
9a19309e69 Slightly better fix 2022-05-24 14:46:29 +02:00
Alexander Tokmakov
229d35408b
Merge pull request #37398 from ClickHouse/fixes_for_transactions 
Fixes for transactions
 2022-05-24 15:28:01 +03:00
alesapin
3ca7a8831b Revert "fix deadlock during fetching part" 
This reverts commit 6ae8a26fae.
 2022-05-24 14:26:06 +02:00
alesapin
a5ba6bca95 Merge branch 'master' into metahys-fix_fetching_part_deadlock 2022-05-24 14:07:47 +02:00
Kruglov Pavel
6c9a524f6b
Merge pull request #37192 from Avogar/formats-with-names 
Improve performance and memory usage for select of subset of columns for some formats
 2022-05-24 13:28:14 +02:00
Amos Bird
093d315756
partition pruning for s3 2022-05-24 18:57:55 +08:00
mergify[bot]
51ff49a0ee
Merge branch 'master' into i_object_storage 2022-05-23 20:29:49 +00:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Alexander Tokmakov
6bc68c0cbc Merge branch 'master' into fixes_for_transactions 2022-05-23 18:49:21 +02:00
Nikolai Kochetov
9756b759c6 Move some resources 2022-05-23 13:46:57 +00:00
Kruglov Pavel
a0aa841909
Fix failed thread stateless tests 2022-05-23 15:15:45 +02:00
avogar
37b66c8a9e Check format name on storage creation 2022-05-23 12:48:48 +00:00
metahys
6ae8a26fae fix deadlock during fetching part 2022-05-23 20:08:04 +08:00
tchepavel
87217ce6cc NATS. Add new setting to doc; clean up code. 2022-05-23 14:57:39 +03:00
tchepavel
d9436ec7dd
Merge branch 'master' into nats-integration 2022-05-23 14:30:09 +03:00
taiyang-li
29e2157469 change as request 2022-05-23 18:42:54 +08:00
Maksim Kita
585b86446e Added hierarchical_index_bytes_allocated column in system.dictionaries 2022-05-23 12:42:00 +02:00
taiyang-li
14f84f02d5 Merge branch 'master' into async_hdfs_read_buffer 2022-05-23 18:36:21 +08:00
Sergei Trifonov
b6bf283f4d
Merge pull request #37243 from Vxider/mv-fix-dependency 
Fix adding dependency when MaterializedView is dropped
 2022-05-23 12:33:54 +02:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names 2022-05-23 12:14:20 +02:00
Kruglov Pavel
754e675ec3
Merge pull request #37253 from Avogar/fix-defaults 
Fix inserting defaults for missing values in columnar formats
 2022-05-23 12:10:14 +02:00
Kseniia Sumarokova
ca3feddf73
Merge pull request #37426 from Vxider/wv-simplify 
Remove unused code in WindowView
 2022-05-23 11:43:51 +02:00
mergify[bot]
747aa5575c
Merge branch 'master' into remove-useless-code-2 2022-05-22 17:41:57 +00:00
Vxider
fcd317a642 update windowview create syntax 2022-05-22 10:18:34 +00:00
alesapin
c8d92b87c8 Merge branch 'master' into i_object_storage 2022-05-22 12:16:10 +02:00
Vxider
5f84f06d6d simplify windowview 2022-05-22 10:13:16 +00:00
Robert Schulze
19d53c14fa
Merge pull request #37382 from ClickHouse/wc++98-compat-extra-semi 
Enable -Wc++98-compat-extra-semi
 2022-05-22 09:40:45 +02:00
kssenii
9aa20193d5 Merge master 2022-05-22 00:56:20 +02:00
Alexey Milovidov
4bfbb0b7ba
Merge pull request #37388 from CurtizJ/explicit-bool-operator 
Mark all `operator bool()` as explicit
 2022-05-22 00:18:19 +03:00
Alexey Milovidov
790f442362
Merge pull request #37407 from ClickHouse/certificates-mem-leak 
Fix possible memory leaks in system.certificates implementation
 2022-05-22 00:15:30 +03:00
Vasily Nemkov
c12f826d22 Implemented changing comment to a ReplicatedMergeTree table 2022-05-21 20:08:30 +03:00
Yakov Olkhovskiy
4c13b52b6a scope guard resources 2022-05-20 19:59:17 -04:00
Alexander Tokmakov
44f2d4529a better waiting, add fault injection 2022-05-20 22:10:01 +02:00
Nikolai Kochetov
56feef01e7 Move some resources 2022-05-20 19:49:31 +00:00
avogar
3bad21cea7 Try fix tests 2022-05-20 17:48:24 +00:00
Alexander Tokmakov
1027798517 handle connection loss on commit 2022-05-20 17:35:29 +02:00
Anton Popov
cb0e6c2718 mark all operators bool() as explicit 2022-05-20 15:29:54 +00:00
avogar
a4cf07708c Fix comments 2022-05-20 14:57:27 +00:00
avogar
566d1b15fd Merge branch 'master' of github.com:ClickHouse/ClickHouse into formats-with-names 2022-05-20 13:54:52 +00:00
avogar
78069765bb Try to remove code duplication 2022-05-20 13:20:52 +00:00
avogar
6e2df9ecc4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-defaults 2022-05-20 13:15:34 +00:00
Robert Schulze
0f6715bd91
Follow-up to PR #37300: semicolon warnings 
In PR #37300, Alexej asked why we the compiler does not warn about
unnecessary semicolons, e.g.

  f()
  {
  }; // <-- here

The answer is surprising: In C++98, above syntax was disallowed but by
most compilers accepted it regardless. C++>11 introduced "empty
declarations" which made the syntax legal.

The previous behavior can be restored using flag
-Wc++98-compat-extra-semi. This finds many useless semicolons which were
removed in this change. Unfortunately, there are also false positives
which would require #pragma-s and HAS_* logic (--> check_flags.cmake) to
suppress. In the end, -Wc++98-compat-extra-semi comes with extra effort
for little benefit. Therefore, this change only fixes some semicolons
but does not enable the flag.
 2022-05-20 15:06:34 +02:00
Vxider
956459eba2 Merged with master 2022-05-20 10:50:21 +00:00
Alexander Tokmakov
12bbb7de87 fix race on TID allocation 2022-05-20 12:41:44 +02:00
alesapin
654b27e307 Merge branch 'master' into i_object_storage 2022-05-20 11:56:13 +02:00
Kseniia Sumarokova
ee295038fa
Merge pull request #37214 from Vxider/wv-engine 
Change ENGINE behavior in WindowView
 2022-05-20 11:14:19 +02:00
Robert Schulze
b475fbc9a7
Merge pull request #37300 from ClickHouse/cmake-cleanup-pt3 
Various cmake cleanups
 2022-05-20 10:02:36 +02:00
alesapin
92c15ec97c Get rid of IDiskRemote 2022-05-19 20:07:15 +02:00
Dmitry Novik
b3ccf96c81 Merge remote-tracking branch 'origin/master' into grouping-function 2022-05-19 17:58:33 +00:00
Alexander Tokmakov
a53cfa9fca
Merge pull request #37355 from ClickHouse/revert-37266-fix-mutations-with-object 
Revert "Fix mutations in tables with columns of type `Object`"
 2022-05-19 16:27:22 +03:00
Vxider
f6cc73235b Merged with master 2022-05-19 13:03:45 +00:00
Vxider
b2c7ba7cbe update 2022-05-19 12:54:41 +00:00
Vxider
24f1a1fde7 Merge branch 'wv-alter' of https://github.com/Vxider/ClickHouse into wv-populate 2022-05-19 12:49:37 +00:00
vxider
797edb0ae1
Update src/Storages/WindowView/StorageWindowView.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-05-19 20:49:02 +08:00
Vxider
1cc347364f Merged with master 2022-05-19 12:45:46 +00:00
Kseniia Sumarokova
d4ad138a04
Merge pull request #37103 from bigo-sg/hive_partition_key_read 
optimization for reading hive file  when all columns to read are partition keys
 2022-05-19 14:24:00 +02:00
Kseniia Sumarokova
b0c7cf910a
Merge pull request #37224 from Vxider/wv-fix-dependency 
Fix addDependency in WindowView
 2022-05-19 13:51:11 +02:00
Kseniia Sumarokova
24d0899cb7
Merge pull request #36945 from Vxider/wv-populate 
POPULATE support for WindowView
 2022-05-19 13:47:52 +02:00
Kseniia Sumarokova
af2d9154e1
Merge pull request #37279 from bigo-sg/hive_local_cache_multi_disk 
Support multi disks for caching hive files
 2022-05-19 13:25:03 +02:00
Kruglov Pavel
468954cc87
Remove redundant include 2022-05-19 12:52:33 +02:00
Alexander Tokmakov
f787dc7097
Revert "Fix mutations in tables with columns of type Object" 2022-05-19 13:24:48 +03:00
Mikhail f. Shiryaev
32769caf04
Merge pull request #37346 from ClickHouse/22.6-prepare 
Update version after release
 2022-05-19 10:54:53 +02:00
Mikhail f. Shiryaev
d0fe794fe5
Update version to 22.6.1.1 2022-05-19 10:06:09 +02:00
mergify[bot]
7ff8098272
Merge branch 'master' into wv-alter 2022-05-19 01:51:50 +00:00
Vitaly Baranov
7c8956ca8d
Merge pull request #36879 from tonickkozlov/tonickkozlov/lazy-marks-loading 
Lazy marks loading
 2022-05-18 22:56:46 +02:00
Vxider
04f1b20229 init inner window names 2022-05-18 14:34:40 +00:00
avogar
3c85e8b455 Fix tests 2022-05-18 12:36:13 +00:00
Vxider
4b5eda4de9 fix cleanup 2022-05-18 11:14:08 +00:00
alesapin
e99c90118e Merge branch 'master' into i_object_storage 2022-05-18 12:36:22 +02:00
avogar
b33ad449a9 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-defaults 2022-05-18 10:26:38 +00:00
Kruglov Pavel
aaace46da2
Merge pull request #37266 from CurtizJ/fix-mutations-with-object 
Fix mutations in tables with columns of type `Object`
 2022-05-18 12:19:55 +02:00
Kseniia Sumarokova
f9b08ae5dd
Merge pull request #37281 from bigo-sg/hive_max_partition_query_check 
Check max partitions to read for each hive table
 2022-05-18 11:11:11 +02:00
Robert Schulze
6b2a72d451
Merge pull request #37228 from mrk-andreev/fix-rm-groupid-from-producer 
Remove group.id from StorageKafka::createWriteBuffer
 2022-05-18 11:04:40 +02:00
Vxider
f102497929 fix parser with join query 2022-05-18 08:05:06 +00:00
mergify[bot]
37d6da7506
Merge branch 'master' into wv-engine 2022-05-18 01:56:56 +00:00
mergify[bot]
4900d2249f
Merge branch 'master' into wv-populate 2022-05-18 01:56:52 +00:00
mergify[bot]
c33ca1f392
Merge branch 'master' into system-certificates 2022-05-17 19:52:28 +00:00
tchepavel
2692d88a42
Merge branch 'master' into nats-integration 2022-05-17 16:59:57 +03:00
tchepavel
a3af94d49e Documentation; integration tests; subscribe on startup 2022-05-17 16:58:09 +03:00
Vxider
037db32e24 set alter query internal 2022-05-17 13:50:55 +00:00
kssenii
e4e6e44e86 Merge master 2022-05-17 15:47:33 +02:00
lgbo-ustc
e4bc7e9979 use splitInto instead of Poco::Tokenizer 2022-05-17 19:31:59 +08:00
lgbo-ustc
1497e08301 update exception msg 2022-05-17 19:27:43 +08:00
Kruglov Pavel
515ae986e5
Fix special build 2022-05-17 12:15:41 +02:00
Kseniia Sumarokova
8cb71d0172
Merge pull request #33332 from Michicosun/MeiliSearch 
MeiliSearch
 2022-05-17 11:47:56 +02:00
alesapin
4d4634b12d
Merge pull request #37164 from kssenii/ficx-async-reads-assertions 
fix async reads from remote fs internal setting not being always turned on
 2022-05-17 11:29:14 +02:00
taiyang-li
14ab7eb5a3 merge master and solve conflict 2022-05-17 16:28:08 +08:00
lgbo-ustc
0b3468a150 TOO_MANY_PARTITIONS 2022-05-17 15:50:03 +08:00
lgbo-ustc
f4f4a2d85b reuse setting max_partitions_to_read 2022-05-17 15:49:14 +08:00
lgbo-ustc
4d64678963 Merge remote-tracking branch 'ck/master' into hive_max_partition_query_check 2022-05-17 15:42:10 +08:00
lgbo-ustc
a161a21992 add max partitions check for each hive table 2022-05-17 15:37:32 +08:00
Vxider
90bac2f004 add shutdown check 2022-05-17 06:20:08 +00:00
lgbo-ustc
bc81302bf4 fixed code-style 2022-05-17 12:10:55 +08:00
lgbo-ustc
3a32a22f39 support using multi disks for caching hive files 2022-05-17 11:55:30 +08:00
Vxider
f74c934a61 Merged with master 2022-05-17 02:53:59 +00:00
mergify[bot]
0d0fc23170
Merge branch 'master' into wv-engine 2022-05-17 02:24:18 +00:00
Vxider
3986e82b19 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into wv-populate 2022-05-17 02:18:17 +00:00
lgbo-ustc
d8ad9ad2a6 update codes 2022-05-17 09:27:03 +08:00
Kseniia Sumarokova
94683786dc
Merge branch 'master' into MeiliSearch 2022-05-16 22:42:09 +02:00
Kseniia Sumarokova
f073efc025
Merge pull request #37225 from Vxider/wv-fix-cleanup 
Fix WindowView cleanup
 2022-05-16 22:39:58 +02:00
kssenii
6a51dea791 Merge master 2022-05-16 22:13:22 +02:00
kssenii
5e3410f60a Better 2022-05-16 22:09:11 +02:00
Vitaly Baranov
72bff5242a
Merge pull request #37168 from vitlibar/backup-improvements-5 
Backups Improvements 5
 2022-05-16 21:25:54 +02:00
Robert Schulze
43945cea1b
Fixing some warnings 2022-05-16 20:59:27 +02:00
Yakov Olkhovskiy
4f6c354c3f refactoring 2022-05-16 14:38:47 -04:00
Anton Popov
b6c5ab4fcf fix mutations in tables with columns of type Object 2022-05-16 18:26:53 +00:00
Dmitry Novik
e5b395e054 Support ROLLUP and CUBE in GROUPING function 2022-05-16 17:33:38 +00:00
Yakov Olkhovskiy
863c52f0f4 refactoring, test added 2022-05-16 13:31:28 -04:00
tchepavel
270c01bcb7 Merge branch 'master' of github.com:ClickHouse/ClickHouse into nats-integration 2022-05-16 19:18:51 +03:00
alesapin
ce6d5f7dcc Merge branch 'master' into i_object_storage 2022-05-16 17:42:22 +02:00
avogar
ce834b1086 Remove code duplication 2022-05-16 14:41:01 +00:00
avogar
2d4b4b9008 Fix inserting defaults for missing values in columnar formats 2022-05-16 14:19:44 +00:00
kssenii
f3c1f3796b Some fixes 2022-05-16 16:16:38 +02:00
Vxider
8f271159bf check null pointer 2022-05-16 13:59:55 +00:00
tchepavel
2f00139347 Fix review issues; add insert queue parameter 2022-05-16 16:35:17 +03:00
Mikhail Artemenko
535b3ea686 refactor code & change exception numbers 2022-05-16 16:13:05 +03:00
Vxider
b3a731c226 fix build 2022-05-16 12:51:19 +00:00
mergify[bot]
b7d6d97e64
Merge branch 'master' into wv-alter 2022-05-16 12:44:46 +00:00
Robert Schulze
ed7df7cabd
Merge pull request #36960 from ClickHouse/clangtidies 
Enable clangtidies
 2022-05-16 14:22:50 +02:00
Vxider
6b3b744e3f fix data race 2022-05-16 11:31:49 +00:00
tchepavel
35cbe48ca3 Fixing; preparing test 2022-05-16 13:03:28 +03:00
tchepavel
58d8bc40aa Merge branch 'master' of github.com:ClickHouse/ClickHouse into nats-integration 2022-05-16 13:01:45 +03:00
Vladimir Makarov
f809e2a79f fix error 2022-05-16 09:31:37 +00:00
kssenii
f7493965b7 Remove additional check for merge tree 2022-05-16 11:19:57 +02:00
kssenii
49201f217f Revert 2022-05-16 11:17:56 +02:00
Vxider
31eee576a5 do not addDependency when table is dropped 2022-05-16 09:04:37 +00:00
Vxider
95e6e5aef7 fix window view dependency 2022-05-16 08:41:32 +00:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies 2022-05-16 10:12:50 +02:00
Vxider
9201df7cde update code style 2022-05-16 02:00:39 +00:00
alesapin
4f1d4dde22 Disable s3 logging by default 2022-05-16 00:16:00 +02:00
Mark Andreev
fa6a5ab6e4 Remove group.id from StorageKafka::createWriteBuffer 2022-05-15 23:51:11 +04:00
Vxider
77ad48fee9 fix build 2022-05-15 17:40:42 +00:00
Vxider
15b35c1910 speed up windowview watch 2022-05-15 17:14:53 +00:00
Vxider
18ee285ce1 fix windowview cleanup 2022-05-15 16:50:34 +00:00
Vxider
93fec4ebc7 remove dependency when drop windowview 2022-05-15 16:12:24 +00:00
Robert Schulze
859f2f3391
Enclose large if block in {} 2022-05-15 14:49:20 +02:00
Vladimir Makarov
69e2f8668e fix 2022-05-15 12:29:33 +00:00
tchepavel
7317c76d3c
Merge branch 'master' into nats-integration 2022-05-15 14:41:22 +03:00
Vxider
033b1e7f82 add lock for cleanup query 2022-05-15 11:18:42 +00:00
Vxider
6c2d78b802 rename var 2022-05-15 11:18:23 +00:00
Vladimir Makarov
d6512d257f fixes 2022-05-15 08:57:57 +00:00
Vladimir Makarov
e502ab2f55 disable mac os and try to fix sanitizer error 2022-05-15 08:25:46 +00:00
Vxider
65779adf8b update code style 2022-05-15 07:07:27 +00:00
Vxider
4ed79cd9e1 fix windowview watch 2022-05-15 06:45:25 +00:00
Yakov Olkhovskiy
6156047c83 some refactoring 2022-05-14 17:40:50 -04:00
Vladimir Makarov
bdb6691405 fix style 2022-05-14 21:25:00 +00:00
VVMak
1b9834548d
Merge branch 'master' into annoy-2 2022-05-15 00:01:28 +03:00
Yakov Olkhovskiy
ef2d8d3d0e bugfix, refactoring 2022-05-14 15:45:07 -04:00
Vladimir Makarov
59e0ef9641 remove extra headers 2022-05-14 18:00:34 +00:00
Vladimir Makarov
1ba0bcaffe fix includes and add docs 2022-05-14 17:53:45 +00:00
Vxider
f459dd5aba change windowview engine behaviour 2022-05-14 16:46:49 +00:00
Vladimir Makarov
3187fdddb8 Merge branch 'ANNIndex-Documentation' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-05-14 14:21:19 +00:00
kssenii
646cf4cc95 Better 2022-05-14 14:26:04 +02:00
Mikhail Artemenko
91edbf0f47 add subquery evaluation in getConfiguration 2022-05-14 14:44:35 +03:00
Vladimir Makarov
699f183ea0 small fix and comment about metrics 2022-05-14 10:47:05 +00:00
Vladimir Makarov
d1880f3c10 fix cmake and style 2022-05-14 10:27:05 +00:00
Vxider
f2ef27e5d6 rename windowview header 2022-05-14 09:21:54 +00:00
Vladimir Makarov
316858ee39 fix style 2022-05-14 08:55:07 +00:00
Vladimir Makarov
8f8c622e7c fix 2022-05-14 08:24:54 +00:00
Vxider
b52cac970d fix alter on replicated engine 2022-05-14 06:35:50 +00:00
mergify[bot]
4e184306c1
Merge branch 'master' into wv-alter 2022-05-14 02:11:46 +00:00
Dmitry Novik
6fc7dfea80 Support ordinary GROUP BY 2022-05-13 23:04:12 +00:00
alesapin
cc304a3178 Merge branch 'master' into i_object_storage 2022-05-13 23:44:00 +02:00
Vladimir Makarov
f4ae3150fa add optimization 2022-05-13 21:34:34 +00:00
alesapin
1c082717a4 Relax log level for some checks in check thread 2022-05-13 23:29:36 +02:00
Hakob Saghatelyan
e3726f9ec7 LIMIT restriction on query support added 2022-05-13 21:01:54 +00:00
Hakob Saghatelyan
3c601e5785 add Set support for ann_index_params setting 2022-05-13 20:59:56 +00:00
Yakov Olkhovskiy
d2ffd0e1c2 take certificate path from poco Context 2022-05-13 16:48:34 -04:00
Vladimir Makarov
48222b1fd5 add select params 2022-05-13 20:42:50 +00:00
avogar
68bb07d166 Better naming 2022-05-13 18:39:19 +00:00
kssenii
58faaf0ea3 Merge master 2022-05-13 18:51:07 +02:00
Yakov Olkhovskiy
262cc90067 maybe_unused 2022-05-13 11:31:42 -04:00
Yakov Olkhovskiy
2e0aaa0f89 #if instead of #ifdef 2022-05-13 11:03:13 -04:00