thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
103,548 Commits 9,338 Branches 1,783 Tags 2.1 GiB
6b6504f0d1
Commit Graph

4741 Commits

Author SHA1 Message Date
Kseniia Sumarokova
b0a362e403
Merge pull request #41839 from ClickHouse/kssenii-patch-5 
Remove unused variable in registerStorageMergeTree.cpp
 2022-09-28 11:58:01 +02:00
Alexey Milovidov
e5ffd74a56
Merge branch 'master' into trying_to_fix_compressed_marks 2022-09-28 02:30:10 +03:00
Alexey Milovidov
12dc33f4fd
Merge branch 'master' into nv/fix-mutate-fetch 2022-09-28 01:28:19 +03:00
Kseniia Sumarokova
85b44c909a
Update MergeTreeDataPartCompact.cpp 2022-09-27 18:16:58 +02:00
Salvatore Mesoraca
fc73b74379 Improve feedback when replacing partition with different primary key 2022-09-27 16:31:47 +02:00
Amos Bird
c39c138416
Better index analysis with NULL literal 2022-09-27 22:00:50 +08:00
alesapin
a20a9242a1 Fix order of arguments 2022-09-27 15:50:25 +02:00
Kseniia Sumarokova
2f237a8a2c
Update registerStorageMergeTree.cpp 2022-09-27 15:30:18 +02:00
alesapin
695133cc5b Better mutations 2022-09-27 15:23:02 +02:00
alesapin
d4d4f92976 Use copy for some operations instead of hardlinks 2022-09-27 14:38:13 +02:00
Antonio Andelic
265c8b3d26 Merge branch 'master' into keeper-multiread 2022-09-27 12:37:22 +00:00
kssenii
ce422052b0 Fix CANNOT_READ_ALL_DATA for compact parts in case compressed_marks=1 2022-09-27 14:27:56 +02:00
Alexander Tokmakov
19062e9d97
Update src/Storages/MergeTree/ReplicatedMergeTreeAttachThread.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-09-27 14:26:45 +03:00
Alexander Tokmakov
0f6a44efef fix missing metadata_version for old tables 2022-09-26 20:11:30 +02:00
Alexander Tokmakov
9501f88f8c
Merge branch 'master' into fix_intersecting_parts 2022-09-26 19:53:59 +03:00
Alexander Tokmakov
49983dafeb fix 2022-09-26 17:13:10 +02:00
alesapin
31f6636a47 Fix endless remove 2022-09-26 16:33:25 +02:00
Kruglov Pavel
3dc54272ed
Merge branch 'master' into improve-combinators 2022-09-26 13:03:32 +02:00
Antonio Andelic
97385ca861 Format 2022-09-26 09:33:50 +00:00
Antonio Andelic
cc3719e463 Use multiread 2022-09-26 09:07:34 +00:00
alesapin
5852df7f65
Merge branch 'master' into fix_part_removal_retries 2022-09-24 17:06:32 +02:00
alesapin
ce2c2b5310 Merge branch 'master' into fix_intersecting_parts 2022-09-24 17:00:54 +02:00
alesapin
34d9794ab7
Merge pull request #41653 from ClickHouse/investigating_more_bugs 
Add very explicit logging on disk choice for fetch
 2022-09-24 16:15:02 +02:00
Alexander Tokmakov
a2b8c4926e fix intersecting parts 2022-09-23 20:51:44 +02:00
Alexander Tokmakov
2c16232b02 fix part removal retries 2022-09-23 14:39:05 +02:00
alesapin
148f018eae Fix style: 2022-09-23 14:32:19 +02:00
alesapin
dff61267b4 More explicit logs 2022-09-23 14:31:05 +02:00
Alexander Tokmakov
981c1e95c6
Update MutateTask.cpp 2022-09-22 18:51:27 +03:00
Nikolai Kochetov
63e757a659
Merge pull request #36379 from amosbird/projection-compatibility-fix 
Make minmax_count projection backward compatible
 2022-09-22 14:15:35 +02:00
Nikolai Kochetov
3b63719ece
Merge pull request #41619 from den-crane/patch-50 
increase default max_suspicious_broken_parts to 100
 2022-09-22 13:25:22 +02:00
alesapin
ca410b7a0b
Merge pull request #41147 from ClickHouse/enable_projections_and_zero_copy 
Enable projections and zero copy replication
 2022-09-22 13:05:09 +02:00
alesapin
a95e291474
Merge branch 'master' into investigating_more_bugs 2022-09-22 12:51:55 +02:00
alesapin
57be648984
Merge pull request #41649 from ClickHouse/supress_one_more_key_doesnt_exists 
Release AWS SDK log level + replace one exception
 2022-09-22 12:38:03 +02:00
Alexey Milovidov
2a75e025f2
Merge pull request #40715 from ClickHouse/compress-marks 
Merging #37693: Compress marks and primary key
 2022-09-22 07:30:34 +03:00
Alexey Milovidov
45afacdae4
Merge pull request #41186 from ClickHouse/fix-three-fourth-of-trash 
Fix more than half of the trash
 2022-09-22 07:28:26 +03:00
alesapin
9c433c14d1 Add very explicit logging on disk choice for fetch 2022-09-21 21:33:28 +02:00
alesapin
499f678112 Release AWS SDK log level + replace one exception 2022-09-21 19:56:42 +02:00
Nikita Taranov
100c055510
Prefetching in aggregation (#39304) 
* impl

* stash

* clean up

* do not apply when HT is small

* make branch static

* also in merge

* do not hardcode look ahead value

* fix

* apply to methods with cheap key calculation

* more tests

* silence tidy

* fix build

* support HashMethodKeysFixed

* apply during merge only for cheap

* stash

* fixes

* rename method

* add feature flag

* cache prefetch threshold value

* fix

* fix

* Update HashMap.h

* fix typo

* 256KB as default l2 size

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2022-09-21 18:59:07 +02:00
Denny Crane
75cf53c06e
increase default max_suspicious_broken_parts to 100 2022-09-21 11:10:19 -03:00
Robert Schulze
186d4e4140
Merge pull request #40818 from Vector-Similarity-Search-for-ClickHouse/annoy 
Add annoy index
 2022-09-21 13:42:43 +02:00
Nicolae Vartolomei
e93a78faee Proper comment for bad code 2022-09-20 19:56:14 +00:00
Nicolae Vartolomei
3d6fcf4ae5 Explain bad code 2022-09-20 19:46:07 +00:00
Nicolae Vartolomei
372545b306 Remove leftover 2022-09-20 19:28:40 +00:00
Nicolae Vartolomei
45f1f572f0 Get rid of scary setting for testing fetching byte identical parts 2022-09-20 19:27:49 +00:00
Nicolae Vartolomei
90666a43a8 CLICKHOUSE-2153: Move new_data_part shared pointer to allow Outdated part to be deleted immediately on mutation failure 2022-09-20 18:23:19 +00:00
alesapin
4bd667c6b7 Drop all WALs 2022-09-20 18:28:15 +02:00
Filatenkov Artur
2139d5be17
Merge branch 'master' into annoy 2022-09-20 15:28:08 +03:00
alesapin
8ba74d8655 Fix bugs found by CI 2022-09-20 14:00:57 +02:00
alesapin
e9d31977f3
Merge branch 'master' into enable_projections_and_zero_copy 2022-09-20 12:25:10 +02:00
Filatenkov Artur
be8221fae1
Correct work with negative distance 2022-09-20 09:48:33 +03:00
Alexey Milovidov
7c299dc561
Update src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp 
Co-authored-by: Robert Schulze <robert@clickhouse.com>
 2022-09-20 05:16:12 +03:00
Alexey Milovidov
e7525da88f
Update src/Storages/MergeTree/MergeTreePartsMover.cpp 
Co-authored-by: Robert Schulze <robert@clickhouse.com>
 2022-09-20 05:16:02 +03:00
Filatenkov Artur
3598a7e680
Fix query type information 2022-09-20 01:03:55 +03:00
alesapin
ee24b21aaf Fix revert order 2022-09-19 21:19:36 +02:00
alesapin
f67e23183f Fix parts removal on drop + add unrelated test 2022-09-19 20:19:49 +02:00
Amos Bird
1a21f0a0f0
normalize ast for replicated metadata 2022-09-19 22:06:41 +08:00
Amos Bird
0bb70d4378
Make minmax_count projection backward compatible 2022-09-19 22:06:40 +08:00
Kruglov Pavel
47f6f09ce0
Merge branch 'master' into improve-combinators 2022-09-19 14:31:12 +02:00
Alexey Milovidov
45b6052b20 Merge branch 'master' into compress-marks 2022-09-19 10:57:05 +02:00
Alexey Milovidov
b87d12494e Fix build 2022-09-19 08:53:20 +02:00
Alexey Milovidov
ab4db2d0c4 Fix 5/6 of trash 2022-09-19 08:50:53 +02:00
Alexey Milovidov
d4b9fe41be
Merge pull request #41457 from ClickHouse/remove-trash-5 
Remove trash from Field
 2022-09-19 06:36:48 +03:00
alesapin
a5a9d89fb9
Merge branch 'master' into enable_projections_and_zero_copy 2022-09-18 17:20:08 +02:00
alesapin
560bc2bc22
Merge pull request #41386 from ClickHouse/fix_redundant_fetch 
Fix redundant fetch in zero copy replication
 2022-09-18 15:57:27 +02:00
Alexey Milovidov
791de6592b Remove trash from Field 2022-09-18 05:16:08 +02:00
Alexey Milovidov
f4a48cd4d6 Remove cruft 2022-09-17 22:16:31 +02:00
alesapin
33216dedf9
Merge pull request #41323 from ClickHouse/remove_bad_exception 
Fix parts removal in case of connection loss
 2022-09-17 19:14:53 +02:00
alesapin
dcb1586977 Add comment 2022-09-17 19:11:58 +02:00
alesapin
db30a3041e
Update src/Storages/MergeTree/MutateFromLogEntryTask.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-09-17 19:10:23 +02:00
avogar
0101cc2e56 Support complex combinators in window transform, arrayReduce*, initializeAggregation and Aggregate functons versionning 2022-09-16 19:07:36 +00:00
Nikolai Kochetov
16f78eb804
Merge pull request #41281 from ClickHouse/fix-key-condition-with-actions-dag-literal-name 
Fix wrong literal name in KeyCondition using ActionsDAG.
 2022-09-16 16:06:38 +02:00
alesapin
b4d2d217f4
Merge branch 'master' into enable_projections_and_zero_copy 2022-09-16 13:53:17 +02:00
alesapin
06ccdb1ecf Fix nasty bug 2022-09-16 13:49:39 +02:00
alesapin
af259a2a68
Update src/Storages/MergeTree/MergeFromLogEntryTask.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-09-16 11:50:07 +02:00
Alexander Tokmakov
8301f5758b Update IMergeTreeDataPart.cpp 2022-09-15 21:32:57 +02:00
alesapin
be81d653e1 Fix comment 2022-09-15 18:56:16 +02:00
alesapin
b098c1be36 Fix data duplication in zero copy replication 2022-09-15 18:54:27 +02:00
alesapin
7570c5cf67 Remove recursive without blobs 2022-09-15 15:31:19 +02:00
alesapin
c585ed47fc Forget to remove 2022-09-15 15:15:28 +02:00
alesapin
e127c8b4a9 Add kringe 2022-09-15 14:49:31 +02:00
alesapin
c04f7afb2a Delete kringe 2022-09-15 14:48:59 +02:00
alesapin
63213465ef Fix comment 2022-09-14 17:28:12 +02:00
alesapin
ad279940d6 Trying print less scary messages in case No such key for outdated parts 2022-09-14 17:25:58 +02:00
alesapin
d8c14ebcf6 Fixup 2022-09-14 16:44:40 +02:00
alesapin
9d3ab9a7f7 Merge branch 'master' into enable_projections_and_zero_copy 2022-09-14 15:16:16 +02:00
Anton Popov
3623807527
Merge pull request #41290 from CurtizJ/dynamic-columns-20 
Fix `ALTER ADD COLUMN`  with columns of type `Object`
 2022-09-14 14:15:29 +02:00
Vladimir C
00d0b37cd0
Merge pull request #41263 from ClickHouse/vdimir/quorum-log-msg-fix 2022-09-14 10:54:21 +02:00
Anton Popov
ee54451741 fix adding of columns of type Object 2022-09-13 22:43:59 +00:00
Nikolai Kochetov
3f0bc94d25 Comment. 2022-09-13 18:50:01 +00:00
Nikolai Kochetov
66d9d92de4 Fix wrong literal name in KeyCondition using ActionsDAG. 2022-09-13 18:47:29 +00:00
alesapin
2cecf9e55f Fix bug 2022-09-13 18:30:55 +02:00
vdimir
f2acb53887
Better log message for replicas number in StorageReplicatedMerge 2022-09-13 13:49:51 +00:00
alesapin
7a1ae1f316 Fix for stateless test 2022-09-13 15:10:49 +02:00
zhongyuankai
f36d70e9dd
Merge branch 'master' into compress_marks_and_primary_key 2022-09-13 20:05:39 +08:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze 
Fix SYSTEM UNFREEZE for ordinary database
 2022-09-13 14:55:11 +03:00
alesapin
cc9f6d47be Merge branch 'master' into enable_projections_and_zero_copy 2022-09-13 12:34:22 +02:00
Kseniia Sumarokova
b2c9c04c7b
Merge pull request #40821 from kssenii/improve-marks-cache-loading 
Allow to load marks in threadpool in advance
 2022-09-13 12:31:58 +02:00
Kseniia Sumarokova
651d850652
Merge pull request #41230 from kssenii/minor-change 
minor change in MergeTreeDataPartCompact
 2022-09-13 12:23:18 +02:00
zhongyuankai
75c0854a78 fix test 2022-09-13 11:18:25 +08:00
zhongyuankai
1477bb132f
Merge branch 'master' into compress_marks_and_primary_key 2022-09-13 11:05:57 +08:00
kssenii
06f80d78f8 Fix 2022-09-12 18:16:53 +02:00
Alexander Tokmakov
da4fbb552d
Merge branch 'master' into fix_parts_removal 2022-09-12 17:49:29 +03:00
alesapin
e3e2c76606 FIX STYLE KEK 2022-09-12 13:52:57 +02:00
Kseniia Sumarokova
df22e8d30c
Merge pull request #41190 from kssenii/fix-clean-up-thread 
Fix cleaning up broken detached part after timeout
 2022-09-12 13:21:45 +02:00
kssenii
02c3d8b0a0 Fix 2022-09-11 16:42:56 +02:00
alesapin
6eead4ea1e Fix fasttest build 2022-09-11 15:16:54 +02:00
alesapin
ac53830000 Fix hung check 2022-09-11 13:37:39 +02:00
kssenii
b7d751b782 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-11 13:23:30 +02:00
Alexey Milovidov
7f1e7b5967 Merge branch 'master' into fix-half-of-trash 2022-09-11 06:20:47 +02:00
Alexey Milovidov
fd235919aa Remove some methods 2022-09-10 05:04:40 +02:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Anton Popov
d1fa2148de
Merge branch 'master' into dynamic-columns-14 2022-09-09 19:32:07 +02:00
Anton Popov
a1b5582a7c
Merge pull request #37152 from CurtizJ/fix-filling-of-missed-nested 
Fix filling of missed Nested columns with multiple levels
 2022-09-09 19:29:01 +02:00
alesapin
50121e42f3 Enable projections and zero copy replication 2022-09-09 18:38:02 +02:00
Nikolay Degterinsky
12e7cd6b19
Merge pull request #41068 from jthmath/bugfix 
fix ignore projection logic error
 2022-09-09 17:10:03 +02:00
alesapin
86a3719bef
Merge pull request #41128 from ClickHouse/less_errors 
Fix bad warnings in part fetches
 2022-09-09 16:58:09 +02:00
alesapin
01e745bbbf Fix parts removal 2022-09-09 13:26:13 +02:00
Alexander Tokmakov
48927ba0ac
Merge branch 'master' into no-hardlinks-while-making-backup-of-mergetree-in-atomic-db 2022-09-09 14:24:44 +03:00
alesapin
23d95e6723 Fix parts removal in case of failures 2022-09-09 13:24:22 +02:00
alesapin
e3fc78aa5f Fix bad warnings in part fetches 2022-09-09 12:14:42 +02:00
Anton Popov
ba41239ecd Merge remote-tracking branch 'upstream/master' into HEAD 2022-09-08 15:20:29 +00:00
Anton Popov
86b29b7f1a fix serilization of Object inside other types 2022-09-08 15:16:39 +00:00
alesapin
62c911924b
Merge pull request #41108 from ClickHouse/more_logging 
More logging for S3
 2022-09-08 17:10:55 +02:00
alesapin
5e00124f8a More logging for S3 2022-09-08 16:18:21 +02:00
Alexander Tokmakov
507ad0c4d9
Merge pull request #40691 from azat/mergetree/transaction-wal-fix 
Move committing InMemory parts to WAL out of NOEXCEPT_SCOPE()
 2022-09-08 15:55:30 +03:00
Vitaly Baranov
122009a2bd Use table lock if database is ordinary and zero-copy-replication is enabled. 2022-09-08 13:54:59 +02:00
alesapin
6da37b92e1 Improve logging one time 2022-09-08 13:11:53 +02:00
Vladimir C
90647cb0c5
Merge pull request #39970 from SachinSetiya/majority_quorum 
Add majority_insert_quorum setting
 2022-09-08 11:31:33 +02:00
Anton Popov
7c12b448b8 Merge remote-tracking branch 'upstream/master' into HEAD 2022-09-08 01:52:52 +00:00
Anton Popov
5a790b15b4 try to fix filling of missed Nested columns with multiple levels 2022-09-08 00:20:11 +00:00
Kseniia Sumarokova
a270eeef91
Merge pull request #41008 from kssenii/refactor-merge-tree-read 
Small refactoring around merge tree readers (get rid of data part ptr)
 2022-09-07 18:27:33 +02:00
alesapin
365438d617
Merge pull request #41016 from ClickHouse/one_more_logging 
Slightly improve diagnostics and remove assertions
 2022-09-07 15:23:17 +02:00
vdimir
6234d564d1
Fix ReplicatedMergeTreeSink, add more info to logs 2022-09-07 11:19:28 +00:00
Sachin
ade4337978
Add majority_insert_quorum setting 
majority_insert_quorum is defined as (number_of_replicas/2)+1. Insert
will be successful only if majority of quorum have applied it. If
insert_quorum and majority_insert_quorum both are specified, max of
both will be used.
 2022-09-07 11:19:24 +00:00
Kruglov Pavel
8513af5c1b
Merge pull request #40638 from azat/mergetree/insert-perf 
Do not obtain storage snapshot for each INSERT block (slightly improves performance)
 2022-09-07 12:53:09 +02:00
Vitaly Baranov
9c847ceec9 No hardlinks while making backup of MergeTree in atomic database. 2022-09-07 11:44:50 +02:00
alesapin
81c98dadd2 Remove redundant change 2022-09-07 11:01:06 +02:00
jthmath
478d2ad06a fix ignore projection 2022-09-07 10:23:52 +08:00
alesapin
43493389b9 Merge branch 'master' into one_more_logging 2022-09-06 19:40:24 +02:00
Anton Popov
9def867795 fix reading of subcolumns from in-memory parts 2022-09-06 17:38:51 +00:00
alesapin
b778b9f37f Improve logging better 2022-09-06 19:25:58 +02:00
zhongyuankai
d286214c4a fix typos 2022-09-07 00:24:06 +08:00
Anton Popov
f0a404e2c8 Merge remote-tracking branch 'upstream/master' into HEAD 2022-09-06 15:51:16 +00:00
zhongyuankai
c7b2f719c6 batter 2022-09-06 23:41:39 +08:00
Anton Popov
ce9b76f416 fix Nested in in-memory parts 2022-09-06 15:01:47 +00:00
Kseniia Sumarokova
3558361a05
Merge branch 'master' into refactor-merge-tree-read 2022-09-06 16:00:43 +02:00
Anton Popov
f76c1482bd try to fix filling of missed Nested columns with multiple levels 2022-09-06 13:56:32 +00:00
Filatenkov Artur
7949b3fb74
Merge branch 'master' into annoy 2022-09-06 11:23:15 +03:00
zhongyuankai
0bf76fe642 Merge branch 'compress-marks' into compress_marks_and_primary_key 2022-09-06 08:01:43 +08:00
zhongyuankai
d232d7e904 Merge branch 'master' into compress_marks_and_primary_key 2022-09-06 07:59:59 +08:00
Alexey Milovidov
940a53e519
Merge pull request #40984 from Lucky-Chang/typo_fix 
Fix some typos and clang-tidy warnings
 2022-09-06 02:37:50 +03:00
Kseniia Sumarokova
33f541042a
Fix clang tidy 2022-09-06 00:04:50 +02:00
alesapin
8977af5150 Remove some outdated assertions 2022-09-05 22:59:25 +02:00
kssenii
4c8aa04b73 Revert "Fix" 
This reverts commit ec55c90f36.
 2022-09-05 20:12:40 +02:00
kssenii
83514fa2ef Refactor 2022-09-05 20:08:22 +02:00
ianton-ru
39e1fc7a0f
Merge branch 'master' into fix-ordinary-system-unfreeze 2022-09-05 17:10:59 +03:00
Kseniia Sumarokova
e12858dca5
Merge pull request #36837 from ClickHouse/stress_s3 
Trying stress on top of S3
 2022-09-05 15:12:55 +02:00
Alexey Milovidov
ad9ae84e76 Fix error 2022-09-05 07:34:44 +02:00
Alexey Milovidov
fd2be1f041 Fix error 2022-09-05 07:31:29 +02:00
Alexey Milovidov
d7127e4b2d Make it slightly more sane 2022-09-05 07:26:58 +02:00
Alexey Milovidov
b4eec0e6f4 Fix strange code 2022-09-05 06:31:19 +02:00
Alexey Milovidov
6bc814a860 Renaming 2022-09-05 04:47:32 +02:00
Alexey Milovidov
b6ab997ca0 Merge branch 'compress_marks_and_primary_key' of github.com:zhongyuankai/ClickHouse into compress-marks 2022-09-05 04:30:08 +02:00
Alexey Milovidov
79f0d12f39 Merge branch 'master' into compress-marks 2022-09-05 04:29:46 +02:00
Luck-Chang
1ac8e739c9 fix some typos and clang-tidy warnings 2022-09-05 09:50:24 +08:00
kssenii
48dc32faf8 Better test 2022-09-04 19:54:34 +02:00
Azat Khuzhin
2e85f9f0ad Remove completely processed WAL files 
Previously all WAL files had been stored, though with the time of use
this can take too much space on disk, and also the startup time will be
increased.

But it is pretty easy to prune old WAL files (the one parts from which
had been completely written to disk already).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-04 14:18:14 +02:00
alesapin
a67703b76b
Merge branch 'master' into stress_s3 2022-09-04 13:00:28 +02:00
Alexey Milovidov
193cd1b3b2
Merge pull request #39138 from nickitat/control_block_size_in_aggregator 
Control block size in aggregator
 2022-09-04 04:51:00 +03:00
Alexey Milovidov
097221a3af
Merge pull request #40726 from amosbird/projection-fix-1 
Disable projection when grouping set is used.
 2022-09-04 04:23:30 +03:00
Yakov Olkhovskiy
fdcced8962
Merge pull request #40850 from canhld94/ch_canh_fix_set 
Fix tryGetPreparedSet crash when same set expression built from different column(s)
 2022-09-02 14:40:35 -04:00
Anton Popov
464818c142 try to fix filling of missed Nested columns with multiple levels 2022-09-02 15:05:58 +00:00
Anton Popov
9dda9658a8 Merge remote-tracking branch 'upstream/master' into HEAD 2022-09-02 12:48:27 +00:00
alesapin
3f02995161 Better exceptions handling 2022-09-02 13:54:41 +02:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Duc Canh Le
65b64010a0 move out-of-bound check into set::areTypesEqual 2022-09-02 11:56:13 +08:00
Duc Canh Le
763328ef53 Merge branch 'master' into ch_canh_fix_set 2022-09-02 11:45:13 +08:00
kssenii
ec55c90f36 Fix 2022-09-01 17:04:34 +02:00
alesapin
f2bcd63618
Merge pull request #40844 from ClickHouse/fix_incorrect_remove_in_zero_copy 
Fix incorrect directory remove in zero copy replication
 2022-09-01 13:21:58 +02:00
Vitaly Baranov
6dcca686cb
Merge pull request #40827 from vitlibar/fix-incremental-backups-for-log-family 
Fix incremental backups for Log family.
 2022-09-01 13:13:32 +02:00
kssenii
82879543d3 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-01 13:13:01 +02:00
Anton Popov
f7bdf07adc
Merge pull request #38715 from CurtizJ/fix-read-in-order-fixed-prefix 
Better support of `optimize_read_in_order` in case of fixed prefix of sorting key
 2022-09-01 12:59:18 +02:00
zhongyuankai
228e984856
Merge branch 'master' into compress_marks_and_primary_key 2022-09-01 11:07:21 +08:00
alesapin
251bdab9a0 Better grep 2022-08-31 20:08:47 +02:00
Kruglov Pavel
86516d3bb4
Merge pull request #40740 from amosbird/row-policy-index-fix-1 
Use index when row_policy_filter is always false
 2022-08-31 18:46:14 +02:00
Duc Canh Le
55c20ce45b fix tryGetPreparedSet crash when same set expression built from different columns 2022-08-31 21:43:08 +08:00
kssenii
3a183236a2 Fix 2022-08-31 15:39:53 +02:00
Filatenkov Artur
9a231cf49a
fix out-of-bounds 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2022-08-31 16:28:10 +03:00
alesapin
c427491bcb Better 2022-08-31 15:05:45 +02:00
alesapin
a4f71ba6b8 Fix one more bug 2022-08-31 14:53:33 +02:00
alesapin
2d976753e1 Fix incorrect directory remove in zero copy replication 2022-08-31 13:50:59 +02:00
Vitaly Baranov
007ae0e6cc Fix incremental backups for Log family. 2022-08-31 12:57:28 +02:00
FArthur-cmd
941cceda7f fix build 2022-08-30 23:09:22 +00:00
Anton Popov
3504781529
Merge branch 'master' into fix-read-in-order-fixed-prefix 2022-08-30 23:32:43 +02:00
kssenii
a5f69517ae Allow to load marks in threadpool 2022-08-30 20:20:05 +02:00
FArthur-cmd
dadc29fb73 add checks for sizes 2022-08-30 15:46:55 +00:00
FArthur-cmd
c7d31c3a90 fix indexes for ColumnArray 2022-08-30 15:32:05 +00:00
FArthur-cmd
e7abc06c89 Revert "Revert "Add Annoy index"" 
This reverts commit 6fdfb964d0.
 2022-08-30 15:26:56 +00:00
zhongyuankai
7cd8079c75 fix test 2022-08-30 22:08:27 +08:00
Alexander Tokmakov
6fdfb964d0
Revert "Add Annoy index" 2022-08-30 15:10:10 +03:00
Anton Popov
ee691961d9 fix build 2022-08-30 11:43:27 +00:00
Kruglov Pavel
624af5bffd
Update src/Storages/MergeTree/KeyCondition.cpp 2022-08-30 13:18:05 +02:00
zhongyuankai
5992cbeaa1 Merge branch 'master' into compress_marks_and_primary_key 2022-08-30 08:51:21 +08:00
zhongyuankai
c1d7f60064 fix test 2022-08-30 08:50:27 +08:00
FArthur-cmd
862b53b06f Merge branch 'annoy-2' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-08-29 16:43:39 +00:00
FArthur-cmd
3305af8db2 fix case when query is already matched 2022-08-29 16:43:24 +00:00
Filatenkov Artur
d73f661732
Merge branch 'master' into annoy-2 2022-08-29 17:33:13 +03:00
Alexander Tokmakov
8d90d30d37
Merge pull request #40589 from ClickHouse/remove_wrong_code_from_mutations 
Remove wrong code for skipping mutations in MergeTree
 2022-08-29 14:18:59 +03:00
Alexander Tokmakov
eda0582ec0
Merge pull request #40641 from ClickHouse/fix_startup_of_dropped_replica 
Do not try to strartup dropped replica
 2022-08-29 14:15:15 +03:00
Amos Bird
15a69bce84
Use index when row_policy_filter is always false 2022-08-29 16:44:32 +08:00
Amos Bird
abf51bccfd
Disable projection when grouping set is used. 2022-08-29 14:35:48 +08:00
Alexey Milovidov
5ab2413601
Merge branch 'master' into compress-marks 2022-08-29 04:50:35 +03:00
Alexey Milovidov
27782ceef8 Remove useless method 2022-08-28 22:33:42 +02:00
Alexey Milovidov
dd412f6145 Fix error 2022-08-28 20:42:11 +02:00
Alexey Milovidov
5ee81598b5 Merge branch 'compress_marks_and_primary_key' of github.com:zhongyuankai/ClickHouse into compress-marks 2022-08-28 04:19:14 +02:00
Azat Khuzhin
3c478da003 Move committing InMemory parts to WAL out of NOEXCEPT_SCOPE() 
Since this commit can definitelly throw (i.e. due to ENOSPC).

Note, that it should be safe, since rollback() will call dropPart() for
those parts.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-26 23:37:59 +02:00
FArthur-cmd
f3d0083e57 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy-2 2022-08-26 13:53:53 +00:00
Vladimir C
e067629e0d
Merge pull request #40239 from vdimir/vdimir/tmp-file-metrics 
More metrics for on-disk temporary data
 2022-08-26 11:28:01 +02:00
Alexander Tokmakov
b8e3a582f7 do not try to strartup dropped replica 2022-08-25 22:14:39 +02:00
Azat Khuzhin
09aaa2b5dd Do not obtain storage snapshot for each INSERT block (improves performance) 
It depends on how much parts do you have, but for some workload with
InMemory only parts w/o merges, I got 5% increase.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-25 21:02:27 +02:00
Alexander Gololobov
6a69e08799
Merge pull request #40559 from ClickHouse/lwd_vertical_merge_fix 
Fix vertical merge of parts with lightweight deleted rows
 2022-08-25 20:47:44 +02:00
Alexander Tokmakov
eae2667a1c
Merge pull request #40148 from ClickHouse/replicate-tables-readonly 
`ReplicatedMergeTree` tables as readonly until attach is finished
 2022-08-25 19:39:19 +03:00
Alexander Tokmakov
a6f55b1430 fix 2022-08-25 16:27:12 +02:00
Vladimir C
ddde5096ef
Merge branch 'master' into vdimir/tmp-file-metrics 2022-08-25 15:23:35 +02:00
Antonio Andelic
797e21761a Address PR comments 2022-08-24 18:32:03 +00:00
Antonio Andelic
da0d6056f4 Merge branch 'master' into replicate-tables-readonly 2022-08-24 17:29:33 +00:00
vdimir
7194df1184
Move back TemporaryFile -> TemporaryFileOnDisk 2022-08-24 16:14:11 +00:00
vdimir
1e24a731f7
Move Disks/TemporaryFileOnDisk -> Common/TemporaryFile 2022-08-24 16:14:10 +00:00
vdimir
cd4038d511
Use TemporaryFileOnDisk instead of Poco::TemporaryFile 2022-08-24 16:14:08 +00:00
Alexander Gololobov
1c2dd50ca5 Fix vertical merge of parts with lightweight deleted rows 2022-08-24 15:18:33 +02:00
FArthur-cmd
f59735d4f9 fix style 2022-08-21 12:45:35 +00:00
FArthur-cmd
fb7785c740 add support queries with parameters, add setting to avoid memory overflow 2022-08-21 11:07:03 +00:00
alesapin
78ba732b31 Merge branch 'master' into data_source_description 2022-08-21 12:11:21 +02:00
FArthur-cmd
1e8910f048 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy-2 2022-08-21 08:45:03 +00:00
Alexey Milovidov
ab91c99495
Merge branch 'master' into control_block_size_in_aggregator 2022-08-20 21:28:27 +03:00
alesapin
e4d486fbba
Merge pull request #40346 from ClickHouse/fix_ttl_vertical_merge_bug 
Fix bug with multiple TTL merges on expired column
 2022-08-20 17:09:32 +02:00
Alexey Milovidov
6f39e3e039
Merge pull request #40419 from ClickHouse/fix-parallel-replicas 
Fix obvious trash in Parallel replicas
 2022-08-20 14:10:14 +03:00
Nikita Mikhaylov
4a453f0776 Fix obvious trash 2022-08-19 18:08:13 +00:00
alesapin
d8664c3227 Add shortcut for backups 2022-08-19 16:58:30 +02:00
Antonio Andelic
d07fd218a2 Fix initialization 2022-08-19 14:56:17 +00:00
Antonio Andelic
eeb716d36b Remove unused error code 2022-08-19 12:03:02 +00:00
Antonio Andelic
ee3df77fda Simplify logic 2022-08-19 11:12:20 +00:00
Antonio Andelic
5e2c8869a9 Merge branch 'master' into replicate-tables-readonly 2022-08-19 09:41:11 +00:00
Antonio Andelic
3f7d28cc6b Address PR comments 2022-08-19 09:39:59 +00:00
Antonio Andelic
a8f3b62bf6 Use global zookeeper 2022-08-19 08:49:51 +00:00
Antonio Andelic
6f533313e9 Restart everything during attach 2022-08-19 08:17:02 +00:00
alesapin
a2e08299f8
Update src/Storages/MergeTree/IMergedBlockOutputStream.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-08-18 15:36:55 +02:00
alesapin
20a2e786c9 Fix bug with multiple TTL merges on expired column 2022-08-18 14:21:20 +02:00
Alexander Tokmakov
6e5e0689a3
Merge branch 'master' into parallel_drop_ranges_execution 2022-08-18 13:43:29 +03:00
Antonio Andelic
7dce8e5ebd Use just atomic for first try 2022-08-18 06:57:40 +00:00
FArthur-cmd
05331603f5 fix size 2022-08-18 06:16:18 +00:00
kssenii
d63d34f7bb Merge master 2022-08-17 15:49:32 +02:00
alesapin
4398a357c8
Merge pull request #40151 from ClickHouse/fix_cannot_quickly_remove_directory 
Fix "Cannot quickly remove directory"
 2022-08-17 12:11:18 +02:00
Antonio Andelic
1936ea6377 Fix test issues 2022-08-17 08:28:53 +00:00
Nikita Taranov
6bdbaccc37 use max_block_size from settings 2022-08-16 18:56:22 +02:00
Antonio Andelic
b0b9f96837 Use setting for retry period, create independent zookeeper sessions 2022-08-16 13:07:51 +00:00
Alexander Tokmakov
387b69caea fix race condition 2022-08-16 14:03:31 +02:00
Alexander Tokmakov
d4fb75c8db Revert "fix race with check thread" 
This reverts commit 45284108ba.
 2022-08-16 12:49:17 +02:00
Antonio Andelic
9ac4ca265a Small initialization fixes 2022-08-16 08:19:02 +00:00
Antonio Andelic
2ed42bca7e Merge branch 'master' into replicate-tables-readonly 2022-08-16 08:17:53 +00:00
kssenii
eb26b219b9 Merge master 2022-08-16 00:56:27 +02:00
Alexander Tokmakov
589c3408d2
Merge pull request #40234 from ClickHouse/better_message_on_restore_covered 
Better error message when restoring covered parts
 2022-08-15 22:01:48 +03:00
Alexander Tokmakov
b3f3b60ac3 allow parallel execution of disjoint drop ranges 2022-08-15 20:32:03 +02:00
FArthur-cmd
1d048432a1 update contrib and update cmake 2022-08-15 14:35:51 +00:00
Alexander Tokmakov
c18f3bef91 fix 2022-08-15 16:03:03 +02:00
Alexander Tokmakov
edaff70010 better error message when restoring covered parts 2022-08-15 13:53:14 +02:00
Nikita Mikhaylov
2282f2f414
Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-08-15 12:25:45 +02:00
Alexey Milovidov
3d5ceb8fee Merge branch 'master' into disable-zero-copy-replication 2022-08-14 01:40:20 +02:00
Alexey Milovidov
201cf69854
Merge pull request #39470 from nicelulu/issues_39469 
Fix issues 39469
 2022-08-13 03:13:28 +03:00
Alexey Milovidov
0e450787f3 Edit the documentation 2022-08-12 21:24:27 +02:00
Alexey Milovidov
a88e713102 Disable zero-copy replication by default 2022-08-12 21:22:11 +02:00
Alexander Tokmakov
46aaf842f8
Merge pull request #40031 from ClickHouse/fix_old_tmp_dirs_cleanup 
Fix old temporary directories cleanup
 2022-08-12 20:31:20 +03:00
Anton Popov
4bd50bb06c
Merge branch 'master' into distinct_sorted_simplify 2022-08-12 17:11:18 +02:00
Antonio Andelic
c6b3a897d2 Test replicated table without ZK, small fixes 2022-08-12 11:56:46 +00:00
Nikita Mikhaylov
00d2c57a68
Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-08-12 13:32:24 +02:00
Alexander Tokmakov
e691888267 fix build 2022-08-12 13:03:57 +02:00
Alexander Tokmakov
d24b9874bc fix Cannot quickly remove directory 2022-08-12 12:47:25 +02:00
Antonio Andelic
1c0f85ccdb Attach table in background 2022-08-12 09:32:13 +00:00
kssenii
768a207176 Merge master 2022-08-11 22:25:53 +02:00
Amos Bird
fa8fab2e8f
Fix KeyCondition with other filters 2022-08-11 19:20:44 +08:00
Jianmei Zhang
d1051d822c
Use getSerializedFileExtension() to get correct file extension for index (#40095) 2022-08-11 12:39:40 +02:00
Alexey Milovidov
38fc0cc783
Merge branch 'master' into issues_39469 2022-08-11 04:45:06 +03:00
zhongyuankai
6ee9191203 fix build 2022-08-11 08:19:08 +08:00
Alexander Tokmakov
fff903ee81 fix 2022-08-10 15:48:56 +02:00
vdimir
b7c5c54181
Fix build 2022-08-10 13:43:55 +00:00
vdimir
5eb4cd39e0
Merge branch 'master' into refactor-prepared-sets 2022-08-10 11:47:49 +00:00
Alexander Tokmakov
54614f5b98 Merge branch 'master' into fix_old_tmp_dirs_cleanup 2022-08-10 13:13:04 +02:00
zhongyuankai
71935b96ef support compress marks and primary key 2022-08-10 18:07:14 +08:00
kssenii
5c3227ba56 Merge master 2022-08-10 12:00:34 +02:00
FArthur-cmd
0ab937afbb small update 2022-08-10 09:55:07 +00:00
Maksim Kita
aff8149f5c
Merge pull request #39998 from kitaisreal/actions-dag-refactoring 
ActionsDAG rename index to outputs
 2022-08-10 11:44:18 +02:00
kssenii
0dda03c94b Fix checks 2022-08-10 00:06:58 +02:00
Alexander Tokmakov
060b270929 fix 2022-08-09 23:16:08 +02:00
Alexander Tokmakov
fe572104aa fix old tmp dirs cleanup 2022-08-09 18:44:51 +02:00
Maksim Kita
a576a55375 Fixed build 2022-08-09 15:03:59 +02:00
zhangxiao871
8bae5f24af Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into yandex-master 2022-08-09 15:49:03 +08:00
Nikolai Kochetov
540ae64f37 Fix big memory usage for fetch. 2022-08-08 13:32:49 +00:00
vdimir
708747ca0b
Merge branch 'master' into refactor-prepared-sets 2022-08-08 14:27:18 +02:00
Alexander Tokmakov
25a9bf5deb
Merge pull request #39922 from ClickHouse/fix_flaky_tests_commit_obsolete_part 
Fix flaky tests (`Tried to commit obsolete part`)
 2022-08-05 14:36:43 +03:00
kssenii
91f4143b6e Merge master 2022-08-05 12:14:08 +02:00
Alexander Tokmakov
4bf7a2ca3a fix warning 2022-08-05 12:05:50 +02:00
Nikolai Kochetov
2fe893ba67
Merge pull request #25563 from ClickHouse/use-dag-in-key-condition 
Use ActionsDAG in KeyCondition
 2022-08-05 11:22:46 +02:00
kssenii
7a9b0bc47f Merge master 2022-08-05 01:48:52 +02:00
Anton Popov
0492ef63a4
Merge pull request #39464 from CurtizJ/fix-custom-serializations 
Fix columns of type `Object` with sparse serialization
 2022-08-05 00:55:41 +02:00
Nikolai Kochetov
60599197b2 Review fixes. 2022-08-04 15:23:10 +00:00
Nikolai Kochetov
658a269d56
Merge branch 'master' into use-dag-in-key-condition 2022-08-04 16:18:40 +02:00
FArthur-cmd
a2eafae157 fix bug with limit in order by 2022-08-04 12:33:18 +00:00
alesapin
1eb28f5f5f
Merge pull request #39860 from ClickHouse/avoid_additional_disk_touch 
Better total part size calculation on mutation
 2022-08-04 14:21:31 +02:00
Vladimir C
a627b00c43
Merge branch 'master' into refactor-prepared-sets 2022-08-04 13:27:38 +02:00
Nikolay Degterinsky
71cb055ecc
Merge pull request #39812 from guowangy/applyFunction-multi-thread 
KeyCondition: optimize applyFunction in multi-thread scenario
 2022-08-04 01:36:47 +02:00
Anton Popov
04f6a85070 fix mutations 2022-08-03 17:34:46 +00:00
Maksim Kita
e6efb47aa3
Merge pull request #39850 from kitaisreal/select-query-has-join-method 
Add hasJoin method into ASTSelectQuery
 2022-08-03 18:50:25 +02:00
Anton Popov
5da32fafb3 fix ALTER MODIFY COLUMN from nested 2022-08-03 15:08:38 +00:00
alesapin
f608e62c27 Fix call 2022-08-03 14:17:31 +02:00
alesapin
56a4d26e87 Better total part size calculation on mutation 2022-08-03 14:15:45 +02:00
Maksim Kita
3e6b663020 ASTSelectQuery added hasJoin method 2022-08-03 11:25:45 +02:00
Anton Popov
a333cc4146 Merge remote-tracking branch 'upstream/master' into HEAD 2022-08-02 12:57:43 +00:00
Igor Nikonov
56697125d3 Merge remote-tracking branch 'origin/master' into distinct_sorted_simplify 2022-08-02 11:18:46 +00:00
Wangyang Guo
91e3e2f18b KeyCondition: optimize applyFunction in multi-thread scenario 
Construct and deconstruct args (ColumnsWithTypeAndName) will inc/dec
ref_count (actually this is a atomic lock inc/dec operation) to share_ptr,
which may share the same DataTypePtr among different threads. This will
have a lock contention issue in large parallel situation.

The patch try to minimize `args` scope and reduce unnecessary
construct/destory of instances. It will improve the performance in
multi-thread cases.
 2022-08-02 16:16:16 +08:00
Robert Schulze
bf574b9154
Merge pull request #39760 from ClickHouse/bit-fiddling 
Use std::popcount, ::countl_zero, ::countr_zero functions
 2022-08-01 17:04:51 +02:00
fastio
e113d697be
Update src/Storages/MergeTree/MergeTreeSettings.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-08-01 11:04:18 +08:00
Dmitry Novik
ab5a147065
Merge pull request #38725 from azat/fix-order-by-projection 
Fix ORDER BY that matches projections ORDER BY
 2022-08-01 00:32:59 +02:00
kssenii
e5f4a619ed Merge master 2022-07-31 20:24:40 +03:00
Robert Schulze
a7734672b9
Use std::popcount, ::countl_zero, ::countr_zero functions 
- Introduced with the C++20 <bit> header

- The problem with __builtin_c(l|t)z() is that 0 as input has an
  undefined result (*) and the code did not always check. The std::
  versions do not have this issue.

- In some cases, we continue to use buildin_c(l|t)z(), (e.g. in
  src/Common/BitHelpers.h) because the std:: versions only accept
  unsigned inputs (and they also check that) and the casting would be
  ugly.

(*) https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
 2022-07-31 15:16:51 +00:00
Igor Nikonov
7b0b38e997 DistinctSortedTransform works only if columns contains sort prefix of 
sort description
 2022-07-29 20:01:07 +00:00
Nikolai Kochetov
59a11b32ad
Merge branch 'master' into use-dag-in-key-condition 2022-07-29 17:01:33 +02:00
Igor Nikonov
ae82bfb7f3 Merge remote-tracking branch 'origin/master' into distinct_sorted_simplify 2022-07-29 13:12:03 +00:00
Anton Popov
57e4fb2e30 Merge remote-tracking branch 'upstream/master' into HEAD 2022-07-29 11:42:11 +00:00
Anton Popov
bb551566e7 fix build 2022-07-29 11:41:53 +00:00
Anton Popov
04fd72cdef fix alter column rename 2022-07-28 13:19:23 +00:00
Vladimir C
115506356c
Merge branch 'master' into refactor-prepared-sets 2022-07-27 19:57:23 +02:00
Nikolai Kochetov
6206e468a4 Ignor materialize function for ActionsDAG in PK analysis. 2022-07-27 17:02:48 +00:00
Anton Popov
56039c8780 fix alter of LowCardinality 2022-07-27 15:35:09 +00:00
Anton Popov
4cf0353892 try to fix Nested 2022-07-27 14:05:16 +00:00
Filatenkov Artur
a218597546
Merge branch 'master' into annoy-2 2022-07-27 15:37:58 +03:00
FArthur-cmd
6b5e92e68a change index policy 2022-07-27 15:36:07 +03:00
vdimir
8eecb9ef82
upd PreparedSets: rename/change signature of methods, add comments 2022-07-27 11:22:16 +00:00
Alexander Tokmakov
8fc075a527
Merge pull request #39619 from ClickHouse/try-do-enable-global-allocations-for-attach 
Replace MemoryTrackerBlockerInThread to LockMemoryExceptionInThread
 2022-07-27 13:13:04 +03:00
Alexander Gololobov
a64aa00869
Merge pull request #37893 from zhangjmruc/feature/sql-standard-delete 
Support SQL standard "delete from ... where ..." syntax and lightweight implementation on merge tree tables
 2022-07-26 23:39:07 +02:00
vdimir
5ce2960f03
Get rid of SelectQueryInfoBase -> SelectQueryInfo 2022-07-26 18:39:09 +00:00
vdimir
d9928ac93d
Add methods to SubqueryForSet, do not use refernce to SetPtr 2022-07-26 18:39:09 +00:00
vdimir
1e3fa2e01f
Refactor PreparedSets/SubqueryForSet 2022-07-26 18:39:02 +00:00
Anton Popov
9321ca34cf try to fix Nested 2022-07-26 17:32:37 +00:00
Nikolai Kochetov
33ce662d3e Replace MemoryTrackerBlockerInThread to LockMemoryExceptionInThread in some places. Reduced MemoryTrackerBlockerInThread level to User. 2022-07-26 15:22:00 +00:00
Kruglov Pavel
c683cb252f
Merge pull request #39227 from amosbird/rename-log1 
Rename log when rename merge tree tables
 2022-07-26 17:12:44 +02:00
Alexander Gololobov
25deba2c1b
Merge branch 'master' into feature/sql-standard-delete 2022-07-25 22:13:20 +02:00
Alexander Tokmakov
778dcf6994
Merge pull request #39230 from Algunenano/improve_replicated_merge_logging 
Improve logging around replicated merges
 2022-07-25 18:42:12 +03:00
Alexander Gololobov
48de02a7b8 Capitalized const name 2022-07-25 16:32:16 +02:00
Alexander Gololobov
7b44950d1d More detailed comment 2022-07-25 16:14:33 +02:00
Nikolai Kochetov
b70be40804
Merge branch 'master' into use-dag-in-key-condition 2022-07-25 14:30:22 +02:00
Nikolai Kochetov
1ce8e714d3 Fixing build. 2022-07-25 12:30:01 +00:00
Alexander Gololobov
460950ecdc
Merge branch 'master' into feature/sql-standard-delete 2022-07-24 21:27:22 +02:00
Igor Nikonov
eacc57b6a4 Simplify DistinctSorted 
+ use ordinary Distinct for deduplication in MergeTree when no sorting provided
 2022-07-24 14:26:25 +00:00
Azat Khuzhin
22d8e532ed Fix ORDER BY that matches projections ORDER BY 
In case of projection is seleted and it is comlete (all parts has such
projection) and ORDER BY matches projection ORDER BY, and
optimize_read_in_order=1 (default), then the sorting is simply not done,
because projections has separate plan, and so
InterpreterSelectQuery::executeOrder() -> SortingStep has
pipeline.getNumStreams() == 0 and it cannot do sorting.

Fix this, by adding sorting when creating plan for reading from
projections.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-24 15:23:13 +03:00
Kseniia Sumarokova
04944ff6f5
Merge pull request #39509 from monadbobo/remove_duplicate_code 
Removed some duplicate code.
 2022-07-24 14:24:40 +03:00
Alexander Gololobov
6692770113 Allow to update "system" virtual columns such as _row_exists 2022-07-24 12:21:47 +02:00
Alexander Gololobov
c8b3c574a4 Disable lightweight delete if table has projections 2022-07-24 12:21:47 +02:00
Alexander Gololobov
be64b45583 Properly handle empty parts list 2022-07-24 12:21:47 +02:00
Alexander Gololobov
594195451e Cleanups 2022-07-24 12:21:18 +02:00
Kseniia Sumarokova
f146183400
Merge pull request #39491 from ClickHouse/remove_unused_constructor 
Remove unused constructor
 2022-07-23 21:16:39 +02:00
mergify[bot]
c01ff2d38a
Merge branch 'master' into improve_replicated_merge_logging 2022-07-23 19:10:10 +00:00
Anton Popov
3d03b2714b try to fix Nested 2022-07-22 23:05:11 +00:00
Nikolai Kochetov
dd65a36c59 Fix another one test. 2022-07-22 19:11:25 +00:00
Nikolai Kochetov
53175e43f3 Better support for projections. 2022-07-22 19:11:25 +00:00
Simon Liu
2305f282dc Removed some duplicate code. 2022-07-23 02:27:17 +08:00
Anton Popov
49627aa554 Merge remote-tracking branch 'upstream/master' into HEAD 2022-07-22 17:16:06 +00:00
Alexander Tokmakov
bed2206ae9
Merge pull request #39460 from ClickHouse/remove_some_dead_and_commented_code 
Remove some dead and commented code
 2022-07-22 13:24:34 +03:00
alesapin
69984b28e0 Remove unused constructor 2022-07-22 12:21:10 +02:00
Vadim Volodin
8f14c60932 remove exception for old partitions, add server configuration option to enable system unfreeze 2022-07-22 13:10:08 +03:00
zhangxiao871
5549b64c06 Fix issues 39469 2022-07-21 23:52:33 +08:00
Anton Popov
10719fd61d fix style check 2022-07-21 15:06:55 +00:00
Anton Popov
e0d2c8fb37 fix json type with sparse columns 2022-07-21 14:47:19 +00:00
Vadim Volodin
85bf022258 review zookeeper and partition matching 2022-07-21 16:57:28 +03:00
Alexander Tokmakov
a8da5d96fc remove some dead and commented code 2022-07-21 15:05:48 +02:00
Alexander Gololobov
1ea9f143ff Leave only _row_exists-based implementation of lightweight delete 2022-07-21 11:26:13 +02:00
Nikolai Kochetov
e15967e9db
Merge pull request #38475 from ClickHouse/additional-filters 
Additional filters for a table (from setting)
 2022-07-21 07:52:04 +02:00
fastio
f382a91d54 Auto optimize partition in background if enabled 2022-07-21 10:49:09 +08:00
Alexey Milovidov
fd691000b7
Merge pull request #39342 from azat/bg-executor-fix 
Fix possible UB in MergeTreeBackgroundExecutor (leads to SIGSEGV on race with DROP/DETACH)
 2022-07-21 05:22:51 +03:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Nikolai Kochetov
4e8cd70b1d
Merge branch 'master' into use-dag-in-key-condition 2022-07-20 17:38:33 +02:00
Dmitry Novik
a09422de7b
Merge pull request #38642 from ClickHouse/group-by-use-nulls 
Add group_by_use_nulls setting
 2022-07-20 14:42:42 +02:00
Alexander Tokmakov
988569bdb1
Merge pull request #39366 from ClickHouse/fix_race_in_replication_queue 
Do not enqueue uneeded parts for check
 2022-07-20 00:35:02 +03:00
Nikolai Kochetov
f570cde815 Fixing build. 2022-07-19 20:19:57 +00:00
Nikolai Kochetov
a00bcd389f Fixing build 2022-07-19 19:33:14 +02:00
Nikolai Kochetov
eaeb30a71a Merge branch 'master' into use-dag-in-key-condition 2022-07-19 18:39:52 +02:00
Dmitry Novik
50989bdb68
Merge branch 'master' into group-by-use-nulls 2022-07-19 14:58:01 +02:00
Alexander Tokmakov
a761da9f1b add comment 2022-07-19 13:58:59 +02:00
kssenii
21ebf8874e Merge remote-tracking branch 'upstream/master' into make-cache-composable 2022-07-19 13:36:42 +02:00
Nikolai Kochetov
d0db7135cb
Merge pull request #38946 from amosbird/nullable-partition-fix1 
Fix incorrect partition pruning when there is a nullable partition
 2022-07-19 12:14:36 +02:00
FArthur-cmd
3b30cd9259 improvements 2022-07-19 00:59:07 +03:00
Alexander Tokmakov
d82f378a9d do not enqueue uneeded parts for check 2022-07-18 23:37:07 +02:00
Alexander Gololobov
614cb9a87f Disable trivial count() optimization if _row_exists column is present 2022-07-18 20:08:01 +02:00
Alexander Gololobov
ae0d00083c Renamed __row_exists to _row_exists 2022-07-18 20:07:36 +02:00
Alexander Gololobov
f324ca9921 Cleanups 2022-07-18 20:07:22 +02:00
Alexander Gololobov
9de72d995a POC lightweight delete using __row_exists virtual column and prewhere-like filtering 2022-07-18 20:06:42 +02:00
Robert Schulze
6df3c9d799
Merge pull request #39300 from ClickHouse/stringref-to-stringview 
First try at reducing the use of StringRef
 2022-07-18 19:20:16 +02:00
Azat Khuzhin
c6f7698f9a Fix possible UB in MergeTreeBackgroundExecutor 
Lambda erase_from_active() captures the item (TaskRuntimeDataPtr), most
of the code path is OK, since it explicitly reset the item->task.

However one is not, when it moves the item to pending list, which will
be cleaned up when the table will be DROP/DETACH'ed, from
MergeTreeBackgroundExecutor::removeTasksCorrespondingToStorage(),
and in this case if IStorage will be already removed, then it will lead
to use-after-free on destroying the lambda, since it captures the item
by value.

And I belive that CI founds this issue here [1]:

<details>

<summary>stack trace</summary>

    4 0x268d1354 in DB::ReplicatedMergeTreeQueue::CurrentlyExecuting::~CurrentlyExecuting() build_docker/../src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp:1510:33
    5 0x268ead1a in std::__1::default_delete<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting>::operator()(DB::ReplicatedMergeTreeQueue::CurrentlyExecuting*) const build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:54:5
    6 0x268ead1a in std::__1::unique_ptr<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting, std::__1::default_delete<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting> >::reset(DB::ReplicatedMergeTreeQueue::CurrentlyExecuting*) build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:315:7
    7 0x268ead1a in std::__1::unique_ptr<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting, std::__1::default_delete<DB::ReplicatedMergeTreeQueue::CurrentlyExecuting> >::~unique_ptr() build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:269:19
    8 0x268ead1a in DB::ReplicatedMergeTreeQueue::SelectedEntry::~SelectedEntry() build_docker/../src/Storages/MergeTree/ReplicatedMergeTreeQueue.h:351:12
    9 0x268ead1a in void std::__1::__destroy_at<DB::ReplicatedMergeTreeQueue::SelectedEntry, 0>(DB::ReplicatedMergeTreeQueue::SelectedEntry*) build_docker/../contrib/libcxx/include/__memory/construct_at.h:56:13
    ...
    16 0x265e9abb in DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>)::'lambda'()::~() build_docker/../src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:127:30
    17 0x265e9abb in DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) build_docker/../src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:235:1
    18 0x265ea730 in DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() build_docker/../src/Storages/MergeTree/MergeTreeBackgroundExecutor.cpp:261:13

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/533c7888038453c047df816f3f65292cca05a54f/stateless_tests__ubsan__actions_.html

And I also I think that the following issue will also be fixed [2]:

<details>

<summary>stack trace</summary>

    [ 680 ] {} <Fatal> : Logical error: 'Memory tracker: allocations not allowed.'.
    ...
    [ 23430 ] {} <Fatal> BaseDaemon: 23. /build/build_docker/../src/Common/formatReadable.cpp:46: formatReadableSizeWithBinarySuffix(double, int) @ 0x1713bb98 in /usr/bin/clickhouse
    ...
    [ 23430 ] {} <Fatal> BaseDaemon: 31. ../src/Common/MemoryTracker.cpp:112: MemoryTracker::logPeakMemoryUsage() @ 0x170e0ab9 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 32. /build/build_docker/../src/Common/MemoryTracker.cpp:98: MemoryTracker::~MemoryTracker() @ 0x170e063a in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 33. /build/build_docker/../src/Storages/MergeTree/MergeList.cpp:144: DB::MergeListElement::~MergeListElement() @ 0x279fb290 in /usr/bin/clickhouse
    ...
    [ 23430 ] {} <Fatal> BaseDaemon: 38. /build/build_docker/../src/Storages/MergeTree/BackgroundProcessList.h:41: DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>::~BackgroundProcessListEntry() @ 0x276ce6c7 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 39. /build/build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:54: std::__1::default_delete<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo> >::operator()(DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>*) const @ 0x276ce60b in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 40. /build/build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:316: std::__1::unique_ptr<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>, std::__1::default_delete<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo> > >::reset(DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>*) @ 0x276ce57c in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 41. /build/build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:269: std::__1::unique_ptr<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>, std::__1::default_delete<DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo> > >::~unique_ptr() @ 0x276ce399 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 42. /build/build_docker/../src/Storages/MergeTree/MutatePlainMergeTreeTask.h:22: DB::MutatePlainMergeTreeTask::~MutatePlainMergeTreeTask() @ 0x27defceb in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 43. /build/build_docker/../contrib/libcxx/include/__memory/construct_at.h:57: void std::__1::__destroy_at<DB::MutatePlainMergeTreeTask, 0>(DB::MutatePlainMergeTreeTask*) @ 0x27dd69c1 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: 44. /build/build_docker/../contrib/libcxx/include/__memory/construct_at.h:82: void std::__1::destroy_at<DB::MutatePlainMergeTreeTask, 0>(DB::MutatePlainMergeTreeTask*) @ 0x27dd6955 in /usr/bin/clickhouse
    [ 23430 ] {} <Fatal> BaseDaemon: Integrity check of the executable skipped because the reference checksum could not be read. (calculated checksum: 91F5937571C11255DFE73230B52CE9C0)
    [ 602 ] {} <Fatal> Application: Child process was terminated by signal 6.

</details>

  [2]: https://s3.amazonaws.com/clickhouse-test-reports/39222/a068c397dfd7943359a8b554566c3c70b78baf8d/stateless_tests__debug__actions__%5B1/3%5D.html

Refs: https://github.com/ClickHouse/ClickHouse/pull/29614#discussion_r720455032 (cc @nikitamikhaylov)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-18 19:01:41 +03:00
kssenii
0aed5a602a Merge master 2022-07-18 16:16:28 +02:00
Anton Popov
5605b60c97
Merge pull request #39039 from CurtizJ/randomize-more-settings 
Add more settings for randomization
 2022-07-18 15:05:41 +02:00
kssenii
a6655bea1c Some fixes 2022-07-18 14:23:08 +02:00
Robert Schulze
1ced1d11fd
Merge pull request #39285 from azat/bg-exe 
[RFC] Remove superior atomic from MergeTreeBackgroundExecutor and annotations for TSA
 2022-07-18 12:46:54 +02:00
Alexey Milovidov
13a8f85ed0
Merge branch 'master' into nullable-partition-fix1 2022-07-18 00:34:58 +03:00
Robert Schulze
13482af4ee
First try at reducing the use of StringRef 
- to be replaced by std::string_view
- suggested in #39262
 2022-07-17 17:26:02 +00:00
kssenii
8ad26b3111 Merge master 2022-07-16 22:50:03 +02:00
Kseniia Sumarokova
3f43641b18
Update MergedColumnOnlyOutputStream.cpp 2022-07-16 16:32:33 +02:00
Azat Khuzhin
accb0e38da Remove superior atomic from MergeTreeBackgroundExecutor and annotations for TSA 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-16 14:00:39 +03:00
alesapin
5cf4bd50de
Merge pull request #39214 from azat/s3-fix-fetching-memory-part 
Fix fetch of in-memory part with allow_remote_fs_zero_copy_replication
 2022-07-15 17:52:05 +02:00
Alexander Tokmakov
5ef7f5f494
Merge pull request #39229 from azat/fix-noexcept-scope 
[RFC] Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception)
 2022-07-15 18:14:56 +03:00
Raúl Marín
d68b3dfd43 Fix destructor order 2022-07-15 15:48:35 +02:00
Raúl Marín
aea045f297 Improve logging around replicated merges 2022-07-15 15:48:35 +02:00
Anton Popov
76eec62469
Merge branch 'master' into randomize-more-settings 2022-07-15 14:12:49 +02:00
Kruglov Pavel
a944a92d4c
Merge pull request #39224 from Avogar/string-view-by-value 
Pass const std::string_view by value, not by reference
 2022-07-15 13:05:56 +02:00
jianmei zhang
197d3796ec Killed mutation has empty commands may cause SegV 2022-07-15 15:37:36 +08:00
jianmei zhang
ca42f649da Rewrite logic for loading deleted mask related to getDeletedMask() 2022-07-15 15:31:10 +08:00
jianmei zhang
0793add9b5 Fix clang tidy compile error, remove std::move for const getDeletedRowsPtr() 2022-07-15 12:32:42 +08:00
jianmei zhang
8b1114fa60 fix clang tidy compile error 2022-07-15 12:32:42 +08:00
jianmei zhang
d37152a5d6 Remove loadDeletedMask() and get deleted mask when needed 2022-07-15 12:32:42 +08:00
jianmei zhang
4268ac8b0e Add supportLightweightDeleteMutate() in IMergeTreeDataPart to disable LWD for part with projections 2022-07-15 12:32:42 +08:00
jianmei zhang
9d1dff515c code changes for strange name and adding exception for mutation entry 2022-07-15 12:32:42 +08:00
jianmei zhang
780cdfb8f0 Code changes based on review opinions. Make ordinary single alter delete lightwight by default 2022-07-15 12:32:42 +08:00
jianmei zhang
6e6f77ef8a Fix compile error in clang tidy build 2022-07-15 12:32:41 +08:00
jianmei zhang
7e433859ea Change deleted rows mask from String to Native UInt8 format 2022-07-15 12:32:41 +08:00
jianmei zhang
dcc7367ac4 Fix code style error 2022-07-15 12:32:41 +08:00
jianmei zhang
6248486790 Fix lightweight delete bugs: add skip files and use source part's columns to avoid metadata updated cases 2022-07-15 12:32:41 +08:00
jianmei zhang
8ad2bb7c33 Code changes due to master new fixes, and update reference for mutations table 2022-07-15 12:32:41 +08:00
jianmei zhang
11fdea6e4b Add missing code for deleted_mask_filter_holder 2022-07-15 12:32:41 +08:00
jianmei zhang
9d27af7ee2 For some columns mutations, skip to apply deleted mask when read some columns. Also add unit test case 2022-07-15 12:32:41 +08:00
jianmei zhang
2c74e9b866 Merge support to apply lightweight delete mask 2022-07-15 12:32:41 +08:00
jianmei zhang
b4a37e1e22 Disable optimizations for count() when lightweight delete exists, add hasLightweightDelete() function in IMergeTreeDataPart 2022-07-15 12:32:41 +08:00
jianmei zhang
8696319d62 Support lightweight delete execution using string as deleted rows mask,also part of select can handle LWD 2022-07-15 12:32:41 +08:00
jianmei zhang
8df7b7a030 Support new mutation type for lightweight 2022-07-15 12:32:41 +08:00
Alexander Tokmakov
435d98738d
Update ReplicatedMergeTreeQueue.cpp 2022-07-14 22:41:01 +03:00
avogar
9291d33080 Pass const std::string_view & by value, not by reference 2022-07-14 16:11:57 +00:00
Anton Popov
d4cb2184c2
Merge branch 'master' into randomize-more-settings 2022-07-14 17:01:07 +02:00
Nikolai Kochetov
b24eeedf53
Merge branch 'master' into nullable-partition-fix1 2022-07-14 16:41:38 +02:00
Azat Khuzhin
b20407fab9 Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception) 
Current implementation of NOEXCEPT_SCOPE will not work, you cannot
rethrow exception outside the catch block, this will simply terminate
(via std::terminate) the program.
In other words NOEXCEPT_SCOPE macro will simply call std::terminate on
exception and will lost original exception.

But if NOEXCEPT_SCOPE will accept the code that should be runned w/o
exceptions, then it can catch exception and log it, rewrite it in this
way.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 17:16:18 +03:00
Amos Bird
96bb6e0cd2
Rename log when rename merge tree tables 2022-07-14 21:22:46 +08:00
FArthur-cmd
ad459bc6e4 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy-2 2022-07-14 14:10:44 +03:00
Azat Khuzhin
3e77b80b88 Fix fetch of in-memory part with allow_remote_fs_zero_copy_replication 
CI founds the following error during trying to fetch in-memory part [1]:

    2022.07.13 08:06:54.231033 [ 141093 ] {} <Error> InterserverIOHTTPHandler: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/disks/s3/store/886/88635b40-e4e3-4fe6-a0a0-1b6755463358/all_0_0_0/data.bin, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

<details>

<summary>stacktrace</summary>

    2022.07.13 08:06:54.101825 [ 189342 ] {8dbd11b3-f38a-4d5d-9ded-148987adb71d} <Debug> executeQuery: (from [::1]:54570) (comment: 01643_replicated_merge_tree_fsync_smoke.sql) select 'memory in_memory_parts_insert_sync'; (stage: Complete)
    2022.07.13 08:06:54.131309 [ 691 ] {} <Debug> test_26u6kx.rep_fsync_r2 (39c3823c-22e5-4c05-9dec-cdffd8872c40): Fetching part all_0_0_0 from /clickhouse/tables/test_26u6kx/rep_fsync/replicas/r1
    2022.07.13 08:06:54.231033 [ 141093 ] {} <Error> InterserverIOHTTPHandler: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/disks/s3/store/886/88635b40-e4e3-4fe6-a0a0-1b6755463358/all_0_0_0/data.bin, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):
    0. DB::Exception::Exception() @ 0xba0191a in /usr/bin/clickhouse
    1. DB::throwFromErrnoWithPath() @ 0xba029ca in /usr/bin/clickhouse
    2. DB::OpenedFile::open() const @ 0x156e7fb0 in /usr/bin/clickhouse
    3. DB::OpenedFile::getFD() const @ 0x156e8003 in /usr/bin/clickhouse
    4. DB::ReadBufferFromFilePReadWithDescriptorsCache::ReadBufferFromFilePReadWithDescriptorsCache() @ 0x156e5f23 in /usr/bin/clickhouse
    5. ? @ 0x156e53f0 in /usr/bin/clickhouse
    6. DB::createReadBufferFromFileBase() @ 0x156e52b5 in /usr/bin/clickhouse
    7. DB::DiskLocal::readFile() const @ 0x15e45ea8 in /usr/bin/clickhouse
    8. DB::MetadataStorageFromDisk::readFileToString() const @ 0x15e6ab8b in /usr/bin/clickhouse
    9. DB::MetadataStorageFromDisk::readMetadataUnlocked() const @ 0x15e6cdeb in /usr/bin/clickhouse
    10. DB::MetadataStorageFromDisk::getSerializedMetadata() const @ 0x15e6cfc4 in /usr/bin/clickhouse
    11. DB::DiskObjectStorage::getSerializedMetadata() const @ 0x15e19e2e in /usr/bin/clickhouse
    12. DB::DiskDecorator::getSerializedMetadata() const @ 0x15e54ed1 in /usr/bin/clickhouse
    13. DB::DiskDecorator::getSerializedMetadata() const @ 0x15e54ed1 in /usr/bin/clickhouse
    14. DB::DataPartsExchange::Service::sendPartFromDiskRemoteMeta() @ 0x1700bb9e in /usr/bin/clickhouse
    15. DB::DataPartsExchange::Service::processQuery(DB::HTMLForm const&, DB::ReadBuffer&, DB::WriteBuffer&, DB::HTTPServerResponse&) @ 0x1700a649 in /usr/bin/clickhouse
    16. DB::InterserverIOHTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::InterserverIOHTTPHandler::Output&) @ 0x17433c53 in /usr/bin/clickhouse
    17. DB::InterserverIOHTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x174344f1 in /usr/bin/clickhouse
    18. DB::HTTPServerConnection::run() @ 0x1768714d in /usr/bin/clickhouse
    19. Poco::Net::TCPServerConnection::start() @ 0x1a398093 in /usr/bin/clickhouse
    20. Poco::Net::TCPServerDispatcher::run() @ 0x1a399411 in /usr/bin/clickhouse
    21. Poco::PooledThread::run() @ 0x1a54b7bb in /usr/bin/clickhouse
    22. Poco::ThreadImpl::runnableEntry(void*) @ 0x1a548ec0 in /usr/bin/clickhouse
    23. ? @ 0x7fdf1c204609 in ?
    24. clone @ 0x7fdf1c129133 in ?
     (version 22.7.1.1781 (official build))

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/8b6e31cc615ca52c80724b6e5097777cb9514f07/stateless_tests__release__s3_storage__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 11:43:35 +03:00
Anton Popov
089009c996 fix reading from StorageLog with mmap 2022-07-13 20:36:18 +00:00
kssenii
615221f73b Minor changes 2022-07-13 19:22:47 +02:00
kssenii
0944183eec Fix merge 2022-07-13 16:50:31 +02:00
Azat Khuzhin
3720a3843c Add merge_algorithm to system.part_log 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 17:16:25 +03:00
Anton Popov
54a0985c42 randomize one more setting 2022-07-13 12:50:12 +00:00
FArthur-cmd
8ddc2578a2 refacroring 2022-07-13 14:48:02 +03:00
kssenii
0d2d3ac2b4 Merge master 2022-07-13 13:26:15 +02:00
Dmitry Novik
5f65b45269
Merge branch 'master' into group-by-use-nulls 2022-07-12 22:36:04 +02:00
Nikolai Kochetov
8efbe6d44d
Merge pull request #39119 from ClickHouse/removed-projection-and-old-part 
Properly remove projection from part in case it was removed from table metadata.
 2022-07-12 16:24:30 +02:00
Robert Schulze
69de9ee0e8
Merge pull request #39108 from ClickHouse/prefix_cmake_commands 
Prefix overridden add_executable() command with "clickhouse_"
 2022-07-12 15:05:48 +02:00
FArthur-cmd
6e2812490b refactoring 2022-07-12 14:29:11 +03:00
Alexander Tokmakov
c99fe6e6c8
Merge branch 'master' into removed-projection-and-old-part 2022-07-12 14:00:55 +03:00
FArthur-cmd
0e89fbff53 refactoring 2022-07-12 13:57:16 +03:00
Alexander Tokmakov
b35ad0dbdc
Merge pull request #39091 from ClickHouse/tavplubix-patch-1 
Follow-up to #39036
 2022-07-12 13:47:37 +03:00
Nikolai Kochetov
945b5d4b3f
Merge pull request #38993 from azat/fix-fsync_part_directory-on-fetch 
Fix fsync_part_directory for fetches
 2022-07-12 11:48:42 +02:00
Nikolai Kochetov
cd4211298e Fix typo. 2022-07-12 09:37:00 +00:00
Nikolai Kochetov
03d9301425 Fix log. 2022-07-12 08:31:09 +00:00
Nikolai Kochetov
631be9219b Properly remove projection from part in case it was removed from table metadata. 2022-07-12 08:28:22 +00:00
FArthur-cmd
783e6e398c merge with other working branch 2022-07-11 23:02:58 +03:00
FArthur-cmd
6ae227a6bb add test 2022-07-11 22:50:10 +03:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_" 
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
 2022-07-11 19:36:18 +02:00
FArthur-cmd
868628d3fd make stable prototype with arrays support 2022-07-11 18:02:50 +03:00
Alexander Tokmakov
e9ddbc247f
Merge pull request #39016 from Algunenano/clearOldLogs_zkerror 
clearOldLogs: Don't report KEEPER_EXCEPTION on concurrent deletes
 2022-07-11 15:16:18 +03:00
Alexander Tokmakov
048d27189e
Update MergeTreeData.cpp 2022-07-11 15:07:31 +03:00
FArthur-cmd
7b7a9521c7 change submodule path 2022-07-10 18:10:47 +03:00
FArthur-cmd
2f9f5f172a Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy_3 2022-07-10 17:25:21 +03:00
Nikolai Kochetov
b262ca7dd6 Refactoring 2022-07-08 22:04:56 +00:00
Alexander Tokmakov
51130d723c
Update MergeTreeData.cpp 2022-07-08 20:17:29 +03:00
Raúl Marín
a965438eea clearOldLogs: Don't report errors on concurrent deletes 2022-07-08 13:57:53 +02:00
Azat Khuzhin
b2f7f9c0f0 Fix fsync_part_directory for fetches 
Before it produces the following error:

    2022.07.07 23:40:03.131044 [ 44869 ] {} <Error> default.rep_data: auto DB::StorageReplicatedMergeTree::processQueueEntry(ReplicatedMergeTreeQueue::SelectedEntryPtr)::(anonymous class)::operator()(DB::StorageReplicatedMergeTree::LogEntryPtr &) const: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/tmp-fetch_36_25458_25458_0, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

    0. Poco::Exception::Exception() @ 0x1764634c in /usr/bin/clickhouse
    1. DB::Exception::Exception() @ 0xb42caba in /usr/bin/clickhouse
    2. DB::throwFromErrnoWithPath() @ 0xb42d79b in /usr/bin/clickhouse
    3. DB::LocalDirectorySyncGuard::LocalDirectorySyncGuard() @ 0x1404bbd4 in /usr/bin/clickhouse
    4. DB::DiskLocal::getDirectorySyncGuard() const @ 0x140466a4 in /usr/bin/clickhouse
    5. DB::DataPartsExchange::Fetcher::downloadPartToDisk() @ 0x15122af1 in /usr/bin/clickhouse
    6. DB::DataPartsExchange::Fetcher::fetchPart() @ 0x1511de39 in /usr/bin/clickhouse

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-08 13:48:51 +03:00
Dmitry Novik
d1df66687b
Merge branch 'master' into group-by-use-nulls 2022-07-07 20:54:38 +02:00
Raúl Marín
33ffb71453 Avoid 'Node exists' errors when creating Ephemeral Locks 2022-07-07 16:10:04 +02:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
FArthur-cmd
8968ff8995 small inmprovements 2022-07-07 11:33:06 +03:00
Amos Bird
19bd3d8d9f
Nullable partition pruning fix 2022-07-07 16:02:38 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9 
Backup Improvements 9
 2022-07-07 02:24:47 +02:00
FArthur-cmd
a3407ba6db Merge branch 'master' of github.com:FArthur-cmd/ClickHouse into annoy-2 2022-07-06 19:42:55 +03:00
Vitaly Baranov
7f84cf3968 Fix style. 2022-07-06 16:36:59 +02:00
Maksim Kita
b94489d52c
Merge pull request #38859 from kitaisreal/merge-tree-merge-disable-batch-optimization 
MergeTree merge disable batch optimization
 2022-07-06 15:59:40 +02:00
FArthur-cmd
c94d6a57ec add arrays 2022-07-06 14:39:49 +03:00
Raúl Marín
86d1c319f4 Remove unnecessary log 2022-07-06 12:23:45 +02:00
Maksim Kita
bdc21737d5 MergeTree merge disable batch optimization 2022-07-05 16:15:00 +02:00
Vitaly Baranov
43d35eec1b Write unfinished mutations to backup. 2022-07-05 14:51:09 +02:00
kssenii
b52084265c Merge master 2022-07-04 21:37:43 +02:00
Dmitry Novik
c66e2c8095
Merge pull request #37155 from amosbird/projection-fix-two 
Fix intermediate header in aggregate projection
 2022-07-04 16:32:34 +02:00
FArthur-cmd
21df8ed12f refactoring 2022-07-04 17:21:50 +03:00
FArthur-cmd
21b99e7304 Merge branch 'master' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-07-04 14:54:21 +03:00
Vitaly Baranov
1dba3d9f9c Make temporary files in better places. 2022-07-04 11:02:58 +02:00
Vitaly Baranov
92e0ee0b6f More detailed error messages. 2022-07-03 14:20:19 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder 2022-07-02 15:39:44 +02:00
Dmitry Novik
81dd90893e Merge remote-tracking branch 'origin/master' into group-by-use-nulls 2022-07-01 16:24:05 +00:00
alesapin
0d7298f3a9 Fix typos 2022-07-01 15:26:27 +02:00
alesapin
57284e6a9d Fix possible deadlocks with MergeTreeData::Transaction 2022-07-01 15:16:32 +02:00
Vitaly Baranov
cadb496894
Merge pull request #38537 from vitlibar/backup-improvements-8 
Backup Improvements 8
 2022-07-01 11:20:46 +02:00
Amos Bird
d73f975849
Fix intermediate header in aggregate projection 2022-07-01 16:47:32 +08:00
alesapin
09faa1641e Some comments 2022-06-30 23:09:32 +02:00
alesapin
979565edf0 Merge branch 'master' into better_data_part_storage_builder 2022-06-30 22:54:42 +02:00
alesapin
eb5046ab26 Simplify everything 2022-06-30 22:51:27 +02:00
Alexander Tokmakov
d2cbdc7c53
Update ReplicatedMergeTreeQueue.cpp 2022-06-30 17:23:53 +03:00
Vitaly Baranov
e367d96964 Fix style. 2022-06-30 15:10:33 +02:00
alesapin
1363b478d4 Merge branch 'fix_disk_tx_write_op' into better_data_part_storage_builder 2022-06-30 14:38:32 +02:00