thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
77,532 Commits 9,338 Branches 1,783 Tags 2.1 GiB
56b613068b
Commit Graph

2703 Commits

Author SHA1 Message Date
Azat Khuzhin
117e9e77c8 Apply max_untracked_memory/memory_profiler_{step,sample_probability} during mutate/merge 2021-10-03 17:39:07 +03:00
Maksim Kita
395119339f Merge branch 'master' into rename-common 2021-10-02 17:15:10 +03:00
alexey-milovidov
86d1ce63cf
Merge pull request #29649 from CurtizJ/merging-29376 
Merging #29376
 2021-10-02 12:02:42 +03:00
Alexey Milovidov
2ad5dbdfe8 Rename "common" to "base" 2021-10-02 11:30:38 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Anton Popov
bb92ff52d8
Update IMergeTreeDataPart.cpp 2021-10-02 00:50:19 +03:00
Anton Popov
d404efbf7e add test for freeze in-memory parts 2021-10-01 22:05:36 +03:00
Anton Popov
f0363c524d Merge branch 'master' of git://github.com/mo-avatar/ClickHouse into merging-29376 2021-10-01 20:29:33 +03:00
Maksim Kita
57666725f9
Merge pull request #29623 from kitaisreal/bloom-filter-indexes-map-support-fix-unsafe-identifier-cast 
BloomFilter indexes map data type support fix unsafe identifier cast
 2021-10-01 19:28:04 +03:00
Nikita Mikhaylov
b2f7b6fae8
Fix race between MergeTask and storage destruction (#29614) 2021-10-01 18:55:04 +03:00
vdimir
7444d64156
Fix for recursive tuples in MergeTreeWhereOptimizer 2021-10-01 17:01:03 +03:00
vdimir
23ce41e3bb
MergeTreeWhereOptimizer supports constant tuple in both sides of equals 2021-10-01 17:01:03 +03:00
vdimir
64ffacd9e0
MergeTreeWhereOptimizer::tryAnalyzeTuple, fix fuzz 2021-10-01 17:01:02 +03:00
vdimir
e326b12a37
Fix style in MergeTreeWhereOptimizer.cpp 2021-10-01 17:01:02 +03:00
凌涛
f660224a5f
add tests 2021-10-01 17:01:01 +03:00
凌涛
ea68944055
fixed index didn't work in tuple where 2021-10-01 17:01:01 +03:00
Maksim Kita
c793c8e7fa BloomFilter indexes map data type support fix unsafe identifier cast 2021-10-01 14:40:00 +03:00
Alexey Milovidov
4271f8c738 Merge remote-tracking branch 'origin/master' into yandex-to-clickhouse-in-configs 2021-10-01 01:33:58 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
Nikolai Kochetov
68f8b9d235 Update ColumnGathererStream 2021-09-29 20:45:01 +03:00
mergify[bot]
6a2ac6e871
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-29 13:29:08 +00:00
Maksim Kita
0279afa738
Merge pull request #29474 from kitaisreal/bloom-filter-indexes-updated-tests 
Bloom filter indexes updated tests
 2021-09-28 22:40:27 +03:00
Maksim Kita
9ecd0a3f4f
Merge pull request #29220 from ClickHouse/full-text-bloom-filter-index-map-data-type 
Merging #28511
 2021-09-28 22:40:17 +03:00
alexey-milovidov
1e6f9ac635
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-28 20:26:44 +03:00
mo-avatar
6c16ec83eb in memory part lost when freeze 2021-09-28 22:52:01 +08:00
Maksim Kita
f2402e1a10 Fixed tests 2021-09-28 12:52:07 +03:00
Maksim Kita
71fb9e6059 Bloom filter indexes updated tests 2021-09-28 12:38:27 +03:00
Azat Khuzhin
9a23b20965 Remove debug from MergeTreeIndexConditionBloomFilter::traverseASTEquals() 2021-09-28 10:28:28 +03:00
alexey-milovidov
995913e89c
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-28 02:27:36 +03:00
Maksim Kita
7861c26eaf
Merge pull request #29431 from kitaisreal/map-bloom-filter-index-map-values-equals-support 
Map bloom filter index mapValues equals function support
 2021-09-27 22:03:53 +03:00
Nikolai Kochetov
236d71ea94
Merge pull request #28582 from ClickHouse/rewrite-pushing-to-views 
Rewrite PushingToViews
 2021-09-27 21:19:11 +03:00
Maksim Kita
e290f19957 Merge branch 'master' into full-text-bloom-filter-index-map-data-type 2021-09-27 17:50:07 +03:00
Maksim Kita
b28d6e76cc Map bloom filter full text index added support for mapKeys, mapValues 2021-09-27 17:26:25 +03:00
Maksim Kita
cd9c796e6f Map bloom filter index mapValues equals function support 2021-09-27 17:23:56 +03:00
Nikita Mikhaylov
3741b8a17e
Merge pull request #29337 from amosbird/projection-fix18 
Fix projection merges and mutations.
 2021-09-27 13:45:13 +03:00
Nikolai Kochetov
78a7665f43 Merge branch 'master' into rewrite-pushing-to-views 2021-09-27 10:56:50 +03:00
Amos Bird
34017c942e
More fixes 2021-09-25 13:38:56 +08:00
Alexey Milovidov
82d57c6e47 Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-25 01:38:39 +03:00
Amos Bird
23d3d894e6
Fix projection merges and mutations. 2021-09-24 22:45:50 +08:00
Nikolai Kochetov
e617aeb7a5
Merge pull request #29139 from amosbird/projection-fix17 
Disable projections when ARRAY JOIN is used.
 2021-09-24 11:15:23 +03:00
alexey-milovidov
b8083929a6
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-23 20:49:40 +03:00
Dmitrii Kovalkov
e3ae32f763 Allow nulls first 2021-09-23 19:39:15 +03:00
Nikolai Kochetov
998d29ebc7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-23 13:10:27 +03:00
Maksim Kita
3d79812427 Merge branch 'master' into full-text-bloom-filter-index-map-data-type 2021-09-23 12:38:03 +03:00
Maksim Kita
0341b99c69
Merge pull request #29280 from kitaisreal/full-text-bloom-filter-added-support-for-array-data-type 
FullText bloom filter index added support for Array data type
 2021-09-23 12:34:06 +03:00
Maksim Kita
d9e265df2c Fixed debug message 2021-09-23 12:13:01 +03:00
Maksim Kita
ebbefd49c3 FullText bloom filter index added support for Array data type 2021-09-23 02:06:54 +03:00
Maksim Kita
d5a7e92b97 Bloom filter map added support for has function 2021-09-23 01:10:14 +03:00
Maksim Kita
1bf375e2b7
Merge pull request #29254 from kitaisreal/map-bloom-filter-index-updated 
Updated bloom filter index for map type to work with mapKeys, mapValues functions
 2021-09-23 00:31:14 +03:00
Maksim Kita
93208fca4b Bloom filter for map data type dded support for IN, mapContains functions 2021-09-22 20:25:10 +03:00
Nikita Mikhaylov
558c74e826 Done 2021-09-22 11:52:47 +00:00
Maksim Kita
92fd6b376f Updated bloom filter index for map type to work with mapKeys function 2021-09-22 12:45:48 +03:00
Alexey Milovidov
e9e77b4403 .tech -> .com 2021-09-22 03:22:57 +03:00
Maksim Kita
84e37c2d87 Fixed tests 2021-09-21 21:28:52 +03:00
Maksim Kita
6dadc521fb MergeTreeIndexFullText updated for data type map 2021-09-21 16:55:48 +03:00
凌涛
e3e3b84f63 fix tests 2021-09-21 16:55:48 +03:00
凌涛
9461d6dbf2 modify code style 2021-09-21 16:55:47 +03:00
凌涛
e84f1000fb optimze code to make sure only map function will be processed 2021-09-21 16:55:47 +03:00
凌涛
a3e657778a bloom filter skipping index support Map with key of String type 2021-09-21 16:55:07 +03:00
凌涛
8f87d44958 bloom filter skipping index support Map with key of String type 2021-09-21 16:55:07 +03:00
Maksim Kita
15c9315ea1 MergeTreeIndexConditionBloomFilter updated for Map data type 2021-09-21 15:07:59 +03:00
fuwhu
8962672e59 add bloom filter skip index for map data type. 2021-09-21 10:59:16 +03:00
Anton Popov
55cf857aba
Merge pull request #28707 from azat/max_suspicious_broken_parts_bytes 
Implement max_suspicious_broken_parts_bytes
 2021-09-20 19:55:07 +03:00
Nikolai Kochetov
a790d391c0 Merge branch 'master' into rewrite-pushing-to-views 2021-09-20 19:43:15 +03:00
Anton Popov
e3e6258b95
Merge pull request #29163 from CurtizJ/small-refactoring 
Slightly better code near reading of subcolumns
 2021-09-19 16:14:52 +03:00
Anton Popov
f50098a059
fix style check 2021-09-19 00:13:00 +03:00
Anton Popov
1c80a95b36 slightly better code near reading of subcolumns 2021-09-18 22:34:54 +03:00
Amos Bird
6b3cd34cff
Disable projections when ARRAY JOIN is used. 2021-09-18 08:15:59 +08:00
Azat Khuzhin
b108c5e1b0 Implement max_suspicious_broken_parts_bytes 
Usually broken parts are found from just created parts, which are
usually small enough, so it make sense to have a limit in bytes for
broken parts.

Default value was set to 1G.
 2021-09-17 22:06:32 +03:00
Nikolai Kochetov
a8443bef4d Fix build. 2021-09-17 20:52:26 +03:00
Nicolae Vartolomei
9a02061d9c
Rewrite part movement between shards logic and add kill support 
See https://github.com/ClickHouse/ClickHouse/pull/24585#issuecomment-857735081
for extra context about the current implementation.
 2021-09-17 18:11:32 +01:00
Nicolae Vartolomei
0381c634d4
Add support for user defined identifier on log entries 
Sometimes we want to push a log entry once and only once. Because it is
not possible to create a sequential node in ZooKeeper and store its name
to a well known location in the same transaction we'll do it in the
other order. First somehow generate a unique identifier, then submit a
log entry with that identifier. Later, we can search through log entries
using the identifier we provided to find the node.

Required for part movement between shards.
 2021-09-17 15:32:35 +01:00
Nikolai Kochetov
a8c3b02598 Merge branch 'master' into rewrite-pushing-to-views 2021-09-17 15:38:11 +03:00
mergify[bot]
c23f1fc160
Merge branch 'master' into fix_part_checker 2021-09-17 07:23:12 +00:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask (#25165) 
Introduced MergeTask and MutateTask
 2021-09-17 00:19:58 +03:00
Nikolai Kochetov
341553febd Fix build. 2021-09-16 20:40:42 +03:00
alesapin
606ab6d3e4 Typo fix 2021-09-16 18:40:25 +03:00
alesapin
8930a0d2bd More precise initialization 2021-09-16 18:39:11 +03:00
alesapin
b4cbc86164 Fix bug 2021-09-16 17:59:55 +03:00
Anton Popov
f6191b98e7 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-15 17:47:22 +03:00
Kruglov Pavel
8a67c3cf44
Merge pull request #28636 from amosbird/nullable-index-fix 
Fix nullable/lowcardinality primary key with constant conversion
 2021-09-15 15:56:19 +03:00
Anton Popov
fc17936c12 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-14 23:02:30 +03:00
Nikolai Kochetov
b997214620 Rename QueryPipeline to QueryPipelineBuilder. 2021-09-14 20:48:18 +03:00
Nikolai Kochetov
7fc830ae2b Rename QueryPipeline to QueryPipelineBuilder. 2021-09-14 19:28:41 +03:00
mergify[bot]
4993c7c527
Merge branch 'master' into fix_part_checker 2021-09-14 14:13:52 +00:00
Nikolai Kochetov
0e267c50b4 Merge branch 'master' into rewrite-pushing-to-views 2021-09-14 16:13:54 +03:00
alesapin
9441378b39 Add logs to queue because I cannot understand anything without them 2021-09-14 12:04:11 +03:00
Kseniia Sumarokova
d3e604e14b
Merge pull request #28603 from kssenii/fix-web-disk-2 
Some fixes for static files disk
 2021-09-14 08:24:24 +03:00
mergify[bot]
c54434084f
Merge branch 'master' into fix_part_checker 2021-09-13 13:19:34 +00:00
alesapin
1d05fab295
Merge pull request #28864 from ClickHouse/fix_replace_range_drop_part_reordering 
Fix reordering of REPLACE_RANGE and DROP PART
 2021-09-13 15:11:29 +03:00
alesapin
fa8b698b7e Merge branch 'fix_part_checker' of github.com:ClickHouse/ClickHouse into fix_part_checker 2021-09-13 11:00:41 +03:00
alesapin
ab41384f63 Move queue initialization to restarting thread 2021-09-13 11:00:07 +03:00
alesapin
91e10adca5 Merge branch 'master' into fix_part_checker 2021-09-13 10:19:04 +03:00
kssenii
9ec6a9b4c9 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2 2021-09-13 08:32:34 +03:00
alexey-milovidov
cdcfdbec7e
Merge pull request #28873 from CurtizJ/fix-subcolumns-compact-parts 
Fix reading of subcolumns from compact parts
 2021-09-12 15:54:07 +03:00
alexey-milovidov
ea13a8b562
Merge pull request #28659 from myrrc/improvement/tostring_to_magic_enum 
Improving CH type system with concepts
 2021-09-12 15:26:29 +03:00
mergify[bot]
267e56df8c
Merge branch 'master' into fix-web-disk-2 2021-09-11 15:45:43 +00:00
Anton Popov
8999a83320 add comment 2021-09-10 20:22:28 +03:00
Anton Popov
a63887413c fix reading of subcolumns from compact parts 2021-09-10 20:19:18 +03:00
Anton Popov
0bb74f8eaf
Merge pull request #28762 from CurtizJ/fix-nested-1 
Fix usage of nested columns with non-array columns with the same prefix [2]
 2021-09-10 17:34:43 +03:00
Alexander Tokmakov
518f542c3e fix reordering of REPLACE_RANGE and DROP PART 2021-09-10 17:32:45 +03:00
alesapin
d1dea0e95f
Update src/Storages/MergeTree/ReplicatedMergeTreePartCheckThread.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2021-09-10 15:32:24 +03:00
Anton Popov
5cff615eca Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-10 13:40:48 +03:00
alexey-milovidov
2236a5df98
Merge pull request #28085 from myrrc/improvement/merge-tree-part-opt-parse 
optional<> semantics for parsing MergeTreePartInfo and DetachedPartInfo
 2021-09-10 00:04:03 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
alesapin
59edf6b5f6 Better logic on part checker 2021-09-09 18:19:12 +03:00
kssenii
0c013ed91b Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2 2021-09-09 15:26:36 +03:00
Nikolai Kochetov
13eb93a9c0
Merge pull request #28700 from amosbird/projection-fix16 
Fix crash on exception with projection aggregate
 2021-09-09 15:24:05 +03:00
Anton Popov
8e5b3b2f6c Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-09 15:12:34 +03:00
Nikolai Kochetov
abfb1d20ea
Merge pull request #25734 from vdimir/join-arrayjoin-type-cast 
Move join type inference to ExpressionAnalyzer from syntax stage + small refactor
 2021-09-09 14:38:43 +03:00
Anton Popov
2ccdb6c22d fix clang-tidy 2021-09-09 12:40:26 +03:00
Nikita Mikhaylov
4db5062d6b
Merge pull request #28374 from nikitamikhaylov/global-merge-executor 
Introduced global executor for background MergeTree-related operations
 2021-09-09 11:30:21 +03:00
alesapin
407c0bbb56
Merge pull request #28734 from ClickHouse/fix_race_on_mutation_pointer 
Fix race on mutation_pointer update
 2021-09-09 10:36:09 +03:00
Amos Bird
91293c7449
Fix crash on exception with projection aggregate 2021-09-09 10:43:56 +08:00
Nikita Mikhaylov
f12504b002 Fix stupid race 2021-09-08 20:30:54 +00:00
Anton Popov
9816014f6c fix nested and scalar columns with dot, which is missed in part 2021-09-08 21:42:34 +03:00
kssenii
6108a4139c Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2 2021-09-08 20:54:07 +03:00
Nikita Mikhaylov
1416b1ccba
Update MergeTreeBackgroundExecutor.cpp 2021-09-08 20:21:25 +03:00
Nikita Mikhaylov
dae9a01b4b Fix build 2021-09-08 16:13:32 +00:00
Nikita Mikhaylov
3b7fa06540 Review fixes 2021-09-08 12:42:03 +00:00
alesapin
f8a6dfb0c4 Fix race on mutation_pointer update 2021-09-08 14:37:16 +03:00
Nikita Mikhaylov
ed37a22c37 Delete debug print 2021-09-08 09:35:18 +00:00
Nikita Mikhaylov
6062dd0021 Better 2021-09-08 00:21:21 +00:00
Azat Khuzhin
db0767a194 Implement detach_not_byte_identical_parts 
Maybe useful for further analysis of non byte-identical parts.
 2021-09-07 23:29:57 +03:00
Nikita Mikhaylov
231f64cf37 Fix unit test 2021-09-07 13:51:30 +00:00
Nikita Mikhaylov
9b38248e64 Remove scheduler thread 2021-09-07 13:13:23 +00:00
Nikita Mikhaylov
8a0765c60d Review fixes (partially) 2021-09-07 12:45:39 +00:00
mergify[bot]
e9fbdec13a
Merge branch 'master' into improvement/merge-tree-part-opt-parse 2021-09-07 09:52:24 +00:00
Nikita Mikhaylov
cfe2207be3 Better 2021-09-06 22:07:41 +00:00
Nikita Mikhaylov
324ab97f50 Fix gcc build 2021-09-06 16:44:46 +00:00
Nikita Mikhaylov
7c39f84a5e Merge upstream/master into global-merge-executor (using imerge) 2021-09-06 15:37:18 +00:00
Mike Kot
8e9aacadd1 Initial: replacing hardcoded toString for enums with magic_enum 2021-09-06 16:24:03 +02:00
Nikita Mikhaylov
17d65b3d7b Allow allocations 2021-09-06 14:04:37 +00:00
Nikita Mikhaylov
ea0fbf81af Renaming 2021-09-06 12:01:16 +00:00
Nikita Mikhaylov
0249015515 added priority queue 2021-09-06 11:37:51 +00:00
Nikolai Kochetov
2178ed21df
Merge pull request #28560 from amosbird/projection-fix15 
Fix wrong header of minmax_count projection
 2021-09-06 13:00:44 +03:00
mergify[bot]
8eba5f29a1
Merge branch 'master' into improvement/merge-tree-part-opt-parse 2021-09-06 09:53:46 +00:00
Amos Bird
8fac97692a
Fix nullable primary key with constant conversion 2021-09-06 15:11:37 +08:00
Nikita Mikhaylov
3eec8a3c2f Use boost::circular_buffer 2021-09-03 22:15:20 +00:00
kssenii
6ee94c923f Fix 2021-09-04 00:51:11 +03:00
Anton Popov
59128412ea Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-03 22:17:11 +03:00
Nikolai Kochetov
66a76ab70f Rewrite PushingToViewsBlockOutputStream part 6 2021-09-03 20:29:36 +03:00
Kruglov Pavel
f559c34113
Merge pull request #28302 from amosbird/binaryconstantwrap 
Always monotonic for non-zero division
 2021-09-03 20:02:54 +03:00
Nikita Mikhaylov
26ab6ebc0a finalize in destructor 2021-09-03 14:20:34 +00:00
Nikita Mikhaylov
b867eabe73 Use Poco::Event 2021-09-03 13:02:49 +00:00
alesapin
dd7493d871
Merge pull request #27038 from excitoon-favorites/altermaterializecolumn 
`ALTER TABLE ... MATERIALIZE COLUMN`
 2021-09-03 15:50:52 +03:00
Nikita Mikhaylov
218f0adca7 Merge upstream/master into global-merge-executor (using imerge) 2021-09-03 12:29:48 +00:00
Nikita Mikhaylov
ca7adb76b9 Owning future 2021-09-03 12:27:49 +00:00
Mike Kot
1f148719ce Review fixes 2021-09-03 13:33:40 +02:00
Amos Bird
bb87d01579
Fix wrong header of minmax_count projection 2021-09-03 18:49:01 +08:00
Nikita Mikhaylov
d777c6cd70 Better [2] 2021-09-03 00:51:12 +00:00
Nikita Mikhaylov
60765b8c6e Better 2021-09-03 00:45:19 +00:00
Nikita Mikhaylov
292a24abe8 Merge upstream/master into global-merge-executor (using imerge) 2021-09-03 00:34:24 +00:00
Nikita Mikhaylov
9257188512 Comments 2021-09-03 00:28:29 +00:00
Nikita Mikhaylov
7baad6adde Fix build 2021-09-02 22:56:03 +00:00
Nikita Mikhaylov
cc7c221fad Own PriorityQueue + prettifying the code 2021-09-02 21:31:32 +00:00
Nikita Mikhaylov
6624fa12ba deleted set 2021-09-02 18:49:37 +00:00
Nikita Mikhaylov
a6fe91ca47 get rid of half of allocations 2021-09-02 17:40:29 +00:00
alesapin
464b8167d1 Update comment 2021-09-02 18:29:26 +03:00
alesapin
ba78a737dc Fix detach/attach for ReplicatedVersionedCollapsingMergeTree after alter 2021-09-02 17:47:00 +03:00
alesapin
b592400e54
Merge pull request #28404 from ClickHouse/remove_outdated_settings 
Remove obsolete settings for replicated fetches.
 2021-09-02 14:03:12 +03:00
Nikita Mikhaylov
7f21cd7f3d save 2021-09-02 10:39:27 +00:00
Nikolai Kochetov
5f0b1bc4e3
Merge pull request #26286 from amosbird/projection-improve2 
Virtual projection for min max indices.
 2021-09-02 13:03:30 +03:00
Anton Popov
5e421ab272 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-02 02:42:50 +03:00
mergify[bot]
5d299fbdee
Merge branch 'master' into remove_outdated_settings 2021-09-01 14:07:48 +00:00
Nikita Mikhaylov
c7727f093d Merge upstream/master into global-merge-executor (using imerge) 2021-08-31 23:34:11 +00:00
Nikita Mikhaylov
ac5316ce98 Stress test + reworker executor 2021-08-31 23:20:23 +00:00
alexey-milovidov
4cc0b0298c
Merge pull request #28269 from amosbird/fixweirdcode 
Better nullable primary key implementation
 2021-09-01 00:48:45 +03:00
Nikita Mikhaylov
28eff0e369 save 2021-08-31 18:07:24 +00:00
alesapin
2c7a51d816
Merge pull request #28366 from azat/remove-Temporary-part-fix-v2 
Fix removing of parts in a Temporary state (v2 followup)
 2021-08-31 19:13:15 +03:00
Nikita Mikhaylov
dbc950caa4 added a test 2021-08-31 14:54:24 +00:00
Nikolai Kochetov
5be05c2ef9 Rewrite PushingToViewsBlockOutputStream part 3. 2021-08-31 16:50:07 +03:00
Mike Kot
4194ed2ec7 Merge remote-tracking branch 'upstream/master' into improvement/merge-tree-part-opt-parse 2021-08-31 14:35:14 +02:00
Nikita Mikhaylov
e39f9aabde delete useless file 2021-08-31 12:29:51 +00:00
alesapin
9a3bbf44e3 Better description 2021-08-31 15:28:17 +03:00
vdimir
e8e26463bf
Change signature of ASTSelectQuery::arrayJoinExpressionList 2021-08-31 15:27:44 +03:00
alesapin
921e51e061 Remove some obsolete settings for replicated fetches 2021-08-31 15:22:56 +03:00
Nikita Mikhaylov
3027efe860 after merge with master 2021-08-31 12:09:35 +00:00
Nikita Mikhaylov
f8d4f04294 Merge upstream/master into global-merge-executor (using imerge) 2021-08-31 11:52:11 +00:00
Nikita Mikhaylov
1adb9bfe23 better 2021-08-31 11:02:39 +00:00
alesapin
525999145e
Merge pull request #26791 from ClickHouse/async-reads 
Experiment with asynchronous readers
 2021-08-31 13:17:45 +03:00
Nikolai Kochetov
8143c2e2b8
Merge pull request #28353 from ClickHouse/another-try-to-fix-BackgroundPoolTask-decrement 
Another try to fix BackgroundPoolTask decrement.
 2021-08-31 11:34:26 +03:00
Alexey Milovidov
ca6f6306f3 Merge branch 'master' into async-reads 2021-08-31 02:52:58 +03:00
Nikita Mikhaylov
c4416906c8 done 2021-08-30 19:37:03 +00:00
Azat Khuzhin
2c0390f326 Fix removing of parts in a Temporary state (v2 followup) 
CI reports [1]:

    <Fatal> : Logical error: 'Deleting data part all_0_0_0_3 doesn't exist'.
    ...
    <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeData.cpp:1254: DB::MergeTreeData::removePartsFinally() @ 0x1f94dffa in /usr/bin/clickhouse

  [1]: https://clickhouse-test-reports.s3.yandex.net/0/855a53ff8160c4638fe345b0d26e062804ba790a/stress_test_(debug).html#fail1

Follow-up for: #28221
 2021-08-30 21:07:26 +03:00
Azat Khuzhin
0828548b88 Fix intersecting parts due to new part had been replaced with an empty part 
AFAICS the problem is that some parts may be replaced with empty parts
(after #25820), and removed by the cleanup thread, due to it is empty
[1] (while it should not be deleted since it can download source part):

    <details>

    ```
    2021.08.18 20:11:22.687933 [ 341 ] {} <Trace> test_dpefxp.alter_table_1 (0758ca24-90e7-452c-8758-ca2490e7252c): Created log entry for mutation -1_115_115_0_146
    ...
    2021.08.18 20:11:22.707609 [ 22825 ] {} <Trace> test_dpefxp.alter_table_6 (766ae414-e113-4965-b66a-e414e1137965): Executing log entry to mutate part -1_115_115_0 to -1_115_115_0_146
    2021.08.18 20:11:22.707643 [ 22825 ] {} <Debug> test_dpefxp.alter_table_6 (766ae414-e113-4965-b66a-e414e1137965): Source part -1_115_115_0 for -1_115_115_0_146 is not ready; will try to fetch it instead
    ...
    2021.08.18 20:11:22.709397 [ 333 ] {} <Trace> test_dpefxp.alter_table_6 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000001579 for part -1_115_115_0 because it is covered by part -1_115_115_0_146 that is currently executing.
    2021.08.18 20:11:22.718861 [ 22825 ] {} <Information> test_dpefxp.alter_table_6 (766ae414-e113-4965-b66a-e414e1137965): DB::Exception: No active replica has part -1_115_115_0_146 or covering part
    ...
    2021.08.18 20:11:27.936829 [ 295 ] {} <Information> test_dpefxp.alter_table_6 (766ae414-e113-4965-b66a-e414e1137965): Going to replace lost part -1_115_115_0_146 with empty part
    2021.08.18 20:11:27.957839 [ 295 ] {} <Information> test_dpefxp.alter_table_6 (766ae414-e113-4965-b66a-e414e1137965): Created empty part -1_115_115_0_146 instead of lost part
    ...
    2021.08.18 20:11:28.731635 [ 257 ] {} <Trace> test_dpefxp.alter_table_6 (ReplicatedMergeTreeCleanupThread): Cleared 190 old blocks from ZooKeeper
    ...
    2021.08.18 20:11:28.734507 [ 257 ] {} <Trace> test_dpefxp.alter_table_6 (766ae414-e113-4965-b66a-e414e1137965): Will try to insert a log entry to DROP_RANGE for part: -1_115_115_0_146
    2021.08.18 20:11:28.779373 [ 22837 ] {} <Debug> test_dpefxp.alter_table_6 (766ae414-e113-4965-b66a-e414e1137965): Removed 1 parts inside -1_115_115_0_146.
    ...
    2021.08.18 20:11:28.792600 [ 273 ] {} <Trace> test_dpefxp.alter_table_6 (766ae414-e113-4965-b66a-e414e1137965): Created log entry /clickhouse/tables/00993_system_parts_race_condition_drop_zookeeper_test_dpefxp/alter_table/log/log-0000003459 for merge -1_111_118_1
    ...
    2021.08.18 20:11:28.910988 [ 354 ] {} <Error> test_dpefxp.alter_table_7 (ReplicatedMergeTreeQueue): Code: 49. DB::Exception: Part -1_111_118_1 intersects next part -1_115_115_0_146. It is a bug. (LOGICAL_ERROR), Stack trace (when copying this message, always include the lines below):
    2021.08.18 20:11:31.282160 [ 305 ] {} <Error> test_dpefxp.alter_table_2 (ReplicatedMergeTreeQueue): Code: 49. DB::Exception: Part -1_111_118_1 intersects next part -1_115_115_0_146. It is a bug. (LOGICAL_ERROR), Stack trace (when copying this message, always include the lines below):
    ```

    </details>

  [1]: https://clickhouse-test-reports.s3.yandex.net/27752/59e3cb18f4e53c453951267b5599afeb664290d8/functional_stateless_tests_(release,_wide_parts_enabled).html
 2021-08-30 21:05:46 +03:00
Nikolai Kochetov
ab45c412d0 Another try to fix BackgroundPoolTask decrement. 2021-08-30 15:45:32 +03:00
alesapin
531079c452
Merge pull request #28221 from azat/remove-Temporary-part-fix 
Fix removing of parts in a Temporary state
 2021-08-30 12:39:08 +03:00
Vladimir Chebotarev
de5daa22ab Revert "MATERIALIZE COLUMN xxx FINAL" 
This reverts commit bf1430c1ec.
 2021-08-30 11:50:26 +03:00
Vladimir Chebotarev
0d6e0181d6 MATERIALIZE COLUMN xxx FINAL 2021-08-30 11:49:59 +03:00
Vladimir Chebotarev
ce0850007f 1. Draw some circles. 2021-08-30 11:49:26 +03:00
alexey-milovidov
77f26e8673
Merge pull request #28299 from amosbird/fixtmprowsource 
Use real tmp file instead of predefined “rows_sources" for vertical merges
 2021-08-29 21:49:59 +03:00
Alexey Milovidov
e4f3d97b52 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-29 21:42:14 +03:00
alexey-milovidov
3d3afbdb35
Merge pull request #28193 from amosbird/alterprimarykey2 
Reserve protocol number for ALTER PRIMARY KEY.
 2021-08-29 20:37:19 +03:00
Amos Bird
9e7ae5dd6d
Use real tmp file instead of predefined one 2021-08-29 16:54:15 +08:00
Amos Bird
4624bf70b0
Always monotonic for non-zero division 2021-08-28 23:33:18 +08:00
Amos Bird
f2374a6916
Better nullable primary key implementation. 2021-08-28 17:48:28 +08:00
Amos Bird
b68857d086
Simplify projection, add minmax_count projection. 2021-08-28 11:25:37 +08:00
Alexey Milovidov
79e0433ba7 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-28 01:19:16 +03:00
Amos Bird
ce96b2c35a
bump 2021-08-27 20:44:26 +08:00
tavplubix
703101fe4d
Merge pull request #27931 from ClickHouse/wait_for_all_replicas_timeouts 
Avoid too long waiting for inactive replicas
 2021-08-27 14:31:36 +03:00
Azat Khuzhin
19458a0c61 Fix removing of parts in a Temporary state 
Parts in a temporary state is not exists in data_parts_by_info, so do
not try to search there and throw LOGICAL_ERROR in case of failure:

<details>

```
    02:45:49.037546 [ 5890 ] {} <Error> test_iy9rta.concurrent_kill_4 (526aa7c8-db2a-4f0e-926a-a7c8db2a9f0e): Code: 40. DB::Exception: Part all_0_0_0_1 from r11 has different columns hash. (CHECKSUM_DOESNT_MATCH) (version 21.10.1.7910 (official build)). Data after mutation is not byte-identical to data on another replicas. We will download merged part from replica to force byte-identical result.
    02:45:49.049422 [ 5890 ] {} <Trace> test_iy9rta.concurrent_kill_4 (526aa7c8-db2a-4f0e-926a-a7c8db2a9f0e): Trying to immediately remove part all_0_0_0_1 (state Temporary)
    02:45:49.060210 [ 5890 ] {} <Fatal> : Logical error: 'Part all_0_0_0_1 doesn't exist'.
    02:47:01.572508 [ 29208 ] {} <Fatal> BaseDaemon: (version 21.10.1.7910 (official build), build id: 9309CECED9A0D32CBB00BB8EC77B91456679868E) (from thread 5890) (no query) Received signal Aborted (6)
    ...
    02:47:06.815000 [ 29208 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeData.cpp:2593: DB::MergeTreeData::tryRemovePartImmediately(std::__1::shared_ptr<DB::IMergeTreeDataPart const>&&) @ 0x1f959e0d in /usr/bin/clickhouse
    02:47:06.816309 [ 29209 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeData.cpp:2593: DB::MergeTreeData::tryRemovePartImmediately(std::__1::shared_ptr<DB::IMergeTreeDataPart const>&&) @ 0x1f959e0d in /usr/bin/clickhouse
    02:47:09.455665 [ 29208 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:1939: DB::StorageReplicatedMergeTree::tryExecutePartMutation(DB::ReplicatedMergeTreeLogEntry const&) @ 0x1f5f2bf6 in /usr/bin/clickhouse
    02:47:09.468738 [ 29209 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:1939: DB::StorageReplicatedMergeTree::tryExecutePartMutation(DB::ReplicatedMergeTreeLogEntry const&) @ 0x1f5f2bf6 in /usr/bin/clickhouse
    02:47:11.776857 [ 29208 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:1581: DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&) @ 0x1f5e484c in /usr/bin/clickhouse
    02:47:11.904232 [ 29209 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:1581: DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&) @ 0x1f5e484c in /usr/bin/clickhouse
    02:47:13.941811 [ 29208 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:3176: DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18::operator() shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) const @ 0x1f65faa5 in /usr/bin/clickhouse
    02:47:14.477158 [ 29209 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Storages/StorageReplicatedMergeTree.cpp:3176: DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18::operator() shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) const @ 0x1f65faa5 in /usr/bin/clickhouse
    02:47:16.475373 [ 29208 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676: decltype(std::__1::forward<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>))(std::__1::forward<std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&>(fp0))) std::__1::__invoke<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMEntry>&>(DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) @ 0x1f65fa32 in /usr/bin/clickhouse
    02:47:16.970325 [ 29209 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676: decltype(std::__1::forward<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>))(std::__1::forward<std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&>(fp0))) std::__1::__invoke<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMEntry>&>(DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) @ 0x1f65fa32 in /usr/bin/clickhouse
    02:47:18.979481 [ 29208 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:317: bool std::__1::__invoke_void_return_wrapper<bool>::__call<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplireeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&>(DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogE x1f65f9f2 in /usr/bin/clickhouse
    02:47:19.450807 [ 29209 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:317: bool std::__1::__invoke_void_return_wrapper<bool>::__call<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplireeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&>(DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::SelectedEntry>)::$_18&, std::__1::shared_ptr<DB::ReplicatedMergeTreeLogE x1f65f9f2 in /usr/bin/clickhouse
    02:47:21.055007 [ 29208 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608: std::__1::__function::__default_alloc_func<DB::StorageReplicatedMergeTree::processQueueEntry(std::__1::shared_ptr<DB::ReplicatedMergeTreeQueue::ry>)::$_18, bool (std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>::operator()(std::__1::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&) @ 0x1f65f9b0 in /usr/bin/clickhouse
    02:47:23.546946 [ 413 ] {} <Fatal> Application: Child process was terminated by signal 6.
```

</details>

CI: https://clickhouse-test-reports.s3.yandex.net/0/4a8b82232c11512232df3ecdf4ffaec287116ad5/stress_test_(debug).html#fail1
 2021-08-27 10:08:44 +03:00
Alexey Milovidov
46732e781f Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-27 01:57:48 +03:00
Mike Kot
1f3701d485 Disk empty name fix 2021-08-26 19:28:33 +02:00
Amos Bird
103e0abe1c
Reserve protocol number for ALTER PRIMARY KEY. 2021-08-26 22:08:58 +08:00
Amos Bird
0169fce78e
Projection bug fixes and refactoring. 2021-08-26 19:09:31 +08:00
Ilya Yatsishin
5553a064be
Merge pull request #28139 from lthaooo/fix_prewhere_total_size_of_queried_columns_bug 
Fix bug in MergeTreeWhereOptimizer
 2021-08-26 10:42:23 +03:00
Anton Popov
6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Alexey Milovidov
7dd0146223 Fix error 2021-08-26 01:56:08 +03:00
Alexey Milovidov
3cb9cf7242 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-26 01:55:46 +03:00
Maksim Kita
2403686bb9
Merge pull request #28135 from amosbird/projection-improve3 
Get rid of useless projection columns during merge
 2021-08-25 22:14:38 +03:00
Mike Kot
407af64280 Merge remote-tracking branch 'upstream/master' into improvement/merge-tree-part-opt-parse 2021-08-25 18:24:43 +02:00
lthaooo
eecd5b21c9 fix total_size_of_queried_columns bug 2021-08-25 23:02:13 +08:00
Amos Bird
a6b2a6ee36
Get rid of useless projection columns during merge 2021-08-25 20:54:27 +08:00
lthaooo
314ba1e682
Merge branch 'master' into materialize_ttl_recalculate_only 2021-08-25 10:56:24 +08:00
Alexey Milovidov
848c2ec23e Tuneable prefetch 2021-08-25 02:38:08 +03:00
Alexey Milovidov
fd27a2faef Remove imperfect code 2021-08-25 02:05:55 +03:00
Alexey Milovidov
7c1d0a3baf Progress on development 2021-08-25 01:24:47 +03:00
Alexey Milovidov
f7ca674b0c Progress on development 2021-08-25 01:07:06 +03:00
Alexey Milovidov
8f57216180 Progress on development 2021-08-25 00:45:58 +03:00
Alexey Milovidov
4d9ad3725d Merge branch 'master' into async-reads 2021-08-25 00:45:40 +03:00
Mike Kot
d621605954 Merge remote-tracking branch 'upstream/master' into improvement/merge-tree-part-opt-parse 2021-08-24 19:43:33 +02:00
Mike Kot
f85e2e027c Optional semantics for [Detached]MergeTreePartInfo 2021-08-24 19:43:09 +02:00
Vitaly Baranov
64f69462c6
Merge pull request #27921 from vitlibar/add-tests-for-field-init 
Allow implicit cast bool to Field
 2021-08-24 17:15:44 +03:00
mergify[bot]
ae22bab2c4
Merge branch 'master' into materialize_ttl_recalculate_only 2021-08-24 08:49:19 +00:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Anton Popov
0db8b524f0
Merge pull request #27984 from BraulioVM/optimize/has-all-bloom-filter 
Make `hasAll` filter condition leverage bloom filters
 2021-08-23 22:19:50 +03:00
Alexander Tokmakov
4a4a0b4823 fix 2021-08-23 16:32:07 +03:00
Alexander Tokmakov
cc9c2fd63b make code better 2021-08-23 15:57:50 +03:00
Yatsishin Ilya
17bb938541 fix style 2021-08-23 13:59:01 +03:00
Amos Bird
a2256b1307
Dedicated Mark/Uncompressed cache for skip indices 2021-08-23 15:27:31 +08:00
Vitaly Baranov
89166a655a Init SettingFieldBool with true or false, not with 1 or 0. 2021-08-23 09:47:36 +03:00
Vitaly Baranov
ff1fad8521 Use setSetting() with implicit conversion true/false -> Field. 2021-08-23 08:57:00 +03:00
Braulio Valdivielso
d48f6122e3 clean up implementation a bit 2021-08-22 10:06:30 +01:00
Braulio Valdivielso
e95563fb82 leverage bloom filter index for 'hasAll' 
Fixes #26151. For now, this is a draft implementation. There is some
cleanup that can be done, but the new test is passing
 2021-08-21 20:14:56 +01:00
Alexander Tokmakov
59eb3aa9a9 avoid too long waiting for inactive replicas 2021-08-20 15:59:57 +03:00
nvartolomei
c09c90125f
Merge branch 'master' into nv/last-queue-update-exception 2021-08-19 10:29:16 +01:00
Nikolai Kochetov
7dcff5c90e
Merge pull request #27742 from amosbird/projection-improvement3 
Improve projection analysis.
 2021-08-19 12:11:20 +03:00
alesapin
0aec151719
Merge pull request #27808 from ClickHouse/fix_replicas_may_diverge 
Fix a couple of bugs that may cause replicas to diverge
 2021-08-19 09:58:07 +03:00
Alexander Tokmakov
09ff66da0e fix a couple of bugs that may cause replicas to diverge 2021-08-18 12:50:46 +03:00
alesapin
5cc98c67a4 Fix 2021-08-18 11:56:18 +03:00
mergify[bot]
f11e396151
Merge branch 'master' into nv/last-queue-update-exception 2021-08-18 07:00:50 +00:00
Nikita Mikhaylov
cc8ad0a4e7
Merge pull request #27733 from Algunenano/i27620_take2 
Refactor arrayJoin check on partition expressions
 2021-08-17 15:57:16 +03:00
Nikolai Kochetov
98eb619b4b
Merge pull request #23367 from Avogar/short-circuit 
Implement short circuit function evaluation
 2021-08-17 12:36:07 +03:00
Amos Bird
ea7dc495cf
Better code. 2021-08-17 12:32:49 +08:00
Raúl Marín
90f224807a Style 2021-08-16 17:31:37 +02:00
Amos Bird
90881aab09
Better code style 2021-08-16 22:17:07 +08:00
Raúl Marín
3ec904a953 Only analyze if tuple has arguments 2021-08-16 14:47:41 +02:00
Amos Bird
b162a2b699
Improve projection analysis. 
Remove duplicate index analysis and avoid possible invalid limit checks
during projection analysis.
 2021-08-16 20:11:57 +08:00
Raúl Marín
34099bc196 Refactor arrayJoin check on partition expressions 2021-08-16 12:46:18 +02:00