alexey-milovidov
dfae1efbbd
Merge pull request #17070 from fastio/master
...
Support multiple ZooKeeper clusters
2020-11-27 10:38:01 +03:00
Amos Bird
022ba2b0a9
Fix unmatched type comparison in KeyCondition
2020-11-26 16:15:50 +08:00
alesapin
960d077612
Better functions
2020-11-26 10:25:57 +03:00
alexey-milovidov
75a78e6c20
Update BackgroundJobsExecutor.h
2020-11-26 07:09:05 +03:00
Azat Khuzhin
0b47f4a9e9
Fix optimize_trivial_count_query with partition predicate
...
Consider the following example:
CREATE TABLE test(p DateTime, k int) ENGINE MergeTree PARTITION BY toDate(p) ORDER BY k;
INSERT INTO test VALUES ('2020-09-01 00:01:02', 1), ('2020-09-01 20:01:03', 2), ('2020-09-02 00:01:03', 3);
- SELECT count() FROM test WHERE toDate(p) >= '2020-09-01' AND p <= '2020-09-01 00:00:00'
In this case rpn will be (FUNCTION_IN_RANGE, FUNCTION_UNKNOWN (due to strict), FUNCTION_AND)
and for optimize_trivial_count_query we cannot use index if there is at least one FUNCTION_UNKNOWN.
since there is no post processing and return count() based on only the first predicate is wrong.
Before this patch FUNCTION_UNKNOWN was allowed for optimize_trivial_count_query, and the result was wrong.
And two examples above just to show the difference, the behaviour hadn't been changed with this patch:
- SELECT * FROM test WHERE toDate(p) >= '2020-09-01' AND p <= '2020-09-01 00:00:00'
In this case will be (FUNCTION_IN_RANGE, FUNCTION_IN_RANGE (due to non-strict), FUNCTION_AND)
so it will prune everything out and nothing will be read.
- SELECT * FROM test WHERE toDate(p) >= '2020-09-01' AND toUnixTimestamp(p)%5==0
In this case will be (FUNCTION_IN_RANGE, FUNCTION_UNKNOWN, FUNCTION_AND)
and all, two, partitions will be scanned, but due to filtering later none of rows will be matched.
2020-11-25 23:09:17 +03:00
tavplubix
6477251ea1
Update BackgroundJobsExecutor.h
2020-11-25 23:06:00 +03:00
Anton Popov
852a08eacb
allow to move conditions to prewhere with compact parts
2020-11-23 21:27:59 +03:00
Amos Bird
172b7e9ed1
global in set index.
2020-11-23 22:05:08 +08:00
tavplubix
5cc9cb01cd
Merge pull request #16751 from amosbird/globalcontext
...
Make global_context consistent.
2020-11-22 18:46:17 +03:00
alesapin
9c8b0da382
Merge pull request #16033 from nvartolomei/nv/parts-uuid
...
Add unique identifiers IMergeTreeDataPart structure
2020-11-22 16:13:19 +03:00
Pavel Kruglov
623dc2df7a
Update comment
2020-11-20 17:32:39 +03:00
Pavel Kruglov
ca3fe49a2a
Make setting global
2020-11-20 17:29:13 +03:00
Nicolae Vartolomei
7c8bc1c04e
Use JSON metadata in WAL
2020-11-20 13:49:17 +00:00
Nicolae Vartolomei
040aba9f85
Add uuid.txt to checksums for parts stored on disk
...
We are breaking backwards compatibility anyway (but agted by a setting)
2020-11-20 13:49:17 +00:00
Nicolae Vartolomei
94293ca3ce
Assign UUIDs to parts only when configured to do so
...
Avoid breaking backwards compatibility by default for now.
2020-11-20 13:49:17 +00:00
Pavel Kruglov
e5588b94aa
Fix style and tests
2020-11-20 13:38:53 +03:00
Amos Bird
1d9d586e20
Make global_context consistent.
2020-11-20 18:23:14 +08:00
Pavel Kruglov
625962fb59
Remove debug logging
2020-11-20 03:19:36 +03:00
Pavel Kruglov
3ee4972586
Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert
2020-11-20 01:23:51 +03:00
Pavel Kruglov
4c30857759
Minor change
2020-11-20 01:22:40 +03:00
alesapin
3f01096c86
Less verbose logging when fetch is impossible
2020-11-19 20:03:20 +03:00
alesapin
2623d35f68
Merge pull request #17120 from ClickHouse/fix_granularity_on_block_borders
...
Fix index granularity calculation on block borders
2020-11-19 18:36:07 +03:00
Nicolae Vartolomei
746f8e45f5
All new parts must have uuids
2020-11-19 13:18:03 +00:00
Nicolae Vartolomei
425dc4b11b
Add unique identifiers IMergeTreeDataPart structure
...
For now uuids are not generated at all, they are present only if the
part is updated manually (as you can see in the integration test).
The only place where they can be seen today by an end user is in
`system.parts` table. I was looking for hiding this column behind an
option but couldn't find an easy way to do that.
Likely this is also required for WAL, but need to think how not to break
compatibility.
Relates to #13574 , https://github.com/ClickHouse/ClickHouse/issues/13574
Next 1: In the upcoming PR the plan is to integrate de-duplication based on
these fingerprints in the query pipeline.
Next 2: We'll enable automatic generation of uuids and come up with a
way for conditionally sending uuids when processing distributed queries
only when part movement is in progress.
2020-11-19 13:14:25 +00:00
Peng Jian
a0683ce460
Support mulitple ZooKeeper clusters
2020-11-19 15:44:47 +08:00
alesapin
7080f424e2
Fix bug for skip indices and make code more complex
2020-11-18 15:04:13 +03:00
Pavel Kruglov
547ec19fb3
Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert
2020-11-18 12:01:59 +03:00
filimonov
258170b325
Update ReplicatedMergeTreeMergeStrategyPicker.cpp
2020-11-18 08:45:44 +01:00
Mikhail Filimonov
234c671e52
After CR fixes
2020-11-18 08:45:44 +01:00
Mikhail Filimonov
97fef77ed1
execute_merges_on_single_replica
2020-11-18 08:45:33 +01:00
Pavel Kruglov
53b8ff42c2
Add test and set setting to 0 in some other tests
2020-11-17 19:56:36 +03:00
alesapin
389b883534
Fix index granularity calculation on block borders
2020-11-17 12:37:13 +03:00
Anton Popov
de5ead0c40
add setting remove_empty_parts
2020-11-15 05:24:47 +03:00
alexey-milovidov
915d96fee0
Merge pull request #16955 from nikitamikhaylov/fix-typos
...
Fix typos
2020-11-14 17:17:37 +03:00
alexey-milovidov
8ef4bfc0b5
Merge pull request #16968 from amosbird/blocktypemismatch
...
Fix missing conversion of unmutated parts
2020-11-14 17:16:51 +03:00
Nikolai Kochetov
ed98b48845
Merge pull request #16618 from ClickHouse/actions-dag-f14
...
Actions dag 2
2020-11-13 18:34:51 +03:00
alesapin
b2a8b6135d
then
2020-11-13 16:24:14 +03:00
alesapin
b8e33bd35b
Add comments
2020-11-13 15:42:26 +03:00
Pavel Kruglov
264e9daf6f
Fix style
2020-11-13 11:16:30 +03:00
Pavel Kruglov
6a57c0a8cf
Move merge in MergeTreeDataWriter
2020-11-13 10:55:56 +03:00
alesapin
8d3858fc22
Merge pull request #16895 from CurtizJ/remove-empty-parts
...
Remove empty parts after they were pruned by TTL [2]
2020-11-13 10:00:18 +03:00
Amos Bird
fdf979d5ce
Fix missing conversion of unmutated parts
2020-11-13 11:45:50 +08:00
nikitamikhaylov
c0940f2ebd
done
2020-11-13 02:27:18 +03:00
Pavel Kruglov
8d5e0784d3
Add setting optimize_on_insert
2020-11-12 23:37:23 +03:00
alesapin
bbd85c3b2d
Merge pull request #16390 from ClickHouse/add_separate_pool_for_fetches
...
Add separate pool for fetches
2020-11-12 22:42:28 +03:00
Anton Popov
4cecd89e84
fix dropping of empty parts
2020-11-12 20:37:54 +03:00
Anton Popov
0aa35d53c7
Merge remote-tracking branch 'upstream/master' into HEAD
2020-11-12 14:31:58 +03:00
alesapin
2cac5e5d97
Merge pull request #16875 from CurtizJ/refactor-alter-partition
...
Avoid code duplication in alterPartition
2020-11-12 13:19:59 +03:00
alesapin
a8d466a645
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-12 12:51:02 +03:00
Nikolai Kochetov
46f70dd0de
Merge branch 'master' into actions-dag-f14
2020-11-12 11:54:44 +03:00
tavplubix
058aa8f85e
Merge pull request #16824 from ClickHouse/replace_stringstreams_with_buffers
...
Replace std::*stringstreams with DB::*Buffers
2020-11-12 01:11:44 +03:00
Anton Popov
0a6d93d929
Update MergeTreeData.h
2020-11-11 21:17:41 +03:00
Anton Popov
2497598a49
Remove empty parts after they were pruned by TTL
2020-11-11 19:19:43 +03:00
Anton Popov
c01870076d
Update MergeTreeData.h
2020-11-11 19:09:01 +03:00
Anton Popov
6f0dc08ba7
avoid code duplication in alterPartition
2020-11-11 16:34:07 +03:00
Nikolai Kochetov
1846bb3cac
Merge branch 'master' into actions-dag-f14
2020-11-11 13:08:57 +03:00
Pavel Kruglov
c648f62629
Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize_final_optimization
2020-11-10 22:58:21 +03:00
Alexander Tokmakov
b94cc5c4e5
remove more stringstreams
2020-11-10 21:22:26 +03:00
Nikolai Kochetov
c6575c9032
Update ExpressionActions constructur
2020-11-10 19:27:55 +03:00
Nikolai Kochetov
1db8e77371
Add comments. Update ActionsDAG::Index
2020-11-10 17:54:59 +03:00
Pavel Kruglov
9120189d8a
Add SelectPartsDecision enum class
2020-11-10 17:42:56 +03:00
Nikolai Kochetov
07fe3a6347
Fix build.
2020-11-10 15:14:05 +03:00
Nikolai Kochetov
195c941c4e
Merge branch 'master' into storage-read-query-plan
2020-11-10 15:02:22 +03:00
alesapin
2e357516a6
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-10 13:53:08 +03:00
Vladimir Chebotarev
059357d51e
ALTER UPDATE/DELETE ... IN PARTITION with partition pruning in ReplicatedMergeTree (#13403 )
...
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-11-10 13:23:46 +03:00
Nikolai Kochetov
363c1e05c0
Try fix tests.
2020-11-10 12:35:05 +03:00
Alexander Tokmakov
5cdfcfb307
remove other stringstreams
2020-11-09 22:12:44 +03:00
Nikolai Kochetov
e41b1ae52b
Empty commit.
2020-11-09 19:35:43 +03:00
Alexander Tokmakov
62ff00ee8b
use WriteBuffer in formatAST(...)
2020-11-09 19:05:40 +03:00
alesapin
ea7fe94aa9
Merge pull request #15511 from nvartolomei/nv/drop-detached-part-quorum-possible-solution
...
ALTER TABLE ... DROP|DETACH PART
2020-11-09 18:00:30 +03:00
alexey-milovidov
e47ed55d32
Merge pull request #14977 from achimbab/indexOf_bloomfilter
...
Fix indexOf() to use BloomFilter
2020-11-09 17:58:42 +03:00
Nikolai Kochetov
99cc9b1ec0
Fix build
2020-11-09 16:20:56 +03:00
Nikolai Kochetov
8c4db34f9d
Update after merge.
2020-11-09 14:58:11 +03:00
Nikolai Kochetov
6717c7a0af
Merge branch 'master' into actions-dag-f14
2020-11-09 14:57:48 +03:00
alesapin
e347f4b72f
Add some comments
2020-11-09 12:14:20 +03:00
alesapin
72046313d4
Merge branch 'master' into nvartolomei-drop-part
2020-11-09 10:02:02 +03:00
alesapin
4e846d2197
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-09 10:00:21 +03:00
alexey-milovidov
0e6ae4aff7
Merge pull request #16253 from amosbird/pf
...
Prune partition in verbatim way.
2020-11-08 18:58:02 +03:00
Alexey Milovidov
3f5839c83c
Add one more case
2020-11-08 16:45:57 +03:00
Alexey Milovidov
8e1a2fc9e4
Fix UB and overflow
2020-11-08 16:41:25 +03:00
alexey-milovidov
8264bd900b
Update MergeTreeIndexConditionBloomFilter.cpp
2020-11-08 16:17:03 +03:00
alexey-milovidov
85f7eadc4f
Update MergeTreeIndexConditionBloomFilter.cpp
2020-11-08 16:14:32 +03:00
alexey-milovidov
dfa35a5e6d
Update MergeTreeIndexConditionBloomFilter.cpp
2020-11-08 16:11:18 +03:00
Alexey Milovidov
cc012e2cc7
Merge branch 'master' into achimbab-indexOf_bloomfilter
2020-11-08 14:42:03 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream
...
Fix "server failed to start" error
2020-11-08 14:27:08 +03:00
Alexey Milovidov
5314185e25
Merge branch 'master' into azat-optimize_skip_unused_shards-optimization
2020-11-08 00:17:59 +03:00
Alexey Milovidov
fd84d16387
Fix "server failed to start" error
2020-11-07 03:14:53 +03:00
alexey-milovidov
7fb53b205c
Merge pull request #16637 from azat/mt-read_in_order-spread-fix
...
Fix spreading for ReadInOrderOptimizer with expression in ORDER BY
2020-11-06 17:36:03 +03:00
Nikolai Kochetov
c10f733587
Merge branch 'master' into storage-read-query-plan
2020-11-06 15:43:46 +03:00
Nikolai Kochetov
9aeb757da4
Merge branch 'master' into actions-dag-f14
2020-11-06 15:04:20 +03:00
Amos Bird
2b0085c106
Pruning is different from counting
2020-11-06 19:58:03 +08:00
Amos Bird
aa436a3cb1
Transform single point
2020-11-06 14:59:55 +08:00
Amos Bird
30bf5e6d26
Prune partition in verbatim way.
2020-11-06 09:56:13 +08:00
alexey-milovidov
ee3e289ec2
Merge pull request #15983 from ClickHouse/no_background_pool_no_more
...
Replace BackgroundProcessingPool with SchedulePool task and ThreadPool.
2020-11-05 21:38:22 +03:00
Anton Popov
bea0984d12
Merge pull request #16642 from kitaisreal/bloom-filter-index-added-big-int-and-uuid-support
...
BloomFilter index added big integers and UUID support
2020-11-05 15:26:08 +03:00
Maksim Kita
7dd749fc4c
BloomFilter index added big integers and UUID support
2020-11-04 23:41:10 +03:00
Alexey Milovidov
1bcf22d42f
Fix 'max_parallel_replicas' without sampling.
2020-11-04 18:59:14 +03:00
alesapin
20043e77bb
Merge branch 'master' into nvartolomei-drop-part
2020-11-04 10:22:23 +03:00
Azat Khuzhin
2389406c21
Fix spreading for ReadInOrderOptimizer with expression in ORDER BY
...
This will fix optimize_read_in_order/optimize_aggregation_in_order with
max_threads>0 and expression in ORDER BY
2020-11-04 07:07:26 +03:00
Nikolai Kochetov
54a9b80a11
Fix build
2020-11-03 22:30:58 +03:00
