alesapin
3f01096c86
Less verbose logging when fetch is impossible
2020-11-19 20:03:20 +03:00
alesapin
2623d35f68
Merge pull request #17120 from ClickHouse/fix_granularity_on_block_borders
...
Fix index granularity calculation on block borders
2020-11-19 18:36:07 +03:00
Nicolae Vartolomei
746f8e45f5
All new parts must have uuids
2020-11-19 13:18:03 +00:00
Nicolae Vartolomei
425dc4b11b
Add unique identifiers IMergeTreeDataPart structure
...
For now uuids are not generated at all, they are present only if the
part is updated manually (as you can see in the integration test).
The only place where they can be seen today by an end user is in
`system.parts` table. I was looking for hiding this column behind an
option but couldn't find an easy way to do that.
Likely this is also required for WAL, but need to think how not to break
compatibility.
Relates to #13574 , https://github.com/ClickHouse/ClickHouse/issues/13574
Next 1: In the upcoming PR the plan is to integrate de-duplication based on
these fingerprints in the query pipeline.
Next 2: We'll enable automatic generation of uuids and come up with a
way for conditionally sending uuids when processing distributed queries
only when part movement is in progress.
2020-11-19 13:14:25 +00:00
Peng Jian
a0683ce460
Support mulitple ZooKeeper clusters
2020-11-19 15:44:47 +08:00
alesapin
7080f424e2
Fix bug for skip indices and make code more complex
2020-11-18 15:04:13 +03:00
Pavel Kruglov
547ec19fb3
Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert
2020-11-18 12:01:59 +03:00
filimonov
258170b325
Update ReplicatedMergeTreeMergeStrategyPicker.cpp
2020-11-18 08:45:44 +01:00
Mikhail Filimonov
234c671e52
After CR fixes
2020-11-18 08:45:44 +01:00
Mikhail Filimonov
97fef77ed1
execute_merges_on_single_replica
2020-11-18 08:45:33 +01:00
Pavel Kruglov
53b8ff42c2
Add test and set setting to 0 in some other tests
2020-11-17 19:56:36 +03:00
alesapin
389b883534
Fix index granularity calculation on block borders
2020-11-17 12:37:13 +03:00
Anton Popov
de5ead0c40
add setting remove_empty_parts
2020-11-15 05:24:47 +03:00
alexey-milovidov
915d96fee0
Merge pull request #16955 from nikitamikhaylov/fix-typos
...
Fix typos
2020-11-14 17:17:37 +03:00
alexey-milovidov
8ef4bfc0b5
Merge pull request #16968 from amosbird/blocktypemismatch
...
Fix missing conversion of unmutated parts
2020-11-14 17:16:51 +03:00
Nikolai Kochetov
ed98b48845
Merge pull request #16618 from ClickHouse/actions-dag-f14
...
Actions dag 2
2020-11-13 18:34:51 +03:00
alesapin
b2a8b6135d
then
2020-11-13 16:24:14 +03:00
alesapin
b8e33bd35b
Add comments
2020-11-13 15:42:26 +03:00
Pavel Kruglov
264e9daf6f
Fix style
2020-11-13 11:16:30 +03:00
Pavel Kruglov
6a57c0a8cf
Move merge in MergeTreeDataWriter
2020-11-13 10:55:56 +03:00
alesapin
8d3858fc22
Merge pull request #16895 from CurtizJ/remove-empty-parts
...
Remove empty parts after they were pruned by TTL [2]
2020-11-13 10:00:18 +03:00
Amos Bird
fdf979d5ce
Fix missing conversion of unmutated parts
2020-11-13 11:45:50 +08:00
nikitamikhaylov
c0940f2ebd
done
2020-11-13 02:27:18 +03:00
Pavel Kruglov
8d5e0784d3
Add setting optimize_on_insert
2020-11-12 23:37:23 +03:00
alesapin
bbd85c3b2d
Merge pull request #16390 from ClickHouse/add_separate_pool_for_fetches
...
Add separate pool for fetches
2020-11-12 22:42:28 +03:00
Alexander Tokmakov
0731bfb7be
Merge branch 'master' into database_replicated
2020-11-12 21:04:18 +03:00
Anton Popov
4cecd89e84
fix dropping of empty parts
2020-11-12 20:37:54 +03:00
Anton Popov
0aa35d53c7
Merge remote-tracking branch 'upstream/master' into HEAD
2020-11-12 14:31:58 +03:00
alesapin
2cac5e5d97
Merge pull request #16875 from CurtizJ/refactor-alter-partition
...
Avoid code duplication in alterPartition
2020-11-12 13:19:59 +03:00
alesapin
a8d466a645
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-12 12:51:02 +03:00
Nikolai Kochetov
46f70dd0de
Merge branch 'master' into actions-dag-f14
2020-11-12 11:54:44 +03:00
tavplubix
058aa8f85e
Merge pull request #16824 from ClickHouse/replace_stringstreams_with_buffers
...
Replace std::*stringstreams with DB::*Buffers
2020-11-12 01:11:44 +03:00
Anton Popov
0a6d93d929
Update MergeTreeData.h
2020-11-11 21:17:41 +03:00
Anton Popov
2497598a49
Remove empty parts after they were pruned by TTL
2020-11-11 19:19:43 +03:00
Anton Popov
c01870076d
Update MergeTreeData.h
2020-11-11 19:09:01 +03:00
Anton Popov
6f0dc08ba7
avoid code duplication in alterPartition
2020-11-11 16:34:07 +03:00
Nikolai Kochetov
1846bb3cac
Merge branch 'master' into actions-dag-f14
2020-11-11 13:08:57 +03:00
Pavel Kruglov
c648f62629
Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize_final_optimization
2020-11-10 22:58:21 +03:00
Alexander Tokmakov
b94cc5c4e5
remove more stringstreams
2020-11-10 21:22:26 +03:00
Nikolai Kochetov
c6575c9032
Update ExpressionActions constructur
2020-11-10 19:27:55 +03:00
Nikolai Kochetov
1db8e77371
Add comments. Update ActionsDAG::Index
2020-11-10 17:54:59 +03:00
Pavel Kruglov
9120189d8a
Add SelectPartsDecision enum class
2020-11-10 17:42:56 +03:00
Anton Popov
9b354be66f
Merge remote-tracking branch 'upstream/master' into HEAD
2020-11-10 16:21:11 +03:00
Nikolai Kochetov
07fe3a6347
Fix build.
2020-11-10 15:14:05 +03:00
Nikolai Kochetov
195c941c4e
Merge branch 'master' into storage-read-query-plan
2020-11-10 15:02:22 +03:00
alesapin
2e357516a6
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-10 13:53:08 +03:00
Vladimir Chebotarev
059357d51e
ALTER UPDATE/DELETE ... IN PARTITION with partition pruning in ReplicatedMergeTree (#13403 )
...
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-11-10 13:23:46 +03:00
Nikolai Kochetov
363c1e05c0
Try fix tests.
2020-11-10 12:35:05 +03:00
Alexander Tokmakov
5cdfcfb307
remove other stringstreams
2020-11-09 22:12:44 +03:00
Nikolai Kochetov
e41b1ae52b
Empty commit.
2020-11-09 19:35:43 +03:00
Alexander Tokmakov
62ff00ee8b
use WriteBuffer in formatAST(...)
2020-11-09 19:05:40 +03:00
alesapin
ea7fe94aa9
Merge pull request #15511 from nvartolomei/nv/drop-detached-part-quorum-possible-solution
...
ALTER TABLE ... DROP|DETACH PART
2020-11-09 18:00:30 +03:00
alexey-milovidov
e47ed55d32
Merge pull request #14977 from achimbab/indexOf_bloomfilter
...
Fix indexOf() to use BloomFilter
2020-11-09 17:58:42 +03:00
Nikolai Kochetov
99cc9b1ec0
Fix build
2020-11-09 16:20:56 +03:00
Nikolai Kochetov
8c4db34f9d
Update after merge.
2020-11-09 14:58:11 +03:00
Nikolai Kochetov
6717c7a0af
Merge branch 'master' into actions-dag-f14
2020-11-09 14:57:48 +03:00
alesapin
e347f4b72f
Add some comments
2020-11-09 12:14:20 +03:00
alesapin
72046313d4
Merge branch 'master' into nvartolomei-drop-part
2020-11-09 10:02:02 +03:00
alesapin
4e846d2197
Merge branch 'master' into add_separate_pool_for_fetches
2020-11-09 10:00:21 +03:00
alexey-milovidov
0e6ae4aff7
Merge pull request #16253 from amosbird/pf
...
Prune partition in verbatim way.
2020-11-08 18:58:02 +03:00
Alexey Milovidov
3f5839c83c
Add one more case
2020-11-08 16:45:57 +03:00
Alexey Milovidov
8e1a2fc9e4
Fix UB and overflow
2020-11-08 16:41:25 +03:00
alexey-milovidov
8264bd900b
Update MergeTreeIndexConditionBloomFilter.cpp
2020-11-08 16:17:03 +03:00
alexey-milovidov
85f7eadc4f
Update MergeTreeIndexConditionBloomFilter.cpp
2020-11-08 16:14:32 +03:00
alexey-milovidov
dfa35a5e6d
Update MergeTreeIndexConditionBloomFilter.cpp
2020-11-08 16:11:18 +03:00
Alexey Milovidov
cc012e2cc7
Merge branch 'master' into achimbab-indexOf_bloomfilter
2020-11-08 14:42:03 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream
...
Fix "server failed to start" error
2020-11-08 14:27:08 +03:00
Alexey Milovidov
5314185e25
Merge branch 'master' into azat-optimize_skip_unused_shards-optimization
2020-11-08 00:17:59 +03:00
Alexey Milovidov
fd84d16387
Fix "server failed to start" error
2020-11-07 03:14:53 +03:00
Anton Popov
245c395a68
Merge remote-tracking branch 'upstream/master' into HEAD
2020-11-06 22:00:32 +03:00
Anton Popov
01a04e06ef
implement nested with multiple nesting
2020-11-06 21:57:04 +03:00
alexey-milovidov
7fb53b205c
Merge pull request #16637 from azat/mt-read_in_order-spread-fix
...
Fix spreading for ReadInOrderOptimizer with expression in ORDER BY
2020-11-06 17:36:03 +03:00
Nikolai Kochetov
c10f733587
Merge branch 'master' into storage-read-query-plan
2020-11-06 15:43:46 +03:00
Nikolai Kochetov
9aeb757da4
Merge branch 'master' into actions-dag-f14
2020-11-06 15:04:20 +03:00
Amos Bird
2b0085c106
Pruning is different from counting
2020-11-06 19:58:03 +08:00
Amos Bird
aa436a3cb1
Transform single point
2020-11-06 14:59:55 +08:00
Amos Bird
30bf5e6d26
Prune partition in verbatim way.
2020-11-06 09:56:13 +08:00
alexey-milovidov
ee3e289ec2
Merge pull request #15983 from ClickHouse/no_background_pool_no_more
...
Replace BackgroundProcessingPool with SchedulePool task and ThreadPool.
2020-11-05 21:38:22 +03:00
Anton Popov
bea0984d12
Merge pull request #16642 from kitaisreal/bloom-filter-index-added-big-int-and-uuid-support
...
BloomFilter index added big integers and UUID support
2020-11-05 15:26:08 +03:00
Maksim Kita
7dd749fc4c
BloomFilter index added big integers and UUID support
2020-11-04 23:41:10 +03:00
Alexey Milovidov
1bcf22d42f
Fix 'max_parallel_replicas' without sampling.
2020-11-04 18:59:14 +03:00
alesapin
20043e77bb
Merge branch 'master' into nvartolomei-drop-part
2020-11-04 10:22:23 +03:00
Azat Khuzhin
2389406c21
Fix spreading for ReadInOrderOptimizer with expression in ORDER BY
...
This will fix optimize_read_in_order/optimize_aggregation_in_order with
max_threads>0 and expression in ORDER BY
2020-11-04 07:07:26 +03:00
Nikolai Kochetov
54a9b80a11
Fix build
2020-11-03 22:30:58 +03:00
Nikolai Kochetov
4984fdc693
Fix build
2020-11-03 22:28:50 +03:00
Nikolai Kochetov
7b61f5d641
Refactor more
2020-11-03 22:05:47 +03:00
alesapin
faca9d9b91
Check max bytes on disk
2020-11-03 18:41:46 +03:00
Nikolai Kochetov
6767a226fc
Merge branch 'master' into actions-dag-f14
2020-11-03 15:21:06 +03:00
Nikolai Kochetov
07a7c46b89
Refactor ExpressionActions [Part 3]
2020-11-03 14:28:28 +03:00
alexey-milovidov
11fc6fd8c9
Merge pull request #16584 from ClickHouse/drop_table_task_better_scheduling
...
Better scheduling of drop table task
2020-11-03 14:22:51 +03:00
Anton Popov
85ab1e9bc3
Merge pull request #15938 from Avogar/select_final
...
Select final
2020-11-03 13:49:47 +03:00
Anton Ivashkin
1742fb3256
Merge master
2020-11-03 12:27:16 +03:00
Anton Ivashkin
78021714f1
S3 zero copy replication: more simple s3 check
2020-11-03 12:20:26 +03:00
alesapin
294489f6ad
Merge branch 'master' into nvartolomei-drop-part
2020-11-03 10:19:03 +03:00
Anton Popov
a3a8e18637
Merge branch 'master' into select_final
2020-11-03 00:00:43 +03:00
alesapin
dac6705995
DROP PART and clear blocks in a single zookeeper op
2020-11-02 20:30:59 +03:00
Alexander Tokmakov
ac32809b6a
fix #16482
2020-11-02 19:40:39 +03:00
alesapin
fbda80ac89
Simplier storage of last index column
2020-11-02 15:19:37 +03:00
alesapin
8b0d9fbc07
Merge branch 'no_background_pool_no_more' into add_separate_pool_for_fetches
2020-11-02 10:54:09 +03:00
alesapin
101aa9e931
Merge branch 'master' into no_background_pool_no_more
2020-11-02 10:49:19 +03:00
alesapin
ae53092536
Merge branch 'master' into system_fetches_table
2020-10-31 18:45:17 +03:00
tavplubix
db146ee615
Merge pull request #16555 from ClickHouse/yet_another_fix_for_00993
...
Better logging in replication queue, fix flacky test
2020-10-31 14:38:15 +03:00
tavplubix
ce6c44eb9f
Update ReplicatedMergeTreeQueue.cpp
2020-10-30 18:47:58 +03:00
Alexander Tokmakov
cf8202d36e
better logging, fix flacky test
2020-10-30 15:41:39 +03:00
alesapin
15ba4dd851
Review fix
2020-10-30 11:52:11 +03:00
alesapin
b34960bffa
Merge branch 'master' into system_fetches_table
2020-10-30 11:33:37 +03:00
Nikolai Kochetov
1c106691b5
Merge pull request #16423 from amosbird/jbodread
...
Balanced reading from JBOD
2020-10-29 19:22:45 +03:00
alesapin
0e07afb821
Merge branch 'no_background_pool_no_more' into add_separate_pool_for_fetches
2020-10-29 12:53:55 +03:00
alesapin
40fc512e79
Merge branch 'master' into no_background_pool_no_more
2020-10-29 12:53:34 +03:00
Nikolai Kochetov
671d2b7f1b
Update MergeTreeReadPool.cpp
...
Fix comment
2020-10-29 10:04:23 +03:00
alexey-milovidov
0faf2bc7e3
Merge pull request #16332 from ClickHouse/fix-queue-processing-of-very-large-entries
...
Fix processing of very large entries in queue
2020-10-29 09:09:29 +03:00
Nikolai Kochetov
10bad32fb4
Refactor code a little bit. Add comment.
2020-10-29 00:52:26 +03:00
Amos Bird
f995ef9797
Balanced reading from JBOD
2020-10-29 04:05:07 +08:00
alesapin
9830ff1003
Merge pull request #16094 from nvartolomei/nv/wal-metadata-compatibility
...
RFC: Add metadata section to MergeTreeWriteAheadLog
2020-10-28 15:54:46 +03:00
alesapin
8e8bdeb5d7
Merge pull request #16434 from ClickHouse/fix_fake_race_on_merges_list
...
Fix fake race condition on system.merges merge_algorithm
2020-10-28 15:51:53 +03:00
alesapin
0931658c2f
Merge branch 'master' into add_separate_pool_for_fetches
2020-10-28 14:20:56 +03:00
alesapin
3c31a5134e
Fix bug
2020-10-28 14:03:26 +03:00
alesapin
c10370f98d
Merge branch 'master' into fix-queue-processing-of-very-large-entries
2020-10-28 11:19:34 +03:00
alesapin
1c9e24b5e2
Merge branch 'master' into system_fetches_table
2020-10-28 10:20:04 +03:00
Alexander Kuzmenkov
6509f1eb39
Merge branch 'master' into minumum
2020-10-27 21:38:23 +03:00
alesapin
3ec5fc7a4c
Add explicit
2020-10-27 18:29:06 +03:00
alesapin
60f2d822d7
Fix fake race condition on system.merges merge_algorithm
2020-10-27 18:27:12 +03:00
alesapin
6253d1466a
Fix comments
2020-10-27 16:09:14 +03:00
alesapin
3d3d246bdc
Rename table
2020-10-27 16:07:50 +03:00
alesapin
e40c509392
More clear callback name
2020-10-27 16:00:40 +03:00
alesapin
f35edfb2a5
Remove some unused fields
2020-10-27 15:57:55 +03:00
alesapin
ee4e6caf70
Remove redundant metric increment
2020-10-27 15:52:49 +03:00
alesapin
3824fcb076
Rename file and some tweaks
2020-10-27 15:47:42 +03:00
alesapin
31d63efdac
Fix style
2020-10-27 15:26:10 +03:00
alesapin
dca35b1913
Add awesome test for fetch
2020-10-27 15:24:10 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell
2020-10-27 12:04:03 +01:00
Anton Popov
55d867a660
avoid double reading of subcolumns
2020-10-26 21:55:52 +03:00
alesapin
880f4bbd05
System fetches
2020-10-26 19:38:35 +03:00
alesapin
9ed4668dbb
Refactor common part of background list
2020-10-26 15:40:55 +03:00
Alexander Tokmakov
636da1318b
Merge branch 'master' into database_replicated
2020-10-26 15:00:43 +03:00
alesapin
47ebc9f74e
Add separate pool for fetches
2020-10-26 14:02:47 +03:00
alesapin
d5d0819698
Merge branch 'master' into no_background_pool_no_more
2020-10-26 12:58:51 +03:00
alexey-milovidov
3aa49204d6
Update MergeTreeSettings.cpp
2020-10-26 07:39:20 +03:00
Alexey Milovidov
d5e277f080
Better exceptions rethrow
2020-10-26 06:35:57 +03:00
Alexey Milovidov
22a89fca01
Fix processing very large entries in queue
2020-10-25 05:57:03 +03:00
Alexey Milovidov
a2d888f19f
Fix processing very large entries in queue
2020-10-25 05:57:03 +03:00
Anton Popov
a249f0c95e
Merge remote-tracking branch 'upstream/master' into HEAD
2020-10-23 22:05:00 +03:00
Anton Popov
600dbc9387
fix rename of columns
2020-10-23 20:57:17 +03:00
alesapin
bc331a8029
Better set
2020-10-23 19:34:41 +03:00
alesapin
b28894c43b
More compatible way
2020-10-23 19:31:51 +03:00
Anton Popov
06dc0155e5
allow to read subhcolumns from other storages
2020-10-23 17:44:23 +03:00
Anton Ivashkin
e3879afa69
S3 zero copy replication: fix virtual method default parameter
2020-10-23 16:27:50 +03:00
alesapin
59ca0d071c
Fix comment
2020-10-23 14:17:37 +03:00
alesapin
69f2f9838e
Add ability to specify PRIMARY KEY without ORDER BY
2020-10-23 14:16:27 +03:00
alesapin
39e47c5338
Less strange lambdas
2020-10-23 11:54:00 +03:00
alesapin
3c559c699f
Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more
2020-10-22 17:16:46 +03:00
alesapin
1d07ece5d1
Fix clang tidy warning
2020-10-22 17:15:56 +03:00
Anton Ivashkin
652c56e74e
Fix style, fix build
2020-10-22 16:07:20 +03:00
alesapin
7a63b373f9
Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more
2020-10-22 15:47:11 +03:00
alesapin
4b65d5469f
Fix some unrelated performance issues in select parts for merge
2020-10-22 15:41:01 +03:00
Nikolai Kochetov
7fa045cff8
Merge branch 'master' into storage-read-query-plan
2020-10-22 13:31:10 +03:00
Anton Ivashkin
f833501c77
Merge branch 'master' into s3_zero_copy_replication
2020-10-22 11:17:21 +03:00
alesapin
945dd3aa59
Merge branch 'master' into blind_fix_of_read_in_order
2020-10-22 10:37:37 +03:00
alexey-milovidov
adeba6bdd8
Merge pull request #15074 from amosbird/btc
...
Extend trivial count optimization.
2020-10-22 02:50:57 +03:00
Kruglov Pavel
0b75885c8a
Merge branch 'master' into select_final
2020-10-21 20:42:39 +03:00
Pavel Kruglov
89fdeb4e15
Fix style, move setting and add checking level>0
2020-10-21 20:35:31 +03:00
Nikolai Kochetov
a0da588f06
Merge branch 'master' into new-block-for-functions-4
2020-10-21 16:16:58 +03:00
alesapin
11ea6d7911
Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more
2020-10-21 11:39:48 +03:00
alesapin
b45d42bb9c
Maybe devirtualization can help
2020-10-21 11:35:36 +03:00
alesapin
9f47c82bbd
Sleep a little when pool is full
2020-10-21 11:31:57 +03:00
alesapin
b1a13d4884
Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more
2020-10-21 10:30:09 +03:00
Nikolai Kochetov
68c9a57788
Merge branch 'master' into new-block-for-functions-4
2020-10-21 09:17:25 +03:00
Alexey Milovidov
2619efadc8
Fix multiple issues with memory tracking
2020-10-21 03:31:34 +03:00
alesapin
8097c696de
Trying another fix
2020-10-20 23:05:20 +03:00
alexey-milovidov
75a02112bc
Merge pull request #15592 from sundy-li/readonly-metrics
...
sub ReadonlyReplica metric when detach readonly tables
2020-10-20 19:17:52 +03:00
Alexander Tokmakov
72dbb43235
Merge branch 'master' into fork-valbaturin-replication-db-engine
2020-10-20 19:14:54 +03:00
alesapin
439bbd9861
Blind performance fix
2020-10-20 18:30:46 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration ( #13956 )
...
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-10-20 18:10:24 +03:00
Nikolai Kochetov
a1d5adf8d5
Merge branch 'master' into new-block-for-functions-4
2020-10-20 16:16:08 +03:00
alesapin
30ca17b09b
Comment to settings
2020-10-20 16:12:25 +03:00
alesapin
7256d01579
Normal backoff value instead of 10 seconds
2020-10-20 16:11:01 +03:00
alesapin
d126aa8a09
Merge branch 'master' into no_background_pool_no_more
2020-10-20 12:54:42 +03:00
alexey-milovidov
124379cccc
Update IMergeTreeDataPart.h
2020-10-20 04:24:30 +03:00
alexey-milovidov
26517ff08d
Update IMergeTreeDataPart.h
2020-10-20 04:23:23 +03:00
Nikolai Kochetov
bc58637ec2
Fixing build.
2020-10-19 21:37:44 +03:00
Nicolae Vartolomei
af76ff1dbf
RFC: Add metadata section to MergeTreeWriteAheadLog
...
Current WAL format doesn't seem to be extendable without breaking
reverse compatibility.
This PR proposes to introduces a metadata section
for each action in the WAL to which optional fields can be added.
Adding new fields is trivial and doesn't have to break backwards
compatibility if these are not critical for correctness.
Metadata contains the minimum version of the reader allowed
to interpret it.
Removing fields is not supported.
2020-10-19 12:16:31 +01:00
alesapin
cb139abe2a
Better schedule after job already running
2020-10-19 11:47:15 +03:00
alesapin
518ba2ceb2
Safe random generator
2020-10-16 16:48:12 +03:00
alesapin
7edbe4d276
Add some comments
2020-10-16 13:12:31 +03:00
Pavel Kovalenko
a474bdcabf
Merge remote-tracking branch 'origin/master' into idempotent-part-move
2020-10-16 13:02:40 +03:00
alesapin
0fb1f74b2e
Trying to schedule better
2020-10-16 11:00:15 +03:00
Kruglov Pavel
5aba639430
Update test
2020-10-15 22:01:18 +03:00
sundy-li
153be93544
sub ReadonlyReplica when detach readonly tables
2020-10-15 22:01:18 +03:00
Pavel Kruglov
6c1e0e3df8
Change default to false
2020-10-15 19:40:20 +03:00
Anton Ivashkin
14a78f87b0
Zero copy replication over S3: fetch instead of merge
2020-10-15 19:14:47 +03:00
alesapin
4c1f5cab13
Sleep less in background task
2020-10-15 19:10:22 +03:00
Pavel Kruglov
d6fc6d35fa
Fix style
2020-10-15 19:08:24 +03:00
alesapin
629506445d
Better backoff policy
2020-10-15 16:57:50 +03:00
Pavel Kovalenko
ed61c5681b
Use 'moving' directory instead of 'detached' when move part to another disk/volume.
2020-10-15 16:55:13 +03:00
alesapin
60d14352f8
Merge branch 'master' into no_background_pool_no_more
2020-10-15 16:34:14 +03:00
Pavel Kruglov
f5fac575f4
don't postprocess single parts
2020-10-15 15:22:41 +03:00
Pavel Kruglov
25c39689c3
Check expired ttl
2020-10-15 15:15:02 +03:00
alesapin
dccf51b316
Remove redundant logging
2020-10-15 14:08:39 +03:00
alesapin
374949051c
Add thread safety
2020-10-15 13:41:36 +03:00
alesapin
e84eda176a
Add test and slightly refactored
2020-10-15 13:22:02 +03:00
alesapin
583d7042e9
Get rid of some deadlocks
2020-10-15 11:29:18 +03:00
alesapin
c5419083e3
Better method name
2020-10-15 10:43:50 +03:00
alesapin
3394c22c03
Fix race and clang-tidy
2020-10-15 10:39:27 +03:00
Mikhail Filimonov
ace2e99cb3
Fix the bug when NOTHING_TO_DO events wrongly increment count_no_work_done
2020-10-14 23:08:47 +02:00
alesapin
3e3505258f
No sleep when we have job
2020-10-14 22:33:36 +03:00
alesapin
ae8ed99805
Missed change
2020-10-14 21:35:03 +03:00
alesapin
7265792510
Less files
2020-10-14 21:33:41 +03:00
alesapin
a535a089d1
Some style fixes
2020-10-14 19:09:18 +03:00
alesapin
fc25b6cd72
Remove background processing pool
2020-10-14 18:26:00 +03:00
Anton Ivashkin
b877459cf7
Zero copy replication over S3: check s3 storage
2020-10-14 18:05:59 +03:00
alesapin
6a4911dc57
Remove some garbage
2020-10-14 18:05:14 +03:00
alesapin
483893cdd4
Slightly refactored code
2020-10-14 17:56:42 +03:00
alesapin
0b14a31ba9
Get rid of background processing pool
2020-10-14 15:44:10 +03:00
alesapin
4014e0f08d
Something very similar to working code
2020-10-14 15:32:35 +03:00
alesapin
efd3126b5d
Moving pool
2020-10-14 10:22:48 +03:00
Pavel Kruglov
51807599e7
fix style
2020-10-13 23:10:37 +03:00
Pavel Kruglov
44c2b138f3
Fix style
2020-10-13 22:53:36 +03:00
Anton Popov
cbe12a532e
allow to extract subcolumns from column
2020-10-13 22:39:22 +03:00
Pavel Kruglov
be0cb31d21
Add tests and comments
2020-10-13 21:55:03 +03:00
Pavel Kruglov
1ccf10505a
add tests and comments
2020-10-13 21:26:40 +03:00
Pavel Kruglov
6d9b310f61
Add optimize_skip_merged_partitions
2020-10-13 21:26:35 +03:00
Pavel Kruglov
8200bab859
Add setting do_not_merge_across_partitions
2020-10-13 17:54:52 +03:00
alesapin
fd35368c59
First simple implementation
2020-10-13 17:25:42 +03:00
Nikolai Kochetov
7e58f99f64
Merge branch 'master' into storage-read-query-plan
2020-10-12 13:12:39 +03:00
alesapin
5b5dd4fb28
Fix order by desc after alter
2020-10-12 11:28:47 +03:00
alexey-milovidov
10e9d14466
Merge pull request #15818 from ClickHouse/style-pragma-once
...
Check for #pragma once in headers
2020-10-11 13:14:09 +03:00
Alexey Milovidov
269b6383f5
Check for #pragma once in headers
2020-10-10 21:37:02 +03:00
Alexey Milovidov
06fcf437fe
Addition to prev. revision
2020-10-10 19:02:13 +03:00
Alexey Milovidov
9f15daf45c
Remove useless metrics to avoid confusion
2020-10-10 18:52:44 +03:00
Nikolai Kochetov
76a04fb4b4
Merge pull request #15762 from ClickHouse/new-block-for-functions
...
Use `ColumnsWithTypeAndName` instead of `Block` for function calls
2020-10-10 08:50:38 +03:00
Nikita Mikhaylov
6ac5431804
Merge pull request #15783 from nikitamikhaylov/copier-backquotes
...
Merging #14939
2020-10-10 02:32:27 +03:00
alexey-milovidov
0fd007ad9e
Merge pull request #15601 from alexelex/alexelex-master
...
Improvement of Quorum Inserts in ClickHouse
2020-10-09 22:43:53 +03:00
alesapin
35165c8ea4
Merge pull request #15756 from CurtizJ/fix-prewhere-with-virtuals
...
Fix prewhere with virtual columns
2020-10-09 18:07:41 +03:00
Anton Ivashkin
9272ed06b4
Move Zookeeper lock for S3 shared part in IMergeTreeDataPart
2020-10-09 17:24:10 +03:00
nikitamikhaylov
88a5302759
Merge branch 'master' of github.com:ClickHouse/ClickHouse into copier-backquotes
2020-10-09 16:26:31 +03:00
Alexandra Latysheva
f6f33168aa
fix compilation error
2020-10-09 12:11:21 +00:00
Anton Ivashkin
766dbfd2be
Merge master
2020-10-09 15:00:14 +03:00
Alexandra Latysheva
0594a77b57
fix thread restart for parallel quorum inserts
2020-10-09 11:20:20 +00:00
Nikolai Kochetov
a7fb2e38a5
Use ColumnWithTypeAndName as function argument instead of Block.
2020-10-09 10:41:28 +03:00
tavplubix
7b5f2b2bb0
Merge pull request #15701 from ClickHouse/better_clone_replica
...
Improvements in lost replica recovery
2020-10-08 23:19:23 +03:00
Anton Ivashkin
acf86568a7
S3 zero copy replication proof of concept
2020-10-08 18:45:10 +03:00
Alexandra Latysheva
6877f4aa9c
for parallel and non-parallel quorum insertion at the same time
2020-10-08 15:35:41 +00:00
Anton Popov
1a6d2bc538
fix prewhere with virtuals columns
2020-10-08 15:59:47 +03:00
Pervakov Grigorii
3a2068f19d
Use tmp disk for vertical merge files
2020-10-08 13:49:41 +03:00
Amos Bird
867216103f
Extend trivial count optimization.
2020-10-08 18:08:17 +08:00
alexey-milovidov
e465ce3d49
Merge pull request #15642 from azat/force_data_skipping_indices
...
Implement force_data_skipping_indices setting
2020-10-07 23:14:51 +03:00
Latysheva Alexandra
f43ad338ed
Merge pull request #12 from alexelex/quorum_inserts_2
...
style check
2020-10-07 18:56:14 +06:00
Alexandra Latysheva
8f8872dc42
style check
2020-10-07 12:24:46 +00:00
Latysheva Alexandra
6c26f77380
Merge pull request #11 from alexelex/quorum_inserts_2
...
changing the condition for updateQuorum, specialized it and reduced t…
2020-10-07 17:49:20 +06:00
Pavel Kovalenko
69c126f1f1
Possibility to move part to another disk/volume if first attempt was failed.
2020-10-07 14:35:28 +03:00
Alexandra Latysheva
a43cac2c1a
changing the condition for updateQuorum, specialized it and reduced the number of calls to the zk
2020-10-07 11:28:48 +00:00
Nikolai Kochetov
7e02152b50
Merge branch 'master' into storage-read-query-plan
2020-10-07 14:27:21 +03:00
Nikolai Kochetov
c5cb05f5f3
Try fix tests.
2020-10-07 14:26:29 +03:00
tavplubix
041015545c
Merge pull request #15537 from ClickHouse/mutation_hangs_after_replace_partition
...
Fix mutation may hang after REPLACE/DROP PARTITION
2020-10-07 14:25:29 +03:00
alexey-milovidov
e00961f882
Merge pull request #15657 from Jokser/disk-s3-write-error-handling
...
Proper error handling during insert into MergeTree with S3
2020-10-07 11:17:32 +03:00
Alexander Tokmakov
ea16370024
improvements in lost replica recovery
2020-10-07 02:29:33 +03:00
Azat Khuzhin
75e612fc16
Use full featured parser for force_data_skipping_indices
2020-10-07 01:44:14 +03:00
Azat Khuzhin
ef6d12967f
Implement force_data_skipping_indices setting
2020-10-07 01:42:31 +03:00
Latysheva Alexandra
85bcc07c0b
Merge pull request #9 from alexelex/quorum_inserts_2
...
Quorum inserts [version 3]
2020-10-07 04:04:41 +06:00
Alexandra Latysheva
644091459f
style fixes and remove small changes
2020-10-06 22:01:30 +00:00
Alexandra Latysheva
f549ecf9d1
insert_quorum_parallel: status nodes in /quorum/parallel and allow merges
2020-10-06 21:49:48 +00:00
Pavel Kovalenko
69b4bc6f33
Proper error handling during insert into MergeTree with S3.
2020-10-06 12:38:00 +03:00
Nikolai Kochetov
f9bf1e3406
Merge branch 'master' into storage-read-query-plan
2020-10-06 09:50:10 +03:00
Latysheva Alexandra
d495a2fb2d
Merge pull request #7 from alexelex/quorum_inserts_2
...
build checker fix (style) and change comments
2020-10-06 02:33:29 +06:00
Alexandra Latysheva
41e0148d72
build checker fix (style) and change comments
2020-10-05 20:30:50 +00:00
Alexander Tokmakov
418766e92b
fix another bug in mutations
2020-10-05 22:16:28 +03:00
Latysheva Alexandra
3b952aff7d
Merge pull request #6 from alexelex/quorum_inserts_2
...
fix test_00814
2020-10-05 23:01:18 +06:00
Alexandra Latysheva
2be62bbd1b
fix test_00814
2020-10-05 16:58:26 +00:00
alesapin
8ec58c17f3
More consistent metadata usage
2020-10-05 19:41:46 +03:00
Latysheva Alexandra
18007622a1
Merge pull request #5 from alexelex/quorum_inserts_2
...
style check fix
2020-10-05 19:00:39 +06:00
Alexandra Latysheva
aa391a47eb
style check fix
2020-10-05 12:59:07 +00:00
Latysheva Alexandra
7e7cedd73f
Merge pull request #4 from alexelex/quorum_inserts_2
...
style check fix
2020-10-05 16:43:14 +06:00
Alexandra Latysheva
0662432c5c
style check
2020-10-05 10:41:49 +00:00
Latysheva Alexandra
a63084adaf
Merge pull request #3 from alexelex/quorum_inserts_2
...
Quorum inserts 2
2020-10-05 03:28:58 +06:00
Alexandra Latysheva
d54b6c9542
add question in comments
2020-10-04 20:32:08 +00:00
Alexandra Latysheva
ff8cbe6745
remove debug things
2020-10-04 20:19:05 +00:00
Alexandra Latysheva
8263e62298
working copy (with some debug info)
2020-10-04 19:55:39 +00:00
alexey-milovidov
9551562889
Merge pull request #15442 from ClickHouse/add_alter_version
...
Add ability to alter version column in VersionedCollapsingMergeTree.
2020-10-03 05:40:15 +03:00
Azat Khuzhin
b838214a35
Pass non-const SelectQueryInfo (and drop mutable qualifiers)
2020-10-02 22:42:35 +03:00
alesapin
77c9f7af79
Merge pull request #15532 from ClickHouse/substitute_defaults_recursively
...
Fix defaults substitution for absent columns which depend on other defaults
2020-10-02 21:42:01 +03:00
Azat Khuzhin
21deb6812c
Drop unused code for numeric_limits<int128> in MergeTreeDataSelectExecutor ( #15519 )
2020-10-02 16:46:20 +03:00
alesapin
17b86dec96
Recursive defaults
2020-10-02 15:38:50 +03:00
Nikolai Kochetov
5ea4dc0850
Try fix tests.
2020-10-02 14:25:16 +03:00
Nikolai Kochetov
ea131989be
Try fix test.
2020-10-01 21:47:20 +03:00
Nikolai Kochetov
ec64def384
Use QueryPlan while reading from MergeTree.
2020-10-01 20:34:22 +03:00
Nicolae Vartolomei
eb1d253355
Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it
2020-10-01 17:59:50 +01:00
alesapin
8bc9ede711
Fix clang tidy
2020-10-01 09:53:47 +03:00
Alexandra Latysheva
e89a56969f
first part
2020-09-30 23:16:27 +00:00
alesapin
97c2579fb9
Use separate function
2020-09-30 22:50:23 +03:00
alesapin
ac9f0ff4da
More strict check for version column
2020-09-30 22:44:35 +03:00
Anton Popov
80cd2523ce
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-30 19:34:29 +03:00
alexey-milovidov
0c07c2d3a3
Merge pull request #15089 from ClickHouse/codecs_for_nested_types
...
Codecs for nested types
2020-09-30 07:18:19 +03:00
alexey-milovidov
412a54e356
Merge pull request #14693 from hustnn/zk_exist_fix2
...
Fix potential memory leak caused by zookeeper exist calling
2020-09-30 04:48:11 +03:00
alexey-milovidov
b658f348a7
Update MergeTreeDataPartWriterWide.cpp
2020-09-30 01:23:42 +03:00
Anton Popov
8ea4c2e26f
Fix TTL in cases, when its expression is a function and is the same as ORDER BY key
2020-09-29 19:31:19 +03:00
alesapin
65a05a970d
Better message on error
2020-09-29 15:07:01 +03:00
tavplubix
4578d89819
Merge pull request #15348 from ClickHouse/replicate_default_args
...
Continuation of #14791
2020-09-28 18:53:02 +03:00
Nikita Mikhaylov
a4832ed5ce
Merge branch 'master' into HyaZz-master
2020-09-28 13:55:45 +03:00
alesapin
bc662ac4e0
Merge pull request #15293 from amosbird/bc3
...
Replace useless multiset with unordered_set
2020-09-28 10:30:04 +03:00
hustnn
a7024134dd
Merge remote-tracking branch 'upstream/master' into zk_exist_fix2
2020-09-28 14:44:55 +08:00
Alexander Tokmakov
34addcf61f
add review suggestions
2020-09-26 22:18:28 +03:00
alesapin
59f2bbe2a3
Merge pull request #15304 from ClickHouse/fix_race_condition_on_rename
...
Fix race condition between write ahead log cleanup and rename
2020-09-26 10:43:26 +03:00
Alexander Tokmakov
68669b8a92
Merge branch 'master' into replicate_default_args
2020-09-25 15:52:17 +03:00
alesapin
f5c38b34ae
Fix race condition during write ahead log cleanup and rename
2020-09-25 15:07:38 +03:00
hexiaoting
4666ae3211
fix build bug for merge_selector2.cpp
2020-09-25 17:59:19 +08:00
Amos Bird
81d08b59e5
Replace useless multiset with unordered_set
2020-09-25 16:38:09 +08:00
alesapin
96c419974c
Merge branch 'master' into codecs_for_nested_types
2020-09-24 10:28:20 +03:00
alesapin
87b64baea5
Fix again
2020-09-23 16:21:06 +03:00
alesapin
bb4b739f43
Fix PVS-Studio warning
2020-09-23 13:11:48 +03:00
filimonov
cc24ef9f83
Better debug message from MergeTreeDataSelectExecutor
...
See #15168
2020-09-22 21:35:29 +02:00
alesapin
7015f272e7
Better comment
2020-09-22 16:48:38 +03:00
alesapin
fd394f699c
Better comments
2020-09-22 15:49:55 +03:00
alesapin
96d06c6ae6
Less changes
2020-09-22 15:16:15 +03:00
hustnn
35f1db0043
Fix potential memory leak caused by zookeeper exist calling
2020-09-22 09:28:12 +08:00
alesapin
e87b961095
Trying to add same logic for compact parts
2020-09-21 20:35:09 +03:00
alesapin
fbd6f10ff7
Use default codec instead of throwing exception
2020-09-21 19:07:14 +03:00
alesapin
8517dda807
First working test
2020-09-21 17:47:10 +03:00
alesapin
4debccb7fe
Almost working version
2020-09-21 17:22:13 +03:00
alesapin
a3e999784b
Buildable code
2020-09-21 14:24:10 +03:00
alesapin
387b97aba5
Merge branch 'master' into codecs_for_nested_types
2020-09-21 11:06:28 +03:00
alesapin
7fc5a2dae1
Some intermediate stage 2
2020-09-21 10:18:23 +03:00
alesapin
09f6a454ae
Some intermediate stage
2020-09-21 10:17:58 +03:00
alexey-milovidov
3e0509af69
Merge pull request #13573 from amosbird/rms
...
specific ReplicatedMergeTree settings
2020-09-21 01:36:33 +03:00
alexey-milovidov
85483f8532
Merge pull request #14853 from ClickHouse/akz/optimized_index_binary_search
...
Optimized marks selection algorithm for continuous marks ranges
2020-09-20 19:48:45 +03:00
alexey-milovidov
f7cae4a57d
Merge pull request #14975 from ClickHouse/fix_merge_tree_settings_check
...
Fix check for merge tree settings sanity on server startup
2020-09-20 19:43:40 +03:00
philip.han
3122455b4d
BloomFilter is used just in specific cases of indexof()
2020-09-20 19:04:33 +09:00
Alexander Tokmakov
de540c6955
minor fixes
2020-09-19 15:59:37 +03:00
alexey-milovidov
eb9ee723c3
Merge pull request #11913 from ClickHouse/compact-parts-by-default
...
Enable compact parts by default for small parts
2020-09-19 15:31:15 +03:00
alexey-milovidov
1fcebce926
Merge pull request #15000 from Jokser/disable-ttl-move-on-insert
...
Option to disable TTL move on data part insert
2020-09-19 13:45:02 +03:00
alexey-milovidov
3463d97f8c
Merge pull request #14973 from amosbird/bm2
...
Ignore key constraints when doing mutations.
2020-09-19 13:44:30 +03:00
alexey-milovidov
4e77c2e48e
Update MergeTreeDataMergerMutator.cpp
2020-09-19 01:29:31 +03:00
alexey-milovidov
322e63971d
Update MergeTreeIndexConditionBloomFilter.cpp
2020-09-18 21:47:32 +03:00
Amos Bird
d842cb704f
Allow mutations to work with key constraints.
2020-09-19 02:40:02 +08:00
Pavel Kovalenko
da04a130ed
Add option to disable ttl move on data part insert - minor fixes.
2020-09-18 20:45:30 +03:00
Anton Popov
13917969ab
better storing info about subcolumns
2020-09-18 20:37:08 +03:00
Pavel Kovalenko
6b4b5898a7
Renaming.
2020-09-18 18:41:14 +03:00
Pavel Kovalenko
0da19ab46d
Add option to disable ttl move on data part insert
2020-09-18 18:30:00 +03:00
Anton Popov
edc2fe2226
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-18 17:51:54 +03:00
alesapin
e38b537017
Merge pull request #11684 from ClickHouse/manual-write-duplicate-parts-to-replicas
...
Don't ignore duplicate parts written to replicas
2020-09-18 16:50:00 +03:00
Alexander Kazakov
1ee2e3d2b3
Review fix
2020-09-18 16:03:48 +03:00
alesapin
e96a3ac5f3
DataType in enumerate streams
2020-09-18 14:37:58 +03:00
Nikolai Kochetov
c7aff19937
Merge pull request #14221 from hagen1778/settings-leaf-limits
...
[settings]: introduce new query complexity settings for leaf-nodes
2020-09-18 14:05:10 +03:00
philip.han
cbedd44a60
Fix indexOf() to use BloomFilter
2020-09-18 17:43:07 +09:00
alesapin
96a693baec
Better message
2020-09-18 10:46:23 +03:00
alesapin
e6914acb3b
Fix check for merge tree settings on server startup
2020-09-18 10:44:17 +03:00
Anton Popov
cb4801e3be
allow to read subcolumns of complex types
2020-09-18 02:12:43 +03:00
roman
ddca262fe6
fix review comments
2020-09-17 20:54:21 +01:00
alesapin
447205c525
More comments
2020-09-17 22:30:45 +03:00
alesapin
dc677b93fb
Comments and fix test
2020-09-17 22:30:17 +03:00
alesapin
b07c00ef7c
Ugly, but probably working implementation
2020-09-17 18:33:50 +03:00
Alexey Milovidov
c1e9ce2cf2
Remove questionable code
2020-09-17 17:38:06 +03:00
alesapin
d09fd8ca8b
Add comment
2020-09-17 17:01:17 +03:00
alesapin
6601f4f786
Better errors, fix merge
2020-09-17 15:10:06 +03:00
alesapin
6c369c8dc6
Fix merge error
2020-09-17 15:01:03 +03:00
alesapin
40b2f203b6
Merge branch 'master' into manual-write-duplicate-parts-to-replicas
2020-09-17 13:21:00 +03:00
roman
b41421cb1c
[settings]: introduce new query complexity settings for leaf-nodes
...
The new setting should allow to control query complexity on leaf nodes
excluding the final merging stage on the root-node. For example, distributed
query that reads 1k rows from 5 shards will breach the `max_rows_to_read=5000`,
while effectively every shard reads only 1k rows. With setting `max_rows_to_read_leaf=1500`
this limit won't be reached and query will succeed since every shard reads
not more that ~1k rows.
2020-09-17 10:37:05 +01:00
alesapin
5f9e7ed169
Merge pull request #14834 from ClickHouse/fix_compression_codec_perf
...
Fix performance degradation during parts selection for merge
2020-09-17 10:09:41 +03:00
alesapin
73544a3781
Merge pull request #14845 from ClickHouse/fix_alias_array
...
Fix recursive column defaults
2020-09-17 10:02:39 +03:00
Anton Popov
8e9828d2ff
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-16 21:07:35 +03:00
Alexey Milovidov
26eb49b48d
Fix typo
2020-09-16 20:02:55 +03:00
alesapin
cbe85327e2
Check stack size
2020-09-16 16:24:07 +03:00
Alexander Kazakov
7465e00163
Optimized marks selection algorithm for continuous marks ranges
2020-09-15 17:22:32 +03:00
alesapin
e20c54998c
Better exception on non physical columns
2020-09-15 16:29:11 +03:00
Anton Popov
3e855daefd
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-15 15:31:29 +03:00
alesapin
8828a78174
Fix alias on default columns
2020-09-15 14:17:58 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build ( #14809 )
2020-09-15 12:55:57 +03:00
Nikolai Kochetov
c1f61984e5
Merge pull request #14513 from amosbird/mf1
...
Binary operator monotonicity
2020-09-15 12:45:23 +03:00
alesapin
6f8da7dfdf
Merge branch 'master' into fix_compression_codec_perf
2020-09-15 09:59:45 +03:00
Alexander Tokmakov
da6beb1a91
Merge branch 'master' into enable_some_tests_with_database_atomic
2020-09-14 23:00:14 +03:00
alesapin
e96256d367
Trying to improve performance
2020-09-14 22:15:25 +03:00
Nikita Mikhaylov
ddea62211d
Merge pull request #11948 from CurtizJ/fsync-parts
...
Experiment. Allow to turn on fsync on inserts, merges and fetches.
2020-09-14 22:37:14 +04:00
Vxider
407bccdefa
Support default arguments for Replicated engine
2020-09-14 15:46:27 +08:00
alesapin
06575970aa
Merge pull request #14705 from amosbird/misc1
...
Add merge_algorithm to system.merges
2020-09-14 09:58:30 +03:00
Amos Bird
5cc8fd395c
Fix empty key segfault
2020-09-13 21:55:16 +08:00
Amos Bird
34b9547ce1
Binary operator monotonicity
2020-09-13 21:55:12 +08:00
Amos Bird
c2d79bc5cc
Add merge_algorithm to system.merges
2020-09-13 10:00:03 +08:00
Anton Popov
a64331d79f
fix syncing of WAL
2020-09-11 16:09:26 +03:00
Anton Popov
f2a5216e97
add waiting for fsync in WAL
2020-09-11 02:28:50 +03:00
Anton Popov
af4089c606
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-11 00:39:21 +03:00
Alexander Tokmakov
f19f02028b
fix misleading exception message about uuid macro
2020-09-10 17:50:10 +03:00
Anton Popov
b9bf67b6ac
allow to randomize part type in MergeTree
2020-09-10 04:27:36 +03:00
alesapin
bd14c79b0b
Merge pull request #14494 from ClickHouse/recompression_in_background
...
Recompression TTL
2020-09-09 17:46:01 +03:00
Alexander Tokmakov
b68782d285
enable more tests with Atomic database
2020-09-09 16:32:50 +03:00
Anton Popov
32135d96f9
Merge pull request #12183 from CurtizJ/polymorphic-parts-2
...
Support codecs in compact parts
2020-09-09 12:34:13 +03:00
alesapin
485b104898
Use max ttl and add introspection to system parts about recompression
2020-09-09 12:15:42 +03:00
Anton Popov
a419267dc6
minor fixes
2020-09-08 19:28:49 +03:00
alesapin
f0f7f2b5d2
Merge branch 'master' into recompression_in_background
2020-09-08 15:43:37 +03:00
alesapin
6ad91d1018
Merge pull request #14490 from ClickHouse/control_ttl_merges_in_pool
...
Control ttl merges in pool
2020-09-08 15:42:20 +03:00
alesapin
8ff0aedebe
Merge branch 'control_ttl_merges_in_pool' into recompression_in_background
2020-09-08 09:34:03 +03:00
alesapin
405a204514
Lower timeout for TTL merges
2020-09-08 09:33:00 +03:00
alesapin
f4846e2ef1
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
...
Co-authored-by: Anton Popov <pad11rus@gmail.com>
2020-09-08 09:30:54 +03:00
alesapin
f274ffc9d1
Add comments
2020-09-07 10:59:14 +03:00
alesapin
ba70de63f8
Fix diff
2020-09-07 10:28:04 +03:00
Anton Popov
3b9be70c8c
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-05 18:53:55 +03:00
Nikolai Kochetov
499405504f
Merge branch 'master' into pipes-and-creating-sets
2020-09-04 23:41:20 +03:00
Anton Popov
c73bb98002
fix clang-tidy
2020-09-04 19:50:40 +03:00
Anton Popov
f1cf03f1bb
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-04 17:40:22 +03:00
alesapin
37a2bd0bfd
less debug
2020-09-04 17:23:32 +03:00
alesapin
79ef72178e
Less garbage
2020-09-04 17:18:49 +03:00
alesapin
f48d654d35
Less garbage
2020-09-04 17:15:34 +03:00
alesapin
fecb2f1311
Less copypaste
2020-09-04 17:08:43 +03:00
alesapin
ea7168580b
Fixes
2020-09-04 16:55:07 +03:00
Anton Popov
68913eab62
better reading from compact parts with differents codecs
2020-09-04 15:48:55 +03:00
alesapin
f8aa6cfe8a
Merge pull request #14438 from ClickHouse/apply_ttl_if_not_calculated
...
Apply TTL if it's not calculated for part
2020-09-04 15:32:50 +03:00
alesapin
f9dd4cc98d
Merge branch 'control_ttl_merges_in_pool' into recompression_in_background
2020-09-04 14:32:25 +03:00
alesapin
e42d0f60da
Fix several bugs
2020-09-04 14:27:27 +03:00
alesapin
4079636f81
Merge
2020-09-04 13:52:51 +03:00
alesapin
69b31ab90d
More comments
2020-09-04 13:29:55 +03:00
alesapin
82c56349a5
Some comments
2020-09-04 13:08:09 +03:00
Nikolai Kochetov
c761970bc5
Merge branch 'master' into pipes-and-creating-sets
2020-09-04 11:38:35 +03:00
alesapin
61ecaebcb1
Simplify settings for TTL merges
2020-09-04 09:55:19 +03:00
Anton Popov
6883ee7eea
create less compressed streams while writing compact parts
2020-09-04 01:38:17 +03:00
Anton Popov
d5da58918e
create less compressed streams while writing compact parts
2020-09-04 01:04:46 +03:00
Anton Popov
98011f6fc3
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-03 17:53:05 +03:00
alesapin
0dab4d9485
Merge branch 'fix_select_parts_for_merge' into control_ttl_merges_in_pool
2020-09-03 17:41:23 +03:00
alesapin
15a02321bf
Merge branch 'fix_select_parts_for_merge' into control_ttl_merges_in_pool
2020-09-03 17:16:29 +03:00
alesapin
55adb6f9f2
Update src/Storages/MergeTree/MergeSelector.h
...
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
2020-09-03 16:55:34 +03:00
alesapin
e48e4309b2
Better names
2020-09-03 16:29:18 +03:00
alesapin
565555e55c
Fix bug in parts selection
2020-09-03 16:04:28 +03:00
alesapin
13248a744b
Fix bug in parts selection
2020-09-03 16:02:24 +03:00
alesapin
f4c7ff0376
Add fixed size of Merge TTLS
2020-09-03 16:00:13 +03:00
alesapin
aa47d0aabc
Merge branch 'apply_ttl_if_not_calculated' into recompression_in_background
2020-09-03 12:07:03 +03:00
alesapin
acc0ee0657
Apply TTL if it's not calculated for part
2020-09-03 11:59:41 +03:00
alesapin
dcbddbb8d0
Add recompression ttls tests and fix bugs
2020-09-02 19:15:41 +03:00
Nikolai Kochetov
92c937db8b
Remove CreatingSetsBlockInputStream
2020-09-02 16:13:13 +03:00
alesapin
128cb7ce22
Don't select already selected parts
2020-09-02 15:16:12 +03:00
alesapin
aac466ab1c
Merge branch 'better_ttl_merges_selection' into recompression_in_background
2020-09-02 13:30:04 +03:00
alesapin
fbb37c37df
Simplier interface
2020-09-02 11:28:46 +03:00
alesapin
b08056fa8c
Better selection of Merges with TTL
2020-09-02 11:18:50 +03:00
Anton Popov
4fe57fc169
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-02 01:29:44 +03:00
Anton Popov
26d75f7602
do fsync for WAL
2020-09-02 01:25:10 +03:00
Anton Popov
3cadc9033a
fsyncs for metadata files of part
2020-09-01 18:53:22 +03:00
alesapin
c37a19f7b0
Better
2020-09-01 14:23:38 +03:00
alesapin
f0dc5a3085
First working test
2020-09-01 13:49:53 +03:00
alesapin
b751319f9e
Merge branch 'master' into recompression_in_background
2020-09-01 10:31:10 +03:00
alesapin
05b10048a6
Merge pull request #14116 from ClickHouse/parts_default_compression
...
Save parts default compression
2020-09-01 09:43:42 +03:00
Anton Popov
25140b9bd5
fsync MergeTree format file
2020-09-01 04:39:36 +03:00
alesapin
46f833b7df
Some changes
2020-08-31 22:50:42 +03:00
alesapin
b20a0bc254
Add recompression flag in ReplicatedEntry
2020-08-31 16:42:42 +03:00
alesapin
067eb4599d
Fix empty columns case
2020-08-31 16:39:27 +03:00
alesapin
adc18f4d3f
Write with recompression TTL
2020-08-31 16:29:31 +03:00
alesapin
42c210fcba
Recompress TTLs in memory metadata
2020-08-31 15:12:51 +03:00
alesapin
034f1a895d
Merge branch 'master' into recompression_in_background
2020-08-31 11:10:14 +03:00
alexey-milovidov
b1efc5df01
Merge pull request #14225 from amosbird/bf1
...
Fix bug in mark inclusion search.
2020-08-30 02:58:22 +03:00
alesapin
2d33a4029b
Merge pull request #14220 from ClickHouse/remove_redundant_flag
...
Disable force TTL on optimize final
2020-08-29 10:40:36 +03:00
Amos Bird
591a4d60d4
Fix bug in mark inclusion search.
2020-08-29 09:46:46 +08:00
alesapin
ebc163c924
Merge branch 'master' into parts_default_compression
2020-08-28 23:11:42 +03:00
alesapin
65c33f0802
Merge pull request #13280 from amosbird/as
...
ALTER MODIFY SAMPLE BY
2020-08-28 19:32:09 +03:00
alesapin
232c264d7d
Disable force TTL on optimise
2020-08-28 16:45:42 +03:00
alesapin
0896b49533
Small fixes
2020-08-28 13:18:56 +03:00
alesapin
77faf9587f
Better interface
2020-08-28 12:07:20 +03:00
Alexey Milovidov
5763737d97
Fixups
2020-08-28 03:53:22 +03:00
Alexey Milovidov
ed1d120de0
Added another test; fixup
2020-08-28 03:28:37 +03:00
Alexey Milovidov
c4e8aaac16
Fixups
2020-08-28 03:07:51 +03:00
Alexey Milovidov
6082697c4d
Support for quorum
2020-08-28 02:39:12 +03:00
Alexey Milovidov
cdba5e727c
Fix mistake
2020-08-28 02:30:07 +03:00
Alexey Milovidov
88db4938f5
Fix error; refinements
2020-08-28 02:22:00 +03:00
Alexey Milovidov
c04dc4e722
Merge branch 'master' into manual-write-duplicate-parts-to-replicas
2020-08-28 01:56:11 +03:00
Amos Bird
05a5a13e08
specific ReplicatedMergeTree settings
2020-08-27 22:37:41 +08:00
Amos Bird
078b14610d
ALTER MODIFY SAMPLE BY
2020-08-27 22:31:30 +08:00
alesapin
32db38b4d2
Fix compression codec in WAL
2020-08-27 17:06:14 +03:00
alesapin
d426751e26
Default codec if part is not stored on disk
2020-08-27 16:32:23 +03:00
alexey-milovidov
9baa0fbf81
Merge pull request #14129 from CurtizJ/fix-performance
...
Slightly optimize very short queries with LowCardinality
2020-08-27 15:07:19 +03:00
alesapin
1225c09a02
Fix default codec for in-memory parts
2020-08-27 11:35:55 +03:00
alesapin
886a7b485e
Fixed codec in clone part
2020-08-27 09:49:41 +03:00
bharatnc
ffb73adb11
fix style check
2020-08-26 15:31:48 -07:00
bharatnc
593f054b10
minor fix exception message
2020-08-26 15:31:48 -07:00
bharatnc
3a35aedb56
add comments explaining index_granularity_bytes safe guard
2020-08-26 15:31:48 -07:00
bharatnc
4ac9b59cec
format file
2020-08-26 15:31:48 -07:00
bharatnc
37416045ba
setting min_index_granularity_bytes
...
This PR adds a new setting called `min_index_granularity_bytes`.
If the `index_granularity_bytes` is > 0 and is lesser than the
`min_index_granularity_bytes`, throw an exception inside the
`registerStorageMergeTree.cpp` file.
2020-08-26 15:31:48 -07:00
alesapin
44eb702fbe
Remove redundant code
2020-08-26 22:41:57 +03:00
alesapin
3a680dc8fb
Fix removal bug
2020-08-26 22:40:04 +03:00
Anton Popov
eeb78bf291
slightly optimize very short queries with LowCardinality
2020-08-26 21:46:18 +03:00
alesapin
1fb93a8bf2
Remove read helpers
2020-08-26 18:57:06 +03:00
alesapin
18eb0dbcaa
Better remove
2020-08-26 18:39:11 +03:00
alesapin
2fc80189af
Add default compression codec to merge tree data part
2020-08-26 18:29:46 +03:00
Alexey Milovidov
0e021c6151
Merge branch 'master' into manual-write-duplicate-parts-to-replicas
2020-08-26 03:24:35 +03:00
Anton Popov
4331158d30
merge with master
2020-08-24 16:09:23 +03:00
Anton Popov
4df5c72878
Merge remote-tracking branch 'upstream/master' into HEAD
2020-08-24 12:57:27 +03:00
alexey-milovidov
7b53a0ef33
Revert "Added allow_merges option for volumes in multi-disk configuration ( #13402 )"
...
This reverts commit 1e2616542a
2020-08-21 18:44:29 +03:00
Vladimir Chebotarev
1e2616542a
Added allow_merges option for volumes in multi-disk configuration ( #13402 )
2020-08-21 12:04:13 +03:00
Artem Zuikov
becc186c91
Add support for extended precision integers and decimals ( #13097 )
2020-08-19 14:52:17 +03:00
alexey-milovidov
23ccb0b6be
Merge pull request #13677 from hagen1778/merge-tree-fail-fast-on-rows-limit
...
[mergeTree]: fail fast if max_rows_to_read limit exceeded on parts scan
2020-08-18 22:24:39 +03:00
roman
35e28b4c6b
[mergeTree]: make exception message more clear
2020-08-17 09:52:04 +01:00
Alexey Milovidov
5059d1f441
Merge branch 'master' into manual-write-duplicate-parts-to-replicas
2020-08-16 15:58:57 +03:00
alexey-milovidov
f3755fecb7
Merge pull request #13793 from amosbird/fnc
...
Fix set index with const column pred
2020-08-16 15:18:31 +03:00
Amos Bird
ae2c50e9e8
Fix set index with const column pred
2020-08-16 15:16:37 +08:00
Alexey Milovidov
9e6d4802ac
Do not drop cache if DROP/RENAME in Atomic database
2020-08-15 09:56:47 +03:00
roman
b637699ccd
[mergeTree]: fail fast if max_rows_to_read limit exceeded on parts scan
...
The motivation behind this change is to skip ranges scan for all selected parts
if it is clear that `max_rows_to_read` is already exceeded. The change is quite
noticeable for queries over big number of parts.
2020-08-14 13:53:48 +01:00
tavplubix
e6df1f4868
Merge pull request #13676 from amosbird/bd
...
Make drop table a little bit more robust
2020-08-14 14:14:58 +03:00
Amos Bird
de4a070d6a
log error message
2020-08-14 09:54:51 +08:00
alexey-milovidov
c06cfda3cb
Merge pull request #13678 from ClickHouse/fix-typo-2
...
Fix typo due to copy-paste
2020-08-14 04:15:23 +03:00
alexey-milovidov
eeefcd6493
Merge pull request #13337 from ClickHouse/refactor-pipes-3
...
Refactor Pipe
2020-08-14 04:12:22 +03:00
Alexey Milovidov
bcd29dc9f8
Fix typo due to copy-paste
2020-08-13 18:41:26 +03:00
Amos Bird
3b92692458
Make drop table a little bit more robust
2020-08-13 22:22:04 +08:00
Alexander Tokmakov
e4da1d8e51
fix deadlock and logical error in replication queue
2020-08-12 00:36:02 +03:00
Nikolai Kochetov
9b67cd9faf
Merge branch 'master' into refactor-pipes-3
2020-08-10 10:50:17 +03:00
alexey-milovidov
fa1c4d1eef
Merge pull request #13538 from CurtizJ/fix-skip-indices
...
Fix secondary indices corruption in compact parts
2020-08-09 20:13:13 +03:00
Anton Popov
2280619ad0
fix secondary indices corruption in compact parts
2020-08-09 04:45:10 +03:00
alexey-milovidov
aefa9b297b
Merge pull request #13449 from ClickHouse/mixed_granularity_parts_by_default
...
Enable mixed granularity parts by default
2020-08-09 01:36:59 +03:00
alexey-milovidov
c0fef71507
Merge pull request #13511 from ClickHouse/codespell
...
Fix typos in code with codespell.
2020-08-08 17:09:16 +03:00
Alexey Milovidov
306aefd33e
Change error code #13416
2020-08-08 17:05:46 +03:00
Alexey Milovidov
12f66fa82c
Fix 99% of typos
2020-08-08 04:01:47 +03:00
Alexey Milovidov
edd89a8610
Fix half of typos
2020-08-08 03:47:03 +03:00
Alexey Milovidov
bbbc52362b
Remove strange file
2020-08-08 03:13:51 +03:00
alesapin
5de2491eda
Enable mixed granularity parts by default
2020-08-07 11:55:51 +03:00
Alexey Milovidov
0ac3f0481f
Probably fix error
2020-08-07 04:27:29 +03:00
Nikolai Kochetov
d83c7cc0e6
Merge branch 'master' into refactor-pipes-3
2020-08-06 17:32:50 +03:00
Nikolai Kochetov
20e63d2271
Refactor Pipe [part 6]
2020-08-06 15:24:05 +03:00
Anton Popov
33699d8d4a
Merge pull request #13282 from amosbird/mha
...
Report broken parts in compact MergeTree
2020-08-05 14:27:32 +03:00
Alexey Milovidov
d91471ea00
Merge branch 'master' into stack-frams-size-limit
2020-08-04 15:28:47 +03:00
alesapin
943eb2f990
bump CI
2020-08-04 12:42:51 +03:00
Amos Bird
019f6d3a76
Report broken parts in compact MergeTree
2020-08-04 16:45:52 +08:00
Nikolai Kochetov
09fbce1b1e
Merge branch 'master' into refactor-pipes-3
2020-08-04 11:32:34 +03:00
alexey-milovidov
c43a27782e
Merge pull request #13153 from ClickHouse/merge-tree-settings-sanity-check
...
Sanity checks for MergeTreeSettings
2020-08-04 01:27:19 +03:00
Alexey Milovidov
1d091738dd
Merge branch 'master' into stack-frams-size-limit
2020-08-03 20:41:47 +03:00
Nikolai Kochetov
2cca4d5fcf
Refactor Pipe [part 2].
2020-08-03 16:54:14 +03:00
Alexander Tokmakov
5b92f43f12
Merge branch 'master' into database_atomic_improvements
2020-08-03 14:35:40 +03:00
Nikolai Kochetov
e411916bde
Refactor Pipe [part 1].
2020-08-03 14:33:11 +03:00
Nikolai Kochetov
cf49a83980
Merge pull request #13201 from azat/fix-logging-in-MergeTreeDataSelectExecutor
...
Fix logging in MergeTreeDataSelectExecutor for multiple threads (attach to thread group)
2020-08-03 10:08:58 +03:00
Alexey Milovidov
77516b48fa
Merge branch 'master' into merge-tree-settings-sanity-check
2020-08-03 00:30:56 +03:00
Alexey Milovidov
0fd7af8a63
Merge branch 'master' into stack-frams-size-limit
2020-08-03 00:06:00 +03:00
Alexey Milovidov
4ed0bf3af1
Better code
2020-08-03 00:01:39 +03:00
Alexey Milovidov
3c489ce159
Fix assertion in KeyCondition
2020-08-02 23:55:20 +03:00
Alexey Milovidov
5f808aa503
Fix bad code
2020-08-02 23:41:52 +03:00
alesapin
75a9053511
Merge pull request #13150 from nvartolomei/nv/freeze-verbose-absolute-path
...
Use absolute paths to backup locations (shadow)
2020-08-02 19:25:24 +03:00
Alexey Milovidov
41111d1e57
Merge branch 'master' into stack-frams-size-limit
2020-08-02 19:18:33 +03:00
Alexey Milovidov
778abb346f
Merge branch 'master' into merge-tree-settings-sanity-check
2020-08-02 17:13:17 +03:00
Azat Khuzhin
e37c42c56c
Fix logging in MergeTreeDataSelectExecutor for multiple threads (attach to thread group)
2020-08-02 13:40:01 +03:00
Azat Khuzhin
101217470e
Use "Not using primary index on part" over "Not using index on part" (add "primary")
2020-08-02 13:40:01 +03:00
Vitaly Baranov
18b21511a9
Merge pull request #13013 from vitlibar/implement-custom-settings
...
Implement custom settings
2020-08-02 05:01:14 +03:00
Alexey Milovidov
d3c0635fc0
Merge branch 'master' into stack-frams-size-limit
2020-08-02 04:35:09 +03:00
Alexey Milovidov
8fe201f1b9
Merge branch 'master' into merge-tree-settings-sanity-check
2020-08-01 19:20:11 +03:00
Nikolai Kochetov
c786f05388
Merge pull request #13189 from ClickHouse/refactor-pipes-2
...
Remove TreeExecutorBIS
2020-08-01 11:56:50 +03:00
Vitaly Baranov
56665a15f7
Rework and rename the template class SettingsCollection => BaseSettings.
2020-07-31 20:54:18 +03:00
Vitaly Baranov
90602b869a
Make SettingsChanges a class.
2020-07-31 19:11:27 +03:00
Vitaly Baranov
4ad99f1472
Make type names of settings shorter: SettingUInt64 => UInt64.
2020-07-31 19:11:27 +03:00
Nikolai Kochetov
39530f837e
Remove TreeExecutorBlockInputStream.
2020-07-31 16:23:19 +03:00
alesapin
45894fcf99
Better comments
2020-07-31 15:22:32 +03:00
alexey-milovidov
9c9ef0db9b
Merge branch 'master' into database_atomic_improvements
2020-07-31 14:56:49 +03:00
alesapin
14928874a1
Fix mutations status check
2020-07-31 14:37:16 +03:00
Alexey Milovidov
2c40539df6
Check ALTERs; update test
2020-07-31 00:42:55 +03:00
Alexey Milovidov
c491b2c153
Lower stack frame size
2020-07-30 23:58:08 +03:00
Alexey Milovidov
b9f49d31df
Sanity checks for MergeTreeSettings
2020-07-30 22:08:13 +03:00
alexey-milovidov
fc35ce69fa
Merge pull request #12666 from ClickHouse/forward_alter_error_to_client
...
Forward exceptions to client if exception happened during ALTER
2020-07-30 21:36:06 +03:00
Nicolae Vartolomei
85174d64a0
Use absolute paths to backup locations (shadow)
...
Tests are frankenstain. Had to hide absolute path `/tmp/clickhouse.test..IP69I/data/shadow/test_01417_single_part_7/data/default/table_for_freeze/7_8_8_0` to make them deterministic and independend of the method they are run (local vs remote target)
2020-07-30 18:03:26 +01:00
alexey-milovidov
4c3241808d
Merge pull request #12728 from ClickHouse/prevent-inconsistent-merge-tree-settings
...
Safe logic for incorrect MergeTreeSettings
2020-07-30 17:55:39 +03:00
alexey-milovidov
e6265bf299
Merge pull request #13100 from ClickHouse/keep-less-amount-of-logs-in-zookeeper
...
Keep less amount of logs in ZooKeeper
2020-07-30 15:41:55 +03:00
alexey-milovidov
a22bdb1989
Merge pull request #13081 from CurtizJ/fix-index_analysis
...
Fix wrong index analysis with functions
2020-07-30 15:30:55 +03:00
Alexey Milovidov
6517ddcbd4
Keep less amount of logs in ZooKeeper
2020-07-30 05:47:05 +03:00
Alexander Kuzmenkov
881220eec2
Merge pull request #12755 from bobrik/ivan/obvious-skip-overall
...
Show marks before applying skipping indices
2020-07-29 20:59:08 +03:00
Anton Popov
4c266d1e5d
fix wrong index analysis with functions
2020-07-29 19:09:38 +03:00
alesapin
7a806eb90f
Merge pull request #13017 from ClickHouse/freeze_partition_with_name
...
Alter table freeze with verbose output
2020-07-29 18:42:48 +03:00
Alexander Kuzmenkov
1b9269ae0c
fixup
2020-07-28 19:58:19 +03:00
Alexander Kuzmenkov
297cf65f1f
Merge remote-tracking branch 'origin/master' into HEAD
2020-07-28 19:56:35 +03:00
alesapin
c75830e878
Alter table freeze with verbose output
2020-07-28 18:10:36 +03:00
alesapin
585d9247ec
Merge pull request #12959 from ClickHouse/test_for_backup
...
Check backup restore in integration test.
2020-07-28 17:59:52 +03:00
Alexander Kazakov
2bde393499
Merging #12548 - Correction to merge_with_ttl_timeout logic by @excitoon ( #12982 )
...
* Fixed `merge_with_ttl_timeout` logic.
* Separate TTL-merge timers for each partition
Co-authored-by: Vladimir Chebotarev <vladimir.chebotarev@gmail.com>
2020-07-28 17:38:34 +03:00
Alexander Kuzmenkov
ba7c33f806
Merge pull request #12754 from bobrik/ivan/obvious-skip
...
Show total granules examined by skipping indices
2020-07-28 17:14:25 +03:00
alexey-milovidov
7a22717180
Merge pull request #12760 from ClickHouse/sticky_mutations_bug
...
Sticking mutations bug
2020-07-28 14:42:54 +03:00
alesapin
1bece3de0a
Remove strange code and test
2020-07-28 12:36:08 +03:00
Alexey Milovidov
798ef1b9f9
Adjustments; added test result
2020-07-27 23:18:23 +03:00
Alexey Milovidov
ec84ed30cb
Merge branch 'master' into prevent-inconsistent-merge-tree-settings
2020-07-27 21:39:25 +03:00
Nikolai Kochetov
abdd160db1
Merge pull request #12589 from bobrik/ivan/parallel-ranges
...
Parallelize PK range and skipping index stages
2020-07-27 19:23:00 +03:00
alexey-milovidov
7dac65ce0f
Merge pull request #12659 from zhang2014/fix/ISSUES-10572
...
ISSUES-10572 fix bloom filter with const column
2020-07-27 14:38:56 +03:00
alesapin
1707f84a44
Less strict check and rare rename bug
2020-07-27 12:42:37 +03:00
alesapin
e0bf5913e7
Fix sticking mutations
2020-07-26 17:21:57 +03:00
Alexander Tokmakov
6bfe88e83f
Merge branch 'master' into database_atomic_improvements
2020-07-26 13:40:05 +03:00
alesapin
c66dc23a47
Fix error style
2020-07-25 23:56:29 +03:00
alesapin
eae7dc852e
Fix build
2020-07-25 23:54:33 +03:00
alesapin
393a9195ca
Better exception message
2020-07-25 17:45:46 +03:00
alesapin
78d357f0d2
Add a test for sticking mutations bug
2020-07-25 17:42:20 +03:00
Ivan Babrou
e835ec0b56
Show marks before applying skipping indices
...
This change makes skipping index efficiency more obvious, changing this:
```
Selected 30 parts by date, 30 parts by key, 592 marks to read from 541 ranges
```
Into this:
```
Selected 30 parts by date, 30 parts by key, 48324 marks by primary key, 592 marks to read from 541 ranges
```
2020-07-24 15:45:38 -07:00
Ivan Babrou
67d4529783
Show total granules examined by skipping indices
...
This change makes skipping index efficiency more obvious, changing this:
```
Index `idx_duration` has dropped 59 granules.
```
Into this:
```
Index `idx_duration` has dropped 59 / 61 granules.
```
2020-07-24 14:50:32 -07:00
Anton Popov
d3fbeb296e
write compressed blocks for every column in granule
2020-07-23 19:56:12 +03:00
Anton Popov
61018c275d
Merge remote-tracking branch 'upstream/master' into polymorphic-parts-4
2020-07-23 18:38:29 +03:00
Anton Popov
1d0493cfcc
better code in ReaderCompact and perf test added
2020-07-23 18:37:44 +03:00
Nikolai Kochetov
dad9d369a1
Merge branch 'master' into bobrik-parallel-randes
2020-07-23 16:21:32 +03:00
Alexey Milovidov
6b4be2b9fe
Safe logic for incorrect MergeTreeSettings #10897
2020-07-23 15:31:09 +03:00
Azat Khuzhin
c97fb4b0e5
Remove unused bits in MergeTreeRangeReader ( #12671 )
...
Refs: #7769
Cc: @amosbird
2020-07-23 11:20:09 +03:00
alesapin
978f286625
Trying to fix FreeBSD build
2020-07-23 11:00:18 +03:00
alesapin
6fdcb59df7
Merge branch 'master' into forward_alter_error_to_client
2020-07-23 10:48:58 +03:00
Alexander Tokmakov
237accd154
Merge branch 'master' into database_atomic_improvements
2020-07-23 06:33:20 +03:00
alesapin
355150afce
Fix deadlock and better code
2020-07-22 22:29:54 +03:00
Artem Zuikov
2afd123eda
Refactoring: extract TreeOptimizer from SyntaxAnalyzer ( #12645 )
2020-07-22 20:13:05 +03:00
alesapin
8c8bdd5070
Fix tests and work with multiple mutations
2020-07-22 18:19:54 +03:00
Nikolai Kochetov
76d784be9c
Merge pull request #12613 from ClickHouse/remove-sort-description-from-streams
...
Remove sort description from streams
2020-07-22 17:35:05 +03:00
alesapin
de846e5e3c
Forward exception to client during alter
2020-07-22 15:36:19 +03:00
Nikolai Kochetov
b27066389a
Do not create ThreadPool for single thread.
2020-07-22 14:51:35 +03:00
zhang2014
580889f1e4
ISSUES-10572 fix bloom filter with const column
2020-07-22 17:42:12 +08:00
Alexander Tokmakov
9bcaaea3e0
Merge branch 'master' into database_atomic_improvements
2020-07-22 05:24:48 +03:00
alesapin
be106b662e
Comment and style
2020-07-21 18:19:41 +03:00
Nikolai Kochetov
486a4932c3
Fix tests.
2020-07-21 17:08:18 +03:00
alesapin
daaa5cc382
Add setting allow_non_metadata_alters
2020-07-21 17:05:30 +03:00
Nikolai Kochetov
755f15def3
Make MergeTreeSetIndex::checkInRange const.
2020-07-21 14:22:45 +03:00
alesapin
379a5a750a
Fix error message about granularity
2020-07-21 14:10:23 +03:00
Nikolai Kochetov
12c5e376c6
Remove mutable from RPNElement.
2020-07-21 14:02:58 +03:00
Nikolai Kochetov
cab9146ff3
Remove SortDescription from IBlockInputStream.
2020-07-21 11:05:52 +03:00
Nikolai Kochetov
d6583698a9
Merge branch 'master' into bobrik-parallel-randes
2020-07-20 19:05:28 +03:00
Nikolai Kochetov
0cc55781d8
Try fix tests.
2020-07-20 18:09:00 +03:00
Alexander Kuzmenkov
40ee6eba19
Merge pull request #12545 from ClickHouse/fix_load_part_without_checksums
...
Fix loading parts without checksums
2020-07-20 14:40:53 +03:00
Ivan Babrou
72622a9b00
Parallelize PK range and skipping index stages
...
This runs PK lookup and skipping index stages on parts
in parallel, as described in #11564 .
While #12277 sped up PK lookups, skipping index stage
may still be a bottleneck in a select query. Here we
parallelize both stages between parts.
On a query that uses a bloom filter skipping index to pick
2,688 rows out of 8,273,114,994 on a two day time span,
this change reduces latency from 10.5s to 1.5s.
2020-07-19 21:49:41 -07:00
Alexander Tokmakov
c27de9b424
fixes
2020-07-17 16:11:44 +03:00
alesapin
ac802a8e96
Merge pull request #12543 from ClickHouse/fix_marks_type_on_mutation
...
Fix incorrect adaptive granularity by default
2020-07-17 11:09:54 +03:00
Alexander Tokmakov
a612080df1
Merge branch 'master' into database_atomic_improvements
2020-07-17 00:43:59 +03:00
Anton Popov
97e8a88b30
Merge pull request #12277 from bobrik/ivan/exact-range-speedup
...
WIP: Optimize PK lookup for queries that match exact PK range
2020-07-16 19:17:50 +03:00
alesapin
b3ec3b1ed1
Fix loading parts without checksums
2020-07-16 13:54:49 +03:00
alesapin
42983c5416
Fix incorrect adaptive granularity by default
2020-07-16 12:10:15 +03:00
Anton Popov
c6423b2b28
lower read buffer size, while reading from compact parts
2020-07-15 14:53:23 +03:00
Alexander Tokmakov
09abe71b60
Merge branch 'master' into database_atomic_improvements
2020-07-15 13:07:08 +03:00
alesapin
614540eddf
Merge pull request #12382 from ClickHouse/clear-all-columns
...
Better errors for CLEAR/DROP columns (possibly in partitions)
2020-07-15 12:52:06 +03:00
alesapin
9e41fbca55
Remove check for drop detached partition
2020-07-14 16:56:30 +03:00
alesapin
014bb070ec
Fix tests
2020-07-14 11:19:39 +03:00
alesapin
1f576ee039
Some intermediate solution
2020-07-13 20:27:52 +03:00
alesapin
4a53264a86
Remove redundant and duplicated code
2020-07-13 19:19:08 +03:00
Alexander Tokmakov
fdb02edc7c
Merge branch 'master' into database_atomic_improvements
2020-07-13 17:28:48 +03:00
Alexander Tokmakov
04c3e7cab6
add implicit uuid macro
2020-07-13 16:18:38 +03:00
Anton Popov
a9530d2883
in-memory parts: fix reading from nested
2020-07-13 12:10:55 +03:00
alexey-milovidov
ae7eff98ed
Merge pull request #12433 from amosbird/np
...
Allow nullable key in MergeTree
2020-07-13 04:36:00 +03:00
Amos Bird
cac5a89169
Allow nullable key in MergeTree
2020-07-12 22:21:51 +08:00
Ivan Babrou
8784994d65
Allow conditions outside of PK with exact range
...
Conditions that are outside of PK are marked as `unknown` in `KeyCondition`,
so it's safe to allow them, as long as they are always combined by `AND`.
2020-07-11 18:59:26 -07:00
Ivan Babrou
d9d8d0242e
Optimize PK lookup for queries that match exact PK range
...
Existing code that looks up marks that match the query has a pathological
case, when most of the part does in fact match the query.
The code works by recursively splitting a part into ranges and then discarding
the ranges that definitely do not match the query, based on primary key.
The problem is that it requires visiting every mark that matches the query,
making the complexity of this sort of look up O(n).
For queries that match exact range on the primary key, we can find
both left and right parts of the range with O(log 2) complexity.
This change implements exactly that.
To engage this optimization, the query must:
* Have a prefix list of the primary key.
* Have only range or single set element constraints for columns.
* Have only AND as a boolean operator.
Consider a table with `(service, timestamp)` as the primary key.
The following conditions will be optimized:
* `service = 'foo'`
* `service = 'foo' and timestamp >= now() - 3600`
* `service in ('foo')`
* `service in ('foo') and timestamp >= now() - 3600 and timestamp <= now`
The following will fall back to previous lookup algorithm:
* `timestamp >= now() - 3600`
* `service in ('foo', 'bar') and timestamp >= now() - 3600`
* `service = 'foo'`
Note that the optimization won't engage when PK has a range expression
followed by a point expression, since in that case the range is not continuous.
Trace query logging provides the following messages types of messages,
each representing a different kind of PK usage for a part:
```
Used optimized inclusion search over index for part 20200711_5710108_5710108_0 with 9 steps
Used generic exclusion search over index for part 20200711_5710118_5710228_5 with 1495 steps
Not using index on part 20200710_5710473_5710473_0
```
Number of steps translates to computational complexity.
Here's a comparison for before and after for a query over 24h of data:
```
Read 4562944 rows, 148.05 MiB in 45.19249672 sec., 100966 rows/sec., 3.28 MiB/sec.
Read 4183040 rows, 135.78 MiB in 0.196279627 sec., 21311636 rows/sec., 691.75 MiB/sec.
```
This is especially useful for queries that read data in order
and terminate early to return "last X things" matching a query.
See #11564 for more thoughts on this.
2020-07-11 12:26:54 -07:00
Anton Popov
fbec940e0f
fix reading and check query for compact parts with different codecs
2020-07-11 02:33:36 +03:00
alexey-milovidov
e22547c29d
Merge pull request #12388 from ClickHouse/bloom-filter-arg-check
...
Check arguments of bloom filter index
2020-07-10 20:54:16 +03:00
alexey-milovidov
caef1d8e24
Update MergeTreeIndexFullText.cpp
2020-07-10 20:53:58 +03:00
Anton Popov
24f627e52c
fix reading from compact parts with different codecs
2020-07-10 18:57:10 +03:00
Alexey Milovidov
47eaffbe63
Additional checks
2020-07-10 11:21:40 +03:00
Alexey Milovidov
4b86f36d37
Check arguments of bloom filter index
2020-07-10 11:13:21 +03:00
Alexey Milovidov
276b3a0215
Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables #11905
2020-07-10 09:30:49 +03:00
alexey-milovidov
c16d8e094b
Merge pull request #12308 from ClickHouse/fix-codec-bad-exception-code
...
Fix wrong exception code in codecs Delta, DoubleDelta #12110
2020-07-10 08:40:46 +03:00
Anton Popov
80a62977f1
fix hashing in DataPartWriterCompact
2020-07-09 21:26:54 +03:00
Anton Popov
9f72baad1f
Merge remote-tracking branch 'upstream/master' into HEAD
2020-07-09 20:51:36 +03:00
Alexey Milovidov
6c458a6287
Fix race condition in ReplicatedMergeTreeQueue
2020-07-09 07:43:28 +03:00
Alexey Milovidov
ea970fd57c
Remove bad ugliness
2020-07-09 04:00:16 +03:00
alesapin
84f8bf1e14
Merge pull request #10697 from CurtizJ/polymorphic-parts
...
Polymorphic parts (in-memory format)
2020-07-08 13:25:45 +03:00
Anton Popov
416d9648c2
remove questionable functionality
2020-07-07 12:58:14 +03:00
Anton Popov
d6434f61dc
support codecs in compact parts
2020-07-07 03:15:02 +03:00
alesapin
3f90f65678
Merge remote-tracking branch 'origin' into CurtizJ-polymorphic-parts
2020-07-06 21:34:17 +03:00
Nikita Mikhaylov
2b539e6ace
fix style
2020-07-06 20:26:15 +03:00
Nikita Mikhaylov
d31ed58f01
done
2020-07-06 17:33:31 +03:00
myrrc
8c3417fbf7
ILIKE operator ( #12125 )
...
* Integrated CachingAllocator into MarkCache
* fixed build errors
* reset func hotfix
* upd: Fixing build
* updated submodules links
* fix 2
* updating grabber allocator proto
* updating lost work
* updating CMake to use concepts
* some other changes to get it building (integration into MarkCache)
* further integration into caches
* updated Async metrics, fixed some build errors
* and some other errors revealing
* added perfect forwarding to some functions
* fix: forward template
* fix: constexpr modifier
* fix: FakePODAllocator missing member func
* updated PODArray constructor taking alloc params
* fix: PODArray overload with n restored
* fix: FakePODAlloc duplicating alloc() func
* added constexpr variable for alloc_tag_t
* split cache values by allocators, provided updates
* fix: memcpy
* fix: constexpr modifier
* fix: noexcept modifier
* fix: alloc_tag_t for PODArray constructor
* fix: PODArray copy ctor with different alloc
* fix: resize() signature
* updating to lastest working master
* syncing with 273267
* first draft version
* fix: update Searcher to case-insensitive
* added ILIKE test
* fixed style errors, updated test, split like and ilike, added notILike
* replaced inconsistent comments
* fixed show tables ilike
* updated missing test cases
* regenerated ya.make
* Update 01355_ilike.sql
Co-authored-by: myrrc <me-clickhouse@myrrec.space>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
2020-07-05 18:57:59 +03:00
alexey-milovidov
e05283d5d3
Merge pull request #12060 from ClickHouse/fix-12053
...
Check wrong type for filter.
2020-07-04 23:46:34 +03:00
Anton Popov
73676f5022
Improve performace of reading in order of sorting key. ( #11696 )
...
* simplify reading in order of sorting key
* add perf test for reading many parts
* Revert "simplify reading in order of sorting key"
This reverts commit 7267d7c46e
2020-07-04 15:48:51 +03:00
alesapin
8dc204350f
Fix version column in replicated version collapsing merge tree ( #12121 )
2020-07-04 10:35:17 +03:00
Anton Popov
ca346ea13c
rename fsync-related settings
2020-07-03 02:41:37 +03:00
Anton Popov
4422df2e37
Merge remote-tracking branch 'upstream/master' into HEAD
2020-07-02 20:18:21 +03:00
Anton Popov
71059e4dff
Merge pull request #12062 from nvartolomei/nv/set-index-tuple-types
...
Try fix pk in tuple performance
2020-07-02 15:51:03 +03:00
Nikolai Kochetov
79cd33a567
Fix tests.
2020-07-02 10:44:47 +03:00
Nicolae Vartolomei
3854ce6d84
Rewrite Set lookup to make it more readable
2020-07-01 15:05:54 +01:00
Anton Popov
d45697a1b5
in-memory-parts: lazy creation of WAL
2020-06-30 21:47:12 +03:00
Nikolai Kochetov
3585700506
Fix tests.
2020-06-30 19:41:43 +03:00
Nicolae Vartolomei
8f1845185e
Try fix pk in tuple performance
...
Possible approach for fixing #10574
The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).
However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.
The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.
2020-06-30 16:33:38 +01:00
Nikolai Kochetov
66da0733ab
Check type of filter for prewhere.
2020-06-30 17:20:27 +03:00
Vladimir Chebotarev
bb8da71eff
Moves task shall be started if new storage policy needs them.
2020-06-30 15:36:44 +03:00
Anton Popov
53e955c6dd
several fixes
2020-06-29 23:36:18 +03:00
Anton Popov
a43cb93be5
remove questionable functionality
2020-06-29 18:46:50 +03:00
Alexey Milovidov
cc597bb30b
Implement for ReplicatedMergeTree
2020-06-28 22:39:31 +03:00
Alexey Milovidov
1f9c27de34
Fix clang-tidy
2020-06-28 22:39:31 +03:00
Alexey Milovidov
b6e31f0a5e
Only allow Enum extension within the same width
2020-06-28 22:39:31 +03:00
Alexey Milovidov
e4d8cc2b44
Slightly more safe
2020-06-28 22:39:31 +03:00
Alexey Milovidov
97ad23b905
Allow to ALTER partition key in some cases
2020-06-28 22:39:31 +03:00
Alexey Milovidov
678d127503
Whitespace
2020-06-28 22:39:31 +03:00
Alexey Milovidov
c4d5ca485e
Remove the usage of obsolete setting
2020-06-28 22:39:31 +03:00
Val
8273248c4e
add log_name_to_exec to dbreplicated
2020-06-28 18:43:30 +03:00
Alexey Milovidov
1462a66d1e
Fix typos
2020-06-27 22:05:00 +03:00
Anton Popov
b2aa565a37
allow to turn on fsync on inserts, merges and fetches
2020-06-27 17:52:16 +03:00
Val
5e076b464e
add replicated db snapshot, integration test, repl alter queries, etc
...
add an option to create replicated tables within replicated db without specifying zk path and replica id
add replicated sch pool
disable replication of alter queries for replicated tables in replicated dbs
snapshot prototype. amend of replicated db workflow
add prototype of integration tests for replicated db
2020-06-27 16:58:15 +03:00
Alexey Milovidov
46caa211d5
Better diagnostics of "Replica {} appears to be already active" message
2020-06-27 16:55:00 +03:00
alesapin
9710a67704
Fix style check
2020-06-26 15:27:12 +03:00
alesapin
6caf2e7a3f
Fix protocol check
2020-06-26 14:38:37 +03:00
alesapin
6f1824f0ea
Correct merge with master
2020-06-26 14:30:23 +03:00
alesapin
e9c47dc89c
Merge branch 'master' into CurtizJ-polymorphic-parts
2020-06-26 14:27:19 +03:00
Alexey Milovidov
8eed47857b
Fix estimation of the number of marks for various thresholds
2020-06-25 23:20:22 +03:00
Alexey Milovidov
8872417d00
Respect direct_io/mmap settings while reading secondary indices
2020-06-25 22:31:54 +03:00
Alexey Milovidov
5608f15749
Revive mmap IO
2020-06-25 22:15:41 +03:00
Anton Popov
f8f67a788e
allow to turn on fsync on inserts and merges
2020-06-25 19:55:45 +03:00
alesapin
63f8f89ca0
bump CI
2020-06-24 19:06:01 +03:00
alesapin
08f5ed6da3
Fix unitialized memory
2020-06-24 17:12:44 +03:00
Nikolai Kochetov
f7f16fdbf7
Merge pull request #11895 from ClickHouse/fix-nullable-prewhere-type-3
...
Fix nullable prewhere type 3
2020-06-24 14:32:08 +03:00
Anton Popov
8c3f496826
add test with alters and in-memory parts
2020-06-24 01:16:29 +03:00
Nikolai Kochetov
658a2d0dc0
Fix anotehr one nullable prewhere column.
2020-06-23 21:52:32 +03:00
alesapin
4185fe9975
Merge branch 'master' into atomic_metadata5
2020-06-23 11:13:05 +03:00
alesapin
292016e045
Add missed lock
2020-06-23 11:04:43 +03:00
Anton Popov
78d28be8cf
check in-memory parts, comments and style-fixes
2020-06-22 21:57:50 +03:00
Nikolai Kochetov
ae21aca3a3
Fix header for nullable prewhere column.
2020-06-22 19:01:59 +03:00
alesapin
c76b4b3481
Clang-tidy fixes
2020-06-22 18:51:11 +03:00
alesapin
b1e8976df4
Merge with master
2020-06-22 12:04:27 +03:00
alesapin
b9e74f4e82
Merge branch 'master' into atomic_metadata5
2020-06-22 12:03:53 +03:00
Alexey Milovidov
0e6fb617aa
Merge branch 'master' into manual-write-duplicate-parts-to-replicas
2020-06-22 01:54:03 +03:00
Alexey Milovidov
ad672f7ce2
Whitespace
2020-06-20 10:26:33 +03:00
Alexey Milovidov
b8769e1ada
Fix style
2020-06-20 10:24:51 +03:00
alesapin
4c0879ae30
Better logging in storages
2020-06-19 20:17:13 +03:00
alesapin
a2e7e9f232
Remove unused constructor and rename method
2020-06-19 18:21:48 +03:00
alesapin
3c47faa9da
Move partition key initialization into registerMergeTree
2020-06-19 18:14:08 +03:00
Alexey Milovidov
5211a42c04
Remove leader election, step 3
2020-06-19 17:18:58 +03:00
alesapin
c666763cc0
Remove unused method better
2020-06-18 20:09:06 +03:00
alesapin
e888dafdc2
Remove unused method
2020-06-18 20:03:42 +03:00
alesapin
d79982f497
Better locks in Storages
2020-06-18 19:10:47 +03:00
alesapin
d4c49816ab
Really atomic metadata
2020-06-18 14:02:31 +03:00
alesapin
ebb36bec8a
Merge branch 'master' into atomic_metadata5
2020-06-18 11:57:16 +03:00
