thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
72,039 Commits 9,338 Branches 1,783 Tags 2.1 GiB
24010689e6
Commit Graph

675 Commits

Author SHA1 Message Date
alesapin
23a7c00ba4
Merge branch 'master' into fix_alter_decimal 2021-03-16 10:24:46 +03:00
Alexey Milovidov
671395e8c8 Most likely improve performance 2021-03-15 22:23:27 +03:00
Mike Kot
406d037ebb Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-15 18:41:47 +03:00
Alexey Milovidov
eaeb4a3bf0 Merge branch 'master' into DateTime64_extended_range 2021-03-15 16:59:32 +03:00
alesapin
69b8ded578 Fix alter modify for decimal columns when type size doesn't change 2021-03-15 11:22:15 +03:00
Azat Khuzhin
402bf77783 Fix concurrent OPTIMIZE and DROP for ReplicatedMergeTree 
Found with fuzzer [1] for 00992_system_parts_race_condition_zookeeper:

    2021.03.13 11:12:30.385188 [ 42042 ] {2d3a8e17-26be-47c1-974f-bd2c9fc7c3af} <Debug> executeQuery: (from [::1]:58192, using production parser) (comment: '/usr/share/clickhouse-test/queries/1_stateful/00153_aggregate_arena_race.sql') CREATE TABLE alter_tabl
    e (a UInt8, b Int16, c Float32, d String, e Array(UInt8), f Nullable(UUID), g Tuple(UInt8, UInt16)) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_3.alter_table', 'r1') ORDER BY a PARTITION BY b % 10 SETTINGS old_parts_lifetime = 1, cleanup_delay_p
    eriod = 1, cleanup_delay_period_random_add = 0;
    ...
    2021.03.13 11:12:30.678387 [ 42042 ] {528cafc5-a02b-4df8-a531-a9a98e37b478} <Debug> executeQuery: (from [::1]:58192, using production parser) (comment: '/usr/share/clickhouse-test/queries/1_stateful/00153_aggregate_arena_race.sql') CREATE TABLE alter_table2 (a UInt8, b Int16, c Float32, d String, e Array(UInt8), f Nullable(UUID), g Tuple(UInt8, UInt16)) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_3.alter_table', 'r2') ORDER BY a PARTITION BY b % 10 SETTINGS old_parts_lifetime = 1, cleanup_delay_period = 1, cleanup_delay_period_random_add = 0;
    ...
    2021.03.13 11:12:40.671994 [ 4193 ] {d96ee93c-69b0-4e89-b411-16c382ae27a8} <Debug> executeQuery: (from [::1]:59714, using production parser) (comment: '/usr/share/clickhouse-test/queries/1_stateful/00153_aggregate_arena_race.sql') OPTIMIZE TABLE alter_table FINAL
    ...
    2021.03.13 11:12:40.990174 [ 2298 ] {a80f9306-3a73-4778-a921-db53249247e3} <Debug> executeQuery: (from [::1]:59768, using production parser) (comment: '/usr/share/clickhouse-test/queries/1_stateful/00153_aggregate_arena_race.sql') DROP TABLE alter_table;
    ...
    2021.03.13 11:12:41.333054 [ 2298 ] {a80f9306-3a73-4778-a921-db53249247e3} <Debug> test_3.alter_table (d4fedaca-e0f6-4c22-9a4f-9f4d11b6b705): Removing part from filesystem 7_0_0_0
    ...
    2021.03.13 11:12:41.335380 [ 2298 ] {a80f9306-3a73-4778-a921-db53249247e3} <Debug> DatabaseCatalog: Waiting for table d4fedaca-e0f6-4c22-9a4f-9f4d11b6b705 to be finally dropped
    ...
    2021.03.13 11:12:41.781032 [ 4193 ] {d96ee93c-69b0-4e89-b411-16c382ae27a8} <Debug> test_3.alter_table (d4fedaca-e0f6-4c22-9a4f-9f4d11b6b705): Waiting for queue-0000000085 to disappear from r2 queue
    ...
    2021.03.13 11:12:41.900039 [ 371 ] {} <Trace> test_3.alter_table2 (ReplicatedMergeTreeQueue): Not executing log entry queue-0000000085 of type MERGE_PARTS for part 7_0_0_1 because part 7_0_0_0 is not ready yet (log entry for that part is being processed).
    2021.03.13 11:12:41.900213 [ 365 ] {} <Trace> test_3.alter_table2 (ReplicatedMergeTreeQueue): Cannot execute alter metadata queue-0000000056 with version 22 because another alter 21 must be executed before
    2021.03.13 11:12:41.900231 [ 13762 ] {} <Trace> test_3.alter_table2 (ae877c49-0d30-416d-9afe-27fd457d8fc4): Executing log entry to merge parts -7_0_0_0 to -7_0_0_1
    2021.03.13 11:12:41.900330 [ 13762 ] {} <Debug> test_3.alter_table2 (ae877c49-0d30-416d-9afe-27fd457d8fc4): Don't have all parts for merge -7_0_0_1; will try to fetch it instead
    ...

  [1]: https://clickhouse-test-reports.s3.yandex.net/21691/eb3710c164b991b8d4f86b1435a65f9eceb8f1f5/stress_test_(address).html#fail1
 2021-03-14 22:35:50 +03:00
alexey-milovidov
53e21443eb
Merge branch 'master' into DateTime64_extended_range 2021-03-14 12:27:17 +03:00
Anton Ivashkin
ace628d565 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-12 12:58:32 +03:00
Anton Ivashkin
109cb634e1 Remove debug lines 2021-03-11 12:08:19 +03:00
Anton Ivashkin
6f7800ecb4 Fix PVS detected errors 2021-03-10 16:27:08 +03:00
Anton Ivashkin
aff13c0c52 Make method StorageReplicatedMergeTree::fetchExistsPart 2021-03-09 20:49:50 +03:00
Anton Ivashkin
265d293934 Use 'merge on single replica' option instead of zookeeper lock 2021-03-09 20:39:55 +03:00
alesapin
d80c2cef06 Slightly better 2021-03-09 11:45:41 +03:00
Amos Bird
2ec20c5d23
update and add tests 2021-03-08 17:38:07 +08:00
Amos Bird
e6522e1ebe
JBOD data balancer 2021-03-08 11:10:35 +08:00
alesapin
5b3161e0b5 Get rid of const_cast 2021-03-05 20:24:06 +03:00
Anton Ivashkin
d08b481660 Fixes by review responces 2021-03-05 19:20:38 +03:00
Nikolai Kochetov
a669f7d641 Merge branch 'master' into refactor-actions-dag 2021-03-05 18:21:14 +03:00
Nikolai Kochetov
9a39459888 Refactor ActionsDAG 2021-03-04 20:38:12 +03:00
Amos Bird
93b661ad5a
partition id pruning 2021-03-04 19:43:03 +08:00
Anton Ivashkin
e69124a0a6 Merge master 2021-03-04 13:26:40 +03:00
Alexey Milovidov
4e8239e098 Merge branch 'master' into DateTime64_extended_range 2021-03-03 23:43:20 +03:00
Mike Kot
6ea574525c Small fixes regarding the review 2021-03-03 16:51:41 +03:00
alesapin
06678d650d Merge branch 'master' into fix_alter_partition_key 2021-03-02 13:43:41 +03:00
Mike Kot
4e1ac185b5 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-01 20:58:07 +03:00
Anton Ivashkin
98065ec56e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-03-01 13:33:40 +03:00
Anton Ivashkin
3c11d44494 Add description for getUniqueId method, fix typos 2021-03-01 13:31:36 +03:00
alesapin
9c8afbeb53 Fix alter modify query for partition key and other metadata fields 2021-03-01 12:59:19 +03:00
Nikolai Kochetov
976dbe8077
Merge pull request #20341 from ClickHouse/filter-push-down 
Filter push down
 2021-03-01 12:35:06 +03:00
alexey-milovidov
b8fba768e5
Merge pull request #21264 from ClickHouse/fix_zookeeper_update 
Fix several bugs with ZooKeeper client
 2021-02-28 01:57:04 +03:00
alesapin
9e93d7f507 Fix tidy and add comments 2021-02-27 11:07:14 +03:00
Nikolai Kochetov
d91b8a3acb
Merge branch 'master' into filter-push-down 2021-02-26 19:33:27 +03:00
Nikolai Kochetov
d328bfa41f Review fixes. Add setting max_optimizations_to_apply. 2021-02-26 19:29:56 +03:00
Anton Ivashkin
5b267b7eec Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557 Fixes by review response 2021-02-26 12:48:57 +03:00
Mike Kot
2b3b335eda Exit fixes 2021-02-25 21:41:09 +03:00
Mike Kot
b2c898f58c Adding the part when found 2021-02-25 21:25:55 +03:00
fastio
fea2836673 get zookeeper from global context 2021-02-25 14:19:58 +08:00
fastio
3ddb729e4a fix metadata leak when drop Replicated*MergeTree 2021-02-25 14:19:58 +08:00
Vasily Nemkov
4fcc23ec9a Fixed build for GCC-10 2021-02-24 17:08:43 +02:00
Vasily Nemkov
2d03d330bc Extended range of DateTime64 to years 1925 - 2238 
The Year 1925 is a starting point because most of the timezones
switched to saner (mostly 15-minutes based) offsets somewhere
during 1924 or before. And that significantly simplifies implementation.

2238 is to simplify arithmetics for sanitizing LUT index access;
there are less than 0x1ffff days from 1925.

* Extended DateLUTImpl internal LUT to 0x1ffff items, some of which
  represent negative (pre-1970) time values.
  As a collateral benefit, Date now correctly supports dates up to 2149
  (instead of 2106).
* Added a new strong typedef ExtendedDayNum, which represents dates
  pre-1970 and post 2149.
* Functions that used to return DayNum now return ExtendedDayNum.
* Refactored DateLUTImpl to untie DayNum from the dual role of being
  a value and an index (due to negative time). Index is now a different
  type LUTIndex with explicit conversion functions from DatNum, time_t,
  and ExtendedDayNum.
* Updated DateLUTImpl to properly support values close to epoch start
  (1970-01-01 00:00), including negative ones.
* Reduced resolution of DateLUTImpl::Values::time_at_offset_change
  to multiple of 15-minutes to allow storing 64-bits of time_t in
  DateLUTImpl::Value while keeping same size.
* Minor performance updates to DateLUTImpl when building month LUT
  by skipping non-start-of-month days.
* Fixed extractTimeZoneFromFunctionArguments to work correctly
  with DateTime64.
* New unit-tests and stateless integration tests for both DateTime
  and DateTime64.
 2021-02-24 17:08:35 +02:00
Alexander Tokmakov
2a36d6cb55 review suggestions 2021-02-20 02:41:58 +03:00
Mike Kot
eb1826a5e3 Trying to figure out why iteration doesn't work 2021-02-19 17:28:29 +03:00
Mike Kot
f1ef382cf9 Added part_checksum to Replicated...Entry serialization. 2021-02-19 16:04:12 +03:00
Alexander Tokmakov
1aac7b3471 Merge branch 'master' into database_replicated 2021-02-17 00:39:56 +03:00
Mike Kot
de88a7ca94 Fixed the UB with maybe-nullptr member access 2021-02-16 18:36:30 +03:00
Mike Kot
ca83775711 Multiple small hotfixes 
Small fixes

Some fix for old bug

Another old code fix
 2021-02-16 16:39:18 +03:00
Mike Kot
fefc7234df Replaced the part lookup algo to "by hash only", comments on test stub 2021-02-16 16:00:26 +03:00
Alexander Tokmakov
bf6f64a3fb Merge branch 'master' into database_replicated 2021-02-16 01:28:19 +03:00
Mike Kot
8182482cbd Add test stub 2021-02-15 21:06:20 +03:00
Mike Kot
0a02fe913a Add some code for the checksum pre-calculation in ...BlockOutputStream 
Added the comment explaining the double-get for the zookeeper header for
the part.
 2021-02-15 20:31:58 +03:00
Mike Kot
73f4740be5 Trying to re-write the solution by adding new command type 
ATTACH_PART into the replicated log.

The LogEntry now also has the pre-calculated part checksum for this
entry type, which is later used while searching in the detached/ folder
 2021-02-15 18:06:48 +03:00
Mike Kot
cd32803709 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-02-15 15:42:17 +03:00
tavplubix
3f86ce4c67
Update StorageReplicatedMergeTree.cpp 2021-02-15 15:04:30 +03:00
Mike Kot
feff4c6a22 Started adding the new "ATTACH_PART" command into the replicated log 
The original ticket idea was to search for the possibly available data
into the /detached folders for the GET_PART command, but
@tavplubix pointed out this would be quite expensive for an every
fetch.

So a new command is going to be introduced, ATTACH_PART, which will
cover ALTER TABLE ATTACH PART and only for which the search will start.
 2021-02-15 01:59:13 +03:00
alesapin
3253638969 Fix backoff for failed background tasks in replicated merge tree 2021-02-11 14:46:18 +03:00
Nicolae Vartolomei
b153e8c190 Add support for custom fetchPart timeouts 2021-02-08 19:44:02 +00:00
Alexander Tokmakov
78c1d69b8c better code 2021-02-08 22:36:17 +03:00
tavplubix
ac477d9850
Merge pull request #19771 from ClickHouse/thread_state_improvements 
Minor code improvements around ThreadStatus
 2021-02-08 22:34:55 +03:00
Anton Ivashkin
4d44d75bc7 Fix build after merge one more time 2021-02-08 14:45:10 +03:00
Anton Ivashkin
e64c63c611 Merge master 2021-02-05 20:10:06 +03:00
Anton Ivashkin
df6c882aab Fix build after merge 2021-02-05 18:52:40 +03:00
Alexander Tokmakov
87502d0220 Merge branch 'thread_state_improvements' into database_replicated 2021-02-03 20:19:35 +03:00
Alexander Tokmakov
d010f97db0 Merge branch 'master' into database_replicated 2021-02-03 20:13:25 +03:00
Maksim Kita
d0151de4bb
Merge pull request #19608 from kreuzerkrieg/Add_IStoragePolicy_interface 
Add IStoragePolicy interface
 2021-02-02 11:03:20 +03:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
alesapin
2881c830e3 Merge branch 'master' into fix_rare_bug_after_part_corruption 2021-01-28 23:16:52 +03:00
tavplubix
10460313ad
Update StorageReplicatedMergeTree.cpp 2021-01-28 17:48:09 +03:00
Alexander Tokmakov
ffaa8e34a6 minor code improvements around ThreadStatus 2021-01-28 16:57:36 +03:00
Alexander Tokmakov
52e5c0aad7 fix thread status 2021-01-28 16:48:17 +03:00
alesapin
5622e6daa6 Fix rare max_number_of_merges_with_ttl_in_pool limit overrun for non-replicated MergeTree 2021-01-27 14:56:12 +03:00
alesapin
01c8b9e1b1 Fix rare bug when some replicated operations (like mutation) cannot process some parts after data corruption 2021-01-27 13:07:18 +03:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
Alexander Tokmakov
3bd4d97353 Merge branch 'master' into database_replicated 2021-01-25 14:19:04 +03:00
Anton Ivashkin
357d98eb36 Merge master 2021-01-20 12:23:03 +03:00
Anton Ivashkin
eba98b04b0 Zero copy replication over S3: Hybrid storage support 2021-01-18 19:16:45 +03:00
Alexander Tokmakov
7f97a11c84 Merge branch 'master' into database_replicated 2021-01-18 17:09:39 +03:00
alesapin
4ee96869a2 Don't wait forever for log update after table was dropped 2021-01-18 15:15:07 +03:00
alesapin
bfc27254b2 Avoid redundant exception while dropping part 2021-01-14 11:07:13 +03:00
Mike Kot
6b949109f1 marked places to edit 2021-01-12 21:46:03 +03:00
alesapin
dead1016d6 Fix deduplication block names parsing 2021-01-12 13:55:02 +03:00
fastio
a1d0c04e68 fix build 2021-01-08 13:10:00 +08:00
fastio
ea047b951f Expand macros for fetchPartition 2021-01-07 22:13:17 +08:00
alexey-milovidov
a08db94343
Revert "Add metrics for part number in MergeTree in ClickHouse" 2021-01-07 16:40:52 +03:00
alexey-milovidov
f91626e7ff
Merge pull request #17838 from weeds085490/dev/add_metrics_for_parts 
Add metrics for part number in MergeTree in ClickHouse
 2021-01-07 15:27:04 +03:00
weeds085490
5f5b86b485 Merge remote-tracking branch 'origin' into dev/add_metrics_for_parts 2021-01-06 17:32:45 +08:00
sundy-li
6cc0668af4 Add one more argument 2021-01-04 16:21:04 +08:00
sundy-li
8d7fe410cd Fix Logger with unmatched arg size 2021-01-04 16:15:13 +08:00
alexey-milovidov
1bcdf37c36
Merge pull request #18614 from CurtizJ/fix-empty-parts 
Fix removing of empty parts in tables with old syntax
 2020-12-30 17:20:31 +03:00
Anton Popov
6336fbf1df fix removing of empty parts in tables with old syntax 2020-12-29 20:16:57 +03:00
alesapin
a50615a22b More correct error code 2020-12-25 16:38:04 +03:00
roverxu
b339b9dfd0 fix consistence 2020-12-22 17:38:15 +08:00
Alexey Milovidov
9be5fa9ef2 Merge branch 'master' into Enmk-Optimize_deduplicate 2020-12-20 09:57:10 +03:00
alesapin
f45993cd1e More logs during quorum insert 2020-12-17 19:13:01 +03:00
Anton Ivashkin
0f0500ca0c Merge master 2020-12-16 18:31:13 +03:00
Nikolai Kochetov
6defcbb662
Merge branch 'master' into optimize-data-on-insert 2020-12-15 16:50:42 +03:00
Azat Khuzhin
5b3ab48861 More forward declaration for generic headers 
The following headers are pretty generic, so use forward declaration as
much as possible:
- Context.h
- Settings.h
- ConnectionTimeouts.h
(Also this shows that some missing some includes -- this has been fixed)

And split ConnectionTimeouts.h into ConnectionTimeoutsContext.h (since
module part cannot be added for it, due to recursive build dependencies
that will be introduced)

Also remove Settings from the RemoteBlockInputStream/RemoteQueryExecutor
and just pass the context, since settings was passed only in speicifc
places, that can allow making a copy of Context (i.e. Copier).

Approx results (How much units will be recompiled after changing file X?):

- ConnectionTimeouts.h
  - mainline: 100

- Context.h:
  - mainline: ~800
  - patched:  415

- Settings.h:
  - mainline: 900-1K
  - patched:  440 (most of them because of the Context.h)
 2020-12-12 17:43:10 +03:00
Kruglov Pavel
e19eb6f17a
Merge branch 'master' into optimize-data-on-insert 2020-12-08 15:57:46 +03:00
Vasily Nemkov
70ea507dae OPTIMIZE DEDUPLICATE BY columns 
Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on.

Following syntax variants are now supported:

OPTIMIZE TABLE table DEDUPLICATE; -- the old one
OPTIMIZE TABLE table DEDUPLICATE BY *;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT (colX, colY);
OPTIMIZE TABLE table DEDUPLICATE BY col1,col2,col3;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex');
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT (colX, colY);

Note that * behaves just like in SELECT: MATERIALIZED, and ALIAS columns are not used for expansion.
Also, it is an error to specify empty list of columns, or write an expression that results in an empty list of columns, or deduplicate by an ALIAS column.
Column transformers other than EXCEPT are not supported.
 2020-12-07 09:44:07 +03:00
alesapin
27c3301083
Merge pull request #17800 from nvartolomei/nv/waitForAllReplicasToProcessLogEntry-foreign-shard 
Update StorageReplicatedMergeTree::waitForAllReplicasToProcessLogEntry to support waiting on foreign shards / tables
 2020-12-05 16:15:46 +03:00
Pavel Kruglov
5ae6c6dab9 Fix build error 2020-12-04 20:40:28 +03:00
Pavel Kruglov
905ba78adc Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-12-04 18:56:46 +03:00
Pavel Kruglov
9dbced0474 Pass setting instead of context 2020-12-04 17:01:59 +03:00
Nicolae Vartolomei
796aee032d Update StorageReplicatedMergeTree::waitForAllReplicasToProcessLogEntry to support waiting on foreign shards / tables 
This is not used anywhere yet but needed for an upcoming PR for part movement between shards.
 2020-12-04 13:01:12 +00:00
Anton Popov
cab9855dd1
Update StorageReplicatedMergeTree.cpp 2020-12-03 16:54:05 +03:00
Anton Popov
cd1917c7a6
Merge branch 'master' into optimize_final_optimization 2020-12-03 16:52:51 +03:00
Alexander Tokmakov
19c8399eb0 Merge branch 'master' into database_replicated 2020-12-01 17:33:07 +03:00
alexey-milovidov
f4a61ac3c3
Merge pull request #17527 from ucasFL/spelling 
fix spelling errors
 2020-11-29 13:45:42 +03:00
feng lv
7e3524caa1 fix spelling errors 2020-11-28 08:17:20 +00:00
Alexander Tokmakov
9e3fd3c170 Merge branch 'master' into database_replicated 2020-11-27 17:08:34 +03:00
alexey-milovidov
dfae1efbbd
Merge pull request #17070 from fastio/master 
Support multiple ZooKeeper clusters
 2020-11-27 10:38:01 +03:00
nikitamikhaylov
72c7cd6693 replace Context& to Settings& 2020-11-25 16:47:32 +03:00
Alexander Tokmakov
5ccf975a4a Merge branch 'master' into database_replicated 2020-11-24 14:38:02 +03:00
Alexander Tokmakov
dad21ee684 maintain metadata in zk 2020-11-24 13:24:39 +03:00
nikitamikhaylov
68bef22fda Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-sequential-consistency 2020-11-23 16:28:35 +03:00
Alexander Tokmakov
7ab4445e99 try another approach 2020-11-20 19:06:27 +03:00
Pavel Kruglov
ca3fe49a2a Make setting global 2020-11-20 17:29:13 +03:00
Nicolae Vartolomei
94293ca3ce Assign UUIDs to parts only when configured to do so 
Avoid breaking backwards compatibility by default for now.
 2020-11-20 13:49:17 +00:00
Pavel Kruglov
3ee4972586 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-11-20 01:23:51 +03:00
Pavel Kruglov
4c30857759 Minor change 2020-11-20 01:22:40 +03:00
Nicolae Vartolomei
746f8e45f5 All new parts must have uuids 2020-11-19 13:18:03 +00:00
Peng Jian
10cefe4f51 fix code style 2020-11-19 15:44:47 +08:00
Peng Jian
3c86c8b3c9 fix test cases 2020-11-19 15:44:47 +08:00
Peng Jian
091f7065cd fix build 2020-11-19 15:44:47 +08:00
Peng Jian
2df0c3e0b0 Add helper function for extracting zookeeper name and path. 2020-11-19 15:44:47 +08:00
Peng Jian
a0683ce460 Support mulitple ZooKeeper clusters 2020-11-19 15:44:47 +08:00
Pavel Kruglov
547ec19fb3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-11-18 12:01:59 +03:00
Mikhail Filimonov
234c671e52
After CR fixes 2020-11-18 08:45:44 +01:00
Mikhail Filimonov
97fef77ed1
execute_merges_on_single_replica 2020-11-18 08:45:33 +01:00
Pavel Kruglov
6a57c0a8cf Move merge in MergeTreeDataWriter 2020-11-13 10:55:56 +03:00
alesapin
8d3858fc22
Merge pull request #16895 from CurtizJ/remove-empty-parts 
Remove empty parts after they were pruned by TTL [2]
 2020-11-13 10:00:18 +03:00
alesapin
bbd85c3b2d
Merge pull request #16390 from ClickHouse/add_separate_pool_for_fetches 
Add separate pool for fetches
 2020-11-12 22:42:28 +03:00
Anton Popov
4cecd89e84 fix dropping of empty parts 2020-11-12 20:37:54 +03:00
alesapin
2cac5e5d97
Merge pull request #16875 from CurtizJ/refactor-alter-partition 
Avoid code duplication in alterPartition
 2020-11-12 13:19:59 +03:00
alesapin
a8d466a645 Merge branch 'master' into add_separate_pool_for_fetches 2020-11-12 12:51:02 +03:00
tavplubix
058aa8f85e
Merge pull request #16824 from ClickHouse/replace_stringstreams_with_buffers 
Replace std::*stringstreams with DB::*Buffers
 2020-11-12 01:11:44 +03:00
Anton Popov
6f0dc08ba7 avoid code duplication in alterPartition 2020-11-11 16:34:07 +03:00
Alexander Tokmakov
7da5e92754 fix 2020-11-11 13:41:12 +03:00
Pavel Kruglov
287d71b363 Fix build 2020-11-11 13:34:32 +03:00
Pavel Kruglov
dd06ff4f94 Minor change 2020-11-11 00:02:11 +03:00
Pavel Kruglov
c648f62629 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize_final_optimization 2020-11-10 22:58:21 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
Pavel Kruglov
9120189d8a Add SelectPartsDecision enum class 2020-11-10 17:42:56 +03:00
Nikolai Kochetov
195c941c4e Merge branch 'master' into storage-read-query-plan 2020-11-10 15:02:22 +03:00
alesapin
2e357516a6 Merge branch 'master' into add_separate_pool_for_fetches 2020-11-10 13:53:08 +03:00
Nikolai Kochetov
046830967c Try fix tests. 2020-11-10 13:26:26 +03:00
Vladimir Chebotarev
059357d51e
ALTER UPDATE/DELETE ... IN PARTITION with partition pruning in ReplicatedMergeTree (#13403) 
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-11-10 13:23:46 +03:00
alesapin
e347f4b72f Add some comments 2020-11-09 12:14:20 +03:00
alesapin
72046313d4 Merge branch 'master' into nvartolomei-drop-part 2020-11-09 10:02:02 +03:00
alesapin
4e846d2197 Merge branch 'master' into add_separate_pool_for_fetches 2020-11-09 10:00:21 +03:00
alexey-milovidov
0e6ae4aff7
Merge pull request #16253 from amosbird/pf 
Prune partition in verbatim way.
 2020-11-08 18:58:02 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream 
Fix "server failed to start" error
 2020-11-08 14:27:08 +03:00
Alexey Milovidov
5314185e25 Merge branch 'master' into azat-optimize_skip_unused_shards-optimization 2020-11-08 00:17:59 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Nikolai Kochetov
c10f733587 Merge branch 'master' into storage-read-query-plan 2020-11-06 15:43:46 +03:00
Amos Bird
2b0085c106
Pruning is different from counting 2020-11-06 19:58:03 +08:00
Amos Bird
30bf5e6d26
Prune partition in verbatim way. 2020-11-06 09:56:13 +08:00
alexey-milovidov
ee3e289ec2
Merge pull request #15983 from ClickHouse/no_background_pool_no_more 
Replace BackgroundProcessingPool with SchedulePool task and ThreadPool.
 2020-11-05 21:38:22 +03:00
Anton Ivashkin
1742fb3256 Merge master 2020-11-03 12:27:16 +03:00
alesapin
da8ac948bf Disallow to drop parts with parallel quorum 2020-11-03 12:24:10 +03:00
alesapin
294489f6ad Merge branch 'master' into nvartolomei-drop-part 2020-11-03 10:19:03 +03:00
alesapin
dac6705995 DROP PART and clear blocks in a single zookeeper op 2020-11-02 20:30:59 +03:00
alesapin
67d22b3e57 Merge branch 'master' into nvartolomei-drop-part 2020-11-02 19:09:09 +03:00
alesapin
7f8560e0c0
Merge pull request #16538 from amosbird/fp2 
Allow fetching duplicate parts to detach
 2020-11-02 16:10:42 +03:00
Amos Bird
18c5abc44e
Allow fetching duplicate parts to detach 2020-10-30 10:55:50 +08:00
alesapin
40fc512e79 Merge branch 'master' into no_background_pool_no_more 2020-10-29 12:53:34 +03:00
alesapin
0931658c2f Merge branch 'master' into add_separate_pool_for_fetches 2020-10-28 14:20:56 +03:00
tavplubix
4a63f1271b
Merge pull request #16244 from ClickHouse/fix_race_data_parts_exchange_endpoint 
Fix race on StorageReplicatedMergeTree::data_parts_exchange_endpoint
 2020-10-28 13:25:55 +03:00
alesapin
47ebc9f74e Add separate pool for fetches 2020-10-26 14:02:47 +03:00
hchen9
824c04f873 Fix conflict code 2020-10-25 18:12:17 -07:00
hchen9
feaab260d8 Fix conflict 2020-10-23 12:54:33 -07:00
alesapin
39e47c5338 Less strange lambdas 2020-10-23 11:54:00 +03:00
alesapin
7a63b373f9 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 15:47:11 +03:00
Nikolai Kochetov
7fa045cff8 Merge branch 'master' into storage-read-query-plan 2020-10-22 13:31:10 +03:00
Anton Ivashkin
f833501c77 Merge branch 'master' into s3_zero_copy_replication 2020-10-22 11:17:21 +03:00
alexey-milovidov
adeba6bdd8
Merge pull request #15074 from amosbird/btc 
Extend trivial count optimization.
 2020-10-22 02:50:57 +03:00
Alexander Tokmakov
ac42199ae5 fix race on StorageReplicatedMergeTree::data_parts_exchange_endpoint 2020-10-21 22:25:22 +03:00
alesapin
ce8ee01d75 Fix background moves start 2020-10-20 14:27:50 +03:00
alesapin
d126aa8a09 Merge branch 'master' into no_background_pool_no_more 2020-10-20 12:54:42 +03:00
Alexander Kuzmenkov
defcd29b31
Merge pull request #16036 from ClickHouse/stress_test_improvemets 
Minor improvements of stress test
 2020-10-19 16:12:30 +03:00
alesapin
e09713ece1 Merge branch 'master' into no_background_pool_no_more 2020-10-19 10:58:31 +03:00
Alexander Tokmakov
026896835c fixes 2020-10-16 15:20:30 +03:00
alesapin
7edbe4d276 Add some comments 2020-10-16 13:12:31 +03:00
Anton Ivashkin
14a78f87b0 Zero copy replication over S3: fetch instead of merge 2020-10-15 19:14:47 +03:00
alesapin
4c1f5cab13 Sleep less in background task 2020-10-15 19:10:22 +03:00
alesapin
2002289003 Fix bug in alter primary key for replicated versioned collapsing merge tree 2020-10-15 16:02:39 +03:00
Pavel Kruglov
25c39689c3 Check expired ttl 2020-10-15 15:15:02 +03:00
alesapin
583d7042e9 Get rid of some deadlocks 2020-10-15 11:29:18 +03:00
alesapin
c5419083e3 Better method name 2020-10-15 10:43:50 +03:00
alesapin
483893cdd4 Slightly refactored code 2020-10-14 17:56:42 +03:00
alesapin
0b14a31ba9 Get rid of background processing pool 2020-10-14 15:44:10 +03:00
alesapin
4014e0f08d Something very similar to working code 2020-10-14 15:32:35 +03:00
alesapin
efd3126b5d Moving pool 2020-10-14 10:22:48 +03:00
Pavel Kruglov
1ccf10505a add tests and comments 2020-10-13 21:26:40 +03:00
Pavel Kruglov
6d9b310f61 Add optimize_skip_merged_partitions 2020-10-13 21:26:35 +03:00
alesapin
fd35368c59 First simple implementation 2020-10-13 17:25:42 +03:00
alesapin
cb7db6d232 Split select and process merges 
(cherry picked from commit 1889eb0ff1)
 2020-10-13 15:04:16 +03:00
Nikolai Kochetov
7e58f99f64 Merge branch 'master' into storage-read-query-plan 2020-10-12 13:12:39 +03:00
Alexey Milovidov
5b482f4191 Cleanups 2020-10-10 19:31:10 +03:00
alexey-milovidov
0fd007ad9e
Merge pull request #15601 from alexelex/alexelex-master 
Improvement of Quorum Inserts in ClickHouse
 2020-10-09 22:43:53 +03:00
Anton Ivashkin
9272ed06b4 Move Zookeeper lock for S3 shared part in IMergeTreeDataPart 2020-10-09 17:24:10 +03:00
Anton Ivashkin
766dbfd2be Merge master 2020-10-09 15:00:14 +03:00
Alexandra Latysheva
0594a77b57 fix thread restart for parallel quorum inserts 2020-10-09 11:20:20 +00:00
tavplubix
7b5f2b2bb0
Merge pull request #15701 from ClickHouse/better_clone_replica 
Improvements in lost replica recovery
 2020-10-08 23:19:23 +03:00
Anton Ivashkin
acf86568a7 S3 zero copy replication proof of concept 2020-10-08 18:45:10 +03:00
Alexandra Latysheva
6877f4aa9c for parallel and non-parallel quorum insertion at the same time 2020-10-08 15:35:41 +00:00
tavplubix
6065a5b65c
Update StorageReplicatedMergeTree.cpp 2020-10-08 16:37:32 +03:00
Pervakov Grigorii
3a2068f19d Use tmp disk for vertical merge files 2020-10-08 13:49:41 +03:00
Amos Bird
867216103f
Extend trivial count optimization. 2020-10-08 18:08:17 +08:00
hchen9
ed2ce4a6fc Resolve conflict in StorageMemory 2020-10-07 17:19:59 -07:00
Alexander Tokmakov
ffb19fa4bc fix 2020-10-07 22:47:31 +03:00
Latysheva Alexandra
f43ad338ed
Merge pull request #12 from alexelex/quorum_inserts_2 
style check
 2020-10-07 18:56:14 +06:00
Alexandra Latysheva
8f8872dc42 style check 2020-10-07 12:24:46 +00:00
Latysheva Alexandra
6c26f77380
Merge pull request #11 from alexelex/quorum_inserts_2 
changing the condition for updateQuorum, specialized it and reduced t…
 2020-10-07 17:49:20 +06:00
Alexandra Latysheva
a43cac2c1a changing the condition for updateQuorum, specialized it and reduced the number of calls to the zk 2020-10-07 11:28:48 +00:00
Nikolai Kochetov
7e02152b50 Merge branch 'master' into storage-read-query-plan 2020-10-07 14:27:21 +03:00
Nikolai Kochetov
c5cb05f5f3 Try fix tests. 2020-10-07 14:26:29 +03:00
tavplubix
041015545c
Merge pull request #15537 from ClickHouse/mutation_hangs_after_replace_partition 
Fix mutation may hang after REPLACE/DROP PARTITION
 2020-10-07 14:25:29 +03:00
Alexander Tokmakov
47e9fe9ff1 fix 2020-10-07 03:05:48 +03:00
Alexander Tokmakov
ea16370024 improvements in lost replica recovery 2020-10-07 02:29:33 +03:00
Latysheva Alexandra
991e51a80e
Merge pull request #10 from alexelex/quorum_inserts_2 
style check fix
 2020-10-07 04:38:30 +06:00
Alexandra Latysheva
9077c92862 style check fix 2020-10-06 22:36:42 +00:00
Latysheva Alexandra
85bcc07c0b
Merge pull request #9 from alexelex/quorum_inserts_2 
Quorum inserts [version 3]
 2020-10-07 04:04:41 +06:00
Alexandra Latysheva
644091459f style fixes and remove small changes 2020-10-06 22:01:30 +00:00
Alexandra Latysheva
f549ecf9d1 insert_quorum_parallel: status nodes in /quorum/parallel and allow merges 2020-10-06 21:49:48 +00:00
Latysheva Alexandra
d495a2fb2d
Merge pull request #7 from alexelex/quorum_inserts_2 
build checker fix (style) and change comments
 2020-10-06 02:33:29 +06:00
Alexandra Latysheva
41e0148d72 build checker fix (style) and change comments 2020-10-05 20:30:50 +00:00
Alexander Tokmakov
418766e92b fix another bug in mutations 2020-10-05 22:16:28 +03:00
alesapin
8ec58c17f3 More consistent metadata usage 2020-10-05 19:41:46 +03:00
Alexander Tokmakov
30a18f7567 make it better 2020-10-05 16:52:03 +03:00
Latysheva Alexandra
7e7cedd73f
Merge pull request #4 from alexelex/quorum_inserts_2 
style check fix
 2020-10-05 16:43:14 +06:00
Alexandra Latysheva
0662432c5c style check 2020-10-05 10:41:49 +00:00
Latysheva Alexandra
a63084adaf
Merge pull request #3 from alexelex/quorum_inserts_2 
Quorum inserts 2
 2020-10-05 03:28:58 +06:00
Alexandra Latysheva
d54b6c9542 add question in comments 2020-10-04 20:32:08 +00:00
Alexandra Latysheva
ff8cbe6745 remove debug things 2020-10-04 20:19:05 +00:00
Alexandra Latysheva
8263e62298 working copy (with some debug info) 2020-10-04 19:55:39 +00:00
hchen9
cbf3af401d Fix comments 2020-10-04 02:42:03 -07:00
Azat Khuzhin
b838214a35 Pass non-const SelectQueryInfo (and drop mutable qualifiers) 2020-10-02 22:42:35 +03:00
hchen9
b3949db00f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into trivial_count_seq_consistency 2020-10-01 21:15:26 -07:00
Nikolai Kochetov
ec64def384 Use QueryPlan while reading from MergeTree. 2020-10-01 20:34:22 +03:00
Nicolae Vartolomei
7197a0c803 Possibly fix the issue with quorum 2020-10-01 11:38:50 +01:00
hchen9
b4494578d7 Support select_sequential_consistency in StorageReplicatedMergeTree.totalRows and totalBytes 2020-09-30 16:50:58 -07:00
Alexandra Latysheva
e89a56969f first part 2020-09-30 23:16:27 +00:00
Alexander Tokmakov
34addcf61f add review suggestions 2020-09-26 22:18:28 +03:00
alesapin
5ffee8808a
Alter remove column properties and TTLs (#14742) 2020-09-20 16:27:33 +03:00
alesapin
e38b537017
Merge pull request #11684 from ClickHouse/manual-write-duplicate-parts-to-replicas 
Don't ignore duplicate parts written to replicas
 2020-09-18 16:50:00 +03:00
Alexey Milovidov
a5560cd62d Better drop replicated tables 2020-09-17 14:27:17 +03:00
alesapin
40b2f203b6 Merge branch 'master' into manual-write-duplicate-parts-to-replicas 2020-09-17 13:21:00 +03:00
Alexander Tokmakov
da6beb1a91 Merge branch 'master' into enable_some_tests_with_database_atomic 2020-09-14 23:00:14 +03:00
bharatnc
3b3b7bd5e7 minor changes to comment 2020-09-10 09:09:48 -07:00