thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-19 04:42:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
65,566 Commits 9,326 Branches 1,783 Tags 2.2 GiB
3ade38df82
Commit Graph

468 Commits

Author SHA1 Message Date
Azat Khuzhin
6e71757814 Fix thread accounting for max_part_removal_threads 2021-04-11 11:43:30 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297) 
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
 2021-04-11 02:33:54 +03:00
Azat Khuzhin
a6703c5ac2 Fix inactive_parts_to_throw_insert=0 with inactive_parts_to_delay_insert>0 2021-04-10 09:57:08 +03:00
alesapin
8d5a787f6b Merge branch 'master' into merge_tree_deduplication 2021-04-05 10:40:03 +03:00
Alexey Milovidov
54caa61af5 Fix comments 2021-04-04 12:33:06 +03:00
alesapin
0204f5dd35 Merge branch 'master' into merge_tree_deduplication 2021-04-03 15:24:26 +03:00
alesapin
b884593186 Followup fix 2021-04-02 20:55:05 +03:00
alesapin
ca30c91868 More comments 2021-04-02 20:54:24 +03:00
alesapin
fc3afe1087 Fix typos 2021-04-02 20:51:57 +03:00
alesapin
b105153d2c Useful comment 2021-04-02 20:48:58 +03:00
alesapin
759dd79c76 Add tests 2021-04-02 19:45:18 +03:00
Mike Kot
c947280dfc Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-04-01 21:38:51 +03:00
Alexey Milovidov
230b2930b7 Merge branch 'master' into virtualutil 2021-03-30 00:43:00 +03:00
Alexey Milovidov
2a8ac01cdb Rename as suggested by Kita 2021-03-28 22:24:28 +03:00
Alexey Milovidov
6f9f7d118d Fixes 2021-03-28 04:27:26 +03:00
Alexey Milovidov
50f712e198 Integrate mmap cache to the infrastructure 2021-03-28 04:10:30 +03:00
alexey-milovidov
f895bc895c
Merge pull request #22011 from ClickHouse/min_max_time_system_parts_datetime64 
Expose DateTime64 minmax part index in system.parts and system.parts_columns
 2021-03-25 16:02:33 +03:00
Mike Kot
285af08949 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-24 22:34:20 +03:00
alesapin
36898bdc4a
Merge pull request #21142 from Jokser/unfreeze-partitions 
Add ALTER TABLE UNFREEZE command.
 2021-03-23 14:05:38 +03:00
Alexey Milovidov
8d0210b510 Expose DateTime64 minmax part index in system.parts and system.parts_columns #18244 2021-03-23 01:16:41 +03:00
Alexey Milovidov
ce67afd9e3 Fix bad code #18244 2021-03-23 00:49:18 +03:00
Nikolai Kochetov
c3c393a7aa Merge branch 'master' into refactor-actions-dag 2021-03-18 14:33:07 +03:00
Mike Kot
831b90f272 Replaced the iteration algorithm in part finder 
Now, instead of iterating through the directories, we iterate though
directories of on of the table disks (which doesn't give us a
substantial boost but is a bit neater to read).

- Updated the system.replication_queue command types.
- Fixed the part ptr being empty (added the checksum loading and
initialization).
- Removed extra logging.
 2021-03-17 16:55:26 +03:00
Mike Kot
2ccdb7ef5c Multiple small code and test updates 
- Updated the docs to make everything clear.
- Multiple small logger fixes.
- Changed the attach_part command -- now it's after check for the
covering parts -- motivation is to do less work with the checksums
fetching.
- Better logging in the integration test.
 2021-03-17 16:52:35 +03:00
Pavel Kovalenko
5627ae9067 Make MergeTreeData::getPartitionMatcher const 2021-03-17 12:12:55 +03:00
Pavel Kovalenko
a43c85b105 Add test for freeze/unfreeze partition with old MergeTree syntax 2021-03-16 16:36:05 +03:00
Pavel Kovalenko
39f3b17041 Merge remote-tracking branch 'origin/master' into unfreeze-partitions 2021-03-16 10:56:24 +03:00
alexey-milovidov
378e10f929
Merge pull request #21351 from fuwhu/refine-load-data-parts 
Refine MergeTreeData::loadDataParts to not parse format version file and detached directory
 2021-03-15 22:38:30 +03:00
Mike Kot
406d037ebb Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-15 18:41:47 +03:00
fuwhu
69f516cd47 Refine MergeTreeData::loadDataParts to not parse format version file and detached directory 2021-03-10 10:10:25 +08:00
Amos Bird
e52b2fb8ae
update 2021-03-10 02:56:46 +08:00
Amos Bird
2ec20c5d23
update and add tests 2021-03-08 17:38:07 +08:00
Amos Bird
2b11a2a855
Fix build 2021-03-08 11:10:35 +08:00
Amos Bird
e6522e1ebe
JBOD data balancer 2021-03-08 11:10:35 +08:00
Nikolai Kochetov
a669f7d641 Merge branch 'master' into refactor-actions-dag 2021-03-05 18:21:14 +03:00
Nikolai Kochetov
9a39459888 Refactor ActionsDAG 2021-03-04 20:38:12 +03:00
Amos Bird
909cb3c243
Fix again.... 2021-03-04 22:27:07 +08:00
Amos Bird
9205fad8c7
Better 2021-03-04 19:43:03 +08:00
Amos Bird
93b661ad5a
partition id pruning 2021-03-04 19:43:03 +08:00
Pavel Kovalenko
c4523ec74c Add 'const auto' for variable. 2021-03-03 10:49:23 +03:00
alesapin
6b1005aea5
Merge pull request #21334 from ClickHouse/fix_alter_partition_key 
Fix alter modify query for partition key and other metadata fields
 2021-03-03 10:15:11 +03:00
Anton Popov
a4c00ab5dc
Merge pull request #21303 from ucasFL/forbid 
Forbid to drop a column if it's referenced by materialized view
 2021-03-03 02:55:06 +03:00
Pavel Kovalenko
483e63ca45 ALTER TABLE UNFREEZE rework. 2021-03-03 00:10:09 +03:00
Pavel Kovalenko
a3c0c35dff ALTER TABLE UNFREEZE rework. 2021-03-02 23:28:42 +03:00
Nikolai Kochetov
4775ea305e Remove index by name from ActionsDAG 2021-03-02 20:51:54 +03:00
alesapin
c29d7c7f49 Shutup clang tidy 2021-03-02 19:13:36 +03:00
alesapin
9ebf1b4fad Get rid of separate minmax index fields 2021-03-02 13:33:54 +03:00
feng lv
a26c9e64a9 fix 
fix
 2021-03-02 03:20:03 +00:00
Mike Kot
4e1ac185b5 Merge remote-tracking branch 'upstream/master' into feature/attach-partition-local 2021-03-01 20:58:07 +03:00
alesapin
9c8afbeb53 Fix alter modify query for partition key and other metadata fields 2021-03-01 12:59:19 +03:00
Azat Khuzhin
8b58dba094 Drop unused HexWriteBuffer 2021-02-28 10:52:09 +03:00
feng lv
51021c1164 forbid to drop a column if it's referenced by materialized view 2021-02-28 05:24:39 +00:00
Pavel Kovalenko
ed44845f2c Fixed backup_path for unfreeze 2021-02-25 22:21:22 +03:00
alesapin
5c6c318737 Restrict mutations for engines which doesn't support them 2021-02-25 13:07:48 +03:00
Pavel Kovalenko
0df783e0ef Fix code style 2021-02-24 18:38:40 +03:00
Pavel Kovalenko
078dfce038 Add ALTER TABLE UNFREEZE command. 2021-02-24 17:28:30 +03:00
Anton Popov
48e188681c do not start mutation for alters with wrong type conversion 2021-02-20 19:11:08 +03:00
Mike Kot
73f4740be5 Trying to re-write the solution by adding new command type 
ATTACH_PART into the replicated log.

The LogEntry now also has the pre-calculated part checksum for this
entry type, which is later used while searching in the detached/ folder
 2021-02-15 18:06:48 +03:00
alesapin
a2bb586170
Merge pull request #20335 from ClickHouse/fix_scheduling_of_non_successful_background_tasks 
Fix backoff for failed background tasks in replicated merge tree
 2021-02-12 10:06:04 +03:00
alexey-milovidov
7b55eb8c58
Merge pull request #20178 from amosbird/delayinactive 
Delay or throw insertion when too many inactive parts
 2021-02-11 20:52:04 +03:00
alesapin
3253638969 Fix backoff for failed background tasks in replicated merge tree 2021-02-11 14:46:18 +03:00
alesapin
56453256b7
Merge pull request #20300 from ClickHouse/restrict_to_drop_version_column 
Don't allow to drop or rename version column
 2021-02-11 11:04:00 +03:00
Anton Popov
365e2fb746
Merge pull request #20231 from fuwhu/refine-loadDataParts-code 
refine code in MergeTreeData::loadDataParts to avoid parsing WAL file as data part
 2021-02-10 22:42:31 +03:00
alesapin
891fce3275 Don't allow to drop or rename version column 2021-02-10 22:23:48 +03:00
Amos Bird
afb5846a24
refactor 2021-02-10 11:22:24 +08:00
HuFuwang
4ac8860cea refine code in MergeTreeData::loadDataParts to avoid parsing WAL file as data part. 2021-02-10 10:13:43 +08:00
tavplubix
86530e603c
Merge pull request #20027 from bharatnc/ncb/event_time_micro_part_log 
add event_time_microseconds field to system.part_log table
 2021-02-09 14:46:40 +03:00
Nicolae Vartolomei
a8f37e7704 Remove dead code 2021-02-08 17:19:35 +00:00
bharatnc
60c8dfa22c PartLog - populate event_time_microseconds for MergeParts 2021-02-08 07:25:44 -08:00
Amos Bird
109a392e0c
Fix ubsan 2021-02-08 19:41:16 +08:00
Amos Bird
ab5c7b75a4
Delay or throw insertion when too many inactive parts 2021-02-07 19:03:55 +08:00
alesapin
011109c82a
Merge pull request #17348 from xjewer/alex/CLICKHOUSE-606_deduplication_UUID 
CLICKHOUSE-606: query deduplication based on parts' UUID
 2021-02-05 22:47:34 +03:00
Pavel Kovalenko
b7151aa754 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskDecorator.h
#	src/Disks/IDisk.h
#	src/Disks/S3/DiskS3.cpp
 2021-02-03 14:22:18 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID 
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
 2021-02-02 16:53:39 +00:00
Nikolai Kochetov
19e4a33f9d
Merge pull request #19544 from amosbird/limitconcurrency 
Per MergeTree table query limit
 2021-02-01 16:09:12 +03:00
alesapin
5622e6daa6 Fix rare max_number_of_merges_with_ttl_in_pool limit overrun for non-replicated MergeTree 2021-01-27 14:56:12 +03:00
Amos Bird
66fe97d8bd
Per MergeTree table query limit 2021-01-26 14:03:31 +08:00
Pavel Kovalenko
1e3a059f64 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata 
# Conflicts:
#	src/Disks/DiskCacheWrapper.cpp
#	src/Disks/S3/DiskS3.cpp
 2021-01-18 13:39:49 +03:00
alexey-milovidov
5f189c5756
Merge pull request #19122 from ClickHouse/data-part-better-code 
Add metrics for part number in MergeTree in ClickHouse
 2021-01-16 00:20:15 +03:00
alexey-milovidov
971ff2ee0a
Merge pull request #19086 from ClickHouse/faster-parts-removal 
Faster parts removal, more safe and efficient interface of IDisk
 2021-01-15 20:37:35 +03:00
Alexey Milovidov
6a2a5e53ed Slightly better code of IMergeTreeDataPart #18955 2021-01-15 15:15:13 +03:00
Alexey Milovidov
8276a1c8d2 Faster parts removal, more safe and efficient interface of IDisk 2021-01-14 19:24:13 +03:00
Anton Popov
ac426c3da6 restrict MODIFY TTL for tables created in old syntax 2021-01-14 15:32:20 +03:00
Pavel Kovalenko
b09862b7b9 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-12 20:18:40 +03:00
Pavel Kovalenko
0856b2c514 Ability to backup-restore metadata files for DiskS3 (fixes and tests) 2021-01-11 20:37:08 +03:00
alexey-milovidov
a08db94343
Revert "Add metrics for part number in MergeTree in ClickHouse" 2021-01-07 16:40:52 +03:00
weeds085490
5f5b86b485 Merge remote-tracking branch 'origin' into dev/add_metrics_for_parts 2021-01-06 17:32:45 +08:00
Nikita Mikhaylov
4165a58ce4
Merge pull request #11617 from nikitamikhaylov/parallel-parsing-input-format 
Parallel formatting
 2020-12-31 21:22:16 +04:00
alexey-milovidov
c2f0451e39
Merge pull request #18362 from ClickHouse/allow_low_cardinality_key_alter 
Add ability to change some types for primary key
 2020-12-31 15:27:52 +03:00
Nikita Mikhailov
60b4a36c4a arcadia fix + live view fix + cleanup 2020-12-30 07:50:58 +03:00
Nikita Mikhailov
c0eef84c50 better 2020-12-29 04:15:26 +03:00
Nikita Mikhailov
d293ae5e8e style 2020-12-28 20:18:53 +03:00
Nikita Mikhailov
dcfbe782c6 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-23 05:20:22 +03:00
roverxu
e38ecee30e fix style 2020-12-23 09:20:45 +08:00
徐炘
81f9623e2e
Merge branch 'master' into dev/add_metrics_for_parts 2020-12-22 19:04:28 +08:00
alesapin
35413635e7 Add ability to change some types for primary key 2020-12-22 13:41:12 +03:00
flynn
4e580f7b7c
Add asynchronous metrics on MergeTree tables data volume (#17639) 
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-12-22 13:34:35 +03:00
root
d6a69ee164 remove code 2020-12-22 18:12:28 +08:00
roverxu
f3fef24d30 fix deadlock 2020-12-21 16:30:27 +08:00
roverxu
8e72b6d31b Merge remote-tracking branch 'origin' into dev/add_metrics_for_parts 2020-12-21 09:25:54 +08:00
Nikita Mikhailov
6a55e018e4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-18 02:25:32 +03:00
Alexey Milovidov
b25f126569 Added a test 2020-12-17 04:52:27 +03:00
Alexey Milovidov
9a017b77df Merge branch 'master' into relax-too-many-parts-1 2020-12-17 04:40:39 +03:00
roverxu
1065ddef65 fix bug when drop data 2020-12-16 22:02:34 +08:00
Nikita Mikhaylov
5a47928431 save 2020-12-15 00:56:47 +03:00
Nikita Mikhaylov
f40f3ced2a fix JSONEachRowArray 2020-12-15 00:56:46 +03:00
Nikita Mikhaylov
e0addac6fc save changes 2020-12-15 00:56:46 +03:00
alesapin
0eec52b1fd
Merge pull request #17737 from ClickHouse/fix_segfault_in_distributed_out_stream 
Fix segfault when 'not enough space'
 2020-12-05 16:14:30 +03:00
detailyang
e7a151fd1e
hotfix:check in_memory_parts_enable_wal 2020-12-04 23:14:27 +08:00
roverxu
2a6c1463a1 Add metrics for part number in MergeTree 2020-12-04 11:42:58 +08:00
Alexander Tokmakov
bfbf150c67 fix segfault when 'not enough space' 2020-12-02 17:49:43 +03:00
Alexey Milovidov
2e6bedce91 Relax "Too many parts" threshold, part 1 2020-11-29 18:08:02 +03:00
feng lv
7e3524caa1 fix spelling errors 2020-11-28 08:17:20 +00:00
tavplubix
5cc9cb01cd
Merge pull request #16751 from amosbird/globalcontext 
Make global_context consistent.
 2020-11-22 18:46:17 +03:00
Amos Bird
1d9d586e20
Make global_context consistent. 2020-11-20 18:23:14 +08:00
Nicolae Vartolomei
746f8e45f5 All new parts must have uuids 2020-11-19 13:18:03 +00:00
Anton Popov
de5ead0c40 add setting remove_empty_parts 2020-11-15 05:24:47 +03:00
alexey-milovidov
915d96fee0
Merge pull request #16955 from nikitamikhaylov/fix-typos 
Fix typos
 2020-11-14 17:17:37 +03:00
Nikolai Kochetov
ed98b48845
Merge pull request #16618 from ClickHouse/actions-dag-f14 
Actions dag 2
 2020-11-13 18:34:51 +03:00
nikitamikhaylov
c0940f2ebd done 2020-11-13 02:27:18 +03:00
Anton Popov
4cecd89e84 fix dropping of empty parts 2020-11-12 20:37:54 +03:00
Anton Popov
0aa35d53c7 Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-12 14:31:58 +03:00
alesapin
2cac5e5d97
Merge pull request #16875 from CurtizJ/refactor-alter-partition 
Avoid code duplication in alterPartition
 2020-11-12 13:19:59 +03:00
Nikolai Kochetov
46f70dd0de Merge branch 'master' into actions-dag-f14 2020-11-12 11:54:44 +03:00
tavplubix
058aa8f85e
Merge pull request #16824 from ClickHouse/replace_stringstreams_with_buffers 
Replace std::*stringstreams with DB::*Buffers
 2020-11-12 01:11:44 +03:00
Anton Popov
2497598a49 Remove empty parts after they were pruned by TTL 2020-11-11 19:19:43 +03:00
Anton Popov
6f0dc08ba7 avoid code duplication in alterPartition 2020-11-11 16:34:07 +03:00
Nikolai Kochetov
1846bb3cac Merge branch 'master' into actions-dag-f14 2020-11-11 13:08:57 +03:00
Nikolai Kochetov
c6575c9032 Update ExpressionActions constructur 2020-11-10 19:27:55 +03:00
Nikolai Kochetov
1db8e77371 Add comments. Update ActionsDAG::Index 2020-11-10 17:54:59 +03:00
Alexander Tokmakov
5cdfcfb307 remove other stringstreams 2020-11-09 22:12:44 +03:00
Nikolai Kochetov
6717c7a0af Merge branch 'master' into actions-dag-f14 2020-11-09 14:57:48 +03:00
alesapin
72046313d4 Merge branch 'master' into nvartolomei-drop-part 2020-11-09 10:02:02 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Nikolai Kochetov
9aeb757da4 Merge branch 'master' into actions-dag-f14 2020-11-06 15:04:20 +03:00
alesapin
faca9d9b91 Check max bytes on disk 2020-11-03 18:41:46 +03:00
Nikolai Kochetov
07a7c46b89 Refactor ExpressionActions [Part 3] 2020-11-03 14:28:28 +03:00
alesapin
dac6705995 DROP PART and clear blocks in a single zookeeper op 2020-11-02 20:30:59 +03:00
alesapin
39e47c5338 Less strange lambdas 2020-10-23 11:54:00 +03:00
alesapin
7a63b373f9 Merge branch 'blind_fix_of_read_in_order' into no_background_pool_no_more 2020-10-22 15:47:11 +03:00
Vladimir Chebotarev
aa5f207fd4
Added disable_merges option for volumes in multi-disk configuration (#13956) 
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-10-20 18:10:24 +03:00
alesapin
7edbe4d276 Add some comments 2020-10-16 13:12:31 +03:00
alesapin
4014e0f08d Something very similar to working code 2020-10-14 15:32:35 +03:00
alesapin
efd3126b5d Moving pool 2020-10-14 10:22:48 +03:00
nikitamikhaylov
88a5302759 Merge branch 'master' of github.com:ClickHouse/ClickHouse into copier-backquotes 2020-10-09 16:26:31 +03:00
alesapin
8ec58c17f3 More consistent metadata usage 2020-10-05 19:41:46 +03:00
alesapin
8bc9ede711 Fix clang tidy 2020-10-01 09:53:47 +03:00
alesapin
97c2579fb9 Use separate function 2020-09-30 22:50:23 +03:00
alesapin
ac9f0ff4da More strict check for version column 2020-09-30 22:44:35 +03:00
alesapin
65a05a970d Better message on error 2020-09-29 15:07:01 +03:00