thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 03:12:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
112,748 Commits 9,326 Branches 1,783 Tags 2.1 GiB
5f218353ed
Commit Graph

1419 Commits

Author SHA1 Message Date
SmitaRKulkarni
b22fd40d92
Merge branch 'master' into 45508_Update_strategy_for_system_sync_replica 2023-01-27 15:00:47 +01:00
Alexander Tokmakov
f375b1246f
Merge pull request #45363 from ClickHouse/fix_hang_during_drop_in_zero_copy_replication 
Fix no shared id during drop for the fourth time
 2023-01-27 15:59:27 +03:00
Smita Kulkarni
39966457d9 Updated to remove waiting ids till the removed log entry - Updated checking of SYSTEM SYNC REPLICA 2023-01-27 13:36:44 +01:00
alesapin
716d25c33f
Merge pull request #45619 from CurtizJ/refactoring-merge-tree-parts 
Refactoring of code near merge tree parts
 2023-01-27 13:36:33 +01:00
Smita Kulkarni
a95609e0bc Updated to validate & remove each element instead of just last element - Updated checking of SYSTEM SYNC REPLICA 2023-01-27 10:35:12 +01:00
Antonio Andelic
714fad1529 Add support for settings alias 2023-01-26 14:06:46 +00:00
alesapin
84af5eb58b Fxi 2023-01-26 11:42:53 +01:00
alesapin
d5036f2ebc Merge branch 'fix_hang_during_drop_in_zero_copy_replication' of github.com:ClickHouse/ClickHouse into fix_hang_during_drop_in_zero_copy_replication 2023-01-26 11:42:30 +01:00
alesapin
1f61abce46 One useful logging line 2023-01-26 11:41:43 +01:00
Smita Kulkarni
8c885ffeae Updated checking of SYSTEM SYNC REPLICA 
Implementation:
* Updated to wait for current last entry to be processed (after pulling shared log) instead of queue size becoming 0.
* Updated Subscriber to notify both queue size and removed log_entry_id.
 2023-01-26 11:40:09 +01:00
Alexander Tokmakov
a584ad0eb1 forbid runtime strings 2023-01-26 10:52:47 +01:00
Alexander Tokmakov
92c62e2de2
Update src/Storages/StorageReplicatedMergeTree.cpp 2023-01-26 00:28:46 +03:00
Alexander Tokmakov
3744fa2c63 format more messages 2023-01-25 21:16:42 +01:00
alesapin
d0428a9b42 Fxi 2023-01-25 20:30:19 +01:00
Anton Popov
8e3698c91f refactoring of code near merge tree parts 2023-01-25 17:34:09 +00:00
alesapin
55af44ca2f Better fix 2023-01-25 14:40:39 +01:00
alesapin
96d8546b1c Merge branch 'master' into fix_hang_during_drop_in_zero_copy_replication 2023-01-25 13:07:14 +01:00
Alexander Tokmakov
6621834e37 fix 2023-01-24 17:10:54 +01:00
Alexander Tokmakov
6ecae8388e Merge branch 'master' into exception_message_patterns4 2023-01-24 14:42:36 +01:00
Alexey Milovidov
dda6a69f9a
Merge branch 'master' into tavplubix-patch-1 2023-01-24 13:04:25 +03:00
Alexey Milovidov
54bc501d39
Update StorageReplicatedMergeTree.cpp 2023-01-24 13:03:28 +03:00
Alexander Tokmakov
414693feb2 fixes 2023-01-24 00:46:03 +01:00
Alexander Tokmakov
3f6594f4c6 forbid old ctor of Exception 2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Alexander Tokmakov
910d6dc0ce
Merge pull request #45342 from ClickHouse/exception_message_patterns 
Save message format strings for DB::Exception
 2023-01-20 18:46:52 +03:00
Frank Chen
f867c4c206 Add an overriden fromCurrentException to simply the code 2023-01-20 16:43:10 +08:00
Frank Chen
38799eb852 Merge two methods as one 2023-01-20 16:35:28 +08:00
Frank Chen
be8ceeb4bb Save stack trace in part_log 2023-01-20 16:22:40 +08:00
Frank Chen
1fe9bde594 Set replica name to zero copy exclusive lock 2023-01-20 15:48:33 +08:00
Alexander Tokmakov
7bb65cc002
Update StorageReplicatedMergeTree.cpp 2023-01-19 16:45:41 +03:00
Han Fei
f661dad0e9
Merge pull request #45106 from hanfei1991/hanfei/async-cache 
support cache for async inserts block ids
 2023-01-19 10:59:25 +01:00
Vitaly Baranov
7cdb2c4c7f
Merge pull request #45351 from vitlibar/fix-backup-with-killed-mutations 
Fix backup with killed mutations
 2023-01-18 13:14:27 +01:00
Sergei Trifonov
c443c1ece0
Merge branch 'master' into hanfei/async-cache 2023-01-18 00:19:49 +01:00
Alexander Tokmakov
5cd90c1a3e Merge branch 'master' into exception_message_patterns 2023-01-17 20:04:04 +01:00
Alexander Tokmakov
72e8615bec formatting of some exception messages 2023-01-17 20:03:56 +01:00
alesapin
e732f510f0
Merge branch 'master' into fix_hang_during_drop_in_zero_copy_replication 2023-01-17 19:24:36 +01:00
Alexander Tokmakov
8b13b85ea0
Merge pull request #44543 from ClickHouse/text_log_add_pattern 
Add a column with a message pattern to text_log
 2023-01-17 20:19:32 +03:00
alesapin
69925647eb Fix style 2023-01-17 15:59:55 +01:00
alesapin
f6131101bb Fix no shared id during drop for the fourth time 2023-01-17 15:51:49 +01:00
Vitaly Baranov
692065e5fe Fix backup if mutations got killed during the backup process. 2023-01-17 11:05:34 +01:00
Alexander Tokmakov
870cfcc36a less fmt::runtime usages 2023-01-17 00:11:59 +01:00
Anton Popov
6863cd152f
Merge pull request #42181 from CurtizJ/optimize-loading-parts 
Do not load inactive parts at startup
 2023-01-16 14:38:50 +01:00
Alexander Tokmakov
2d7773fccc Merge branch 'master' into text_log_add_pattern 2023-01-13 20:33:46 +01:00
Han Fei
bcf813fedc Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
 2023-01-13 20:19:30 +01:00
Han Fei
a258a39eb1 Merge branch 'master' into hanfei/async-cache 2023-01-13 20:17:58 +01:00
Alexander Tokmakov
9d5ec474a3
Merge pull request #43998 from evillique/make_system_replicas_parallel 
Make `system.replicas` parallel
 2023-01-13 16:33:36 +03:00
Alexander Tokmakov
51d94314d6
Merge pull request #45235 from ClickHouse/more_verbose_logs_about_replication_log_entries 
More verbose logs about replication log entries
 2023-01-13 15:05:21 +03:00
Raúl Marín
9f121d5d9c
Fix background_fetches_pool_size config reload (#45189) 2023-01-12 23:41:25 +01:00
Alexander Tokmakov
d9b8cf9c04 more verbose logs about replication log entries 2023-01-12 20:32:39 +01:00
Anton Popov
6c4f53bdca Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-11 01:00:57 +00:00
Alexander Tokmakov
881b17492f Merge branch 'master' into fix_get_part_name 2023-01-10 21:39:35 +01:00
Han Fei
bef6168c05 support cache for async inserts block ids 2023-01-10 13:19:12 +01:00
alesapin
616e37ba43
Merge pull request #44847 from ClickHouse/fix_rare_race 
Fix rare race which can lead to queue hang
 2023-01-10 12:09:10 +01:00
Alexander Gololobov
742887ce53
Merge pull request #44718 from ClickHouse/lwd_mutation_always_sync 
Make lightweight deletes always synchronous
 2023-01-09 15:58:02 +01:00
Anton Popov
1e5d6e44a5 fix deadlock at shutdown 2023-01-09 13:07:46 +00:00
Nikita Mikhaylov
857799fbca
Parallel distributed insert select with s3Cluster [3] (#44955) 
* Revert "Revert "Resurrect parallel distributed insert select with s3Cluster (#41535)""

This reverts commit b8d9066004.

* Fix build

* Better

* Fix test

* Automatic style fix

Co-authored-by: robot-clickhouse <robot-clickhouse@users.noreply.github.com>
 2023-01-09 13:30:32 +01:00
alesapin
da83e97b4c Fix startup issue 2023-01-06 15:36:53 +01:00
Anton Popov
fcdbc8123d better loading of outdated parts 2023-01-06 11:59:54 +00:00
Anton Popov
ed00399eb6 Merge remote-tracking branch 'upstream/master' into HEAD 2023-01-05 14:12:02 +00:00
Nikolay Degterinsky
673ba769e5 Merge remote-tracking branch 'upstream/master' into make_system_replicas_parallel 2023-01-05 02:36:07 +00:00
alesapin
2a5b4067e4 Review fixes 2023-01-04 15:15:14 +01:00
alesapin
8ddda46d1b
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-01-04 14:59:46 +01:00
alesapin
39331e07d5 Improve logging 2023-01-02 17:05:08 +01:00
alesapin
6fc63878b4 Fix rare race which can lead to queue hang 2023-01-02 16:57:25 +01:00
Alexander Gololobov
2c64f0042a Added force_wait parameter to IStorage::mutate() 2022-12-30 16:23:03 +01:00
alesapin
5dfefb43a3 Fix deadlock in attach thread 2022-12-29 18:52:20 +01:00
Anton Popov
2adf0e9db3 do not wait loading parts at shutdown 2022-12-28 16:07:14 +00:00
Alexander Tokmakov
ca989e9212 less runtime format strings 2022-12-23 19:50:34 +01:00
Anton Popov
0722020cae Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-23 18:15:55 +00:00
alesapin
2fecf420cb
Merge pull request #44513 from ClickHouse/small_polishing 
Minor code polishing
 2022-12-23 13:47:26 +01:00
alesapin
fa467b3398 Minor code polishing 2022-12-22 14:31:42 +01:00
Nikolay Degterinsky
b9d0d25eb2 Better StorageReplicatedMergeTree::getStatus() 2022-12-21 02:11:25 +00:00
Alexander Tokmakov
cdc3912743 fix incorrect usages of getPartName() 2022-12-20 22:44:27 +01:00
Alexander Tokmakov
3f851d5a09 parallel parts cleanup with zero copy replication 2022-12-20 22:04:26 +01:00
alesapin
2818520572 Fix bug with merge/mutate pool size increase 2022-12-20 14:51:11 +01:00
Han Fei
623af800c2
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2022-12-14 14:29:04 +01:00
Han Fei
34a589a7d8 create async_blocks zk path for old replicated tables and add a flag "async_insert_deduplicate" 2022-12-14 14:06:12 +01:00
Alexander Tokmakov
371e85d854
Merge pull request #44157 from ClickHouse/fix_session_expired_on_startup 
Ignore "session expired" errors after BC check
 2022-12-13 15:10:36 +03:00
Anton Popov
0b396bab14 Merge remote-tracking branch 'upstream/master' into HEAD 2022-12-12 18:08:27 +00:00
Alexander Tokmakov
469406c494 ignore session expired errors after BC check 2022-12-12 14:02:14 +01:00
Han Fei
6740ae9fd7
Merge pull request #43304 from hanfei1991/hanfei/aysnc-insert 
Support dedup for async insert.
 2022-12-09 12:50:48 +01:00
Han Fei
3c11aca040 address comments 2022-12-07 23:40:52 +01:00
Anton Popov
df6d2f2163 Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-29 23:21:23 +00:00
Alexander Tokmakov
58d8025a15
Update StorageReplicatedMergeTree.cpp 2022-11-29 16:42:15 +03:00
Alexander Tokmakov
0342e6b236 fix 2022-11-28 16:02:37 +01:00
Alexander Tokmakov
a47e0b8f61 Merge branch 'master' into merge_predicate_uncommitted_blocks_hint 2022-11-28 15:45:34 +01:00
Alexander Tokmakov
f4944664b7 fix 2022-11-25 18:53:46 +01:00
Han Fei
ee5be80d9f Merge branch 'master' into hanfei/aysnc-insert 2022-11-25 17:27:56 +01:00
Alexander Tokmakov
5cc99312d7 provide hint for loading uncommitted blocks in merge predicate 2022-11-25 16:41:20 +01:00
Alexey Milovidov
69d1be50bc
Merge pull request #43446 from ClickHouse/fix-typo-3 
Fix typo
 2022-11-25 15:22:28 +01:00
Sema Checherinda
15a6ce254b
Merge pull request #41145 from CheSema/lock-free-drop-partition 
Delete parts by replacing them with empty parts
 2022-11-25 12:54:30 +01:00
Anton Popov
c65be55da1 Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-24 14:30:23 +00:00
Anton Popov
b05a20da1f fix drop in case of zero-copy 2022-11-24 14:29:30 +00:00
Han Fei
28564df6f1
Merge branch 'master' into hanfei/aysnc-insert 2022-11-24 09:12:43 +01:00
Anton Popov
f4d91cd722 fix drop with zero-copy 2022-11-24 03:31:04 +00:00
Sema Checherinda
fb86019db9 Delete parts over replacing them with empty parts 2022-11-23 15:16:08 +00:00
Anton Popov
44b4345996 fix wait in drop 2022-11-23 04:21:27 +00:00
Anton Popov
e3db29aebf better waiting for outdated data parts 2022-11-22 15:15:20 +00:00
Han Fei
32c7fbec55 use another dedup path for async inserts 2022-11-22 15:12:00 +01:00
Alexander Tokmakov
a7efdd7f4e
Update StorageReplicatedMergeTree.cpp 2022-11-21 21:52:48 +03:00
Anton Popov
92ac9385b9 wait loading of outdated parts for partition commands 2022-11-21 17:01:46 +00:00
Alexey Milovidov
56f8c0b84b Fix typo 2022-11-21 15:10:15 +01:00
Han Fei
b80a2d6c89 address comments 2022-11-18 17:22:05 +01:00
Han Fei
14701670b5 Merge branch 'master' into hanfei/aysnc-insert 2022-11-16 17:24:36 +01:00
Antonio Andelic
11aba7597e Check if initialization is done on write 2022-11-16 13:33:22 +00:00
Antonio Andelic
0e5e58bed9 Remove exception if shared ID already created 2022-11-15 12:18:24 +00:00
Alexander Tokmakov
c64109a2de
Merge branch 'master' into fetch-remove-covered 2022-11-14 18:23:16 +03:00
Han Fei
11f93dec3f tmp commit 2022-11-14 15:53:45 +01:00
Igor Nikonov
f8c907c463
Revert "Revert " Keeper retries during insert (clean)"" 2022-11-10 13:14:04 +01:00
Alexander Tokmakov
fb8c527a2f
Revert " Keeper retries during insert (clean)" 2022-11-10 14:12:24 +03:00
Igor Nikonov
379113f8f1 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-11-09 13:03:32 +00:00
Alexander Tokmakov
eb19df0094 fix 2022-11-08 15:21:25 +01:00
Azat Khuzhin
2bd4a71150
Merge branch 'master' into fetch-remove-covered 2022-11-08 00:55:00 +01:00
Alexander Tokmakov
9210e586d2 fix 2022-11-07 20:27:18 +01:00
Alexander Tokmakov
e67d05604c Merge branch 'master' into fix_intersecting_parts2 2022-11-02 19:30:27 +01:00
Alexander Tokmakov
d8b3a2af84 fix race between INSERT and DROP 2022-11-02 19:26:50 +01:00
Alexander Tokmakov
00c9e50ee3 fix race between INSERT and ALTER PARTITION 2022-11-02 16:25:26 +01:00
Alexander Tokmakov
4f50a999bb remove abandonable_lock part 1 2022-11-01 21:35:25 +01:00
Igor Nikonov
e76c3c381f Better handling tables in read-only mode during insert 
+ throw immediately if table has no metadata in zk
+ stop retries in case of shutdown
+ check if table is readonly at the begining of every retry
 2022-11-01 18:47:51 +00:00
Igor Nikonov
9cc64a0fcc Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-31 18:21:21 +00:00
Igor Nikonov
2d5050ea66 Fix review comments 2022-10-31 18:20:58 +00:00
Alexander Tokmakov
c68ab231f9 fix accessing part in Deleting state 2022-10-30 17:30:51 +01:00
Igor Nikonov
3682913172 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-27 22:01:53 +00:00
Azat Khuzhin
8db31beee6 Remove covered parts for fetched part 
Here is an example that I found on production, simplified.

Consider the following queue (nothing of this had been processed on this
replica):

- GET_PART all_0_0_0 (queue-0000000001)
- GET_PART all_1_1_0 (queue-0000000002)
...
- GET_PART all_0_1_1 (queue-0000000003)
- GET_PART all_2_2_0 (queue-0000000004)
...
- MERGE_PARTS from [all_0_1_1, all_2_2_0] to all_0_2_2 (queue-0000000005)

And now queue-0000000005 started to executing (either because
of reording, or because at that time GET_PART fails), and it
does not have any required parts, so it will fetch them, but
not all_0_0_0 and all_1_1_0, so this replica delay will set to
the time of min(queue-0000000001, queue-0000000002), while it
is not true, since it already have parts that covers those
parts.

and since MERGE_PARTS takes 30min, it increased the replica delay
eventually to 30min, for the time range of 30min, which is pretty huge.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-26 11:10:50 +03:00
Anton Popov
c8199bc125
Merge branch 'master' into refactor-data-part 2022-10-25 14:31:05 +02:00
Igor Nikonov
54013009d3 Code migration from #41734 to align diff and git blame. 
Initial implementation was different and it changed the entire
ReplicatedMergeTreeSink::commitPart() which change history provided by git blame.
Then RetriesControl.retryLoop() was introduced later which significantly reduces
the diff since it's like while() used before.
So, check outing the current version will keep more original history in
git blame, which is useful here
 2022-10-24 08:20:11 +00:00
Anton Popov
b40d9200d2 better semantic of constsness of DataPartStorage 2022-10-23 15:24:20 +00:00
Igor Nikonov
c0d1be9cab
Merge pull request #42560 from ClickHouse/igor/replicated_merge_tree_some_polishing 
Some polishing: replicated merge tree
 2022-10-23 15:15:24 +02:00
Anton Popov
56e5daba0c remove DataPartStorageBuilder 2022-10-23 00:23:15 +00:00
Igor Nikonov
730a3ba849
Merge branch 'master' into igor/replicated_merge_tree_some_polishing 2022-10-22 19:54:09 +02:00
Igor Nikonov
851f556d65 Remove unused parameter 2022-10-21 14:35:37 +02:00
Igor Nikonov
13d31eefbc Replicated merge tree polishing 
+ make read-only members const
+ reduce memory allocations
 2022-10-21 14:02:52 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
alesapin
24d02b5516
Merge pull request #42441 from ClickHouse/check_readonly_in_background_tasks 
Deactivate tasks in ReplicatedMergeTree until startup
 2022-10-20 15:36:20 +02:00
Alexander Tokmakov
d668a82829
Merge branch 'master' into improve_replica_recovery 2022-10-20 14:25:09 +03:00
Antonio Andelic
77eb353839
Merge pull request #42243 from ClickHouse/use-multiread-more 
Use MultiRead where possible
 2022-10-20 09:04:54 +02:00
Alexander Tokmakov
84ef32e33f Merge branch 'master' into improve_replica_recovery 2022-10-19 16:28:01 +02:00
Alexey Milovidov
7befe2825c
Update StorageReplicatedMergeTree.cpp 2022-10-18 17:36:11 +03:00
Alexey Milovidov
0623ad4e37
Update StorageReplicatedMergeTree.cpp 2022-10-18 17:35:44 +03:00
alesapin
9769057767 Deactivate all tasks 2022-10-18 15:53:52 +02:00
alesapin
ab8da5a539 Deactivate mutationsFinalizingTask until startup 2022-10-18 15:50:02 +02:00
Alexander Tokmakov
57f6e27300 Merge branch 'master' into improve_replica_recovery 2022-10-17 18:37:31 +02:00
alesapin
051e3a0db5 Fix if 2022-10-17 12:27:05 +02:00
alesapin
e0c2518416 Fix another trash in zero-copy replication 2022-10-14 16:50:48 +02:00
Antonio Andelic
5e6a7c1678 Use multiread where possible 2022-10-11 09:27:46 +00:00
alesapin
f695b75eef
Merge pull request #42147 from ClickHouse/more_logs_for_parent_parts 
Fix checking parent for old-format parts
 2022-10-08 12:08:15 +02:00
Alexander Tokmakov
ec2bca51ca Merge branch 'master' into improve_replica_recovery 2022-10-07 16:35:57 +02:00
Alexander Tokmakov
b8d9066004 Revert "Resurrect parallel distributed insert select with s3Cluster (#41535)" 
This reverts commit 860e34e760.
 2022-10-07 15:53:30 +02:00
alesapin
f1d93d52d1 Fix checking parent for old-format parts 2022-10-07 14:49:25 +02:00
Alexander Tokmakov
bb78bf1c70 improve replica recovery 2022-10-06 21:09:12 +02:00