Commit Graph

313 Commits

Author SHA1 Message Date
Amos Bird
d4cdf04683
Add missing locks (#34025) 2022-03-02 16:23:29 +01:00
alesapin
25375bc76b Remove unused param 2022-02-03 11:21:19 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-01 14:30:03 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts
Fix flushing of in-memory parts
2022-01-21 02:18:03 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
2022-01-10 22:01:41 +03:00
Alexander Tokmakov
187c43eba8 rename Committed state to Active 2021-12-30 23:45:38 +03:00
Anton Popov
4ebf61b809 proper checking of atomic flags 2021-12-29 01:03:55 +03:00
Anton Popov
a83832c03e fix flushing of in-memory parts 2021-12-27 18:54:28 +03:00
Alexander Tokmakov
1b02136941 fix skipping of some mutations 2021-12-15 21:19:29 +03:00
nautaa
0da3881ff4 flush all InMemoryDataParts when wal is not enabled 2021-12-14 16:31:17 +08:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
Vladimir Chebotarev
e8e3411ebc Minor fixes for StorageMergeTree. 2021-12-01 11:18:07 +03:00
tavplubix
78224ef273
Merge pull request #31738 from ClickHouse/fix_segfault_on_attach_partition
Fix segfault on attach parititon
2021-11-30 14:32:50 +03:00
Vladimir Chebotarev
100e61ab0e Rebase fix. 2021-11-29 10:15:00 +03:00
Vladimir Chebotarev
bb2f21b1ad Rebase fix. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
e414fa01e0 Rebase fix. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
c6e73d7b60 Synced clearOldMutations() with rest of changes. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
ec1f909d74 Fixed wrong lock order. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
12f36206cf Deadlock fix. 2021-11-25 17:11:54 +03:00
Vladimir Chebotarev
84ffcb479b Additional comments. 2021-11-25 17:11:52 +03:00
Vladimir Chebotarev
e0a453aadf Minor fix. 2021-11-25 17:11:29 +03:00
Vladimir Chebotarev
863f3565ad Test fixes. 2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
390416b24d Rebase fix. 2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
30cd133418 Fixed duplication commands bug. 2021-11-25 17:11:28 +03:00
Vladimir Chebotarev
aa3eecbd6d Fixed bug with frozen mutations. 2021-11-25 17:11:27 +03:00
Vladimir Chebotarev
6599124dba Attempt to skip mutations of different partitions in StorageMergeTree. 2021-11-25 17:10:11 +03:00
Alexander Tokmakov
1ab8b5d4df fix segfault on attach parititon 2021-11-25 13:04:39 +03:00
tavplubix
67753e5b83
Update StorageMergeTree.cpp 2021-11-19 20:05:45 +03:00
Alexander Tokmakov
8d7413b0d6 remove strange multimap from mutations 2021-11-19 19:54:43 +03:00
Alexander Tokmakov
51bd98294e better merge selecting backoff 2021-11-09 15:26:51 +03:00
tavplubix
ac4a9bcf23
Fix 00975_move_partition_merge_tree (#30717)
* Revert "Fix race between MOVE PARTITION and merges/mutations for MergeTree"

This reverts commit e9ce859b02.

* fix

* Update StorageMergeTree.cpp

Co-authored-by: alesapin <alesapin@gmail.com>
2021-10-27 18:19:41 +03:00
Alexander Tokmakov
2e7e195e77 change alter_lock to std::timed_mutex 2021-10-26 13:37:00 +03:00
alexey-milovidov
7beffb04ad
Merge pull request #29495 from ClickHouse/improvement_for_strange_mutations
Minor improvement for strange mutations
2021-10-17 20:57:06 +03:00
Nikolai Kochetov
bfcbf5abe0 Merge branch 'master' into removing-data-streams-folder 2021-10-17 10:42:37 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
Azat Khuzhin
07e8b2b3c7 Do not try to remove temporary paths that is currently in written by merge/mutation
v2: rebase against MergeTask
v3: rebase due to conflicts in src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
v4:
- rebase due to conflicts in src/Storages/MergeTree/MergeTask.cpp
- drop common/scope_guard_safe.h (not used)
2021-10-16 00:43:52 +03:00
Azat Khuzhin
e9ce859b02 Fix race between MOVE PARTITION and merges/mutations for MergeTree
From time to time 00975_move_partition_merge_tree test got failed, the
reason is that there is merge in progress that writes data to the source
table again:

<details>

    2021.10.12 13:20:40.243839 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> executeQuery: (from [::1]:38060) (comment: 00975_move_partition_merge_tree.sql) ALTER TABLE test_move_partition_src MOVE PARTITION 1 TO TABLE test_move_partition_dest;
    2021.10.12 13:20:40.244482 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_2_2_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_21_21_0
    ...
    2021.10.12 13:20:40.373487 [ 378 ] {} <Trace> test_t1tzb2.test_move_partition_src (467d7145-b47e-444e-867d-7145b47ea44e) (MergerMutator): Merged 6 parts: from 1_2_2_0 to 1_11_11_0
    ...
    2021.10.12 13:20:40.379750 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_15_15_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_28_28_0

</details>

And also remove cleaning of mutations since this will cause deadlock
after doing MOVE PARTITION under
currently_processing_in_background_mutex.

CI: https://clickhouse-test-reports.s3.yandex.net/0/a59c6b1c8eb47ebf77189a491ee0c3980b38e91c/functional_stateless_tests_
Fixes: 00975_move_partition_merge_tree
2021-10-13 00:14:24 +03:00
Alexander Tokmakov
5365cc686f Merge branch 'master' into improvement_for_strange_mutations 2021-10-07 13:26:58 +03:00
Alexander Tokmakov
bb32432943 more fixes 2021-10-05 00:13:18 +03:00
Azat Khuzhin
117e9e77c8 Apply max_untracked_memory/memory_profiler_{step,sample_probability} during mutate/merge 2021-10-03 17:39:07 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
Nikolai Kochetov
236d71ea94
Merge pull request #28582 from ClickHouse/rewrite-pushing-to-views
Rewrite PushingToViews
2021-09-27 21:19:11 +03:00
Amos Bird
34017c942e
More fixes 2021-09-25 13:38:56 +08:00
Nikolai Kochetov
a790d391c0 Merge branch 'master' into rewrite-pushing-to-views 2021-09-20 19:43:15 +03:00
Vladimir Chebotarev
db516e8c91
Removed sub-optimal mutation notifications in StorageMergeTree when merges are still going (#27552)
* Removed redundand mutation notifications in `StorageMergeTree` when merges are still going.
2021-09-17 16:19:36 +03:00
Nikolai Kochetov
a8c3b02598 Merge branch 'master' into rewrite-pushing-to-views 2021-09-17 15:38:11 +03:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask (#25165)
Introduced MergeTask and MutateTask
2021-09-17 00:19:58 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
kssenii
0c013ed91b Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2 2021-09-09 15:26:36 +03:00