thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-18 05:32:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
ClickHouse® is a real-time analytics DBMS
67,003 Commits 9,136 Branches 1,774 Tags 2 GiB
C++ 79.4%
Python 8.7%
Shell 4.3%
C 3.4%
Jinja 1.9%
Other 2%
fc46effe16
Go to file
Azat Khuzhin fc46effe16 Fix race between DETACH and merges 
CI reports failure of the 01442_merge_detach_attach test [1]:

<details>

    2021-06-21 02:25:43 01442_merge_detach_attach:                                              [ FAIL ] 122.37 sec. - result differs with reference:
    2021-06-21 02:25:43 --- /usr/share/clickhouse-test/queries/0_stateless/01442_merge_detach_attach.reference	2021-06-21 00:43:12.000000000 +0300
    2021-06-21 02:25:43 +++ /tmp/clickhouse-test/0_stateless/01442_merge_detach_attach.stdout	2021-06-21 02:25:43.211212197 +0300
    2021-06-21 02:25:43 @@ -0,0 +1 @@
    2021-06-21 02:25:43 +2

    2021.06.21 02:25:08.930896 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Debug> executeQuery: (from [::1]:36540, using production parser) (comment: '/usr/share/clickhouse-test/queries/0_stateless/01442_merge_detach_attach.sh') ALTER TABLE t DETACH PARTITION tuple()
    2021.06.21 02:25:08.931245 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> executeQuery: (from [::1]:36542, using production parser) (comment: '/usr/share/clickhouse-test/queries/0_stateless/01442_merge_detach_attach.sh') OPTIMIZE TABLE t FINAL
    2021.06.21 02:25:08.931826 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Trace> ContextAccess (default): Access granted: ALTER DELETE ON test_89nl0v.t
    2021.06.21 02:25:08.932159 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Trace> ContextAccess (default): Access granted: OPTIMIZE ON test_89nl0v.t
    2021.06.21 02:25:08.932889 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Information> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249): Detaching all_143_143_0
    2021.06.21 02:25:08.932921 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Selected 2 parts from all_143_143_0 to all_144_144_0
    2021.06.21 02:25:08.933530 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 4.60 TiB.
    2021.06.21 02:25:08.933705 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Information> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249): Detaching all_144_144_0
    2021.06.21 02:25:08.934215 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Merging 2 parts: from all_143_143_0 to all_144_144_0 into Compact
    2021.06.21 02:25:08.934280 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Information> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249): Detached 2 parts.
    2021.06.21 02:25:08.934948 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Selected MergeAlgorithm: Horizontal
    2021.06.21 02:25:08.936090 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_143_143_0, total 1 rows starting from the beginning of the part, column x
    2021.06.21 02:25:08.937621 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> MergeTreeSequentialSource: Reading 2 marks from part all_144_144_0, total 1 rows starting from the beginning of the part, column x
    2021.06.21 02:25:08.938124 [ 100543 ] {16eb4fe5-2d6b-4c81-a6be-c6a3b293edd3} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.
    2021.06.21 02:25:08.939928 [ 100543 ] {} <Debug> TCPHandler: Processed in 0.018537432 sec.
    2021.06.21 02:25:08.942140 [ 100543 ] {} <Debug> TCPHandler: Done processing connection.
    2021.06.21 02:25:08.948343 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Merge sorted 2 rows, containing 1 columns (1 merged, 0 gathered) in 0.014203821 sec., 140.80718139154246 rows/sec., 140.81 B/sec.
    2021.06.21 02:25:08.952021 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Trace> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249): Renaming temporary part tmp_merge_all_143_144_1 to all_143_144_1.
    2021.06.21 02:25:08.952869 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Warning> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Unexpected number of parts removed when adding all_143_144_1: 0 instead of 2
    2021.06.21 02:25:08.953264 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Trace> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (MergerMutator): Merged 2 parts: from all_143_143_0 to all_144_144_0
    2021.06.21 02:25:08.953913 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> MemoryTracker: Peak memory usage: 4.01 MiB.
    2021.06.21 02:25:08.958369 [ 100540 ] {bd4b710d-cd27-44ea-9dc9-e45d6ace0bd7} <Debug> MemoryTracker: Peak memory usage (for query): 4.01 MiB.
    ...
    2021.06.21 02:25:09.216075 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> executeQuery: (from [::1]:36544, using production parser) (comment: '/usr/share/clickhouse-test/queries/0_stateless/01442_merge_detach_attach.sh') SELECT count() FROM t HAVING count() > 0
    2021.06.21 02:25:09.229491 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> ContextAccess (default): Access granted: SELECT(x) ON test_89nl0v.t
    2021.06.21 02:25:09.232000 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
    2021.06.21 02:25:09.239907 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (SelectExecutor): Key condition: unknown
    2021.06.21 02:25:09.240358 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> test_89nl0v.t (efd364af-6721-4249-afd3-64af67214249) (SelectExecutor): Selected 1/1 parts by partition key, 1 parts by primary key, 1/1 marks by primary key, 1 marks to read from 1 ranges
    2021.06.21 02:25:09.241560 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> MergeTreeSelectProcessor: Reading 1 ranges from part all_143_144_1, approx. 2 rows starting from 0
    2021.06.21 02:25:09.256053 [ 58403 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> AggregatingTransform: Aggregating
    2021.06.21 02:25:09.256410 [ 58403 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> Aggregator: Aggregation method: without_key
    2021.06.21 02:25:09.257576 [ 58403 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> AggregatingTransform: Aggregated. 2 to 1 rows (from 2.00 B) in 0.013910412 sec. (143.777 rows/sec., 143.78 B/sec.)
    2021.06.21 02:25:09.257911 [ 58403 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Trace> Aggregator: Merging aggregated data
    2021.06.21 02:25:09.262595 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Information> executeQuery: Read 2 rows, 2.00 B in 0.045805192 sec., 43 rows/sec., 43.66 B/sec.
    2021.06.21 02:25:09.263337 [ 100543 ] {1699b404-b132-4b96-b04a-1c28cd062a79} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.

</details>

  [1]: https://clickhouse-test-reports.s3.yandex.net/25513/b96df135aee40b1a54a4fc3f0d4db89e11385564/functional_stateless_tests_(memory).html#fail1

The problem here is that there is a tiny race window between DETACH
and OPTIMIZE.

DETACH cancel merges, and checks that there no merges for parts
currently under lock, and later changes the state of parts. After it
allows merges.

However OPTIMIZE get parts for processing under lock and only when it
starts writing temporary part it checks that merges are not canceled.

So suppose the following scenario:

      T#OPTIMIZE                       T#DETACH

                                       - cancel merges
                                           - lock
                                             - check that no merges are in progress
                                           - unlock
    - lock
      - get parts

          *<--Here, state of the parts are not changed yet,
              and lock is not held,
              so OPTIMIZE may (and will) get those parts. -->*

      - write temporary part
        - check merges not canceled
    - unlock
                                           - change the parts state
                                       - allows merges

Plus this patch will also fail merges earlier in case of concurrent
DETACH.

Refs: #13746
Refs: #23315
2021-06-30 21:13:31 +03:00
.github Update 40_bug-report.md 2021-05-31 17:06:28 +03:00
base Add comment 2021-06-29 02:19:09 +03:00
benchmark Convert to python3 (#15007) 2020-10-02 19:54:07 +03:00
cmake Add comments for VERSION_REVISION vs DBMS_TCP_PROTOCOL_VERSION 2021-06-28 22:01:14 +03:00
contrib ARM fix startup crash 2021-06-30 12:27:20 +03:00
debian Auto version update to [21.8.1.1] [54453] 2021-06-28 00:50:20 +03:00
docker Merge pull request #25428 from ClickHouse/zookeeper_snapshots 2021-06-29 12:58:45 +03:00
docs Fix typo, applied changes from @brijrajparmar27 2021-06-30 15:49:01 +03:00
programs Merge pull request #25428 from ClickHouse/zookeeper_snapshots 2021-06-29 12:58:45 +03:00
src Fix race between DETACH and merges 2021-06-30 21:13:31 +03:00
tests Mark 01442_merge_detach_attach as long 2021-06-30 21:12:18 +03:00
utils Update version_date.tsv after release 21.3.13.9 2021-06-23 14:34:44 +03:00
website Update website/templates/index/quickstart.html 2021-06-29 15:14:12 +03:00
.arcignore Added .arcignore 2020-05-21 09:17:03 +03:00
.clang-format Fixed wrong code around Memory Profiler 2020-03-03 03:24:44 +03:00
.clang-tidy Enable clang-tidy for programs and utils 2020-05-18 04:19:50 +03:00
.editorconfig Changed tabs to spaces in editor configs and in style guide [#CLICKHOUSE-3]. 2017-04-01 11:35:09 +03:00
.gitattributes Union merge for arcadia_skip_list.txt to avoid frequent conflicts 2021-03-10 08:50:32 +03:00
.gitignore Add *.log/*.stderr/*.stdout into gitignore 2021-06-08 09:14:47 +03:00
.gitmodules Merge branch 'master' into arrow 2021-06-18 12:18:57 +00:00
.potato.yml Fix yamllint issues 2021-02-20 23:25:21 +03:00
.pylintrc Add pylintrc config 2021-01-26 23:35:56 +03:00
.vimrc Changed tabs to spaces in editor configs and in style guide [#CLICKHOUSE-3]. 2017-04-01 11:35:09 +03:00
.yamllint Drop truthy.check-keys from yamllint (does not supported on CI) 2021-02-21 06:15:36 +03:00
AUTHORS Update AUTHORS 2020-01-23 17:36:05 +03:00
CHANGELOG.md Update CHANGELOG.md 2021-06-19 13:32:18 +03:00
CMakeLists.txt Fix locating objcopy in macOS 2021-06-26 17:39:02 +04:00
CODE_OF_CONDUCT.md Add minimal code of conduct #9676 2020-03-16 12:44:28 +03:00
CONTRIBUTING.md Update CONTRIBUTING.md 2020-01-27 21:03:30 +03:00
docker-compose.yml Updated docker-compose.yml #1025 2017-07-26 20:05:32 +03:00
format_sources allow several <graphite> targets (#603) 2017-03-21 23:08:09 +04:00
LICENSE Update LICENSE 2021-01-25 17:00:13 +03:00
PreLoad.cmake Check if XCODE_IDE is true and avoid enforcing ninja in that case 2021-01-06 03:06:03 +04:00
README.md Update README.md (#25507) 2021-06-19 11:50:57 +03:00
release Remove rotten parts of release script 2021-04-25 02:11:31 +03:00
SECURITY.md Update SECURITY.md (outdated long time ago) 2021-01-21 16:48:28 +03:00
uncrustify.cfg Better .clang-format and uncrustify.cfg 2018-11-29 15:45:34 +03:00
ya.make Changes required for auto-sync with Arcadia 2020-04-16 15:31:57 +03:00

README.md

ClickHouse — open source distributed column-oriented DBMS

ClickHouse® is an open-source column-oriented database management system that allows generating analytical data reports in real time.

Useful Links

  • Official website has quick high-level overview of ClickHouse on main page.
  • Tutorial shows how to set up and query small ClickHouse cluster.
  • Documentation provides more in-depth information.
  • YouTube channel has a lot of content about ClickHouse in video format.
  • Slack and Telegram allow to chat with ClickHouse users in real-time.
  • Blog contains various ClickHouse-related articles, as well as announcements and reports about events.
  • Code Browser with syntax highlight and navigation.
  • Contacts can help to get your questions answered if there are any.
  • You can also fill this form to meet Yandex ClickHouse team in person.

Upcoming Events