Commit Graph

2941 Commits

Author SHA1 Message Date
taiyang-li
1b4b727d79 fix conflict 2022-01-10 10:43:13 +08:00
Igor Nikonov
102b9cdcfb Merge remote-tracking branch 'origin/master' into deduplication_token_7461 2022-01-09 19:15:33 +01:00
Igor Nikonov
0857a8d061 insert_deduplication_token support in non-replicated MergeTree 2022-01-09 19:06:09 +01:00
alesapin
733ed7cb54
Merge pull request #32061 from ianton-ru/MDB-15474
Change ZooKeeper path for zero-copy locks for shared data
2022-01-08 17:17:11 +03:00
Azat Khuzhin
28e64020bd Fix query cancellation in case of allow_experimental_parallel_reading_from_replicas
CI found one hanged query [1], where the problem was that becase of
allow_experimental_parallel_reading_from_replicas the Cancel packet was
read by receivePartitionMergeTreeReadTaskResponseAssumeLocked() and so
the executor was not cancelled, while the code in
MergeTreeBaseSelectProcessor was not ready for this:

<details>

    {
        "is_initial_query": 0,
        "elapsed": 1727.714379573,
        "is_cancelled": 0,
        "read_rows": "196577",
        "read_bytes": "1179462",
        "written_rows": "0",
        "written_bytes": "0",
        "query": "SELECT `CounterID`, `EventDate` FROM `test`.`hits` ORDER BY `CounterID` DESC, `EventDate` ASC LIMIT 50",
        ...
     }

In logs:

    2021.12.31 12:11:55.384735 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Debug> executeQuery: (from [::ffff:127.0.0.1]:58094, initial_query_id: e2966ca5-e836-44ef-8f8e-d1c1b32a>
    2021.12.31 12:11:55.454379 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Trace> ContextAccess (default): Access granted: SELECT(EventDate, CounterID) ON test.hits
    2021.12.31 12:11:55.457583 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Trace> datasets.hits_v1: Parallel reading from replicas enabled true
    2021.12.31 12:11:55.459739 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Trace> InterpreterSelectQuery: FetchColumns -> WithMergeableStateAfterAggregationAndLimit
    2021.12.31 12:11:55.471048 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Debug> datasets.hits_v1 (SelectExecutor): Key condition: unknown
    2021.12.31 12:11:55.476514 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Debug> datasets.hits_v1 (SelectExecutor): MinMax index condition: unknown
    2021.12.31 12:11:55.488302 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Debug> datasets.hits_v1 (SelectExecutor): Selected 2/2 parts by partition key, 2 parts by primary key, >
    2021.12.31 12:11:55.494020 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Debug> MergeTreeReverseSelectProcessor: Reading 1 ranges in reverse order from part 201403_20_20_0, app>
    2021.12.31 12:11:55.497644 [ 101532 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Debug> MergeTreeReverseSelectProcessor: Reading 138 ranges in reverse order from part 201403_19_19_2, a>
    2021.12.31 12:11:55.536372 [ 170601 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Compact part, average mark size is 83886080
    2021.12.31 12:11:55.558783 [ 171701 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069
    2021.12.31 12:11:55.563960 [ 171701 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069
    2021.12.31 12:11:55.577512 [ 171701 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069
    2021.12.31 12:11:55.585660 [ 171701 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069
    2021.12.31 12:11:55.613694 [ 170601 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069
    2021.12.31 12:11:55.730597 [ 170601 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069
    2021.12.31 12:11:55.743554 [ 171701 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Debug> MergingSortedTransform: Merge sorted 3 blocks, 65567 rows in 0.243999671 sec., 268717.5754429603>
    2021.12.31 12:11:55.744196 [ 170601 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069
    2021.12.31 12:11:55.890923 [ 170601 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069
    2021.12.31 12:11:55.891222 [ 170601 ] {7e5f551e-5960-4fda-9447-9bfdae4660c1} <Test> MergeTreeBaseSelectProcessor: Reading from Wide part, average mark size is 3069

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/33341/0685fd99855bacd0bce02507c00a3bd7709eea61/stress_test__address__actions_.html
2022-01-07 20:04:57 +03:00
taiyang-li
cf413f16a8 remove function checkPartMetadataCache 2022-01-07 18:37:08 +08:00
mergify[bot]
6fcd5a7d41
Merge branch 'master' into MDB-15474 2022-01-07 10:12:25 +00:00
taiyang-li
b91e0a533c fix style 2022-01-05 20:05:22 +08:00
taiyang-li
c0a9c2b916 refactor metadatacache to reduce using of USE_ROCKSDB 2022-01-05 19:51:50 +08:00
taiyang-li
abefefc719 support multi disk 2022-01-04 18:17:23 +08:00
李扬
2df2442ad0
Merge branch 'master' into hive_table 2022-01-04 01:26:16 -06:00
taiyang-li
37c7c282d7 fix unit test 2022-01-04 13:41:11 +08:00
taiyang-li
94d1f7ccb1 enable continue if cache data corrupted 2022-01-04 12:36:04 +08:00
taiyang-li
92b3f198aa throw exception if clickhouse use merge tree metdata cache but not compiled with rocksdb 2022-01-04 11:52:15 +08:00
taiyang-li
8360b47d82 remove redudant judge 2022-01-04 11:48:07 +08:00
taiyang-li
e13e1f5d7e add unit test && use enum_name 2022-01-04 11:44:10 +08:00
taiyang-li
3f6d830536 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2022-01-04 10:00:34 +08:00
taiyang-li
1e102bc1b2 merge master 2022-01-01 09:01:06 +08:00
alexey-milovidov
3f1810c049
Merge pull request #33327 from ClickHouse/rename_committed_to_active
Rename Committed data part state to Active
2021-12-31 14:40:35 +03:00
taiyang-li
770ff59155 fix building 2021-12-31 13:32:00 +08:00
taiyang-li
05dfc4eb25 fix building and stateless test 2021-12-31 11:13:38 +08:00
Alexander Tokmakov
187c43eba8 rename Committed state to Active 2021-12-30 23:45:38 +03:00
Vladimir C
794e906ffd
Merge pull request #33218 from amosbird/betternullablekey 2021-12-30 17:13:44 +03:00
alesapin
cbdba89d65 Merge branch 'master' into ianton-ru-MDB-15474 2021-12-30 11:21:19 +03:00
alesapin
7a3c874651 Merge branch 'master' into ianton-ru-MDB-15474 2021-12-29 19:01:43 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
taiyang-li
e5ff05c50d fix building 2021-12-29 16:20:29 +08:00
taiyang-li
63dc6821d2 fix ut and some bug 2021-12-29 12:31:54 +08:00
taiyang-li
98f37afc80 fix building 2021-12-28 19:29:01 +08:00
taiyang-li
2a1fe52b9f rename symbols && fix uts && add setting use_metadata_cache 2021-12-28 18:06:13 +08:00
taiyang-li
7dab7caa9d wrap cache related code with USE_ROCKSDB 2021-12-28 11:57:43 +08:00
taiyang-li
30ffa57bd5 remove unused log 2021-12-28 10:17:16 +08:00
Anton Ivashkin
0465aef15d Fixes by code review responces 2021-12-27 19:27:06 +03:00
OnePiece
555b20bc2e
Update MergeTreePartsMover.cpp 2021-12-27 21:38:51 +08:00
OnePiece
47f0e3c72f
Update MergeTreePartsMover.cpp 2021-12-27 20:46:51 +08:00
Amos Bird
1cbd05ae9e
Correctly prevent nullable primary keys if necessary. 2021-12-27 17:32:37 +08:00
taiyang-li
9036b18c2f merge master 2021-12-27 15:12:48 +08:00
taiyang-li
e7401d2a5e wrap rocksdb metacache related code with USE_ROCKSDB 2021-12-27 11:50:59 +08:00
taiyang-li
94a7a09598 Merge remote-tracking branch 'origin/master' into rocksdb_metacache 2021-12-27 10:14:24 +08:00
alexey-milovidov
0f8295de98
Merge pull request #33046 from ClickHouse/fix-prefetch-for-lc
Fix preferch for LowCardinality.
2021-12-23 08:56:16 +03:00
Dmitry Novik
1c83f31e51
Merge pull request #31885 from amosbird/optimizeprojectionmat
Optimize single part projection materialization
2021-12-22 19:36:40 +03:00
mergify[bot]
1e0642065b
Merge branch 'master' into deduplication_token_7461 2021-12-22 15:27:28 +00:00
Nikolai Kochetov
bb24a494ca Fix preferch for LowCardinality. 2021-12-22 15:26:16 +03:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3
Fix optimization with lazy seek for async reads from remote fs
2021-12-22 09:38:23 +03:00
Anton Ivashkin
e88b97dafb Fix typos 2021-12-21 19:56:29 +03:00
Amos Bird
f00cc3b944
optimize single part projection materialization 2021-12-21 22:39:06 +08:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474 2021-12-21 17:38:36 +03:00
Anton Ivashkin
33cbfc89ef Move logic for replicated part to StorageReplicatedMergeTree class 2021-12-21 17:29:50 +03:00
Anton Ivashkin
0c0bf66334 Merge master 2021-12-21 17:27:54 +03:00
kssenii
ebcb66a361 Fix 2021-12-21 10:58:02 +03:00
taiyang-li
2597925724 merge master 2021-12-21 15:55:39 +08:00
mergify[bot]
e21c04ff9c
Merge branch 'master' into fix-data-part-modification-time 2021-12-20 18:58:47 +00:00
Anton Ivashkin
c724b074ae Remove zero-copy version converter 2021-12-20 20:23:25 +03:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment
Loops remove postfix increment
2021-12-20 19:51:07 +03:00
Anton Ivashkin
f0b9a4327a Use table UUID in zero-copy shared label in ZooKeeper 2021-12-20 16:19:00 +03:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
Maksim Kita
3feab5a975 Containers iteration fix erase 2021-12-20 13:42:31 +03:00
save-my-heart
1abe978455 fix wrong modification_time after move part 2021-12-20 16:29:06 +08:00
Igor Nikonov
100ee92c64 insert_deduplication_token setting for INSERT statement
The setting allows a user to provide own deduplication semantic in Replicated*MergeTree
If provided, it's used instead of data digest to generate block ID
So, for example, by providing a unique value for the setting in each INSERT statement,
user can avoid the same inserted data being deduplicated

Inserting data within the same INSERT statement are split into blocks
according to the *insert_block_size* settings
(max_insert_block_size, min_insert_block_size_rows, min_insert_block_size_bytes).
Each block with the same INSERT statement will get an ordinal number.
The ordinal number is added to insert_deduplication_token to get block dedup token
i.e. <token>_0, <token>_1, ... Deduplication is done per block
So, to guarantee deduplication for two same INSERT queries,
dedup token and number of blocks to have to be the same

Issue: #7461
2021-12-19 13:15:45 +00:00
liyang
37ba8004ff Speep up mergetree starting up process 2021-12-18 16:39:59 +08:00
kssenii
40c266840c Fixes 2021-12-17 18:16:29 +03:00
alesapin
6bd7e425c6
Merge pull request #22535 from CurtizJ/sparse-serialization
Sparse serialization and ColumnSparse
2021-12-17 15:26:17 +03:00
taiyang-li
d033fc4c24 merge master and fix conflict 2021-12-17 15:11:21 +08:00
mergify[bot]
4722bc6649
Merge branch 'master' into fix_skipped_mutations 2021-12-16 13:25:00 +00:00
Alexander Tokmakov
1b02136941 fix skipping of some mutations 2021-12-15 21:19:29 +03:00
Anton Popov
ec46cbef20
Merge pull request #32756 from CurtizJ/fix-index-hypothesis
Fix race in skipping index of type `hypothesis`
2021-12-15 15:26:35 +03:00
Anton Popov
16312e7e4a Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-14 18:58:17 +03:00
Anton Popov
84d6715e3d fix race in skipping index of type hypothesis 2021-12-14 18:28:55 +03:00
nautaa
a96da58360 fix 2021-12-14 20:11:43 +08:00
nautaa
3469fb8165 fix 2021-12-14 19:40:35 +08:00
nautaa
0da3881ff4 flush all InMemoryDataParts when wal is not enabled 2021-12-14 16:31:17 +08:00
taiyang-li
8234d1176f merge master 2021-12-14 10:39:21 +08:00
tavplubix
60fae3af76
Follow-up to #32140 (#32667)
* Update LeaderElection.h

* Update LeaderElection.h
2021-12-13 16:41:46 +03:00
Anton Popov
a241103714
Merge pull request #32359 from CurtizJ/fix-non-materialized-skip-index
Fix usage of non-materialized skip indexes
2021-12-13 14:43:58 +03:00
Nikolai Kochetov
22e6fc1685
Merge pull request #32067 from amosbird/projection-fix23
Fix detaching parts with projections
2021-12-13 12:00:17 +03:00
mergify[bot]
47b18c959f
Merge branch 'master' into fix-non-materialized-skip-index 2021-12-13 07:03:02 +00:00
李扬
8675086104
Merge branch 'master' into hive_table 2021-12-12 09:01:46 -06:00
alexey-milovidov
0486c5f341
Merge pull request #32220 from azat/mt-fix-minmax-for-empty-part
[RFC] Fix minmax partition index during merge in case of empty part in future parts
2021-12-12 14:45:10 +03:00
tavplubix
25427719d4
Try fix 'Directory tmp_merge_<part_name>' already exists (#32201)
* try fix 'directory tmp_merge_<part_name>' already exists

* fix

* fix

* fix

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2021-12-10 16:29:51 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
Anton Popov
ea67abf3f0 fix custom serializations in native protocol 2021-12-08 21:59:36 +03:00
Anton Popov
61a5f8a61a add comments 2021-12-08 18:56:30 +03:00
Anton Popov
d8367334a3 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-08 18:26:19 +03:00
Mikhail f. Shiryaev
03927f5fc6
Revert "graphite: split tagged/plain rollup rules (for merges perfomance)"
This reverts commit 303552f515.
2021-12-08 15:49:21 +01:00
Nikita Mikhaylov
398d2f5984 Fix 2021-12-08 12:36:55 +00:00
Anton Popov
dd7387729b fix usage of non-materialized skip indexes 2021-12-07 20:21:27 +03:00
tavplubix
4f46ac6b30
Remove LeaderElection (#32140)
* remove LeaderElection

* try fix tests

* Update test.py

* Update test.py
2021-12-07 19:55:55 +03:00
Mikhail f. Shiryaev
5fc20b3e6a
Merge pull request #25122 from msaf1980/rollup_rules_type
GraphiteMergeTree Rollup rules type
2021-12-07 10:18:08 +01:00
msaf1980
303552f515
graphite: split tagged/plain rollup rules (for merges perfomance) 2021-12-06 16:49:08 +01:00
Nikolai Kochetov
f064f2cdaa
Use seq_cst semantic for MergeTreeBackgroundExecutor mertic. (#32125) 2021-12-06 15:21:39 +03:00
Azat Khuzhin
82e0e9d27c Fix minmax partition index during merge in case of empty part in future parts
It is possible to get empty part in future parts, during merging, that
can be created in case of there is information about this part in
zookeeper, but no replicas has it
(StorageReplicatedMergeTree::createEmptyPartInsteadOfLost())

And this may incorrectly initialize min value, which will be updated
after one more merge (since during that merge there will be no more
information about empty part) or OPTIMIZE.
2021-12-04 21:49:49 +03:00
tavplubix
17f8193435
Merge pull request #32157 from ClickHouse/fix_active_replicas_count_quorum
Fix active replicas count in quorum inserts
2021-12-03 12:39:36 +03:00
Anton Popov
f6be3d16fd
Merge pull request #24820 from kssenii/versioning
Versioning of aggregate function states
2021-12-03 01:41:44 +03:00
tavplubix
4606376734
Update ReplicatedMergeTreeSink.cpp 2021-12-02 21:51:23 +03:00
Alexander Tokmakov
252192bc6d fix active replicas count in quorum inserts 2021-12-02 21:02:11 +03:00
Anton Ivashkin
98bae1b313 Fix tests 2021-12-02 16:25:45 +03:00
taiyang-li
2f4e7e1d4e merge master 2021-12-02 19:48:21 +08:00
tavplubix
5400c5899d
Merge pull request #32063 from ClickHouse/fix_detach_dir_exists
Fix 'directory exists' error when detaching part
2021-12-02 14:04:21 +03:00
kssenii
8edf146687 Merge branch 'master' of github.com:ClickHouse/ClickHouse into versioning 2021-12-02 07:57:14 +00:00
Anton Ivashkin
e0a16a483e Merge master 2021-12-01 19:03:31 +03:00