thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 03:42:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
104,177 Commits 9,315 Branches 1,783 Tags 2.1 GiB
71da1b1379
Commit Graph

4858 Commits

Author SHA1 Message Date
Sema Checherinda
700181ca2e
Merge branch 'master' into memory/mutations-merges-fix 2022-11-28 14:14:57 +01:00
Kseniia Sumarokova
f35aa891d2
Merge pull request #43595 from azat/part.log.uuid-resubmit 
Add table_uuid to system.parts (resubmit)
 2022-11-28 12:02:44 +01:00
Robert Schulze
dce0b78198
Merge pull request #42778 from FArthur-cmd/annoy_cosine 
Add cosine distance for annoy
 2022-11-28 09:53:17 +01:00
Nikita Taranov
8ed5cfc265
Memory bound merging for distributed aggregation in order (#40879) 
* impl

* fix style

* make executeQueryWithParallelReplicas similar to executeQuery

* impl for parallel replicas

* cleaner code for remote sorting properties

* update test

* fix

* handle when nodes of old versions participate

* small fixes

* temporary enable for testing

* fix after merge

* Revert "temporary enable for testing"

This reverts commit cce7f8884c.

* review fixes

* add bc test

* Update src/Core/Settings.h
 2022-11-28 00:41:31 +01:00
Alexander Tokmakov
fc3a0bfdfa fix 2022-11-27 01:12:09 +01:00
Azat Khuzhin
6a057dec70 Add table_uuid to system.parts (resubmit) 
Can be useful if the table constantly recreated, i.e. in some tests.

v2: fix tests (disable for ordinary database)
v3: adjust test after merging replacing with empty parts pr
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit d0ce500f89)
 2022-11-26 20:16:38 +01:00
Han Fei
ee5be80d9f Merge branch 'master' into hanfei/aysnc-insert 2022-11-25 17:27:56 +01:00
Alexander Tokmakov
5cc99312d7 provide hint for loading uncommitted blocks in merge predicate 2022-11-25 16:41:20 +01:00
Nikolai Kochetov
660c942258 Merge branch 'master' into read-from-mt-in-io-pool 2022-11-25 13:56:48 +00:00
Alexander Tokmakov
ace36e2267
Merge branch 'master' into fix_parts_to_do_detection 2022-11-25 15:15:56 +03:00
HarryLeeIBM
fc9880ea5a Fix Bloom filter serialization for s390x 2022-11-24 14:37:05 -08:00
HarryLeeIBM
a35c5a914f Revert "Fix bloom filter serialization endian issue for s390x" 
This reverts commit 8f7970d598.
 2022-11-24 13:44:38 -08:00
HarryLeeIBM
8f7970d598 Fix bloom filter serialization endian issue for s390x 2022-11-24 13:27:02 -08:00
Nikolai Kochetov
d53439390f Review fixes. 2022-11-24 16:04:35 +00:00
alesapin
52821a331a
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:04:03 +01:00
alesapin
188069117e
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:02:26 +01:00
alesapin
5355f82448
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:02:19 +01:00
Sema Checherinda
6fd7dcf26f
Merge branch 'master' into lock-free-drop-partition 2022-11-24 12:34:19 +01:00
alesapin
f64f0461fe
Merge pull request #43288 from ClickHouse/Enable_compress_marks_and_primary_key 
Enable compress marks and primary key for CI
 2022-11-24 12:24:04 +01:00
Raúl Marín
4d3217cb18
Merge branch 'master' into perf_experiment 2022-11-24 10:32:12 +01:00
Han Fei
28564df6f1
Merge branch 'master' into hanfei/aysnc-insert 2022-11-24 09:12:43 +01:00
Han Fei
d0f5fcdf09 update ut and refine logs 2022-11-23 21:07:59 +01:00
Nikolai Kochetov
62debf42b4 Aggregating in order [in progress]. 2022-11-23 18:38:12 +00:00
Sema Checherinda
9f2c00d434 add tags to tests, fix error message 2022-11-23 15:16:10 +00:00
Sema Checherinda
7d748602c0 always write creation_csn 2022-11-23 15:16:10 +00:00
Sema Checherinda
e29d7c2e00 fix typos 2022-11-23 15:16:10 +00:00
Sema Checherinda
b4a21cab68 removal_csn can be PrehistoricCSN iff removal_tid is also PrehistoricCSN when cl removind an old part 2022-11-23 15:16:10 +00:00
Sema Checherinda
eccb62e4c3 work with the latest remarks 2022-11-23 15:16:09 +00:00
Sema Checherinda
a29437e179 delete verbose logging about cleaning old parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
f49936b9a0 remove tmp dir from inmemory part when freeze, more logs about old parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
3a8024002d fix race in read full path from deleting part, add more logs in grabOldParts 2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208 fix isolation breach, droping outdated part should't be ignored 2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25 fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part 2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104 less pedantic checks in operations, rely on existed tx functionality 2022-11-23 15:16:09 +00:00
Sema Checherinda
a4d1adf945 in order to restore partition do not write empty inmemory part 2022-11-23 15:16:09 +00:00
Sema Checherinda
00e7b4e195 fix race on remove, add creation csn for removed parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0 no groupping parts for cover with empty 2022-11-23 15:16:08 +00:00
Sema Checherinda
597421014e fix style check 2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a fix several critical races, fix tests 2022-11-23 15:16:08 +00:00
Sema Checherinda
8977e499c2 address 01825_type_json_18 test 2022-11-23 15:16:08 +00:00
Sema Checherinda
6d929266e0 address 01825_type_json_schema_race_long test 2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070 address the review comments 2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9 Delete parts over replacing them with empty parts 2022-11-23 15:16:08 +00:00
Alexander Tokmakov
c0133199c8
Revert "Add table_uuid to system.parts" 2022-11-23 16:41:43 +03:00
Alexey Milovidov
bedfe68503
Merge branch 'master' into nv/fix-mutate-fetch 2022-11-22 20:06:56 +01:00
Azat Khuzhin
2cfbc27f13 Fix incorrect memory accounting because of merges/mutations 
In case of merges/mutations, separate MemoryTracker object is used, to
track memory separately, to report this metrics.

However, the tail of those allocations hadn't been taken into account,
and in case of lots of mutations/merges it can be significant.

100% reproducible with the following example:

Start the server w/o frequent sync with RSS:

    $ clickhouse-server -C /src/programs/server/config.xml -- --asynchronous_metrics_update_period_s=86400

Run the following snippet:

    create table data (key Int, value Nullable(Decimal(38, 0))) engine=ReplicatedMergeTree('/ch/data', '1') order by key
    insert into data values (1, -13028236692093846346337460743176821145)
    alter table data modify column value UInt256
    Code: 341. DB::Exception: Received from localhost:9000. DB::Exception: Exception happened during execution of mutation 'mutation_2.txt' with part 'all_1_1_0' reason: 'Code: 407. DB::Exception: Convert overflow: while executing 'FUNCTION _CAST(value :: 1, 'UInt256' :: 2) -> _CAST(value, 'UInt256') UInt256 : 3': (while reading from part ./store/dd9/dd9ef183-1270-4877-af22-8a000bdcc1b8/all_1_1_0/): While executing MergeTreeInOrder. (DECIMAL_OVERFLOW) (version 22.12.1.1)'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED)

And track the memory:

    select value/1e9 from system.metrics where metric like 'MemoryTracking'

Reported-by: @filimonov
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Azat Khuzhin
5da2f52722 Use Int64 over UInt64 for prev_untracked_memory* in MemoryTrackerThreadSwitcher 
Since those types originally is Int64

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Azat Khuzhin
05f30aaad4 Fix logger name for MutateFromLogEntryTask 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Raúl Marín
4aa29b6a63 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-11-22 19:09:00 +01:00
Han Fei
32c7fbec55 use another dedup path for async inserts 2022-11-22 15:12:00 +01:00
Sergei Trifonov
59f422f10c
Merge branch 'master' into storage-system-moves 2022-11-22 14:46:00 +01:00
Kseniia Sumarokova
392963012a
Merge pull request #43404 from azat/part.log.uuid 
Add table_uuid to system.parts
 2022-11-22 14:25:34 +01:00
Nikolai Kochetov
9e84a351c7 Fixing special build. Disable setting by default. 2022-11-22 11:24:24 +00:00
Raúl Marín
8776b2f566 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-11-22 11:45:36 +01:00
Anton Popov
0479d8d9f5
Merge pull request #43387 from ClickHouse/fix_default_value_used_in_row_level_filter 
Fix default value used in row level filter
 2022-11-22 01:46:45 +01:00
Sergei Trifonov
94a7614c45
Merge branch 'master' into storage-system-moves 2022-11-21 20:10:12 +01:00
Nikolai Kochetov
c305afd77a Fix race. 2022-11-21 18:26:31 +00:00
serxa
faef2e9b00 fix moves list entry creation 2022-11-21 17:08:54 +00:00
serxa
c425da06b2 fix column name 2022-11-21 16:16:53 +00:00
Azat Khuzhin
d0ce500f89 Add table_uuid to system.parts 
Can be useful if the table constantly recreated, i.e. in some tests.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-21 12:11:51 +01:00
alesapin
b78b602ea1
Merge branch 'master' into Enable_compress_marks_and_primary_key 2022-11-21 12:05:04 +01:00
Nikolai Kochetov
be1a8054c7
Merge branch 'master' into read-from-mt-in-io-pool 2022-11-21 11:27:41 +01:00
Nikolai Kochetov
4a1774c5ac Fixing build. 2022-11-21 10:27:09 +00:00
Raúl Marín
ed0c174c0c Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-11-21 11:02:31 +01:00
zhongyuankai
a3a3261745
Merge branch 'master' into optimize_ttl 2022-11-19 16:32:26 +08:00
zhongyuankai
ce9b4866ea
Update MergeTreeDataMergerMutator.cpp 2022-11-19 16:31:47 +08:00
Alexander Gololobov
f004eea413 Add columns required fro defaults calculation 2022-11-18 23:42:45 +01:00
alesapin
c7d648174b
Merge pull request #43328 from ClickHouse/better-format-version-check 
Check content of `format_version` file in `MergeTreeData`
 2022-11-18 22:34:37 +01:00
Nikolai Kochetov
5da1d893fd
Merge branch 'master' into read-from-mt-in-io-pool 2022-11-18 21:10:45 +01:00
Nikolai Kochetov
03c35e146b Refactor a bit. 2022-11-18 20:09:20 +00:00
serxa
8e65b799c2 fix test + add one more column 2022-11-18 20:04:11 +00:00
Han Fei
b80a2d6c89 address comments 2022-11-18 17:22:05 +01:00
serxa
a05a1b0133 Merge branch 'storage-system-moves' of github.com:ClickHouse/ClickHouse into storage-system-moves 2022-11-18 15:35:00 +00:00
Sergei Trifonov
77100896e3
Merge branch 'master' into storage-system-moves 2022-11-18 16:33:43 +01:00
alesapin
0f3eb51aed Disable compressed marks by default 2022-11-18 15:54:48 +01:00
Han Fei
74c9ba8df0
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-11-18 11:16:38 +01:00
Han Fei
3dc72b6c56
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-11-18 11:15:51 +01:00
Han Fei
8c823bcaab
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-11-18 11:15:45 +01:00
Han Fei
035e933196
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-11-18 11:15:35 +01:00
Han Fei
f8ac75a054
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-11-18 11:15:14 +01:00
Han Fei
d41094207d
Update src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-11-18 11:14:56 +01:00
Nikolai Kochetov
679591cf0a Try fix race. 2022-11-17 19:51:47 +00:00
alesapin
29a08f64af Merge branch 'master' into Enable_compress_marks_and_primary_key 2022-11-17 15:53:37 +01:00
Antonio Andelic
d324e8cc3e Write to first non-readonly disk 2022-11-17 14:52:35 +00:00
Antonio Andelic
d41af8b741 Check and compare content of format_version file 2022-11-17 13:41:36 +00:00
alesapin
06f395d946 Add comments 2022-11-17 13:10:14 +01:00
alesapin
a79d83549f Merge branch 'master' into fix_parts_to_do_detection 2022-11-17 12:30:05 +01:00
Raúl Marín
97d6fc3071 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-11-17 11:48:46 +01:00
Han Fei
221c4e5912 tiny changes 2022-11-16 17:59:08 +01:00
Han Fei
14701670b5 Merge branch 'master' into hanfei/aysnc-insert 2022-11-16 17:24:36 +01:00
Nikolai Kochetov
17b75d3f5c Fix metrics and progress. 2022-11-16 14:10:56 +00:00
Smita Kulkarni
040f2b62d9 Enable compress marks and primary key 
Implementation:
* Set compress_marks and compress_primary_key to true by default.
* Updated getIndexExtensionFromFilesystem to only check for compressed file extension , else return default. When both are present, this function was returning uncompressed file extension giving rise to error as file got removed.
Testing:
* Updated tests where bytes size or compressed data is checked to check with compress_marks and compress_primary_key set to false.
 2022-11-16 14:23:58 +01:00
filimonov
dcb76f6bbf
Allow autoremoval of old parts if detach_not_byte_identical_parts enabled 
Allow autoremoval of old & detached parts if detach_not_byte_identical_parts enabled.

See also #28708 #37975
 2022-11-16 14:16:42 +01:00
alesapin
2012f4bd36 Fix use-after-free 2022-11-16 11:50:51 +01:00
Alexey Milovidov
04687c0d69
Merge pull request #43206 from ClickHouse/miscellaneous-changes-parallel-replicas 
Miscellaneous changes
 2022-11-16 05:30:03 +01:00
Nikolai Kochetov
1b1d23c100 Clumsy attempt to fix progress. 2022-11-15 21:23:18 +00:00
Alexander Tokmakov
69b067b81c
Merge pull request #43244 from ClickHouse/fix-replicated-merge-tree-shared-id 
Don't throw if shared ID already created in `StorageReplicatedMergeTree`
 2022-11-15 23:31:51 +03:00
Nikolai Kochetov
cf1afbfb84 Temporarily enable allow_asynchronous_read_from_io_pool_for_merge_tree by default to see thest results. 2022-11-15 17:47:26 +00:00
Nikolai Kochetov
29098045d8 Try read from MergeTree in background pool. 2022-11-15 17:38:37 +00:00
Sema Checherinda
7e73b187cc
Merge pull request #43159 from Algunenano/over-read 
Fix several buffer over-reads
 2022-11-15 15:04:24 +01:00
alesapin
a12f2d14dc Simpler check 2022-11-15 14:46:38 +01:00
Anton Popov
8c66c759d4
Merge pull request #43133 from amosbird/issue_43111 
Fix Nullable keys in hyperrectangle
 2022-11-15 14:30:13 +01:00
alesapin
4ec11d0f58 Remove redundant branch 2022-11-15 13:58:53 +01:00
Antonio Andelic
0e5e58bed9 Remove exception if shared ID already created 2022-11-15 12:18:24 +00:00
zhongyuankai
35690c1972 Merge branch 'master' into optimize_ttl 2022-11-15 08:49:37 +08:00
zhongyuankai
16f697f61d fix test 2022-11-15 08:47:51 +08:00
Alexey Milovidov
6e01b2b2a1
Merge pull request #43203 from ClickHouse/fix-typo-2 
Fix typo
 2022-11-15 01:39:34 +01:00
alesapin
6a50a32cfc Fix parts_to_do 2022-11-15 00:53:20 +01:00
alesapin
f53ada2bbd Revert "Fix bug in replication queue which can lead to premature mutation finish" 
This reverts commit 3b5424811c.
 2022-11-14 22:59:09 +01:00
alesapin
3b5424811c Fix bug in replication queue which can lead to premature mutation finish 2022-11-14 21:53:16 +01:00
Alexander Tokmakov
c64109a2de
Merge branch 'master' into fetch-remove-covered 2022-11-14 18:23:16 +03:00
zhongyuankai
ea11d80536 Priority is given to deleting completely expired Parts 2022-11-14 23:03:56 +08:00
Han Fei
11f93dec3f tmp commit 2022-11-14 15:53:45 +01:00
Raúl Marín
4988e387f2 Merge remote-tracking branch 'blessed/master' into over-read 2022-11-14 12:28:31 +01:00
Alexey Milovidov
4edd8b0135 Remove unused method 2022-11-14 08:07:27 +01:00
Alexey Milovidov
17693de053 Better logs 2022-11-14 08:01:48 +01:00
Alexey Milovidov
9c48664f11 Better logs 2022-11-14 07:48:50 +01:00
Alexey Milovidov
582db233c4 Better logs 2022-11-14 07:22:17 +01:00
Alexey Milovidov
99f84d8095 Better logs 2022-11-14 07:22:02 +01:00
Alexey Milovidov
df01689ccc Better logs 2022-11-14 07:13:42 +01:00
Alexey Milovidov
a77032fafa Miscellaneous changes 2022-11-14 06:54:58 +01:00
Alexey Milovidov
15202833b1 Miscellaneous changes 2022-11-14 06:48:49 +01:00
Alexey Milovidov
53ee7446dc Miscellaneous changes 2022-11-14 06:46:37 +01:00
Alexey Milovidov
cc1855ea33 Miscellaneous changes 2022-11-14 06:45:52 +01:00
Alexey Milovidov
85dc692f7c Miscellaneous changes 2022-11-14 06:37:55 +01:00
Alexey Milovidov
60c96072e1 Miscellaneous changes 2022-11-14 06:37:19 +01:00
Alexey Milovidov
e080161ba5 Miscellaneous changes 2022-11-14 06:32:18 +01:00
Alexey Milovidov
bfea3deed9 Miscellaneous changes 2022-11-14 06:09:03 +01:00
Alexey Milovidov
b9471e492a Fix typo 2022-11-13 22:52:21 +01:00
Alexey Milovidov
127631ee47
Merge branch 'master' into perf_experiment 2022-11-12 18:58:25 +01:00
Filatenkov Artur
c7c52a570f
Update MergeTreeIndexAnnoy.cpp 2022-11-12 12:43:25 +03:00
FArthur-cmd
5eebc27d24 improve test 2022-11-12 09:23:49 +00:00
Nikolai Kochetov
9106910de8 Try read from MergeTree in background pool. 2022-11-11 20:47:11 +00:00
Igor Nikonov
ee5756b5b4
Merge pull request #43122 from ClickHouse/revert-43116-revert-42607-igor/insert_zk_retries_retry 
Revert "Revert " Keeper retries during insert (clean)""
 2022-11-11 18:22:38 +01:00
Raúl Marín
54db7c6520 Enforce checking read output 2022-11-11 10:56:18 +01:00
Amos Bird
661c541e57
Fix Nullable keys in hyperrectangle. 2022-11-11 11:14:05 +08:00
Alexander Tokmakov
bb00a246f1 priority for DROP_RANGEs, assertion for others 2022-11-10 20:53:05 +01:00
Alexander Tokmakov
e415fc6e95
Revert "Optimize TTL merge, completely expired parts can be removed in time" 2022-11-10 19:38:55 +03:00
Igor Nikonov
f8c907c463
Revert "Revert " Keeper retries during insert (clean)"" 2022-11-10 13:14:04 +01:00
Alexander Tokmakov
fb8c527a2f
Revert " Keeper retries during insert (clean)" 2022-11-10 14:12:24 +03:00
Igor Nikonov
1b9be27fdb
Merge pull request #42607 from ClickHouse/igor/insert_zk_retries_retry 
Keeper retries during insert (clean)
 2022-11-10 12:01:00 +01:00
Antonio Andelic
1954ff385d
Merge pull request #43076 from canhld94/ch_canh_minor 
Better logging for mark range filtering on projection parts
 2022-11-10 09:33:12 +01:00
alesapin
9e1627fcdd
Merge pull request #43060 from ClickHouse/fix_projection_part 
Fix projection part removal with zero-copy replication
 2022-11-09 18:15:49 +01:00
Igor Nikonov
379113f8f1 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-11-09 13:03:32 +00:00
Alexander Tokmakov
1c6a617207
Merge branch 'master' into fix_intersecting_parts2 2022-11-09 14:38:18 +03:00
Maksim Kita
768ada118b Analyzer AST key condition crash fix 2022-11-09 11:41:39 +01:00
Vladimir C
82bf992099
Merge pull request #42869 from zhongyuankai/optimze_ttl_merge 2022-11-09 11:26:46 +01:00
Duc Canh Le
b9a3d090ed use const string & 2022-11-09 13:07:24 +08:00
Duc Canh Le
901e1e2fef better logging for mark range filtering on projection parts 2022-11-09 12:01:51 +08:00
alesapin
761274d857 Fix wrong logic 2022-11-09 01:14:03 +01:00
Igor Nikonov
345304abe0 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-11-08 20:26:50 +00:00
alesapin
a31a79020f Add comment 2022-11-08 16:27:26 +01:00
alesapin
6f059be748 Fix bug with temporary projection parts deletion 2022-11-08 15:37:03 +01:00
Alexander Tokmakov
eb19df0094 fix 2022-11-08 15:21:25 +01:00
Maksim Kita
0aa4fd3c2d Fixed tests 2022-11-08 14:29:17 +01:00
Robert Schulze
2c568df714
Merge pull request #42968 from ClickHouse/bitcast 
Less use of CH-specific bit_cast()
 2022-11-08 10:54:10 +01:00
Robert Schulze
bb507356ef
Merge pull request #42209 from canhld94/ch_canh_fix_prefix_not_like 
Add `notLike` to key condition atom map
 2022-11-08 10:07:21 +01:00
Antonio Andelic
7067f9e0e1
Merge pull request #42947 from azat/mergetree/part-load-tiny 
Attach thread pool for loading parts to the query
 2022-11-08 09:17:32 +01:00
zhongyuankai
37099e57f4
Merge branch 'master' into optimze_ttl_merge 2022-11-08 12:26:46 +08:00
Azat Khuzhin
2bd4a71150
Merge branch 'master' into fetch-remove-covered 2022-11-08 00:55:00 +01:00
serxa
ea17b30358 Merge branch 'storage-system-moves' of github.com:ClickHouse/ClickHouse into storage-system-moves 2022-11-07 20:33:31 +00:00
serxa
59408ebf57 fill more fields 2022-11-07 20:33:25 +00:00
Sergei Trifonov
6d3b96ee42
Merge branch 'master' into storage-system-moves 2022-11-07 21:31:27 +01:00
alesapin
257e766e0e Trying to fix two flaky tests 2022-11-07 20:49:41 +01:00
Alexander Tokmakov
9210e586d2 fix 2022-11-07 20:27:18 +01:00
Alexander Tokmakov
b8174a63a8 Merge branch 'master' into fix_intersecting_parts2 2022-11-07 20:26:36 +01:00
Kruglov Pavel
48163ff9c2
Merge pull request #42938 from amosbird/issue_42771 
Fix NOT_FOUND_COLUMN_IN_BLOCK in some rare case
 2022-11-07 13:10:12 +01:00
Duc Canh Le
bd2bd7149d fix typo 2022-11-07 18:22:50 +08:00
Duc Canh Le
f7e65ebd72 make comments more clear 2022-11-07 16:25:36 +08:00
Robert Schulze
cdaf0becfe
Merge branch 'master' into bitcast 2022-11-07 09:24:52 +01:00
Robert Schulze
f3f50530d1
Merge branch 'master' into ch_canh_fix_prefix_not_like 2022-11-07 09:14:02 +01:00
Duc Canh Le
69d6d42443 NOT LIKE only work for 'prefix%' 2022-11-06 15:26:19 +08:00
zhongyuankai
ef6065b331 fix test 2022-11-06 13:40:26 +08:00
Igor Nikonov
e94b9cd003 Fixes 
+ enable retries in integration test
+ typo in fault injection traces
 2022-11-05 22:28:32 +00:00
Robert Schulze
9c066e964d
Less use of CH-specific bit_cast() 
Converted usage of CH-custom bit_cast to std::bit_cast if possible, i.e.
when
  sizeof(From) == sizeof(To).
(The CH-custom bit_cast is able to deal with sizeof(From) != sizeof(To).)

Motivation for this came from #42847 where it is not clear how the
internal bit_cast should behave on big endian systems, so we better
avoid that situation as much as possible.
 2022-11-04 15:52:48 +00:00
Amos Bird
d901ead1bc
Done 2022-11-04 17:26:44 +08:00
zhongyuankai
049a593272 Merge branch 'master' into optimze_ttl_merge 2022-11-04 17:22:38 +08:00
zhongyuankai
e773eb2fd1 batter 2022-11-04 17:19:34 +08:00
Azat Khuzhin
7032b2b98a Attach thread pool for loading parts to the query 
Detach should be done via CurrentThread::defaultThreadDeleter()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-04 09:06:12 +01:00
Azat Khuzhin
cdd09fd72a Improve messages for loading data parts (add part type) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-04 09:05:06 +01:00
Amos Bird
7186898ffa
Experiment 2022-11-04 00:29:17 +08:00
Antonio Andelic
66ffdda973
Merge branch 'master' into optimize-entire-partition 2022-11-03 09:52:13 +01:00
Nikolay Degterinsky
30ad1a6826
Merge branch 'master' into perf_experiment 2022-11-03 02:18:21 +03:00
Alexander Tokmakov
e67d05604c Merge branch 'master' into fix_intersecting_parts2 2022-11-02 19:30:27 +01:00
Igor Nikonov
9a315b7899 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-11-02 15:32:54 +00:00
Alexander Tokmakov
00c9e50ee3 fix race between INSERT and ALTER PARTITION 2022-11-02 16:25:26 +01:00
Alexander Tokmakov
acbad99b24 remove abandonable_lock part 2 2022-11-02 14:28:49 +01:00
Antonio Andelic
c4b717b343
Update src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2022-11-02 08:24:54 +01:00
Duc Canh Le
ec382d7903 add comment 2022-11-02 09:50:41 +08:00
Duc Canh Le
c6598dc66d
Merge branch 'master' into ch_canh_fix_prefix_not_like 2022-11-02 09:42:01 +08:00
Alexander Tokmakov
4f50a999bb remove abandonable_lock part 1 2022-11-01 21:35:25 +01:00
Igor Nikonov
e76c3c381f Better handling tables in read-only mode during insert 
+ throw immediately if table has no metadata in zk
+ stop retries in case of shutdown
+ check if table is readonly at the begining of every retry
 2022-11-01 18:47:51 +00:00
Anton Popov
88674ef964
Merge pull request #42794 from CurtizJ/better-code-in-check-part 
Slightly better code with projections
 2022-11-01 16:05:27 +01:00
zhongyuankai
c1e2b2af74 Optimize TTL merge, completely expired parts can be removed in time 2022-11-01 22:58:15 +08:00
Anton Popov
876dca48da
Merge pull request #36969 from CurtizJ/dynamic-columns-14 
Support `Object` type inside other types
 2022-11-01 15:20:02 +01:00
Duc Canh Le
07d9720ed5
Merge branch 'master' into ch_canh_fix_prefix_not_like 2022-11-01 22:16:09 +08:00
Maksim Kita
fe48a1ce7e
Merge pull request #42681 from kitaisreal/merge-tree-indexes-use-actions-dag 
MergeTree indexes use ActionsDAG
 2022-11-01 13:55:17 +03:00
Duc Canh Le
3545db0a6c
Merge branch 'master' into ch_canh_fix_prefix_not_like 2022-11-01 16:54:26 +08:00
Yakov Olkhovskiy
bab0e06e3d
Merge pull request #42458 from Clark0/match-func-use-index 
match function can use index if it's a condition on string prefix #37333
 2022-10-31 17:33:58 -04:00
Igor Nikonov
9cc64a0fcc Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-31 18:21:21 +00:00
Igor Nikonov
2d5050ea66 Fix review comments 2022-10-31 18:20:58 +00:00
Anton Popov
2ae3cfa9e0
Merge branch 'master' into dynamic-columns-14 2022-10-31 16:15:19 +01:00
Alexander Tokmakov
599ccb9939
Update MergeTreeData.h 2022-10-31 16:18:17 +03:00
Alexander Tokmakov
b2b9479afa
Update src/Storages/MergeTree/MergeTreeData.h 
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
 2022-10-31 16:15:35 +03:00
Maksim Kita
bca22ec5f5 Fixed code review issues 2022-10-31 12:30:00 +01:00
Maksim Kita
115fcaffc5 Fixed tests 2022-10-31 11:30:56 +01:00
Duc Canh Le
d309e8793a
Merge branch 'master' into ch_canh_fix_prefix_not_like 2022-10-31 15:13:39 +08:00
Alexander Tokmakov
f53df7870c fix race between drop and failed insert 2022-10-30 17:59:47 +01:00
Alexander Tokmakov
c68ab231f9 fix accessing part in Deleting state 2022-10-30 17:30:51 +01:00
FArthur-cmd
607b81989d fix build 2022-10-30 14:57:10 +00:00
Anton Popov
8de75bf29c better code with projections 2022-10-29 14:39:59 +00:00
clarkcaoliu
ca5dbe88db match function can use index if prefix 2022-10-29 18:25:29 +08:00
Filatenkov Artur
21c024b043
Merge branch 'master' into annoy_cosine 2022-10-28 20:15:53 +03:00
FArthur-cmd
f187d4e1d4 add cosine distance for annoy and fix docs for cosineDistance 2022-10-28 17:03:35 +00:00
Igor Nikonov
d09deaf10f Fix style check 2022-10-28 15:17:05 +00:00
Igor Nikonov
937dd44399 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-28 15:09:23 +00:00
Igor Nikonov
c53b96a11c Add more comments 2022-10-28 15:09:01 +00:00
Alexey Milovidov
f64e632707
Merge pull request #42469 from amosbird/issue_42456 
Fix misbehavior of key analysis
 2022-10-28 16:18:00 +02:00
Antonio Andelic
5fae8b0cc9 Merge branch 'master' into optimize-entire-partition 2022-10-28 11:44:10 +00:00
Sergei Trifonov
836595c7ae
Merge branch 'master' into storage-system-moves 2022-10-28 11:39:38 +02:00
Duc Canh Le
848f5b5d65
Merge branch 'master' into ch_canh_fix_prefix_not_like 2022-10-28 13:40:17 +08:00
Duc Canh Le
c080964391 correct 'notLike' in key condition 2022-10-28 13:39:01 +08:00
Duc Canh Le
69311bd5cd Merge branch 'master' into ch_canh_fix_prefix_not_like 2022-10-28 11:59:50 +08:00
Raúl Marín
59686c3796
Merge branch 'master' into perf_experiment 2022-10-28 00:51:56 +02:00
Igor Nikonov
3682913172 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-27 22:01:53 +00:00
Igor Nikonov
a601c166ad Add clean up for faults injected before zk request 
+ for multi-op request, it create responses in multi response
+ some polishing
 2022-10-27 21:35:13 +00:00
Anton Popov
7715afa595
Merge pull request #42618 from CurtizJ/refactor-data-part 
Try to save `IDataPartStorage` interface
 2022-10-27 22:20:44 +02:00
Igor Nikonov
f773436de5 Fix flaky test 2022-10-27 13:34:07 +00:00
Raúl Marín
891484b462 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-27 13:17:07 +02:00
Nikolai Kochetov
24e1aaf324
Merge pull request #42576 from ClickHouse/table-function-merge-invalid-where-type 
Invalid type in where for Merge table (logical error)
 2022-10-27 13:03:27 +02:00
Antonio Andelic
10aa7091ca
Merge pull request #42605 from ClickHouse/correctly-handle-prewhere-type 
Correctly handle non-native integers in PREWHERE
 2022-10-27 09:25:38 +02:00
Igor Nikonov
649e8d302c Sync with retiring branch igor/insert_zk_retries 2022-10-26 17:41:44 +00:00
Igor Nikonov
8435cba352 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-26 17:35:47 +00:00
Maksim Kita
f04f663818 Fixed tests 2022-10-26 18:55:14 +02:00
Nikolai Kochetov
1520bcd53c Fix error code. 2022-10-26 15:50:10 +00:00
Anton Popov
ca4df62e37
Merge branch 'master' into refactor-data-part 2022-10-26 17:49:20 +02:00
Maksim Kita
1c17e9d454 Fixed tests 2022-10-26 15:54:37 +02:00
Nikolai Kochetov
db422fa903
Merge pull request #42410 from ClickHouse/fix-multi-arg-monotonicity 
Fix possible LOGICAL_ERROR in binary ariphmetics monotonicity.
 2022-10-26 14:11:23 +02:00
Kruglov Pavel
219553df8d
Merge pull request #42580 from hanfei1991/fei/limit_max_rows 
fix behaviour of max_rows_to_read for trival limit queries
 2022-10-26 14:09:59 +02:00
Maksim Kita
c47ca522f4 Fixed style check 2022-10-26 13:27:14 +02:00
Antonio Andelic
35a9672704 Merge branch 'master' into optimize-entire-partition 2022-10-26 11:09:25 +00:00
Maksim Kita
1b6293f6db MergeTree indexes use ActionsDAG 2022-10-26 12:44:37 +02:00
Antonio Andelic
97aaebfa18 Address PR comments 2022-10-26 10:06:56 +00:00
Antonio Andelic
979d180931
Merge branch 'master' into correctly-handle-prewhere-type 2022-10-26 11:13:16 +02:00
Azat Khuzhin
8db31beee6 Remove covered parts for fetched part 
Here is an example that I found on production, simplified.

Consider the following queue (nothing of this had been processed on this
replica):

- GET_PART all_0_0_0 (queue-0000000001)
- GET_PART all_1_1_0 (queue-0000000002)
...
- GET_PART all_0_1_1 (queue-0000000003)
- GET_PART all_2_2_0 (queue-0000000004)
...
- MERGE_PARTS from [all_0_1_1, all_2_2_0] to all_0_2_2 (queue-0000000005)

And now queue-0000000005 started to executing (either because
of reording, or because at that time GET_PART fails), and it
does not have any required parts, so it will fetch them, but
not all_0_0_0 and all_1_1_0, so this replica delay will set to
the time of min(queue-0000000001, queue-0000000002), while it
is not true, since it already have parts that covers those
parts.

and since MERGE_PARTS takes 30min, it increased the replica delay
eventually to 30min, for the time range of 30min, which is pretty huge.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-26 11:10:50 +03:00
Anton Popov
79889198b1 fix style check 2022-10-25 23:03:37 +00:00
Anton Popov
d8b09b430f add comments 2022-10-25 22:27:11 +00:00
Anton Popov
c4d4f2dbed better interface 2022-10-25 22:14:06 +00:00
Sergei Trifonov
34b43f8b4b
Merge branch 'master' into storage-system-moves 2022-10-25 22:56:45 +02:00
serxa
55a86bc2de Add active moves system table 2022-10-25 13:56:46 +00:00