thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
103,548 Commits 9,338 Branches 1,783 Tags 2.1 GiB
6b6504f0d1
Commit Graph

4741 Commits

Author SHA1 Message Date
Alexey Milovidov
f880644d3a
Merge branch 'master' into nv/fix-mutate-fetch 2022-12-09 00:51:54 +03:00
Sergei Trifonov
bae1519970
Merge branch 'master' into merge_tree_settings_constraints 2022-12-08 22:25:16 +01:00
Maksim Kita
99de697ea2 Analyzer added indexes support 2022-12-08 17:30:32 +01:00
Sema Checherinda
46f96064ea do not hardlink serialization.json in new part 2022-12-08 12:48:34 +01:00
Han Fei
1bde6d012d fix 2022-12-08 09:57:33 +01:00
serxa
77d551788a Fix relaxed "too many parts" threshold 2022-12-08 00:40:43 +00:00
Han Fei
3c11aca040 address comments 2022-12-07 23:40:52 +01:00
Han Fei
7a3dcd196e
Update src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-12-07 22:30:03 +01:00
Han Fei
6d6bcd9ba2
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-12-07 22:28:55 +01:00
serxa
2306bd74d7 fix for ALTER TABLE ... MODIFY SETTING name=value 2022-12-06 20:46:10 +00:00
serxa
1b5b43ec87 fix checks on CREATE + added resolveSetting.h + improved system table 2022-12-06 19:09:18 +00:00
Nikolai Kochetov
9a4b3493fd Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-12-06 17:09:54 +01:00
Nikolai Kochetov
0ed82f3cc0
Merge branch 'master' into aggregating-in-order-from-query-plan 2022-12-06 16:36:49 +01:00
Yakov Olkhovskiy
031d2802d0
Merge pull request #43642 from HarryLeeIBM/hlee-s390x-bloom-filter 
Fix endian issue in bloom filter serialization for s390x
 2022-12-06 08:23:44 -05:00
Nikolai Kochetov
f75250eaf4 Fix style. 2022-12-05 21:27:54 +00:00
Nikolai Kochetov
b797157437 Respect adaptive index granularity settings while reading from remote disk. 2022-12-05 20:10:31 +00:00
Sema Checherinda
be00335ca4 can_remove_anything is false for temporary parts 2022-12-05 15:56:04 +01:00
Sema Checherinda
005cded791
Merge branch 'master' into logical-error-race-tmp-dirs 2022-12-04 18:18:42 +01:00
Sergei Trifonov
e27e79b45d
Merge branch 'master' into merge_tree_settings_constraints 2022-12-02 20:51:31 +01:00
serxa
006af1b6a4 constraints on merge tree settings 2022-12-02 19:30:14 +00:00
Vladimir C
7d6950d397
Revert "Temporary files evict fs cache" 2022-12-02 14:50:56 +01:00
Sema Checherinda
ce8ab95b65
Merge pull request #43868 from CheSema/data-race-loging-part-status 
do not read part status just for logging
 2022-12-02 14:48:55 +01:00
Kruglov Pavel
c35b2a6495
Add a limit for string size in RowBinary format (#43842) 2022-12-02 13:57:11 +01:00
Sema Checherinda
fd2c7657be do not read part status just for logging 2022-12-01 16:08:05 +01:00
Yakov Olkhovskiy
5083768e74
use #if instead of if consteaxpr - clang's idiosyncrasy 2022-12-01 09:04:36 -05:00
alesapin
5e0ece2373
Merge pull request #43764 from ClickHouse/fix_projection_and_zero_copy 
Fix bug with projections and zero-copy replication
 2022-12-01 13:44:26 +01:00
Sema Checherinda
baaf5c6186 do not leave tmp part on disk, do not go to the keeper for remove it 2022-12-01 13:24:29 +01:00
vdimir
7a814d2854
wip: temporary files evict fs cache 2022-12-01 11:49:24 +00:00
Nikolai Kochetov
7649a3c4d7 Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-12-01 12:12:23 +01:00
Yakov Olkhovskiy
58d156a39e
style fix 2022-11-30 23:15:35 -05:00
Yakov Olkhovskiy
7a32488900
more clear code and some optimization 2022-11-30 23:05:21 -05:00
Alexander Tokmakov
431f6551cb
Merge branch 'master' into fix_assertion_in_thread_status 2022-11-30 23:05:15 +03:00
Han Fei
53e961e7c1 fix build check 2022-11-30 19:09:57 +01:00
alesapin
7b4025ba7c Fix bug 2022-11-30 12:43:46 +01:00
Han Fei
564279be83 fix integration tests 2022-11-30 12:00:09 +01:00
Alexander Tokmakov
24940ad5bb
Merge pull request #43675 from ClickHouse/merge_predicate_uncommitted_blocks_hint 
Provide hint for loading uncommitted blocks in merge predicate
 2022-11-29 17:55:11 +03:00
Sema Checherinda
75369274c1
Merge pull request #43516 from azat/memory/mutations-merges-fix 
Fix incorrect memory accounting because of merges/mutations
 2022-11-29 13:03:04 +01:00
alesapin
428286e548
Merge branch 'master' into fix_projection_and_zero_copy 2022-11-29 12:51:08 +01:00
alesapin
7914e26744 Fix another part of code 2022-11-29 12:50:20 +01:00
Nikolai Kochetov
51439e2c19
Merge pull request #43260 from ClickHouse/read-from-mt-in-io-pool 
Read from MergeTree in I/O pool
 2022-11-29 12:09:03 +01:00
Alexander Tokmakov
e45105bf44 detach threads from thread group 2022-11-28 21:31:55 +01:00
alesapin
4ea833afa1 Fix bug with projections and zero-copy replication 2022-11-28 17:29:03 +01:00
Han Fei
f23ee45d84
Merge branch 'master' into hanfei/aysnc-insert 2022-11-28 16:58:45 +01:00
Nikolai Kochetov
6d0646ed8f
Merge branch 'master' into aggregating-in-order-from-query-plan 2022-11-28 16:53:29 +01:00
Alexander Tokmakov
0342e6b236 fix 2022-11-28 16:02:37 +01:00
Alexander Tokmakov
a47e0b8f61 Merge branch 'master' into merge_predicate_uncommitted_blocks_hint 2022-11-28 15:45:34 +01:00
Sema Checherinda
700181ca2e
Merge branch 'master' into memory/mutations-merges-fix 2022-11-28 14:14:57 +01:00
Kseniia Sumarokova
f35aa891d2
Merge pull request #43595 from azat/part.log.uuid-resubmit 
Add table_uuid to system.parts (resubmit)
 2022-11-28 12:02:44 +01:00
Robert Schulze
dce0b78198
Merge pull request #42778 from FArthur-cmd/annoy_cosine 
Add cosine distance for annoy
 2022-11-28 09:53:17 +01:00
Nikita Taranov
8ed5cfc265
Memory bound merging for distributed aggregation in order (#40879) 
* impl

* fix style

* make executeQueryWithParallelReplicas similar to executeQuery

* impl for parallel replicas

* cleaner code for remote sorting properties

* update test

* fix

* handle when nodes of old versions participate

* small fixes

* temporary enable for testing

* fix after merge

* Revert "temporary enable for testing"

This reverts commit cce7f8884c.

* review fixes

* add bc test

* Update src/Core/Settings.h
 2022-11-28 00:41:31 +01:00
Alexander Tokmakov
fc3a0bfdfa fix 2022-11-27 01:12:09 +01:00
Azat Khuzhin
6a057dec70 Add table_uuid to system.parts (resubmit) 
Can be useful if the table constantly recreated, i.e. in some tests.

v2: fix tests (disable for ordinary database)
v3: adjust test after merging replacing with empty parts pr
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit d0ce500f89)
 2022-11-26 20:16:38 +01:00
Han Fei
ee5be80d9f Merge branch 'master' into hanfei/aysnc-insert 2022-11-25 17:27:56 +01:00
Alexander Tokmakov
5cc99312d7 provide hint for loading uncommitted blocks in merge predicate 2022-11-25 16:41:20 +01:00
Nikolai Kochetov
660c942258 Merge branch 'master' into read-from-mt-in-io-pool 2022-11-25 13:56:48 +00:00
Alexander Tokmakov
ace36e2267
Merge branch 'master' into fix_parts_to_do_detection 2022-11-25 15:15:56 +03:00
HarryLeeIBM
fc9880ea5a Fix Bloom filter serialization for s390x 2022-11-24 14:37:05 -08:00
HarryLeeIBM
a35c5a914f Revert "Fix bloom filter serialization endian issue for s390x" 
This reverts commit 8f7970d598.
 2022-11-24 13:44:38 -08:00
HarryLeeIBM
8f7970d598 Fix bloom filter serialization endian issue for s390x 2022-11-24 13:27:02 -08:00
Nikolai Kochetov
d53439390f Review fixes. 2022-11-24 16:04:35 +00:00
alesapin
52821a331a
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:04:03 +01:00
alesapin
188069117e
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:02:26 +01:00
alesapin
5355f82448
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:02:19 +01:00
Sema Checherinda
6fd7dcf26f
Merge branch 'master' into lock-free-drop-partition 2022-11-24 12:34:19 +01:00
alesapin
f64f0461fe
Merge pull request #43288 from ClickHouse/Enable_compress_marks_and_primary_key 
Enable compress marks and primary key for CI
 2022-11-24 12:24:04 +01:00
Han Fei
28564df6f1
Merge branch 'master' into hanfei/aysnc-insert 2022-11-24 09:12:43 +01:00
Han Fei
d0f5fcdf09 update ut and refine logs 2022-11-23 21:07:59 +01:00
Nikolai Kochetov
62debf42b4 Aggregating in order [in progress]. 2022-11-23 18:38:12 +00:00
Sema Checherinda
9f2c00d434 add tags to tests, fix error message 2022-11-23 15:16:10 +00:00
Sema Checherinda
7d748602c0 always write creation_csn 2022-11-23 15:16:10 +00:00
Sema Checherinda
e29d7c2e00 fix typos 2022-11-23 15:16:10 +00:00
Sema Checherinda
b4a21cab68 removal_csn can be PrehistoricCSN iff removal_tid is also PrehistoricCSN when cl removind an old part 2022-11-23 15:16:10 +00:00
Sema Checherinda
eccb62e4c3 work with the latest remarks 2022-11-23 15:16:09 +00:00
Sema Checherinda
a29437e179 delete verbose logging about cleaning old parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
f49936b9a0 remove tmp dir from inmemory part when freeze, more logs about old parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
3a8024002d fix race in read full path from deleting part, add more logs in grabOldParts 2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208 fix isolation breach, droping outdated part should't be ignored 2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25 fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part 2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104 less pedantic checks in operations, rely on existed tx functionality 2022-11-23 15:16:09 +00:00
Sema Checherinda
a4d1adf945 in order to restore partition do not write empty inmemory part 2022-11-23 15:16:09 +00:00
Sema Checherinda
00e7b4e195 fix race on remove, add creation csn for removed parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0 no groupping parts for cover with empty 2022-11-23 15:16:08 +00:00
Sema Checherinda
597421014e fix style check 2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a fix several critical races, fix tests 2022-11-23 15:16:08 +00:00
Sema Checherinda
8977e499c2 address 01825_type_json_18 test 2022-11-23 15:16:08 +00:00
Sema Checherinda
6d929266e0 address 01825_type_json_schema_race_long test 2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070 address the review comments 2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9 Delete parts over replacing them with empty parts 2022-11-23 15:16:08 +00:00
Alexander Tokmakov
c0133199c8
Revert "Add table_uuid to system.parts" 2022-11-23 16:41:43 +03:00
Alexey Milovidov
bedfe68503
Merge branch 'master' into nv/fix-mutate-fetch 2022-11-22 20:06:56 +01:00
Azat Khuzhin
2cfbc27f13 Fix incorrect memory accounting because of merges/mutations 
In case of merges/mutations, separate MemoryTracker object is used, to
track memory separately, to report this metrics.

However, the tail of those allocations hadn't been taken into account,
and in case of lots of mutations/merges it can be significant.

100% reproducible with the following example:

Start the server w/o frequent sync with RSS:

    $ clickhouse-server -C /src/programs/server/config.xml -- --asynchronous_metrics_update_period_s=86400

Run the following snippet:

    create table data (key Int, value Nullable(Decimal(38, 0))) engine=ReplicatedMergeTree('/ch/data', '1') order by key
    insert into data values (1, -13028236692093846346337460743176821145)
    alter table data modify column value UInt256
    Code: 341. DB::Exception: Received from localhost:9000. DB::Exception: Exception happened during execution of mutation 'mutation_2.txt' with part 'all_1_1_0' reason: 'Code: 407. DB::Exception: Convert overflow: while executing 'FUNCTION _CAST(value :: 1, 'UInt256' :: 2) -> _CAST(value, 'UInt256') UInt256 : 3': (while reading from part ./store/dd9/dd9ef183-1270-4877-af22-8a000bdcc1b8/all_1_1_0/): While executing MergeTreeInOrder. (DECIMAL_OVERFLOW) (version 22.12.1.1)'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED)

And track the memory:

    select value/1e9 from system.metrics where metric like 'MemoryTracking'

Reported-by: @filimonov
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Azat Khuzhin
5da2f52722 Use Int64 over UInt64 for prev_untracked_memory* in MemoryTrackerThreadSwitcher 
Since those types originally is Int64

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Azat Khuzhin
05f30aaad4 Fix logger name for MutateFromLogEntryTask 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Han Fei
32c7fbec55 use another dedup path for async inserts 2022-11-22 15:12:00 +01:00
Sergei Trifonov
59f422f10c
Merge branch 'master' into storage-system-moves 2022-11-22 14:46:00 +01:00
Kseniia Sumarokova
392963012a
Merge pull request #43404 from azat/part.log.uuid 
Add table_uuid to system.parts
 2022-11-22 14:25:34 +01:00
Nikolai Kochetov
9e84a351c7 Fixing special build. Disable setting by default. 2022-11-22 11:24:24 +00:00
Anton Popov
0479d8d9f5
Merge pull request #43387 from ClickHouse/fix_default_value_used_in_row_level_filter 
Fix default value used in row level filter
 2022-11-22 01:46:45 +01:00
Sergei Trifonov
94a7614c45
Merge branch 'master' into storage-system-moves 2022-11-21 20:10:12 +01:00
Nikolai Kochetov
c305afd77a Fix race. 2022-11-21 18:26:31 +00:00