thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
102,712 Commits 9,328 Branches 1,783 Tags 2.1 GiB
124446463c
Commit Graph

10367 Commits

Author SHA1 Message Date
Alexander Tokmakov
fc3a0bfdfa fix 2022-11-27 01:12:09 +01:00
Azat Khuzhin
6a057dec70 Add table_uuid to system.parts (resubmit) 
Can be useful if the table constantly recreated, i.e. in some tests.

v2: fix tests (disable for ordinary database)
v3: adjust test after merging replacing with empty parts pr
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit d0ce500f89)
 2022-11-26 20:16:38 +01:00
Alexander Tokmakov
f4944664b7 fix 2022-11-25 18:53:46 +01:00
Alexander Tokmakov
5cc99312d7 provide hint for loading uncommitted blocks in merge predicate 2022-11-25 16:41:20 +01:00
Alexey Milovidov
69d1be50bc
Merge pull request #43446 from ClickHouse/fix-typo-3 
Fix typo
 2022-11-25 15:22:28 +01:00
Anton Popov
c811f34a41
Simplify AsynchronousInsertQueue and fix race (#43233) 2022-11-25 15:02:22 +01:00
Nikolai Kochetov
660c942258 Merge branch 'master' into read-from-mt-in-io-pool 2022-11-25 13:56:48 +00:00
Alexander Tokmakov
ace36e2267
Merge branch 'master' into fix_parts_to_do_detection 2022-11-25 15:15:56 +03:00
Sema Checherinda
15a6ce254b
Merge pull request #41145 from CheSema/lock-free-drop-partition 
Delete parts by replacing them with empty parts
 2022-11-25 12:54:30 +01:00
Antonio Andelic
9f0fdd780b Modify ALTER before replicating 2022-11-25 10:16:20 +00:00
HarryLeeIBM
fc9880ea5a Fix Bloom filter serialization for s390x 2022-11-24 14:37:05 -08:00
HarryLeeIBM
a35c5a914f Revert "Fix bloom filter serialization endian issue for s390x" 
This reverts commit 8f7970d598.
 2022-11-24 13:44:38 -08:00
HarryLeeIBM
8f7970d598 Fix bloom filter serialization endian issue for s390x 2022-11-24 13:27:02 -08:00
Nikolai Kochetov
d53439390f Review fixes. 2022-11-24 16:04:35 +00:00
Alexander Tokmakov
6531ff765d
Merge pull request #43458 from ClickHouse/tavplubix-patch-5 
Fix createTableSharedID again
 2022-11-24 16:21:50 +03:00
alesapin
52821a331a
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:04:03 +01:00
alesapin
188069117e
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:02:26 +01:00
alesapin
5355f82448
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-11-24 14:02:19 +01:00
Sema Checherinda
6fd7dcf26f
Merge branch 'master' into lock-free-drop-partition 2022-11-24 12:34:19 +01:00
alesapin
f64f0461fe
Merge pull request #43288 from ClickHouse/Enable_compress_marks_and_primary_key 
Enable compress marks and primary key for CI
 2022-11-24 12:24:04 +01:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus 2022-11-24 12:12:08 +01:00
Kseniia Sumarokova
ffc2d027c9
Revert the change 2022-11-24 12:03:16 +01:00
Antonio Andelic
6e21e63871 Add test for storage args with UDFs 2022-11-24 09:35:17 +00:00
Nikolai Kochetov
62debf42b4 Aggregating in order [in progress]. 2022-11-23 18:38:12 +00:00
Antonio Andelic
c52ca6e10e Merge branch 'master' into substitute-udf-create 2022-11-23 16:08:37 +00:00
xiedeyantu
304b6ebf3a s3 table function can support select nested column using {column_name}.{subcolumn_name} 2022-11-23 23:36:12 +08:00
Sema Checherinda
9f2c00d434 add tags to tests, fix error message 2022-11-23 15:16:10 +00:00
Sema Checherinda
7d748602c0 always write creation_csn 2022-11-23 15:16:10 +00:00
Sema Checherinda
e29d7c2e00 fix typos 2022-11-23 15:16:10 +00:00
Sema Checherinda
b4a21cab68 removal_csn can be PrehistoricCSN iff removal_tid is also PrehistoricCSN when cl removind an old part 2022-11-23 15:16:10 +00:00
Sema Checherinda
eccb62e4c3 work with the latest remarks 2022-11-23 15:16:09 +00:00
Sema Checherinda
a29437e179 delete verbose logging about cleaning old parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
e54ca534b0 test 01167_isolation_hermitage uses fallback to recreate table 2022-11-23 15:16:09 +00:00
Sema Checherinda
f49936b9a0 remove tmp dir from inmemory part when freeze, more logs about old parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
3a8024002d fix race in read full path from deleting part, add more logs in grabOldParts 2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208 fix isolation breach, droping outdated part should't be ignored 2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25 fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part 2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104 less pedantic checks in operations, rely on existed tx functionality 2022-11-23 15:16:09 +00:00
Sema Checherinda
a4d1adf945 in order to restore partition do not write empty inmemory part 2022-11-23 15:16:09 +00:00
Sema Checherinda
00e7b4e195 fix race on remove, add creation csn for removed parts 2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0 no groupping parts for cover with empty 2022-11-23 15:16:08 +00:00
Sema Checherinda
597421014e fix style check 2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a fix several critical races, fix tests 2022-11-23 15:16:08 +00:00
Sema Checherinda
8977e499c2 address 01825_type_json_18 test 2022-11-23 15:16:08 +00:00
Sema Checherinda
f2b9f80f2f fix tidy build 2022-11-23 15:16:08 +00:00
Sema Checherinda
6d929266e0 address 01825_type_json_schema_race_long test 2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070 address the review comments 2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9 Delete parts over replacing them with empty parts 2022-11-23 15:16:08 +00:00
Alexander Tokmakov
c0133199c8
Revert "Add table_uuid to system.parts" 2022-11-23 16:41:43 +03:00
Antonio Andelic
981e8a3a27 Modifiy visitor 2022-11-23 13:32:36 +00:00
Alexander Tokmakov
ad74961075
Merge branch 'master' into tavplubix-patch-5 2022-11-23 15:57:11 +03:00
Alexander Gololobov
1dec7910c3
Merge pull request #43336 from azat/buffer/fix-rollback 
Fix possible "No column to rollback" logical error during INSERT into Buffer
 2022-11-23 13:00:32 +01:00
kssenii
7ea1f4a3dd Merge remote-tracking branch 'origin/named-collections-sql-commands' into named-collections-sql-commands 2022-11-23 12:02:00 +01:00
kssenii
6044a9257c Merge remote-tracking branch 'upstream/master' into named-collections-sql-commands 2022-11-23 12:00:55 +01:00
Antonio Andelic
84884df595 Substitute UDFs in CREATE queries 2022-11-23 09:28:08 +00:00
Azat Khuzhin
2cfbc27f13 Fix incorrect memory accounting because of merges/mutations 
In case of merges/mutations, separate MemoryTracker object is used, to
track memory separately, to report this metrics.

However, the tail of those allocations hadn't been taken into account,
and in case of lots of mutations/merges it can be significant.

100% reproducible with the following example:

Start the server w/o frequent sync with RSS:

    $ clickhouse-server -C /src/programs/server/config.xml -- --asynchronous_metrics_update_period_s=86400

Run the following snippet:

    create table data (key Int, value Nullable(Decimal(38, 0))) engine=ReplicatedMergeTree('/ch/data', '1') order by key
    insert into data values (1, -13028236692093846346337460743176821145)
    alter table data modify column value UInt256
    Code: 341. DB::Exception: Received from localhost:9000. DB::Exception: Exception happened during execution of mutation 'mutation_2.txt' with part 'all_1_1_0' reason: 'Code: 407. DB::Exception: Convert overflow: while executing 'FUNCTION _CAST(value :: 1, 'UInt256' :: 2) -> _CAST(value, 'UInt256') UInt256 : 3': (while reading from part ./store/dd9/dd9ef183-1270-4877-af22-8a000bdcc1b8/all_1_1_0/): While executing MergeTreeInOrder. (DECIMAL_OVERFLOW) (version 22.12.1.1)'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED)

And track the memory:

    select value/1e9 from system.metrics where metric like 'MemoryTracking'

Reported-by: @filimonov
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Azat Khuzhin
5da2f52722 Use Int64 over UInt64 for prev_untracked_memory* in MemoryTrackerThreadSwitcher 
Since those types originally is Int64

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Azat Khuzhin
05f30aaad4 Fix logger name for MutateFromLogEntryTask 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Alexey Milovidov
b7e4547f53
Merge branch 'master' into fix-typo-3 2022-11-22 18:30:06 +01:00
Sergei Trifonov
59f422f10c
Merge branch 'master' into storage-system-moves 2022-11-22 14:46:00 +01:00
Kruglov Pavel
98d6b96c82
Merge pull request #42033 from mark-polokhov/BSONEachRow 
Add BSONEachRow input/output format
 2022-11-22 14:45:21 +01:00
Kseniia Sumarokova
392963012a
Merge pull request #43404 from azat/part.log.uuid 
Add table_uuid to system.parts
 2022-11-22 14:25:34 +01:00
Alexander Tokmakov
91b2cdb94a
Merge branch 'master' into tavplubix-patch-5 2022-11-22 16:17:55 +03:00
Kseniia Sumarokova
ff630e3f7c
Update src/Storages/NamedCollectionUtils.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-11-22 14:04:44 +01:00
Nikolai Kochetov
9e84a351c7 Fixing special build. Disable setting by default. 2022-11-22 11:24:24 +00:00
Antonio Andelic
843401c3a3
Merge pull request #43323 from ClickHouse/fix-describe-delta-lake-hudi 
Fix `DESCRIBE` for `deltaLake` and `hudi` table functions
 2022-11-22 09:09:39 +01:00
Anton Popov
0479d8d9f5
Merge pull request #43387 from ClickHouse/fix_default_value_used_in_row_level_filter 
Fix default value used in row level filter
 2022-11-22 01:46:45 +01:00
kssenii
1073626f8e Mask secrets in log 2022-11-21 22:45:43 +01:00
kssenii
155ddd5521 Fix review comments 2022-11-21 21:50:24 +01:00
kssenii
a279bac767 Merge remote-tracking branch 'upstream/master' into named-collections-sql-commands 2022-11-21 21:49:34 +01:00
Sergei Trifonov
94a7614c45
Merge branch 'master' into storage-system-moves 2022-11-21 20:10:12 +01:00
Alexander Tokmakov
a7efdd7f4e
Update StorageReplicatedMergeTree.cpp 2022-11-21 21:52:48 +03:00
Nikolai Kochetov
c305afd77a Fix race. 2022-11-21 18:26:31 +00:00
Kruglov Pavel
581e57be9f
Merge pull request #43236 from xiedeyantu/fix_hdfscluster_connection_fail 
fix skip_unavailable_shards does not work using hdfsCluster table function
 2022-11-21 19:22:16 +01:00
serxa
faef2e9b00 fix moves list entry creation 2022-11-21 17:08:54 +00:00
serxa
c425da06b2 fix column name 2022-11-21 16:16:53 +00:00
Alexey Milovidov
56f8c0b84b Fix typo 2022-11-21 15:10:15 +01:00
Azat Khuzhin
d0ce500f89 Add table_uuid to system.parts 
Can be useful if the table constantly recreated, i.e. in some tests.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-21 12:11:51 +01:00
alesapin
b78b602ea1
Merge branch 'master' into Enable_compress_marks_and_primary_key 2022-11-21 12:05:04 +01:00
Nikolai Kochetov
be1a8054c7
Merge branch 'master' into read-from-mt-in-io-pool 2022-11-21 11:27:41 +01:00
Nikolai Kochetov
4a1774c5ac Fixing build. 2022-11-21 10:27:09 +00:00
Antonio Andelic
9a8d82f0fc Merge branch 'master' into fix-describe-delta-lake-hudi 2022-11-21 08:23:33 +00:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus 2022-11-21 09:10:10 +01:00
Kseniia Sumarokova
c12cfab953
Merge pull request #43343 from azat/disks/write-once 
Allow to "drop tables" from s3_plain disk (so as from web disk)
 2022-11-20 16:04:50 +01:00
Azat Khuzhin
177cbbac4b Avoid race condition for updating system.distribution_queue values 
Previously it was possible to have a race while updating
files_count/bytes_count, since INSERT updates it those counters from one
thread and the same metrics are updated from filesystem in a separate
thread, and even though the access is synchronized with the mutex it
avoids the race only for accessing the variables not the logical race,
since it is possible that getFiles() from a separate thread will
increment counters and later addAndSchedule() will increment them again.

Here you can find an example of this race [1].

  [1]: https://pastila.nl/?00950e00/41a3c7bbb0a7e75bd3f2922c58b02334

Note, that I analyzed logs from production system with lots of async
Distributed INSERT and everything is OK there, even though the logs
contains the following:

    2022.11.20 02:21:15.459483 [ 11528 ] {} <Trace> v21.dist_out.DirectoryMonitor: Files set to 35 (was 34)
    2022.11.20 02:21:15.459515 [ 11528 ] {} <Trace> v21.dist_out.DirectoryMonitor: Bytes set to 4035418 (was 3929008)
    2022.11.20 02:21:15.819488 [ 11528 ] {} <Trace> v21.dist_out.DirectoryMonitor: Files set to 1 (was 2)
    2022.11.20 02:21:15.819502 [ 11528 ] {} <Trace> v21.dist_out.DirectoryMonitor: Bytes set to 190072 (was 296482)

As you may see it first increases the counters and next update
decreases (and 4035418-3929008 == 296482-190072)

Refs: #23885
Reported-by: @tavplubix
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 14:13:01 +01:00
Azat Khuzhin
4f6703c972 Add is_read_only/is_write_once/is_remote/is_broken to system.disks 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:11:39 +01:00
Azat Khuzhin
c029549859 Allow to drop tables from s3_plain disk (so as from web disk) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:10:27 +01:00
zhongyuankai
a3a3261745
Merge branch 'master' into optimize_ttl 2022-11-19 16:32:26 +08:00
zhongyuankai
ce9b4866ea
Update MergeTreeDataMergerMutator.cpp 2022-11-19 16:31:47 +08:00
Alexander Gololobov
f004eea413 Add columns required fro defaults calculation 2022-11-18 23:42:45 +01:00
alesapin
c7d648174b
Merge pull request #43328 from ClickHouse/better-format-version-check 
Check content of `format_version` file in `MergeTreeData`
 2022-11-18 22:34:37 +01:00
Kruglov Pavel
82d6f5357c
Merge pull request #43156 from azat/dist/insert-apply-timeouts 
Apply connection timeouts settings for Distributed async INSERT from the query
 2022-11-18 21:27:57 +01:00
Nikolai Kochetov
5da1d893fd
Merge branch 'master' into read-from-mt-in-io-pool 2022-11-18 21:10:45 +01:00
Nikolai Kochetov
03c35e146b Refactor a bit. 2022-11-18 20:09:20 +00:00
serxa
8e65b799c2 fix test + add one more column 2022-11-18 20:04:11 +00:00
serxa
a05a1b0133 Merge branch 'storage-system-moves' of github.com:ClickHouse/ClickHouse into storage-system-moves 2022-11-18 15:35:00 +00:00
Sergei Trifonov
77100896e3
Merge branch 'master' into storage-system-moves 2022-11-18 16:33:43 +01:00
Sergei Trifonov
d05223e70b
Merge pull request #43335 from ClickHouse/revert-43306-revert-43014-disk-s3-throttler 
Revert "Revert "S3 request per second rate throttling""
 2022-11-18 16:22:30 +01:00
Antonio Andelic
25697fd87d
Merge pull request #43289 from ClickHouse/fix-another-race-rmt 
Fix race on write in `ReplicatedMergeTree`
 2022-11-18 16:09:50 +01:00
alesapin
0f3eb51aed Disable compressed marks by default 2022-11-18 15:54:48 +01:00