alesapin
c8d9ccf3cd
Lock table for share during startup for database ordinary
2022-12-19 18:51:46 +01:00
alesapin
38fd4e0162
Fix race which can lead to merged parts premature removal
2022-12-19 15:25:01 +01:00
Han Fei
98efee2d75
Merge branch 'master' into hanfei/fix-async-flaky
2022-12-19 14:14:29 +01:00
Han Fei
1cb98eb2d6
address comments
2022-12-19 14:05:50 +01:00
Han Fei
ee250174ab
fix flack test "02481_async_insert_dedup.python"
2022-12-18 22:44:51 +01:00
Alexey Milovidov
3e8dcdc155
Add earlier check
2022-12-18 08:01:43 +01:00
Dmitry Novik
875a24a650
Merge remote-tracking branch 'origin/master' into refector-function-node
2022-12-16 16:07:30 +00:00
Anton Popov
b7e76e3539
do not hold data parts during insert
2022-12-15 23:57:10 +00:00
Nikolai Kochetov
e99849d031
Merge pull request #38953 from ClickHouse/add-allocation-ptr-to-trace-log
...
Track allocation ptr in system.trace_log. Add aggregate function flamegraph
2022-12-15 18:16:05 +01:00
Dmitry Novik
7ad3ff8a9e
Merge pull request #44244 from ClickHouse/part-log-removal-info
...
Add more information about part removal into system.parts table
2022-12-15 18:06:55 +01:00
Alexander Tokmakov
688e488e93
Merge pull request #44220 from ClickHouse/correct-readonlyreplica-metric
...
Fix `ReadonlyReplica` metric
2022-12-15 19:05:20 +03:00
Dmitry Novik
b87739d31c
Add HAS_SKIPPED_MUTATION_PARENT state
2022-12-15 16:05:15 +00:00
Kruglov Pavel
b63ff31061
Merge pull request #44192 from Avogar/fix-bad-cast
...
Fix possible Bad cast from type DB::IAST const* to DB::ASTLiteral const*
2022-12-15 14:02:41 +01:00
Dmitry Novik
5b7a990425
Add more information about part removal into system.parts table
2022-12-14 22:30:43 +00:00
Dmitry Novik
1df038e39c
Merge remote-tracking branch 'origin/master' into refector-function-node
2022-12-14 14:55:30 +00:00
Anton Popov
c6b6b0ad7d
Merge pull request #44173 from CurtizJ/fix-fetch-projection
...
Fix `LOGICAL_ERROR` if fetch of projection was stopped
2022-12-14 15:55:26 +01:00
Maksim Kita
9b5ebf88cf
Fixed tests
2022-12-14 15:54:01 +01:00
Anton Popov
3593377f23
Merge pull request #44159 from ClickHouse/prevent-delete-nested-if-empty-part
...
Prevent dropping nested column if it creates empty part
2022-12-14 15:04:15 +01:00
Antonio Andelic
d0905fe408
Fix typo
2022-12-14 13:43:23 +01:00
Antonio Andelic
ff42a68940
Fix ReadonlyReplica
2022-12-14 12:02:12 +00:00
Alexander Tokmakov
816eba6149
Merge pull request #44202 from ClickHouse/fix_an_assertion_in_transactions
...
Fix an assertion in transactions
2022-12-14 14:25:09 +03:00
Maksim Kita
ddad879cb4
Analyzer support Set index
2022-12-14 11:16:53 +01:00
Alexander Tokmakov
f4c8281ccc
avoid outdating part twice
2022-12-13 21:24:26 +01:00
Nikolay Degterinsky
9b6d31b95d
Merge branch 'master' into perf_experiment
2022-12-13 17:15:07 +01:00
Antonio Andelic
2c57f2bd1f
Remove wrong assert
2022-12-13 14:11:08 +00:00
avogar
a1e5cb8c79
Fix possible Bad cast from type DB::IAST const* to DB::ASTLiteral const*
2022-12-13 13:12:44 +00:00
Alexander Tokmakov
a4df1cff52
Update ReplicatedMergeTreeQueue.cpp
2022-12-13 16:04:47 +03:00
Alexander Tokmakov
371e85d854
Merge pull request #44157 from ClickHouse/fix_session_expired_on_startup
...
Ignore "session expired" errors after BC check
2022-12-13 15:10:36 +03:00
Dmitry Novik
3d2fccab87
Merge branch 'master' into refector-function-node
2022-12-12 21:36:39 +01:00
Anton Popov
659a79e53f
fix LOGICAL_ERROR if fetch of projection was stopped
2022-12-12 18:31:20 +00:00
Nikolai Kochetov
e451137040
Add comments.
2022-12-12 18:26:07 +00:00
Vitaly Baranov
7d701e9b32
Merge pull request #44065 from vitlibar/fix-race-in-s3-multipart-upload
...
Fix race in s3 multipart upload
2022-12-12 15:28:05 +01:00
Alexander Tokmakov
d54f739836
ensure force_restore_data is removed from zk
2022-12-12 14:27:35 +01:00
Antonio Andelic
e18ac19ab3
Prevent dropping nested column if it can create empty part
2022-12-12 11:55:33 +00:00
Sema Checherinda
02133f68dd
Merge branch 'master' into mutation-breaks-source-part
2022-12-09 17:20:57 +01:00
Nikolai Kochetov
42d1b38ada
Merge pull request #43965 from ClickHouse/respect-adaptive-granularity-for-disk-remote
...
Respect adaptive granularity for disk remote.
2022-12-09 16:54:34 +01:00
Nikolai Kochetov
9e04473e8c
Fix comment.
2022-12-09 15:44:59 +00:00
Sergei Trifonov
7c37e48092
Merge pull request #43903 from ClickHouse/merge_tree_settings_constraints
...
Constraints on merge tree settings
2022-12-09 14:16:14 +01:00
Maksim Kita
3a3c6eb458
Merge pull request #43341 from kitaisreal/analyzer-support-indexes
...
Analyzer added indexes support
2022-12-09 16:05:40 +03:00
Han Fei
6740ae9fd7
Merge pull request #43304 from hanfei1991/hanfei/aysnc-insert
...
Support dedup for async insert.
2022-12-09 12:50:48 +01:00
Vitaly Baranov
517e84cfb9
Finalize buffers after writing.
2022-12-09 03:02:45 +01:00
Alexey Milovidov
f880644d3a
Merge branch 'master' into nv/fix-mutate-fetch
2022-12-09 00:51:54 +03:00
Sergei Trifonov
bae1519970
Merge branch 'master' into merge_tree_settings_constraints
2022-12-08 22:25:16 +01:00
Dmitry Novik
2826942ca3
Fixup assert condition
2022-12-08 16:41:54 +00:00
Maksim Kita
99de697ea2
Analyzer added indexes support
2022-12-08 17:30:32 +01:00
Sema Checherinda
46f96064ea
do not hardlink serialization.json in new part
2022-12-08 12:48:34 +01:00
Han Fei
1bde6d012d
fix
2022-12-08 09:57:33 +01:00
serxa
77d551788a
Fix relaxed "too many parts" threshold
2022-12-08 00:40:43 +00:00
Han Fei
3c11aca040
address comments
2022-12-07 23:40:52 +01:00
Han Fei
7a3dcd196e
Update src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-12-07 22:30:03 +01:00
Han Fei
6d6bcd9ba2
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-12-07 22:28:55 +01:00
Dmitry Novik
25ecb75ca8
Merge remote-tracking branch 'origin/master' into refector-function-node
2022-12-07 18:36:50 +00:00
Dmitry Novik
a438b6efd8
Try to fix InvertedDAG construction
2022-12-07 18:27:37 +00:00
serxa
2306bd74d7
fix for ALTER TABLE ... MODIFY SETTING name=value
2022-12-06 20:46:10 +00:00
serxa
1b5b43ec87
fix checks on CREATE + added resolveSetting.h + improved system table
2022-12-06 19:09:18 +00:00
Nikolai Kochetov
9a4b3493fd
Merge branch 'master' into add-allocation-ptr-to-trace-log
2022-12-06 17:09:54 +01:00
Nikolai Kochetov
0ed82f3cc0
Merge branch 'master' into aggregating-in-order-from-query-plan
2022-12-06 16:36:49 +01:00
Yakov Olkhovskiy
031d2802d0
Merge pull request #43642 from HarryLeeIBM/hlee-s390x-bloom-filter
...
Fix endian issue in bloom filter serialization for s390x
2022-12-06 08:23:44 -05:00
Nikolai Kochetov
f75250eaf4
Fix style.
2022-12-05 21:27:54 +00:00
Nikolai Kochetov
b797157437
Respect adaptive index granularity settings while reading from remote disk.
2022-12-05 20:10:31 +00:00
Sema Checherinda
be00335ca4
can_remove_anything is false for temporary parts
2022-12-05 15:56:04 +01:00
Sema Checherinda
005cded791
Merge branch 'master' into logical-error-race-tmp-dirs
2022-12-04 18:18:42 +01:00
Sergei Trifonov
e27e79b45d
Merge branch 'master' into merge_tree_settings_constraints
2022-12-02 20:51:31 +01:00
serxa
006af1b6a4
constraints on merge tree settings
2022-12-02 19:30:14 +00:00
Dmitry Novik
15b8c48ca9
Cleanup code
2022-12-02 19:15:26 +00:00
Dmitry Novik
48e8aeac01
Clean up code
2022-12-02 19:15:26 +00:00
Dmitry Novik
2c70dbc76a
Refactor FunctionNode
2022-12-02 19:15:26 +00:00
Vladimir C
7d6950d397
Revert "Temporary files evict fs cache"
2022-12-02 14:50:56 +01:00
Sema Checherinda
ce8ab95b65
Merge pull request #43868 from CheSema/data-race-loging-part-status
...
do not read part status just for logging
2022-12-02 14:48:55 +01:00
Kruglov Pavel
c35b2a6495
Add a limit for string size in RowBinary format ( #43842 )
2022-12-02 13:57:11 +01:00
Sema Checherinda
fd2c7657be
do not read part status just for logging
2022-12-01 16:08:05 +01:00
Yakov Olkhovskiy
5083768e74
use #if instead of if consteaxpr - clang's idiosyncrasy
2022-12-01 09:04:36 -05:00
alesapin
5e0ece2373
Merge pull request #43764 from ClickHouse/fix_projection_and_zero_copy
...
Fix bug with projections and zero-copy replication
2022-12-01 13:44:26 +01:00
Sema Checherinda
baaf5c6186
do not leave tmp part on disk, do not go to the keeper for remove it
2022-12-01 13:24:29 +01:00
vdimir
7a814d2854
wip: temporary files evict fs cache
2022-12-01 11:49:24 +00:00
Nikolai Kochetov
7649a3c4d7
Merge branch 'master' into add-allocation-ptr-to-trace-log
2022-12-01 12:12:23 +01:00
Yakov Olkhovskiy
58d156a39e
style fix
2022-11-30 23:15:35 -05:00
Yakov Olkhovskiy
7a32488900
more clear code and some optimization
2022-11-30 23:05:21 -05:00
Alexander Tokmakov
431f6551cb
Merge branch 'master' into fix_assertion_in_thread_status
2022-11-30 23:05:15 +03:00
Han Fei
53e961e7c1
fix build check
2022-11-30 19:09:57 +01:00
alesapin
7b4025ba7c
Fix bug
2022-11-30 12:43:46 +01:00
Han Fei
564279be83
fix integration tests
2022-11-30 12:00:09 +01:00
Alexander Tokmakov
24940ad5bb
Merge pull request #43675 from ClickHouse/merge_predicate_uncommitted_blocks_hint
...
Provide hint for loading uncommitted blocks in merge predicate
2022-11-29 17:55:11 +03:00
Sema Checherinda
75369274c1
Merge pull request #43516 from azat/memory/mutations-merges-fix
...
Fix incorrect memory accounting because of merges/mutations
2022-11-29 13:03:04 +01:00
alesapin
428286e548
Merge branch 'master' into fix_projection_and_zero_copy
2022-11-29 12:51:08 +01:00
alesapin
7914e26744
Fix another part of code
2022-11-29 12:50:20 +01:00
Nikolai Kochetov
51439e2c19
Merge pull request #43260 from ClickHouse/read-from-mt-in-io-pool
...
Read from MergeTree in I/O pool
2022-11-29 12:09:03 +01:00
Alexander Tokmakov
e45105bf44
detach threads from thread group
2022-11-28 21:31:55 +01:00
alesapin
4ea833afa1
Fix bug with projections and zero-copy replication
2022-11-28 17:29:03 +01:00
Han Fei
f23ee45d84
Merge branch 'master' into hanfei/aysnc-insert
2022-11-28 16:58:45 +01:00
Nikolai Kochetov
6d0646ed8f
Merge branch 'master' into aggregating-in-order-from-query-plan
2022-11-28 16:53:29 +01:00
Alexander Tokmakov
0342e6b236
fix
2022-11-28 16:02:37 +01:00
Alexander Tokmakov
a47e0b8f61
Merge branch 'master' into merge_predicate_uncommitted_blocks_hint
2022-11-28 15:45:34 +01:00
Sema Checherinda
700181ca2e
Merge branch 'master' into memory/mutations-merges-fix
2022-11-28 14:14:57 +01:00
Kseniia Sumarokova
f35aa891d2
Merge pull request #43595 from azat/part.log.uuid-resubmit
...
Add table_uuid to system.parts (resubmit)
2022-11-28 12:02:44 +01:00
Robert Schulze
dce0b78198
Merge pull request #42778 from FArthur-cmd/annoy_cosine
...
Add cosine distance for annoy
2022-11-28 09:53:17 +01:00
Nikita Taranov
8ed5cfc265
Memory bound merging for distributed aggregation in order ( #40879 )
...
* impl
* fix style
* make executeQueryWithParallelReplicas similar to executeQuery
* impl for parallel replicas
* cleaner code for remote sorting properties
* update test
* fix
* handle when nodes of old versions participate
* small fixes
* temporary enable for testing
* fix after merge
* Revert "temporary enable for testing"
This reverts commit cce7f8884c
2022-11-28 00:41:31 +01:00
Alexander Tokmakov
fc3a0bfdfa
fix
2022-11-27 01:12:09 +01:00
Azat Khuzhin
6a057dec70
Add table_uuid to system.parts (resubmit)
...
Can be useful if the table constantly recreated, i.e. in some tests.
v2: fix tests (disable for ordinary database)
v3: adjust test after merging replacing with empty parts pr
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit d0ce500f89
2022-11-26 20:16:38 +01:00
Han Fei
ee5be80d9f
Merge branch 'master' into hanfei/aysnc-insert
2022-11-25 17:27:56 +01:00
Alexander Tokmakov
5cc99312d7
provide hint for loading uncommitted blocks in merge predicate
2022-11-25 16:41:20 +01:00
Nikolai Kochetov
660c942258
Merge branch 'master' into read-from-mt-in-io-pool
2022-11-25 13:56:48 +00:00
Alexander Tokmakov
ace36e2267
Merge branch 'master' into fix_parts_to_do_detection
2022-11-25 15:15:56 +03:00
HarryLeeIBM
fc9880ea5a
Fix Bloom filter serialization for s390x
2022-11-24 14:37:05 -08:00
HarryLeeIBM
a35c5a914f
Revert "Fix bloom filter serialization endian issue for s390x"
...
This reverts commit 8f7970d598
2022-11-24 13:44:38 -08:00
HarryLeeIBM
8f7970d598
Fix bloom filter serialization endian issue for s390x
2022-11-24 13:27:02 -08:00
Nikolai Kochetov
d53439390f
Review fixes.
2022-11-24 16:04:35 +00:00
alesapin
52821a331a
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2022-11-24 14:04:03 +01:00
alesapin
188069117e
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2022-11-24 14:02:26 +01:00
alesapin
5355f82448
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2022-11-24 14:02:19 +01:00
Sema Checherinda
6fd7dcf26f
Merge branch 'master' into lock-free-drop-partition
2022-11-24 12:34:19 +01:00
alesapin
f64f0461fe
Merge pull request #43288 from ClickHouse/Enable_compress_marks_and_primary_key
...
Enable compress marks and primary key for CI
2022-11-24 12:24:04 +01:00
Raúl Marín
4d3217cb18
Merge branch 'master' into perf_experiment
2022-11-24 10:32:12 +01:00
Han Fei
28564df6f1
Merge branch 'master' into hanfei/aysnc-insert
2022-11-24 09:12:43 +01:00
Han Fei
d0f5fcdf09
update ut and refine logs
2022-11-23 21:07:59 +01:00
Nikolai Kochetov
62debf42b4
Aggregating in order [in progress].
2022-11-23 18:38:12 +00:00
Sema Checherinda
9f2c00d434
add tags to tests, fix error message
2022-11-23 15:16:10 +00:00
Sema Checherinda
7d748602c0
always write creation_csn
2022-11-23 15:16:10 +00:00
Sema Checherinda
e29d7c2e00
fix typos
2022-11-23 15:16:10 +00:00
Sema Checherinda
b4a21cab68
removal_csn can be PrehistoricCSN iff removal_tid is also PrehistoricCSN when cl removind an old part
2022-11-23 15:16:10 +00:00
Sema Checherinda
eccb62e4c3
work with the latest remarks
2022-11-23 15:16:09 +00:00
Sema Checherinda
a29437e179
delete verbose logging about cleaning old parts
2022-11-23 15:16:09 +00:00
Sema Checherinda
f49936b9a0
remove tmp dir from inmemory part when freeze, more logs about old parts
2022-11-23 15:16:09 +00:00
Sema Checherinda
3a8024002d
fix race in read full path from deleting part, add more logs in grabOldParts
2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208
fix isolation breach, droping outdated part should't be ignored
2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25
fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part
2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104
less pedantic checks in operations, rely on existed tx functionality
2022-11-23 15:16:09 +00:00
Sema Checherinda
a4d1adf945
in order to restore partition do not write empty inmemory part
2022-11-23 15:16:09 +00:00
Sema Checherinda
00e7b4e195
fix race on remove, add creation csn for removed parts
2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0
no groupping parts for cover with empty
2022-11-23 15:16:08 +00:00
Sema Checherinda
597421014e
fix style check
2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a
fix several critical races, fix tests
2022-11-23 15:16:08 +00:00
Sema Checherinda
8977e499c2
address 01825_type_json_18 test
2022-11-23 15:16:08 +00:00
Sema Checherinda
6d929266e0
address 01825_type_json_schema_race_long test
2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070
address the review comments
2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9
Delete parts over replacing them with empty parts
2022-11-23 15:16:08 +00:00
Alexander Tokmakov
c0133199c8
Revert "Add table_uuid to system.parts"
2022-11-23 16:41:43 +03:00
Alexey Milovidov
bedfe68503
Merge branch 'master' into nv/fix-mutate-fetch
2022-11-22 20:06:56 +01:00
Azat Khuzhin
2cfbc27f13
Fix incorrect memory accounting because of merges/mutations
...
In case of merges/mutations, separate MemoryTracker object is used, to
track memory separately, to report this metrics.
However, the tail of those allocations hadn't been taken into account,
and in case of lots of mutations/merges it can be significant.
100% reproducible with the following example:
Start the server w/o frequent sync with RSS:
$ clickhouse-server -C /src/programs/server/config.xml -- --asynchronous_metrics_update_period_s=86400
Run the following snippet:
create table data (key Int, value Nullable(Decimal(38, 0))) engine=ReplicatedMergeTree('/ch/data', '1') order by key
insert into data values (1, -13028236692093846346337460743176821145)
alter table data modify column value UInt256
Code: 341. DB::Exception: Received from localhost:9000. DB::Exception: Exception happened during execution of mutation 'mutation_2.txt' with part 'all_1_1_0' reason: 'Code: 407. DB::Exception: Convert overflow: while executing 'FUNCTION _CAST(value :: 1, 'UInt256' :: 2) -> _CAST(value, 'UInt256') UInt256 : 3': (while reading from part ./store/dd9/dd9ef183-1270-4877-af22-8a000bdcc1b8/all_1_1_0/): While executing MergeTreeInOrder. (DECIMAL_OVERFLOW) (version 22.12.1.1)'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED)
And track the memory:
select value/1e9 from system.metrics where metric like 'MemoryTracking'
Reported-by: @filimonov
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-22 19:40:35 +01:00
Azat Khuzhin
5da2f52722
Use Int64 over UInt64 for prev_untracked_memory* in MemoryTrackerThreadSwitcher
...
Since those types originally is Int64
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-22 19:40:35 +01:00
Azat Khuzhin
05f30aaad4
Fix logger name for MutateFromLogEntryTask
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-22 19:40:35 +01:00
Raúl Marín
4aa29b6a63
Merge remote-tracking branch 'blessed/master' into perf_experiment
2022-11-22 19:09:00 +01:00
Han Fei
32c7fbec55
use another dedup path for async inserts
2022-11-22 15:12:00 +01:00
Sergei Trifonov
59f422f10c
Merge branch 'master' into storage-system-moves
2022-11-22 14:46:00 +01:00
Kseniia Sumarokova
392963012a
Merge pull request #43404 from azat/part.log.uuid
...
Add table_uuid to system.parts
2022-11-22 14:25:34 +01:00
Nikolai Kochetov
9e84a351c7
Fixing special build. Disable setting by default.
2022-11-22 11:24:24 +00:00
Raúl Marín
8776b2f566
Merge remote-tracking branch 'blessed/master' into perf_experiment
2022-11-22 11:45:36 +01:00
Anton Popov
0479d8d9f5
Merge pull request #43387 from ClickHouse/fix_default_value_used_in_row_level_filter
...
Fix default value used in row level filter
2022-11-22 01:46:45 +01:00
Sergei Trifonov
94a7614c45
Merge branch 'master' into storage-system-moves
2022-11-21 20:10:12 +01:00
Nikolai Kochetov
c305afd77a
Fix race.
2022-11-21 18:26:31 +00:00
