Sema Checherinda
dc52a40c04
decrease stop max-tries back to default, do not clear old parts at shutdown
2022-12-05 16:05:14 +01:00
Sema Checherinda
be00335ca4
can_remove_anything is false for temporary parts
2022-12-05 15:56:04 +01:00
Guillaume Tassery
3537929f73
Merge branch 'master' into gt/add_expression
2022-12-05 15:46:15 +01:00
Anton Popov
c2e92fd274
better code in StorageS3
2022-12-05 14:43:41 +00:00
Alexander Tokmakov
52d3e5471b
Merge pull request #43406 from azat/dist/async-insert-stat
...
Avoid race condition for updating system.distribution_queue values
2022-12-05 12:53:12 +03:00
kssenii
caf1d9653d
Fix memory leak
2022-12-04 19:29:53 +01:00
Sema Checherinda
005cded791
Merge branch 'master' into logical-error-race-tmp-dirs
2022-12-04 18:18:42 +01:00
kssenii
dca3444283
Merge remote-tracking branch 'upstream/master' into named-collections-sql-commands
2022-12-04 15:56:02 +01:00
Sergei Trifonov
e27e79b45d
Merge branch 'master' into merge_tree_settings_constraints
2022-12-02 20:51:31 +01:00
serxa
006af1b6a4
constraints on merge tree settings
2022-12-02 19:30:14 +00:00
Dmitry Novik
15b8c48ca9
Cleanup code
2022-12-02 19:15:26 +00:00
Dmitry Novik
48e8aeac01
Clean up code
2022-12-02 19:15:26 +00:00
Dmitry Novik
2c70dbc76a
Refactor FunctionNode
2022-12-02 19:15:26 +00:00
chen
b6eddbac0d
fix s3Cluster function returns NOT_FOUND_COLUMN_IN_BLOCK error ( #43629 )
...
* fix s3Cluster function returns NOT_FOUND_COLUMN_IN_BLOCK error
* Update StorageS3Cluster.cpp
* Update 01801_s3_cluster_count.sql
* fix
2022-12-02 15:43:29 +01:00
Vitaly Baranov
e1f7f04752
Referential dependencies for RESTORE ( #43834 )
...
* Rename DDLDependencyVisitor -> DDLLoadingDependencyVisitor.
* Move building a loading graph to TablesLoader.
* Implement referential dependencies for tables and use them
when restoring tables from a backup.
* Remove StorageID::operator < (because of its inconsistency with ==).
* Add new tests.
* Fix test.
* Fix memory leak.
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
2022-12-02 15:05:46 +01:00
Vladimir C
7d6950d397
Revert "Temporary files evict fs cache"
2022-12-02 14:50:56 +01:00
Sema Checherinda
ce8ab95b65
Merge pull request #43868 from CheSema/data-race-loging-part-status
...
do not read part status just for logging
2022-12-02 14:48:55 +01:00
Kruglov Pavel
c35b2a6495
Add a limit for string size in RowBinary format ( #43842 )
2022-12-02 13:57:11 +01:00
Sema Checherinda
fd2c7657be
do not read part status just for logging
2022-12-01 16:08:05 +01:00
Yakov Olkhovskiy
5083768e74
use #if instead of if consteaxpr - clang's idiosyncrasy
2022-12-01 09:04:36 -05:00
alesapin
5e0ece2373
Merge pull request #43764 from ClickHouse/fix_projection_and_zero_copy
...
Fix bug with projections and zero-copy replication
2022-12-01 13:44:26 +01:00
Vladimir C
117d01f2ac
Merge pull request #43457 from ClickHouse/vdimir/tmp-data-in-fs-cache
2022-12-01 13:41:40 +01:00
Sema Checherinda
baaf5c6186
do not leave tmp part on disk, do not go to the keeper for remove it
2022-12-01 13:24:29 +01:00
vdimir
9bdf01a55c
Do not use in FileSegment::getPathInLocalCache in StorageSystemFilesystemCache
2022-12-01 11:49:26 +00:00
vdimir
35c4e8ac84
wip: temporary files evict fs cache
2022-12-01 11:49:25 +00:00
vdimir
7a814d2854
wip: temporary files evict fs cache
2022-12-01 11:49:24 +00:00
Antonio Andelic
1f93183dc7
Merge pull request #43087 from ClickHouse/keeper-prometheus
...
Add support for embedded Prometheus endpoint in Keeper
2022-12-01 12:46:47 +01:00
Nikolai Kochetov
7649a3c4d7
Merge branch 'master' into add-allocation-ptr-to-trace-log
2022-12-01 12:12:23 +01:00
Yakov Olkhovskiy
58d156a39e
style fix
2022-11-30 23:15:35 -05:00
Yakov Olkhovskiy
7a32488900
more clear code and some optimization
2022-11-30 23:05:21 -05:00
Alexander Tokmakov
431f6551cb
Merge branch 'master' into fix_assertion_in_thread_status
2022-11-30 23:05:15 +03:00
Han Fei
53e961e7c1
fix build check
2022-11-30 19:09:57 +01:00
Antonio Andelic
2f501d5327
Merge branch 'master' into keeper-prometheus
2022-11-30 13:24:37 +01:00
alesapin
7b4025ba7c
Fix bug
2022-11-30 12:43:46 +01:00
flynn
e70167752a
fix
2022-11-30 11:31:06 +00:00
Han Fei
564279be83
fix integration tests
2022-11-30 12:00:09 +01:00
flynn
5057509fe6
fix
2022-11-30 10:43:51 +00:00
flynn
6a625e4b9b
Use disk operation to serialize and deserialize meta files of StorageFileLog
2022-11-30 10:34:00 +00:00
Anton Popov
fe5fff0347
Merge pull request #43329 from xiedeyantu/support_nested_column
...
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-29 22:27:19 +01:00
Anton Popov
2a1fd48e91
fix tests
2022-11-29 17:33:35 +00:00
Alexander Tokmakov
24940ad5bb
Merge pull request #43675 from ClickHouse/merge_predicate_uncommitted_blocks_hint
...
Provide hint for loading uncommitted blocks in merge predicate
2022-11-29 17:55:11 +03:00
Alexander Tokmakov
58d8025a15
Update StorageReplicatedMergeTree.cpp
2022-11-29 16:42:15 +03:00
Sema Checherinda
75369274c1
Merge pull request #43516 from azat/memory/mutations-merges-fix
...
Fix incorrect memory accounting because of merges/mutations
2022-11-29 13:03:04 +01:00
alesapin
428286e548
Merge branch 'master' into fix_projection_and_zero_copy
2022-11-29 12:51:08 +01:00
alesapin
7914e26744
Fix another part of code
2022-11-29 12:50:20 +01:00
Nikolai Kochetov
51439e2c19
Merge pull request #43260 from ClickHouse/read-from-mt-in-io-pool
...
Read from MergeTree in I/O pool
2022-11-29 12:09:03 +01:00
Anton Popov
486da48ae7
fix tests
2022-11-28 21:15:41 +00:00
Alexander Tokmakov
e45105bf44
detach threads from thread group
2022-11-28 21:31:55 +01:00
alesapin
4ea833afa1
Fix bug with projections and zero-copy replication
2022-11-28 17:29:03 +01:00
Anton Popov
e0dd533811
fix scheduling of async tasks in StorageS3
2022-11-28 16:13:01 +00:00
Alexey Milovidov
f5c8077819
Merge pull request #43539 from ClickHouse/substitute-udf-create
...
Substitute UDFs in `CREATE` queries
2022-11-28 17:05:05 +01:00
Han Fei
f23ee45d84
Merge branch 'master' into hanfei/aysnc-insert
2022-11-28 16:58:45 +01:00
Nikolai Kochetov
6d0646ed8f
Merge branch 'master' into aggregating-in-order-from-query-plan
2022-11-28 16:53:29 +01:00
Alexander Tokmakov
0342e6b236
fix
2022-11-28 16:02:37 +01:00
Alexander Tokmakov
a47e0b8f61
Merge branch 'master' into merge_predicate_uncommitted_blocks_hint
2022-11-28 15:45:34 +01:00
Anton Popov
5e8e1788ae
fix files pruning in StorageS3
2022-11-28 13:56:25 +00:00
Sema Checherinda
700181ca2e
Merge branch 'master' into memory/mutations-merges-fix
2022-11-28 14:14:57 +01:00
Kseniia Sumarokova
f35aa891d2
Merge pull request #43595 from azat/part.log.uuid-resubmit
...
Add table_uuid to system.parts (resubmit)
2022-11-28 12:02:44 +01:00
Robert Schulze
dce0b78198
Merge pull request #42778 from FArthur-cmd/annoy_cosine
...
Add cosine distance for annoy
2022-11-28 09:53:17 +01:00
Antonio Andelic
678958603b
Merge branch 'master' into keeper-prometheus
2022-11-28 09:48:57 +01:00
Nikita Taranov
8ed5cfc265
Memory bound merging for distributed aggregation in order ( #40879 )
...
* impl
* fix style
* make executeQueryWithParallelReplicas similar to executeQuery
* impl for parallel replicas
* cleaner code for remote sorting properties
* update test
* fix
* handle when nodes of old versions participate
* small fixes
* temporary enable for testing
* fix after merge
* Revert "temporary enable for testing"
This reverts commit cce7f8884c
2022-11-28 00:41:31 +01:00
Alexander Tokmakov
fc3a0bfdfa
fix
2022-11-27 01:12:09 +01:00
Azat Khuzhin
6a057dec70
Add table_uuid to system.parts (resubmit)
...
Can be useful if the table constantly recreated, i.e. in some tests.
v2: fix tests (disable for ordinary database)
v3: adjust test after merging replacing with empty parts pr
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit d0ce500f89
2022-11-26 20:16:38 +01:00
Anton Popov
65a78bcd91
improve performance of storage S3
2022-11-26 15:24:01 +00:00
Alexey Milovidov
a2c894bcfd
Just in case
2022-11-26 01:02:29 +03:00
Alexander Tokmakov
f4944664b7
fix
2022-11-25 18:53:46 +01:00
Han Fei
ee5be80d9f
Merge branch 'master' into hanfei/aysnc-insert
2022-11-25 17:27:56 +01:00
Alexander Tokmakov
5cc99312d7
provide hint for loading uncommitted blocks in merge predicate
2022-11-25 16:41:20 +01:00
Alexey Milovidov
69d1be50bc
Merge pull request #43446 from ClickHouse/fix-typo-3
...
Fix typo
2022-11-25 15:22:28 +01:00
Anton Popov
c811f34a41
Simplify AsynchronousInsertQueue and fix race ( #43233 )
2022-11-25 15:02:22 +01:00
Nikolai Kochetov
660c942258
Merge branch 'master' into read-from-mt-in-io-pool
2022-11-25 13:56:48 +00:00
Alexander Tokmakov
ace36e2267
Merge branch 'master' into fix_parts_to_do_detection
2022-11-25 15:15:56 +03:00
Sema Checherinda
15a6ce254b
Merge pull request #41145 from CheSema/lock-free-drop-partition
...
Delete parts by replacing them with empty parts
2022-11-25 12:54:30 +01:00
Antonio Andelic
9f0fdd780b
Modify ALTER before replicating
2022-11-25 10:16:20 +00:00
凌涛
99632c95c9
code style
2022-11-25 10:12:32 +08:00
Alexander Gololobov
038cac1892
add engine_full to system table databases
2022-11-25 10:11:12 +08:00
HarryLeeIBM
fc9880ea5a
Fix Bloom filter serialization for s390x
2022-11-24 14:37:05 -08:00
HarryLeeIBM
a35c5a914f
Revert "Fix bloom filter serialization endian issue for s390x"
...
This reverts commit 8f7970d598
2022-11-24 13:44:38 -08:00
HarryLeeIBM
8f7970d598
Fix bloom filter serialization endian issue for s390x
2022-11-24 13:27:02 -08:00
Nikolai Kochetov
d53439390f
Review fixes.
2022-11-24 16:04:35 +00:00
Alexander Tokmakov
6531ff765d
Merge pull request #43458 from ClickHouse/tavplubix-patch-5
...
Fix createTableSharedID again
2022-11-24 16:21:50 +03:00
alesapin
52821a331a
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2022-11-24 14:04:03 +01:00
alesapin
188069117e
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2022-11-24 14:02:26 +01:00
alesapin
5355f82448
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp
...
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
2022-11-24 14:02:19 +01:00
Sema Checherinda
6fd7dcf26f
Merge branch 'master' into lock-free-drop-partition
2022-11-24 12:34:19 +01:00
alesapin
f64f0461fe
Merge pull request #43288 from ClickHouse/Enable_compress_marks_and_primary_key
...
Enable compress marks and primary key for CI
2022-11-24 12:24:04 +01:00
Antonio Andelic
a6f38cb1cc
Merge branch 'master' into keeper-prometheus
2022-11-24 12:12:08 +01:00
Kseniia Sumarokova
ffc2d027c9
Revert the change
2022-11-24 12:03:16 +01:00
Antonio Andelic
6e21e63871
Add test for storage args with UDFs
2022-11-24 09:35:17 +00:00
Guillaume Tassery
58c5422692
cosmetic
2022-11-24 10:19:04 +01:00
Han Fei
28564df6f1
Merge branch 'master' into hanfei/aysnc-insert
2022-11-24 09:12:43 +01:00
Han Fei
d0f5fcdf09
update ut and refine logs
2022-11-23 21:07:59 +01:00
Nikolai Kochetov
62debf42b4
Aggregating in order [in progress].
2022-11-23 18:38:12 +00:00
Antonio Andelic
c52ca6e10e
Merge branch 'master' into substitute-udf-create
2022-11-23 16:08:37 +00:00
xiedeyantu
304b6ebf3a
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-23 23:36:12 +08:00
Sema Checherinda
9f2c00d434
add tags to tests, fix error message
2022-11-23 15:16:10 +00:00
Sema Checherinda
7d748602c0
always write creation_csn
2022-11-23 15:16:10 +00:00
Sema Checherinda
e29d7c2e00
fix typos
2022-11-23 15:16:10 +00:00
Sema Checherinda
b4a21cab68
removal_csn can be PrehistoricCSN iff removal_tid is also PrehistoricCSN when cl removind an old part
2022-11-23 15:16:10 +00:00
Sema Checherinda
eccb62e4c3
work with the latest remarks
2022-11-23 15:16:09 +00:00
Sema Checherinda
a29437e179
delete verbose logging about cleaning old parts
2022-11-23 15:16:09 +00:00
Sema Checherinda
e54ca534b0
test 01167_isolation_hermitage uses fallback to recreate table
2022-11-23 15:16:09 +00:00
Sema Checherinda
f49936b9a0
remove tmp dir from inmemory part when freeze, more logs about old parts
2022-11-23 15:16:09 +00:00
Sema Checherinda
3a8024002d
fix race in read full path from deleting part, add more logs in grabOldParts
2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208
fix isolation breach, droping outdated part should't be ignored
2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25
fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part
2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104
less pedantic checks in operations, rely on existed tx functionality
2022-11-23 15:16:09 +00:00
Sema Checherinda
a4d1adf945
in order to restore partition do not write empty inmemory part
2022-11-23 15:16:09 +00:00
Sema Checherinda
00e7b4e195
fix race on remove, add creation csn for removed parts
2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0
no groupping parts for cover with empty
2022-11-23 15:16:08 +00:00
Sema Checherinda
597421014e
fix style check
2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a
fix several critical races, fix tests
2022-11-23 15:16:08 +00:00
Sema Checherinda
8977e499c2
address 01825_type_json_18 test
2022-11-23 15:16:08 +00:00
Sema Checherinda
f2b9f80f2f
fix tidy build
2022-11-23 15:16:08 +00:00
Sema Checherinda
6d929266e0
address 01825_type_json_schema_race_long test
2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070
address the review comments
2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9
Delete parts over replacing them with empty parts
2022-11-23 15:16:08 +00:00
Alexander Tokmakov
c0133199c8
Revert "Add table_uuid to system.parts"
2022-11-23 16:41:43 +03:00
Antonio Andelic
981e8a3a27
Modifiy visitor
2022-11-23 13:32:36 +00:00
Alexander Tokmakov
ad74961075
Merge branch 'master' into tavplubix-patch-5
2022-11-23 15:57:11 +03:00
Alexander Gololobov
1dec7910c3
Merge pull request #43336 from azat/buffer/fix-rollback
...
Fix possible "No column to rollback" logical error during INSERT into Buffer
2022-11-23 13:00:32 +01:00
kssenii
7ea1f4a3dd
Merge remote-tracking branch 'origin/named-collections-sql-commands' into named-collections-sql-commands
2022-11-23 12:02:00 +01:00
kssenii
6044a9257c
Merge remote-tracking branch 'upstream/master' into named-collections-sql-commands
2022-11-23 12:00:55 +01:00
Antonio Andelic
84884df595
Substitute UDFs in CREATE queries
2022-11-23 09:28:08 +00:00
Alexey Milovidov
c1283a6bb5
Update StorageSystemDataSkippingIndices.cpp
2022-11-22 22:17:20 +03:00
Alexey Milovidov
bedfe68503
Merge branch 'master' into nv/fix-mutate-fetch
2022-11-22 20:06:56 +01:00
Azat Khuzhin
2cfbc27f13
Fix incorrect memory accounting because of merges/mutations
...
In case of merges/mutations, separate MemoryTracker object is used, to
track memory separately, to report this metrics.
However, the tail of those allocations hadn't been taken into account,
and in case of lots of mutations/merges it can be significant.
100% reproducible with the following example:
Start the server w/o frequent sync with RSS:
$ clickhouse-server -C /src/programs/server/config.xml -- --asynchronous_metrics_update_period_s=86400
Run the following snippet:
create table data (key Int, value Nullable(Decimal(38, 0))) engine=ReplicatedMergeTree('/ch/data', '1') order by key
insert into data values (1, -13028236692093846346337460743176821145)
alter table data modify column value UInt256
Code: 341. DB::Exception: Received from localhost:9000. DB::Exception: Exception happened during execution of mutation 'mutation_2.txt' with part 'all_1_1_0' reason: 'Code: 407. DB::Exception: Convert overflow: while executing 'FUNCTION _CAST(value :: 1, 'UInt256' :: 2) -> _CAST(value, 'UInt256') UInt256 : 3': (while reading from part ./store/dd9/dd9ef183-1270-4877-af22-8a000bdcc1b8/all_1_1_0/): While executing MergeTreeInOrder. (DECIMAL_OVERFLOW) (version 22.12.1.1)'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED)
And track the memory:
select value/1e9 from system.metrics where metric like 'MemoryTracking'
Reported-by: @filimonov
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-22 19:40:35 +01:00
Azat Khuzhin
5da2f52722
Use Int64 over UInt64 for prev_untracked_memory* in MemoryTrackerThreadSwitcher
...
Since those types originally is Int64
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-22 19:40:35 +01:00
Azat Khuzhin
05f30aaad4
Fix logger name for MutateFromLogEntryTask
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-22 19:40:35 +01:00
Alexey Milovidov
b7e4547f53
Merge branch 'master' into fix-typo-3
2022-11-22 18:30:06 +01:00
Han Fei
32c7fbec55
use another dedup path for async inserts
2022-11-22 15:12:00 +01:00
Sergei Trifonov
59f422f10c
Merge branch 'master' into storage-system-moves
2022-11-22 14:46:00 +01:00
Kruglov Pavel
98d6b96c82
Merge pull request #42033 from mark-polokhov/BSONEachRow
...
Add BSONEachRow input/output format
2022-11-22 14:45:21 +01:00
Kseniia Sumarokova
392963012a
Merge pull request #43404 from azat/part.log.uuid
...
Add table_uuid to system.parts
2022-11-22 14:25:34 +01:00
Alexander Tokmakov
91b2cdb94a
Merge branch 'master' into tavplubix-patch-5
2022-11-22 16:17:55 +03:00
Kseniia Sumarokova
ff630e3f7c
Update src/Storages/NamedCollectionUtils.cpp
...
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-11-22 14:04:44 +01:00
Nikolai Kochetov
9e84a351c7
Fixing special build. Disable setting by default.
2022-11-22 11:24:24 +00:00
Antonio Andelic
843401c3a3
Merge pull request #43323 from ClickHouse/fix-describe-delta-lake-hudi
...
Fix `DESCRIBE` for `deltaLake` and `hudi` table functions
2022-11-22 09:09:39 +01:00
Anton Popov
0479d8d9f5
Merge pull request #43387 from ClickHouse/fix_default_value_used_in_row_level_filter
...
Fix default value used in row level filter
2022-11-22 01:46:45 +01:00
kssenii
1073626f8e
Mask secrets in log
2022-11-21 22:45:43 +01:00
kssenii
155ddd5521
Fix review comments
2022-11-21 21:50:24 +01:00
kssenii
a279bac767
Merge remote-tracking branch 'upstream/master' into named-collections-sql-commands
2022-11-21 21:49:34 +01:00
Sergei Trifonov
94a7614c45
Merge branch 'master' into storage-system-moves
2022-11-21 20:10:12 +01:00
Alexander Tokmakov
a7efdd7f4e
Update StorageReplicatedMergeTree.cpp
2022-11-21 21:52:48 +03:00
Nikolai Kochetov
c305afd77a
Fix race.
2022-11-21 18:26:31 +00:00
Kruglov Pavel
581e57be9f
Merge pull request #43236 from xiedeyantu/fix_hdfscluster_connection_fail
...
fix skip_unavailable_shards does not work using hdfsCluster table function
2022-11-21 19:22:16 +01:00
serxa
faef2e9b00
fix moves list entry creation
2022-11-21 17:08:54 +00:00
kssenii
5e01441f61
Show progress bar while reading from s3 table function
2022-11-21 17:56:02 +01:00
serxa
c425da06b2
fix column name
2022-11-21 16:16:53 +00:00
Alexey Milovidov
56f8c0b84b
Fix typo
2022-11-21 15:10:15 +01:00
Azat Khuzhin
d0ce500f89
Add table_uuid to system.parts
...
Can be useful if the table constantly recreated, i.e. in some tests.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-21 12:11:51 +01:00
alesapin
b78b602ea1
Merge branch 'master' into Enable_compress_marks_and_primary_key
2022-11-21 12:05:04 +01:00
Nikolai Kochetov
be1a8054c7
Merge branch 'master' into read-from-mt-in-io-pool
2022-11-21 11:27:41 +01:00
Nikolai Kochetov
4a1774c5ac
Fixing build.
2022-11-21 10:27:09 +00:00
Antonio Andelic
9a8d82f0fc
Merge branch 'master' into fix-describe-delta-lake-hudi
2022-11-21 08:23:33 +00:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus
2022-11-21 09:10:10 +01:00
Guillaume Tassery
7267473aec
Merge branch 'master' into gt/add_expression
2022-11-21 09:07:12 +01:00
Guillaume Tassery
1f73c8dabd
Merge branch 'gt/add_expression' of github.com:YiuRULE/ClickHouse into gt/add_expression
2022-11-21 09:06:25 +01:00
Guillaume Tassery
702768469b
cosmetic
2022-11-21 09:04:44 +01:00
Kseniia Sumarokova
c12cfab953
Merge pull request #43343 from azat/disks/write-once
...
Allow to "drop tables" from s3_plain disk (so as from web disk)
2022-11-20 16:04:50 +01:00
Azat Khuzhin
177cbbac4b
Avoid race condition for updating system.distribution_queue values
...
Previously it was possible to have a race while updating
files_count/bytes_count, since INSERT updates it those counters from one
thread and the same metrics are updated from filesystem in a separate
thread, and even though the access is synchronized with the mutex it
avoids the race only for accessing the variables not the logical race,
since it is possible that getFiles() from a separate thread will
increment counters and later addAndSchedule() will increment them again.
Here you can find an example of this race [1].
[1]: https://pastila.nl/?00950e00/41a3c7bbb0a7e75bd3f2922c58b02334
Note, that I analyzed logs from production system with lots of async
Distributed INSERT and everything is OK there, even though the logs
contains the following:
2022.11.20 02:21:15.459483 [ 11528 ] {} <Trace> v21.dist_out.DirectoryMonitor: Files set to 35 (was 34)
2022.11.20 02:21:15.459515 [ 11528 ] {} <Trace> v21.dist_out.DirectoryMonitor: Bytes set to 4035418 (was 3929008)
2022.11.20 02:21:15.819488 [ 11528 ] {} <Trace> v21.dist_out.DirectoryMonitor: Files set to 1 (was 2)
2022.11.20 02:21:15.819502 [ 11528 ] {} <Trace> v21.dist_out.DirectoryMonitor: Bytes set to 190072 (was 296482)
As you may see it first increases the counters and next update
decreases (and 4035418-3929008 == 296482-190072)
Refs: #23885
Reported-by: @tavplubix
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-20 14:13:01 +01:00
Alexey Milovidov
81ca8e91e6
Update StorageSystemDataSkippingIndices.cpp
2022-11-20 14:27:52 +03:00
Azat Khuzhin
4f6703c972
Add is_read_only/is_write_once/is_remote/is_broken to system.disks
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-19 10:11:39 +01:00
Azat Khuzhin
c029549859
Allow to drop tables from s3_plain disk (so as from web disk)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-19 10:10:27 +01:00
zhongyuankai
a3a3261745
Merge branch 'master' into optimize_ttl
2022-11-19 16:32:26 +08:00
zhongyuankai
ce9b4866ea
Update MergeTreeDataMergerMutator.cpp
2022-11-19 16:31:47 +08:00
Alexander Gololobov
f004eea413
Add columns required fro defaults calculation
2022-11-18 23:42:45 +01:00
alesapin
c7d648174b
Merge pull request #43328 from ClickHouse/better-format-version-check
...
Check content of `format_version` file in `MergeTreeData`
2022-11-18 22:34:37 +01:00
Kruglov Pavel
82d6f5357c
Merge pull request #43156 from azat/dist/insert-apply-timeouts
...
Apply connection timeouts settings for Distributed async INSERT from the query
2022-11-18 21:27:57 +01:00
Nikolai Kochetov
5da1d893fd
Merge branch 'master' into read-from-mt-in-io-pool
2022-11-18 21:10:45 +01:00
Nikolai Kochetov
03c35e146b
Refactor a bit.
2022-11-18 20:09:20 +00:00
serxa
8e65b799c2
fix test + add one more column
2022-11-18 20:04:11 +00:00
Han Fei
b80a2d6c89
address comments
2022-11-18 17:22:05 +01:00
serxa
a05a1b0133
Merge branch 'storage-system-moves' of github.com:ClickHouse/ClickHouse into storage-system-moves
2022-11-18 15:35:00 +00:00
Sergei Trifonov
77100896e3
Merge branch 'master' into storage-system-moves
2022-11-18 16:33:43 +01:00
Sergei Trifonov
d05223e70b
Merge pull request #43335 from ClickHouse/revert-43306-revert-43014-disk-s3-throttler
...
Revert "Revert "S3 request per second rate throttling""
2022-11-18 16:22:30 +01:00
Antonio Andelic
25697fd87d
Merge pull request #43289 from ClickHouse/fix-another-race-rmt
...
Fix race on write in `ReplicatedMergeTree`
2022-11-18 16:09:50 +01:00
alesapin
0f3eb51aed
Disable compressed marks by default
2022-11-18 15:54:48 +01:00
Vitaly Baranov
a348332eab
Merge pull request #43227 from vitlibar/improve-masking-sensitive-info
...
Improve masking sensitive info
2022-11-18 15:37:50 +01:00
Han Fei
74c9ba8df0
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-11-18 11:16:38 +01:00
Han Fei
3dc72b6c56
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-11-18 11:15:51 +01:00
Han Fei
8c823bcaab
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-11-18 11:15:45 +01:00
Han Fei
035e933196
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-11-18 11:15:35 +01:00
Han Fei
f8ac75a054
Update src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-11-18 11:15:14 +01:00
Han Fei
d41094207d
Update src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-11-18 11:14:56 +01:00
Mikhail f. Shiryaev
f942ab1bd2
Update version to 22.12.1.1
2022-11-17 22:15:12 +01:00
Nikolai Kochetov
679591cf0a
Try fix race.
2022-11-17 19:51:47 +00:00
Azat Khuzhin
aa77beabc9
Fix possible "No column to rollback" logical error during INSERT into Buffer
...
In case of concurrent INSERT and SELECT, and memory pressure it is
possible to get this error, if you are very lucky.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-17 19:12:58 +01:00
Vitaly Baranov
050df6ac7f
Move InDepthNodeVisitor.h back to src/Interpreters.
2022-11-17 18:16:32 +01:00
Azat Khuzhin
e67fc54fc9
StorageBuffer: log exception before rollback
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-17 18:10:11 +01:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling""
2022-11-17 17:35:04 +01:00
Vitaly Baranov
2b9ed89721
Rename formatWithHiddenSecrets() -> formatWithSecretsHidden().
2022-11-17 17:30:37 +01:00
kssenii
4e2fd5e22a
Slightly better loading of metadata
2022-11-17 16:35:05 +01:00
alesapin
29a08f64af
Merge branch 'master' into Enable_compress_marks_and_primary_key
2022-11-17 15:53:37 +01:00
Antonio Andelic
d324e8cc3e
Write to first non-readonly disk
2022-11-17 14:52:35 +00:00
kssenii
5d9b34b59d
More tests
2022-11-17 15:34:35 +01:00
Antonio Andelic
d41af8b741
Check and compare content of format_version file
2022-11-17 13:41:36 +00:00
alesapin
06f395d946
Add comments
2022-11-17 13:10:14 +01:00
Kruglov Pavel
a406557263
Merge branch 'master' into dist/insert-apply-timeouts
2022-11-17 12:51:47 +01:00
Antonio Andelic
d5848d53cc
Fix DESCRIBE for deltaLake and hudi table functions
2022-11-17 11:46:17 +00:00
alesapin
a79d83549f
Merge branch 'master' into fix_parts_to_do_detection
2022-11-17 12:30:05 +01:00
Guillaume Tassery
c47f90d197
add expression index in system.data_skipping_indices
2022-11-16 20:54:55 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Sergei Trifonov
159743edd6
Merge pull request #43014 from ClickHouse/disk-s3-throttler
...
S3 request per second rate throttling
2022-11-16 18:51:06 +01:00
Han Fei
221c4e5912
tiny changes
2022-11-16 17:59:08 +01:00
kssenii
f0ce28349d
Better
2022-11-16 17:58:21 +01:00
Han Fei
14701670b5
Merge branch 'master' into hanfei/aysnc-insert
2022-11-16 17:24:36 +01:00
Nikolai Kochetov
17b75d3f5c
Fix metrics and progress.
2022-11-16 14:10:56 +00:00
Antonio Andelic
11aba7597e
Check if initialization is done on write
2022-11-16 13:33:22 +00:00
Smita Kulkarni
040f2b62d9
Enable compress marks and primary key
...
Implementation:
* Set compress_marks and compress_primary_key to true by default.
* Updated getIndexExtensionFromFilesystem to only check for compressed file extension , else return default. When both are present, this function was returning uncompressed file extension giving rise to error as file got removed.
Testing:
* Updated tests where bytes size or compressed data is checked to check with compress_marks and compress_primary_key set to false.
2022-11-16 14:23:58 +01:00
filimonov
dcb76f6bbf
Allow autoremoval of old parts if detach_not_byte_identical_parts enabled
...
Allow autoremoval of old & detached parts if detach_not_byte_identical_parts enabled.
See also #28708 #37975
2022-11-16 14:16:42 +01:00
alesapin
2012f4bd36
Fix use-after-free
2022-11-16 11:50:51 +01:00
Alexey Milovidov
04687c0d69
Merge pull request #43206 from ClickHouse/miscellaneous-changes-parallel-replicas
...
Miscellaneous changes
2022-11-16 05:30:03 +01:00
Vitaly Baranov
d365570b27
Hide passwords in SHOW CREATE TABLE and system.tables.
2022-11-16 01:35:11 +01:00
Nikolai Kochetov
1b1d23c100
Clumsy attempt to fix progress.
2022-11-15 21:23:18 +00:00
Alexander Tokmakov
69b067b81c
Merge pull request #43244 from ClickHouse/fix-replicated-merge-tree-shared-id
...
Don't throw if shared ID already created in `StorageReplicatedMergeTree`
2022-11-15 23:31:51 +03:00
Nikolai Kochetov
cf1afbfb84
Temporarily enable allow_asynchronous_read_from_io_pool_for_merge_tree by default to see thest results.
2022-11-15 17:47:26 +00:00
Nikolai Kochetov
29098045d8
Try read from MergeTree in background pool.
2022-11-15 17:38:37 +00:00
kssenii
234f6ee54d
Add commands CREATE/ALTER/DROP NAMED COLLECTION
2022-11-15 15:51:56 +01:00
Sema Checherinda
7e73b187cc
Merge pull request #43159 from Algunenano/over-read
...
Fix several buffer over-reads
2022-11-15 15:04:24 +01:00
alesapin
a12f2d14dc
Simpler check
2022-11-15 14:46:38 +01:00
Anton Popov
8c66c759d4
Merge pull request #43133 from amosbird/issue_43111
...
Fix Nullable keys in hyperrectangle
2022-11-15 14:30:13 +01:00
alesapin
4ec11d0f58
Remove redundant branch
2022-11-15 13:58:53 +01:00
Antonio Andelic
0e5e58bed9
Remove exception if shared ID already created
2022-11-15 12:18:24 +00:00
Antonio Andelic
96838f5be7
Merge branch 'master' into keeper-prometheus
2022-11-15 12:00:02 +00:00
Kseniia Sumarokova
59cf5def67
Merge branch 'master' into disk-s3-throttler
2022-11-15 12:13:37 +01:00
xiedeyantu
ec6698395e
fix skip_unavailable_shards does not work using hdfsCluster table function
2022-11-15 13:25:15 +08:00
zhongyuankai
35690c1972
Merge branch 'master' into optimize_ttl
2022-11-15 08:49:37 +08:00
zhongyuankai
16f697f61d
fix test
2022-11-15 08:47:51 +08:00
Alexey Milovidov
6e01b2b2a1
Merge pull request #43203 from ClickHouse/fix-typo-2
...
Fix typo
2022-11-15 01:39:34 +01:00
Alexey Milovidov
3cab3ce484
Merge pull request #43198 from FArthur-cmd/MySQL_clickhouse_connect
...
add columns to information schema
2022-11-15 01:11:29 +01:00
alesapin
6a50a32cfc
Fix parts_to_do
2022-11-15 00:53:20 +01:00
alesapin
f53ada2bbd
Revert "Fix bug in replication queue which can lead to premature mutation finish"
...
This reverts commit 3b5424811c
2022-11-14 22:59:09 +01:00
alesapin
3b5424811c
Fix bug in replication queue which can lead to premature mutation finish
2022-11-14 21:53:16 +01:00
Kruglov Pavel
386b69acae
Merge pull request #43131 from xiedeyantu/fix_s3cluster_connection_fail
...
fix skip_unavailable_shards does not work using s3Cluster table function
2022-11-14 20:48:00 +01:00
Vitaly Baranov
8e99f5fea3
Move maskSensitiveInfoInQueryForLogging() to src/Parsers/
2022-11-14 18:55:19 +01:00
Alexander Tokmakov
c64109a2de
Merge branch 'master' into fetch-remove-covered
2022-11-14 18:23:16 +03:00
zhongyuankai
ea11d80536
Priority is given to deleting completely expired Parts
2022-11-14 23:03:56 +08:00
Han Fei
11f93dec3f
tmp commit
2022-11-14 15:53:45 +01:00
Raúl Marín
4988e387f2
Merge remote-tracking branch 'blessed/master' into over-read
2022-11-14 12:28:31 +01:00
Kseniia Sumarokova
b7df9aa3da
Merge pull request #43155 from zzsmdfj/add_oss_function_and_StorageOSS
...
to add_oss_function_and_StorageOSS
2022-11-14 12:23:57 +01:00
Kseniia Sumarokova
e58a7140be
Merge pull request #43194 from ClickHouse/async-metrics-documentation
...
Add documentation for `system.asynchronous_metrics`.
2022-11-14 12:21:34 +01:00
Kseniia Sumarokova
79206c6eb0
Merge pull request #43147 from kssenii/named-collections-refactoring
...
Add generic implementation for arbitrary structured named collections, access type and system.named_collections
2022-11-14 11:57:17 +01:00
Antonio Andelic
ee182776da
Merge branch 'master' into keeper-prometheus
2022-11-14 09:14:13 +01:00
Alexey Milovidov
4edd8b0135
Remove unused method
2022-11-14 08:07:27 +01:00
Alexey Milovidov
17693de053
Better logs
2022-11-14 08:01:48 +01:00
Alexey Milovidov
9c48664f11
Better logs
2022-11-14 07:48:50 +01:00
Alexey Milovidov
582db233c4
Better logs
2022-11-14 07:22:17 +01:00
Alexey Milovidov
99f84d8095
Better logs
2022-11-14 07:22:02 +01:00
Alexey Milovidov
df01689ccc
Better logs
2022-11-14 07:13:42 +01:00
