Azat Khuzhin
3b2ceee753
Apply connection timeouts settings for Distributed async INSERT from the query
...
Previosly connection related settings (connect_timeout_with_failover_ms,
connect_timeout_with_failover_secure_ms) was applied from the query only
for the case insert_distributed_sync=1, and in case of async INSERT it
uses global settings.
Note that this changes how connections is allocated, so now
split_batch_on_failure will create it's own connection, and this can
introduce more duplicates since in case of split_batch_on_failure is
enabled it may send files to different server, but this should not be a
problem because:
- it does not resend batch if it has only one file, when deduplication
will work
- and in all other cases deduplication will not work since checksum
should be different
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-11 09:54:07 +01:00
Azat Khuzhin
d1cee3e1ff
Do not resend batch as separate files if there is only one file
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-11 09:48:44 +01:00
zzsmdfj
3835373644
to add_oss_function_and_StorageOSS
2022-11-11 16:40:10 +08:00
Amos Bird
661c541e57
Fix Nullable keys in hyperrectangle.
2022-11-11 11:14:05 +08:00
kssenii
63de577172
Better, add tests
2022-11-10 22:47:30 +01:00
avogar
9e89af28c6
Refactor BSONEachRow format, fix bugs, support more data types, support parallel parsing and schema inference
2022-11-10 20:15:14 +00:00
Alexander Tokmakov
bb00a246f1
priority for DROP_RANGEs, assertion for others
2022-11-10 20:53:05 +01:00
kssenii
a5d06cd909
Better Impl
2022-11-10 19:59:52 +01:00
Alexander Tokmakov
e415fc6e95
Revert "Optimize TTL merge, completely expired parts can be removed in time"
2022-11-10 19:38:55 +03:00
kssenii
a47aac3e2b
Better system table structure
2022-11-10 17:07:17 +01:00
kssenii
bed7ac9442
Support arbitrary structure collections
2022-11-10 17:05:24 +01:00
kssenii
c4f522b414
Support arbintrary structure collections
2022-11-10 16:32:15 +01:00
Igor Nikonov
f8c907c463
Revert "Revert " Keeper retries during insert (clean)""
2022-11-10 13:14:04 +01:00
Alexander Tokmakov
fb8c527a2f
Revert " Keeper retries during insert (clean)"
2022-11-10 14:12:24 +03:00
Igor Nikonov
1b9be27fdb
Merge pull request #42607 from ClickHouse/igor/insert_zk_retries_retry
...
Keeper retries during insert (clean)
2022-11-10 12:01:00 +01:00
Antonio Andelic
1954ff385d
Merge pull request #43076 from canhld94/ch_canh_minor
...
Better logging for mark range filtering on projection parts
2022-11-10 09:33:12 +01:00
Nikolai Kochetov
f2f5c170e4
Merge branch 'master' into read-in-order-from-query-plan
2022-11-09 22:36:36 +01:00
alesapin
9e1627fcdd
Merge pull request #43060 from ClickHouse/fix_projection_part
...
Fix projection part removal with zero-copy replication
2022-11-09 18:15:49 +01:00
kssenii
e94e087187
Add generic named collections class and factory
2022-11-09 14:08:16 +01:00
Igor Nikonov
379113f8f1
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-11-09 13:03:32 +00:00
Alexander Tokmakov
1c6a617207
Merge branch 'master' into fix_intersecting_parts2
2022-11-09 14:38:18 +03:00
Maksim Kita
768ada118b
Analyzer AST key condition crash fix
2022-11-09 11:41:39 +01:00
Vladimir C
82bf992099
Merge pull request #42869 from zhongyuankai/optimze_ttl_merge
2022-11-09 11:26:46 +01:00
Duc Canh Le
b9a3d090ed
use const string &
2022-11-09 13:07:24 +08:00
Duc Canh Le
901e1e2fef
better logging for mark range filtering on projection parts
2022-11-09 12:01:51 +08:00
alesapin
761274d857
Fix wrong logic
2022-11-09 01:14:03 +01:00
Igor Nikonov
345304abe0
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-11-08 20:26:50 +00:00
alesapin
a31a79020f
Add comment
2022-11-08 16:27:26 +01:00
alesapin
6f059be748
Fix bug with temporary projection parts deletion
2022-11-08 15:37:03 +01:00
serxa
2de26daa56
fix build
2022-11-08 14:31:29 +00:00
SmitaRKulkarni
658036d179
Merge pull request #42708 from ClickHouse/42414_Enable_functions_in_readonly_mode
...
42414 enable functions in readonly mode
2022-11-08 15:22:31 +01:00
Alexander Tokmakov
eb19df0094
fix
2022-11-08 15:21:25 +01:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
SmitaRKulkarni
4f2e5596e3
Merge pull request #42933 from yakkomajuri/patch-2
...
fix(typo): Passible -> Possible
2022-11-08 14:54:50 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
Maksim Kita
0aa4fd3c2d
Fixed tests
2022-11-08 14:29:17 +01:00
Kseniia Sumarokova
d2b777c059
Merge pull request #41054 from rubin-do/hudi
...
Add Hudi and DeltaLake table engines
2022-11-08 12:03:06 +01:00
Kseniia Sumarokova
92b5ca326c
Merge pull request #42303 from xiedeyantu/system_table_add_column
...
system.detached_parts add column bytes_on_disk and path
2022-11-08 12:00:56 +01:00
Robert Schulze
2c568df714
Merge pull request #42968 from ClickHouse/bitcast
...
Less use of CH-specific bit_cast()
2022-11-08 10:54:10 +01:00
Robert Schulze
bb507356ef
Merge pull request #42209 from canhld94/ch_canh_fix_prefix_not_like
...
Add `notLike` to key condition atom map
2022-11-08 10:07:21 +01:00
Antonio Andelic
7067f9e0e1
Merge pull request #42947 from azat/mergetree/part-load-tiny
...
Attach thread pool for loading parts to the query
2022-11-08 09:17:32 +01:00
zhongyuankai
37099e57f4
Merge branch 'master' into optimze_ttl_merge
2022-11-08 12:26:46 +08:00
Azat Khuzhin
2bd4a71150
Merge branch 'master' into fetch-remove-covered
2022-11-08 00:55:00 +01:00
Kseniia Sumarokova
4fc7926c16
Merge branch 'master' into system_table_add_column
2022-11-07 23:36:57 +01:00
Kseniia Sumarokova
d695240d53
Update StorageHudi.cpp
2022-11-07 23:07:00 +01:00
serxa
ea17b30358
Merge branch 'storage-system-moves' of github.com:ClickHouse/ClickHouse into storage-system-moves
2022-11-07 20:33:31 +00:00
serxa
59408ebf57
fill more fields
2022-11-07 20:33:25 +00:00
Sergei Trifonov
6d3b96ee42
Merge branch 'master' into storage-system-moves
2022-11-07 21:31:27 +01:00
alesapin
257e766e0e
Trying to fix two flaky tests
2022-11-07 20:49:41 +01:00
Alexander Tokmakov
9210e586d2
fix
2022-11-07 20:27:18 +01:00
Alexander Tokmakov
b8174a63a8
Merge branch 'master' into fix_intersecting_parts2
2022-11-07 20:26:36 +01:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
xiedeyantu
d4cdc38b71
better
2022-11-07 22:49:58 +08:00
Kseniia Sumarokova
891e167509
Update StorageHudi.cpp
2022-11-07 15:17:16 +01:00
Kruglov Pavel
48163ff9c2
Merge pull request #42938 from amosbird/issue_42771
...
Fix NOT_FOUND_COLUMN_IN_BLOCK in some rare case
2022-11-07 13:10:12 +01:00
kssenii
897811371d
Minor changes
2022-11-07 12:45:26 +01:00
kssenii
819be6bb32
Merge remote-tracking branch 'upstream/master' into hudi
2022-11-07 11:30:53 +01:00
Duc Canh Le
bd2bd7149d
fix typo
2022-11-07 18:22:50 +08:00
SmitaRKulkarni
c563e124c2
Merge branch 'master' into 42414_Enable_functions_in_readonly_mode
2022-11-07 09:50:05 +01:00
Duc Canh Le
f7e65ebd72
make comments more clear
2022-11-07 16:25:36 +08:00
Robert Schulze
cdaf0becfe
Merge branch 'master' into bitcast
2022-11-07 09:24:52 +01:00
Robert Schulze
f3f50530d1
Merge branch 'master' into ch_canh_fix_prefix_not_like
2022-11-07 09:14:02 +01:00
Antonio Andelic
e08f94d0f8
Merge pull request #42659 from ClickHouse/optimize-entire-partition
...
Support for optimizing old parts for entire partition only
2022-11-07 09:06:11 +01:00
Duc Canh Le
69d6d42443
NOT LIKE only work for 'prefix%'
2022-11-06 15:26:19 +08:00
zhongyuankai
ef6065b331
fix test
2022-11-06 13:40:26 +08:00
Duc Canh Le
a8bce28c04
Merge branch 'master' into ch_canh_fix_prefix_not_like
2022-11-06 13:04:08 +08:00
Igor Nikonov
e94b9cd003
Fixes
...
+ enable retries in integration test
+ typo in fault injection traces
2022-11-05 22:28:32 +00:00
Igor Nikonov
859022660e
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-11-04 23:31:51 +00:00
Daniil Rubin
7500cc38c0
Apply clang-format
2022-11-04 20:51:44 +00:00
Daniil Rubin
37048e986a
Add comments, fix review issues
2022-11-04 20:49:24 +00:00
Daniil Rubin
d6bbd65dde
Fix broken include
2022-11-04 16:03:12 +00:00
Robert Schulze
9c066e964d
Less use of CH-specific bit_cast()
...
Converted usage of CH-custom bit_cast to std::bit_cast if possible, i.e.
when
sizeof(From) == sizeof(To).
(The CH-custom bit_cast is able to deal with sizeof(From) != sizeof(To).)
Motivation for this came from #42847 where it is not clear how the
internal bit_cast should behave on big endian systems, so we better
avoid that situation as much as possible.
2022-11-04 15:52:48 +00:00
Vladimir C
f359411668
Merge branch 'master' into optimze_ttl_merge
2022-11-04 16:52:20 +01:00
Vitaly Baranov
52b1f4aed9
Merge pull request #42484 from vitlibar/mask-sensitive-info-in-logs
...
Mask some information in logs
2022-11-04 14:09:38 +01:00
zhongyuankai
1345f7e73d
Merge branch 'master' into optimze_ttl_merge
2022-11-04 20:54:46 +08:00
alesapin
82729a88c8
Merge pull request #42833 from vitlibar/backup-s3-increase-upload-part-size-exponentially
...
Increase the size of upload part exponentially for backup to S3.
2022-11-04 13:41:10 +01:00
Kseniia Sumarokova
ed44b20694
Merge pull request #42224 from kssenii/fit-http-buffer-retries
...
Fix http buffer retries
2022-11-04 11:50:17 +01:00
Amos Bird
d901ead1bc
Done
2022-11-04 17:26:44 +08:00
zhongyuankai
049a593272
Merge branch 'master' into optimze_ttl_merge
2022-11-04 17:22:38 +08:00
zhongyuankai
e773eb2fd1
batter
2022-11-04 17:19:34 +08:00
Azat Khuzhin
7032b2b98a
Attach thread pool for loading parts to the query
...
Detach should be done via CurrentThread::defaultThreadDeleter()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-04 09:06:12 +01:00
Azat Khuzhin
cdd09fd72a
Improve messages for loading data parts (add part type)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-04 09:05:06 +01:00
Daniil Rubin
99d57325bb
Merge branch 'master' into hudi
2022-11-03 19:29:23 +00:00
Daniil Rubin
a573d8aef7
Apply clang-format
2022-11-03 18:17:39 +00:00
Daniil Rubin
7b57e94d9f
Fix StorageHudi
2022-11-03 18:14:10 +00:00
Daniil Rubin
a9f8948c8d
Fix build
2022-11-03 17:28:41 +00:00
Amos Bird
7186898ffa
Experiment
2022-11-04 00:29:17 +08:00
Yakko Majuri
e68a3b0c7a
fix(typo): Passible -> Possible
2022-11-03 11:25:07 -03:00
kssenii
4fe4a07600
Add test
2022-11-03 12:14:08 +01:00
Antonio Andelic
66ffdda973
Merge branch 'master' into optimize-entire-partition
2022-11-03 09:52:13 +01:00
Alexander Tokmakov
e67d05604c
Merge branch 'master' into fix_intersecting_parts2
2022-11-02 19:30:27 +01:00
Alexander Tokmakov
d8b3a2af84
fix race between INSERT and DROP
2022-11-02 19:26:50 +01:00
Vitaly Baranov
32194c1200
Add max limitation for the size of an uploaded part.
2022-11-02 17:53:54 +01:00
Igor Nikonov
9a315b7899
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-11-02 15:32:54 +00:00
Vitaly Baranov
e0133688bc
Merge branch 'master' into mask-sensitive-info-in-logs
2022-11-02 16:26:13 +01:00
Alexander Tokmakov
00c9e50ee3
fix race between INSERT and ALTER PARTITION
2022-11-02 16:25:26 +01:00
Alexander Tokmakov
acbad99b24
remove abandonable_lock part 2
2022-11-02 14:28:49 +01:00
Antonio Andelic
c4b717b343
Update src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
...
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
2022-11-02 08:24:54 +01:00
Duc Canh Le
ec382d7903
add comment
2022-11-02 09:50:41 +08:00
Duc Canh Le
c6598dc66d
Merge branch 'master' into ch_canh_fix_prefix_not_like
2022-11-02 09:42:01 +08:00
Alexander Tokmakov
4f50a999bb
remove abandonable_lock part 1
2022-11-01 21:35:25 +01:00
Nikolai Kochetov
478d3079c8
Merge branch 'master' into read-in-order-from-query-plan
2022-11-01 21:02:22 +01:00
Nikolai Kochetov
9ffebf4a0c
Fixing read-in-order for special storages.
2022-11-01 19:51:52 +00:00
Igor Nikonov
e76c3c381f
Better handling tables in read-only mode during insert
...
+ throw immediately if table has no metadata in zk
+ stop retries in case of shutdown
+ check if table is readonly at the begining of every retry
2022-11-01 18:47:51 +00:00
Anton Popov
88674ef964
Merge pull request #42794 from CurtizJ/better-code-in-check-part
...
Slightly better code with projections
2022-11-01 16:05:27 +01:00
zhongyuankai
c1e2b2af74
Optimize TTL merge, completely expired parts can be removed in time
2022-11-01 22:58:15 +08:00
Anton Popov
876dca48da
Merge pull request #36969 from CurtizJ/dynamic-columns-14
...
Support `Object` type inside other types
2022-11-01 15:20:02 +01:00
Duc Canh Le
07d9720ed5
Merge branch 'master' into ch_canh_fix_prefix_not_like
2022-11-01 22:16:09 +08:00
Kruglov Pavel
21d50f76ea
Merge pull request #41979 from Avogar/s3-cluster-schema-inference
...
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-11-01 14:00:21 +01:00
Vitaly Baranov
b9f2f17331
Add test and logging.
2022-11-01 12:23:20 +01:00
Maksim Kita
fe48a1ce7e
Merge pull request #42681 from kitaisreal/merge-tree-indexes-use-actions-dag
...
MergeTree indexes use ActionsDAG
2022-11-01 13:55:17 +03:00
Duc Canh Le
3545db0a6c
Merge branch 'master' into ch_canh_fix_prefix_not_like
2022-11-01 16:54:26 +08:00
Yakov Olkhovskiy
bab0e06e3d
Merge pull request #42458 from Clark0/match-func-use-index
...
match function can use index if it's a condition on string prefix #37333
2022-10-31 17:33:58 -04:00
Igor Nikonov
9cc64a0fcc
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-10-31 18:21:21 +00:00
Igor Nikonov
2d5050ea66
Fix review comments
2022-10-31 18:20:58 +00:00
Anton Popov
2ae3cfa9e0
Merge branch 'master' into dynamic-columns-14
2022-10-31 16:15:19 +01:00
Alexander Tokmakov
599ccb9939
Update MergeTreeData.h
2022-10-31 16:18:17 +03:00
Alexander Tokmakov
b2b9479afa
Update src/Storages/MergeTree/MergeTreeData.h
...
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
2022-10-31 16:15:35 +03:00
SmitaRKulkarni
54035270b2
Merge branch 'master' into 42414_Enable_functions_in_readonly_mode
2022-10-31 14:15:29 +01:00
Maksim Kita
bca22ec5f5
Fixed code review issues
2022-10-31 12:30:00 +01:00
Maksim Kita
115fcaffc5
Fixed tests
2022-10-31 11:30:56 +01:00
Vitaly Baranov
5d2a222fe4
Mask sensitive information in logs.
2022-10-31 10:50:33 +01:00
Duc Canh Le
d309e8793a
Merge branch 'master' into ch_canh_fix_prefix_not_like
2022-10-31 15:13:39 +08:00
Alexander Tokmakov
f53df7870c
fix race between drop and failed insert
2022-10-30 17:59:47 +01:00
Alexander Tokmakov
c68ab231f9
fix accessing part in Deleting state
2022-10-30 17:30:51 +01:00
Anton Popov
8de75bf29c
better code with projections
2022-10-29 14:39:59 +00:00
clarkcaoliu
ca5dbe88db
match function can use index if prefix
2022-10-29 18:25:29 +08:00
Igor Nikonov
d09deaf10f
Fix style check
2022-10-28 15:17:05 +00:00
Igor Nikonov
937dd44399
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-10-28 15:09:23 +00:00
Igor Nikonov
c53b96a11c
Add more comments
2022-10-28 15:09:01 +00:00
Alexey Milovidov
f64e632707
Merge pull request #42469 from amosbird/issue_42456
...
Fix misbehavior of key analysis
2022-10-28 16:18:00 +02:00
Antonio Andelic
5fae8b0cc9
Merge branch 'master' into optimize-entire-partition
2022-10-28 11:44:10 +00:00
serxa
acc00218d9
fix build
2022-10-28 11:23:49 +00:00
Sergei Trifonov
836595c7ae
Merge branch 'master' into storage-system-moves
2022-10-28 11:39:38 +02:00
Duc Canh Le
848f5b5d65
Merge branch 'master' into ch_canh_fix_prefix_not_like
2022-10-28 13:40:17 +08:00
Duc Canh Le
c080964391
correct 'notLike' in key condition
2022-10-28 13:39:01 +08:00
Duc Canh Le
69311bd5cd
Merge branch 'master' into ch_canh_fix_prefix_not_like
2022-10-28 11:59:50 +08:00
Igor Nikonov
3682913172
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-10-27 22:01:53 +00:00
Igor Nikonov
a601c166ad
Add clean up for faults injected before zk request
...
+ for multi-op request, it create responses in multi response
+ some polishing
2022-10-27 21:35:13 +00:00
Anton Popov
7715afa595
Merge pull request #42618 from CurtizJ/refactor-data-part
...
Try to save `IDataPartStorage` interface
2022-10-27 22:20:44 +02:00
Igor Nikonov
f773436de5
Fix flaky test
2022-10-27 13:34:07 +00:00
Nikolai Kochetov
24e1aaf324
Merge pull request #42576 from ClickHouse/table-function-merge-invalid-where-type
...
Invalid type in where for Merge table (logical error)
2022-10-27 13:03:27 +02:00
Antonio Andelic
10aa7091ca
Merge pull request #42605 from ClickHouse/correctly-handle-prewhere-type
...
Correctly handle non-native integers in PREWHERE
2022-10-27 09:25:38 +02:00
SmitaRKulkarni
e860ecc80c
Merge branch 'master' into 42414_Enable_functions_in_readonly_mode
2022-10-27 08:26:44 +02:00
Nikolai Kochetov
313cbb1058
Fix test.
2022-10-26 17:49:36 +00:00
Igor Nikonov
649e8d302c
Sync with retiring branch igor/insert_zk_retries
2022-10-26 17:41:44 +00:00
Igor Nikonov
8435cba352
Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry
2022-10-26 17:35:47 +00:00
Maksim Kita
f04f663818
Fixed tests
2022-10-26 18:55:14 +02:00
Smita Kulkarni
6a8fa2d4a5
Added new field allow_readonly in system.table_functions to allow using table functions in readonly mode.
...
Implementation:
* Added a new field allow_readonly to table system.table_functions.
* Updated to use new field allow_readonly to allow using table functions in readonly mode.
* Added TableFunctionProperties similar to AggregateFunctionProperties.
* The functions allowed in readonly mode are in this set table_functions_allowed_in_readonly_mode.
Testing:
* Added a test for filesystem tests/queries/0_stateless/02473_functions_in_readonly_mode.sh
Documentation:
* Updated the english documentation for Table Functions.
2022-10-26 18:45:23 +02:00
Nikolai Kochetov
1520bcd53c
Fix error code.
2022-10-26 15:50:10 +00:00
