Han Fei
4c0efb0262
Merge pull request #53240 from hanfei1991/hanfei/statistic
...
use statistic to order prewhere conditions better
2023-11-29 12:26:44 +01:00
Nikolai Kochetov
731a1bfcf8
Cleanup
2023-11-29 11:11:58 +00:00
vdimir
470a092e3d
Merge pull request #51745 from ClickHouse/vdimir/setting_join_algorithm_order
...
Setting JoinAlgorithm respect specified order
2023-11-29 11:24:39 +01:00
Raúl Marín
b102145a4f
Merge pull request #57301 from Algunenano/fix_reports
...
Fix code reports
2023-11-29 11:11:33 +01:00
Robert Schulze
bc3da5032e
Merge pull request #57330 from rschu1ze/fix-no-lib-build
...
Follow-up to #56490 : Fix build with `cmake -DENABLE_LIBRARIES=0`
2023-11-29 09:42:12 +01:00
Antonio Andelic
f8811351fa
Merge pull request #57267 from ClickHouse/fix-readonly-replica
...
Fix `ReadonlyReplica` metric for all cases
2023-11-29 08:32:25 +01:00
Sichen Zhao
1b8b1c0a63
increase default replicated_deduplication_window from 100 to 1k
2023-11-28 14:24:06 -08:00
Robert Schulze
5f5e8633c7
Merge pull request #57220 from ucasfl/alias_column_in_indices
...
Support use alias column in indices
2023-11-28 21:31:34 +01:00
Robert Schulze
7d26990752
Follow-up to #56490 : Fix build with -DENABLE_LIBRARIES=0
2023-11-28 19:59:43 +00:00
vdimir
b748405976
Intorduce *List definition for muli enum settings
2023-11-28 19:09:02 +00:00
vdimir
a6eb2ad930
Fix JoinStep::describeJoinActions
2023-11-28 19:09:01 +00:00
Nikolai Kochetov
4d906d6e3c
Cleanup
2023-11-28 18:35:19 +00:00
avogar
4d9a1b50f9
Add information about new _size virtual column in file/s3/url/hdfs/azure table functions
2023-11-28 18:15:07 +00:00
Nikolai Kochetov
84194f10f2
Fixing tests.
2023-11-28 18:04:30 +00:00
Nikolai Kochetov
08a7575984
Re-implement filtering a bit.
2023-11-28 16:17:35 +00:00
Han Fei
e4421e2808
remove wrong code
2023-11-28 16:46:55 +01:00
Nikita Taranov
52f644c0df
Merge branch 'master' into optimize_reading3
2023-11-28 16:36:38 +01:00
Han Fei
6733438adc
Merge branch 'master' into hanfei/statistic
2023-11-28 16:35:00 +01:00
Han Fei
8e84c734bf
rename some code
2023-11-28 16:32:47 +01:00
zvonand
c306d21b54
merge master + resolve conflicts
2023-11-28 15:51:21 +01:00
Mikhail Koviazin
daaa575106
Remove unused Strings from MergeTreeData
...
Compiler does not emit a warning when there is an unused variable of
a type with a non-trivial constructor. std::string is one of them.
Hence, these two variables were overlooked. This commit removes them
2023-11-28 16:33:44 +02:00
Alexander Tokmakov
e40c71a74c
Merge pull request #56533 from ClickHouse/rmt_check_shutdown_flags_in_retry_loops
...
ReplicatedMergeTree: check shutdown flags in retry loops
2023-11-28 15:24:40 +01:00
Alexander Sapin
2cd9f5f3cc
Fix known issue
2023-11-28 15:13:12 +01:00
Alexander Sapin
e6855cef4d
Followup
2023-11-28 14:01:31 +01:00
Alexander Sapin
b5404f4a9f
Followup
2023-11-28 13:34:05 +01:00
Alexander Sapin
e1aa62d9cd
Make alter materialized view non expiremental and deprecate allow_experimental_alter_materialized_view_structure obsolete
2023-11-28 13:17:55 +01:00
Antonio Andelic
636ac3f349
Fix assert
2023-11-28 12:07:05 +00:00
vdimir
af81cb8252
Setting JoinAlgorithm respect specified order
2023-11-28 11:01:35 +00:00
Antonio Andelic
9dd4783675
Merge branch 'master' into fix-readonly-replica
2023-11-28 10:43:29 +00:00
Robert Schulze
d459fa1fc6
Merge remote-tracking branch 'ClickHouse/master' into alias_column_in_indices
2023-11-28 10:15:25 +00:00
Raúl Marín
ff0df49e2b
Fix comment in MergeTreeDataPartWide::checkConsistency
2023-11-28 10:09:03 +00:00
Nikolai Kochetov
7e1f625179
Revert system tables to master.
2023-11-28 10:07:36 +00:00
Julia Kartseva
57a4492275
Merge pull request #56490 from jkartseva/fuzz_json
...
Fuzz JSON table function
2023-11-27 21:47:53 -08:00
Ubuntu
a85cf758b0
Fuzz JSON table function
...
Create a table function `fuzzJSON`
An example query:
```
CREATE NAMED COLLECTION json_fuzzer AS json_str={};
SELECT *
FROM fuzzJSON(json_fuzzer, json_str = '{"students" : ["Alice", "Bob"]}', random_seed = 666, max_output_length = 128, probability = 0.9)
LIMIT 3
Query id: 7f802052-efb0-41b4-87fa-03b7dd290e9d
┌─json──────────────────────────────────────────────────────────────────────────────────┐
│ {"ade0yX":[9200406572736542991, true, "sm"]} │
│ {"students":["Alice", "eSN3WO#a6NYTBe0$#OWwyIQ"], "cVoP2BuQugQ":17384271928263249403} │
│ {"students":["Alice", "Bob", "T+-k4+PJGkL%XRRaF2BoeN@&A"]} │
└───────────────────────────────────────────────────────────────────────────────────────┘
```
Next step:
* Generate invalid string
Fixes #35962
add Object('json')
use named collection
2023-11-28 01:58:24 +00:00
Han Fei
7bd2ea3add
Merge branch 'master' into hanfei/statistic
2023-11-27 23:31:23 +01:00
Antonio Andelic
544f6081eb
Merge pull request #57259 from ClickHouse/ignore-column-comments
...
Ignore comments when comparing column descriptions
2023-11-27 20:43:39 +01:00
Nikolai Kochetov
e1b00b3ee0
Merge branch 'master' into SystemTablesFilterEngine
2023-11-27 18:54:21 +00:00
Antonio Andelic
9077550bb1
Better Readonly metric
2023-11-27 14:25:45 +00:00
Antonio Andelic
9707796869
Ignore comments when comparing column descriptions
2023-11-27 12:25:41 +00:00
vdimir
bbc202a891
Merge pull request #56816 from AVMusorin/improve-settings-rocksdb
...
Show total_bytes and total_rows in system tables for RocksDB storage
2023-11-27 11:42:15 +01:00
Kruglov Pavel
b10e46b2bc
Merge pull request #57126 from Avogar/size-virtual-column
...
Add _size virtual column to s3/file/hdfs/url/azureBlobStorage engines
2023-11-27 10:12:18 +01:00
flynn
aa15a4e46d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into alias_column_in_indices
2023-11-27 03:52:00 +00:00
flynn
f995ac87c5
review fix
2023-11-27 03:50:34 +00:00
Azat Khuzhin
4a02de4674
Add ability to disable checksums for S3 to avoid excessive input file read
...
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
PayloadSigningPolicy::Never)
So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.
By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.
And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.
Here is an example stacktrace of this excessive read:
<details>
<summary>stacktrace</summary>
(lldb) bt
* thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
* frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()
</details>
This new behaviour could be enabled with `s3_disable_checksum=true`.
Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
2023-11-26 19:20:19 +01:00
Alexey Milovidov
a9bb76378f
Merge pull request #56724 from canhld94/ch_replicated_column_mismatch
...
Do not allow tables on different replicas have different aggregate functions in SimpleAggregateFunction column
2023-11-26 17:20:10 +01:00
Alexey Milovidov
1cc33f3430
Merge pull request #56164 from azat/not-byte-identical-message
...
Add more details to "Data after merge is not byte-identical to data on another replicas"
2023-11-26 04:14:15 +01:00
Alexey Milovidov
304d6375be
Merge pull request #56225 from azat/rocksdb-compact
...
Allow manual compaction of rocksdb via OPTIMIZE query
2023-11-26 03:59:54 +01:00
Robert Schulze
5b7d2a903d
Merge pull request #57152 from rschu1ze/fine-granular-plan-opt-settings
...
Fine-granular enablement/disabling of plan-level optimizations
2023-11-25 20:44:23 +01:00
flynn
9f174b1c5e
Support use alias column in indices
2023-11-25 15:40:22 +00:00
Vitaly Baranov
2e7f314599
Merge pull request #50209 from ilejn/merge_row_policy
...
Engine Merge obeys row policy
2023-11-25 10:34:22 +01:00
Sema Checherinda
bd36dce81b
Merge pull request #57129 from ClickHouse/fix_something
...
Fix something in ReplicatedMergeTree
2023-11-24 20:19:25 +01:00
Alexander Tokmakov
deda8483be
Merge pull request #56536 from ClickHouse/move-partition-inactive-replica
...
Fix race between REPLACE_RANGE and GET_PART (set actual part name when fetching)
2023-11-24 20:09:24 +01:00
Aleksandr Musorin
c3b617bd7b
Move class functions upper
2023-11-24 19:00:00 +01:00
Kseniia Sumarokova
13187bf4f1
Merge pull request #56959 from melvynator/master
...
Add experimental flag for MaterializedPostgreSQL table engine.
2023-11-24 16:16:42 +01:00
Kseniia Sumarokova
c25c6ec403
Merge pull request #57031 from aiven-sal/aiven-sal/fix_namedcoll2
...
NamedCollections: make exception message more informative.
2023-11-24 16:15:16 +01:00
alesapin
a3c2896e9d
Merge pull request #56846 from aalexfvk/add_cancellation_point_for_moving_background_operation
...
Add cancellation hook for moving background operation
2023-11-24 12:12:47 +01:00
Alexey Milovidov
d72bc854d1
Merge pull request #57074 from ClickHouse/less-allocation-in-arenas
...
Avoid excessive allocation in Arena
2023-11-23 23:02:26 +01:00
Igor Nikonov
0434044a7e
Merge pull request #57054 from ClickHouse/pr-cleanup-narrow-dependency
...
Parallel replicas: cleanup, narrow dependency
2023-11-23 20:12:37 +01:00
Robert Schulze
77d48e4d55
"optimise" --> "optimize". Sorry, friends from the UK.
2023-11-23 18:41:35 +00:00
Aleksei Filatov
1a03f5f7f4
Merge remote-tracking branch 'upstream/master' into add_cancellation_point_for_moving_background_operation
2023-11-23 16:43:33 +03:00
serxa
663c8cd944
Merge branch 'master' into dashboards-table
2023-11-23 10:29:48 +00:00
serxa
0fc25fbda8
Merge branch 'master' into async-loader-integration
2023-11-23 09:32:43 +00:00
Alexander Tokmakov
a0ce9f46cf
fix
2023-11-22 23:15:20 +01:00
Alexey Milovidov
2ba2f48051
Merge branch 'master' of github.com:ClickHouse/ClickHouse into less-allocation-in-arenas
2023-11-22 22:11:59 +01:00
avogar
70f45c11bd
Fix
2023-11-22 19:56:42 +00:00
Alexander Tokmakov
ef3e558a65
fix no such key for empty parts
2023-11-22 20:51:10 +01:00
Alexander Tokmakov
d9eb68ba35
fix unexpected outdated parts
2023-11-22 20:29:57 +01:00
Alexander Tokmakov
9f1a9826af
fix replace/move partition
2023-11-22 20:04:14 +01:00
Alexander Tokmakov
86f7732c24
Update StorageReplicatedMergeTree.cpp
2023-11-22 19:41:27 +01:00
avogar
031f030102
Remove unused method
2023-11-22 18:27:32 +00:00
Kruglov Pavel
b84e3cf683
Merge branch 'master' into size-virtual-column
2023-11-22 19:25:00 +01:00
avogar
4a86f4a7b9
Fix style changes
2023-11-22 18:24:34 +00:00
avogar
007353a2dd
Add _size virtual column to s3/file/hdfs/url/azureBlobStorage engines
2023-11-22 18:12:36 +00:00
Alexander Tokmakov
d277bfb225
setActualPartName for executeReplaceRange
2023-11-22 18:48:25 +01:00
vdimir
ffbe85d3a0
Merge pull request #56668 from ClickHouse/vdimir/analyzer_s3_partition_pruning
...
Analyzer: filtering by virtual columns for StorageS3
2023-11-22 16:44:44 +01:00
Duc Canh Le
44a3dc1cd8
add new method to check if two types are exactly identical
...
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
2023-11-22 15:02:34 +00:00
Dmitry Novik
7539928814
Merge pull request #56079 from ZhiguoZh/20231027-combine-filter-avx512
...
Optimize DB::combineFilters with AVX512_VBMI2 intrinsic
2023-11-22 15:42:33 +01:00
zvonand
5153798aeb
Introduced fileCluster table function
...
Added fileCluster function
Added test and docs
2023-11-22 15:06:04 +01:00
Kruglov Pavel
9fb1acc947
Merge pull request #56810 from Avogar/iceberg-metadata-files
...
Iceberg metadata files
2023-11-22 14:40:04 +01:00
Alexey Milovidov
9cf7ebeced
Minor changes
2023-11-22 12:13:47 +01:00
Ilya Golshtein
c2816ec589
merge_row_policy: make clang-tidy happy
2023-11-21 22:06:18 +00:00
Dmitry Novik
1e0b4f979c
Add extra test
2023-11-21 22:55:55 +01:00
Kseniia Sumarokova
04aa0b10fc
Merge pull request #56990 from ClickHouse/s3queue-fix-incorrect-ref-count-from-exception-in-ctor
...
S3Queue: fix metadata reference increment
2023-11-21 21:37:23 +01:00
Alexander Tokmakov
990e3ea1c8
Merge branch 'master' into rmt_check_shutdown_flags_in_retry_loops
2023-11-21 18:01:16 +01:00
vdimir
a139ae97eb
Merge pull request #52918 from ClickHouse/vdimir/s3_blob_log
...
Add system table with blob storage operations log
2023-11-21 17:40:42 +01:00
Igor Nikonov
63a1625b77
Merge remote-tracking branch 'origin/master' into pr-cleanup-narrow-dependency
2023-11-21 16:05:48 +00:00
Igor Nikonov
ce98dfb251
Settings pointer to reference
2023-11-21 16:04:54 +00:00
Igor Nikonov
66f6a6575f
Cleanup iteration: settings usage
2023-11-21 13:29:04 +00:00
Igor Nikonov
83c6ac4f6c
Remove storage as parameter dependency, only columns are needed
2023-11-21 12:14:12 +00:00
alesapin
227bb30d02
Merge pull request #56986 from kevinmingtarja/implicit-constraint-collapsing-merged-tree
...
Add implicit constraint for CollapsingMergeTree sign column
2023-11-21 12:43:51 +01:00
kssenii
474f1ae0b6
Better
2023-11-21 11:42:23 +01:00
Aleksandr Musorin
ee0ae17ce2
Added optimize_trivial_approximate_count_query documentatiosn for RocksDB
2023-11-21 11:31:56 +01:00
Salvatore Mesoraca
f96cd4cfb8
Update src/Storages/NamedCollectionsHelpers.cpp
...
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
2023-11-21 11:14:48 +01:00
Salvatore Mesoraca
970e34f32e
NamedCollections: make exception message more informative.
...
Fix test 00002_log_and_exception_messages_formatting
2023-11-21 10:21:28 +01:00
vdimir
15234474d7
Implement system table blob_storage_log
2023-11-21 09:18:25 +00:00
Amos Bird
ee1cec4d38
Allow RENAME of index columns.
2023-11-21 15:42:54 +08:00
Amos Bird
6581aaf9e8
Prevent incompatible ALTER of projection columns.
2023-11-21 15:42:54 +08:00
kevinyhzou
3adc8fdf78
Fix ci
2023-11-21 11:22:12 +08:00
Kseniia Sumarokova
2beee5d9cb
Merge branch 'master' into s3queue-fix-empty-key-check
2023-11-20 20:17:47 +01:00
jsc0218
3d5265c1d3
Merge remote-tracking branch 'origin/master' into SystemTablesFilterEngine
...
merge master
2023-11-20 18:10:54 +00:00