Commit Graph

15306 Commits

Author SHA1 Message Date
Han Fei
4c0efb0262
Merge pull request #53240 from hanfei1991/hanfei/statistic
use statistic to order prewhere conditions better
2023-11-29 12:26:44 +01:00
Nikolai Kochetov
731a1bfcf8 Cleanup 2023-11-29 11:11:58 +00:00
vdimir
470a092e3d
Merge pull request #51745 from ClickHouse/vdimir/setting_join_algorithm_order
Setting JoinAlgorithm respect specified order
2023-11-29 11:24:39 +01:00
Raúl Marín
b102145a4f
Merge pull request #57301 from Algunenano/fix_reports
Fix code reports
2023-11-29 11:11:33 +01:00
Robert Schulze
bc3da5032e
Merge pull request #57330 from rschu1ze/fix-no-lib-build
Follow-up to #56490: Fix build with `cmake -DENABLE_LIBRARIES=0`
2023-11-29 09:42:12 +01:00
Antonio Andelic
f8811351fa
Merge pull request #57267 from ClickHouse/fix-readonly-replica
Fix `ReadonlyReplica` metric for all cases
2023-11-29 08:32:25 +01:00
Sichen Zhao
1b8b1c0a63 increase default replicated_deduplication_window from 100 to 1k 2023-11-28 14:24:06 -08:00
Robert Schulze
5f5e8633c7
Merge pull request #57220 from ucasfl/alias_column_in_indices
Support use alias column in indices
2023-11-28 21:31:34 +01:00
Robert Schulze
7d26990752
Follow-up to #56490: Fix build with -DENABLE_LIBRARIES=0 2023-11-28 19:59:43 +00:00
vdimir
b748405976
Intorduce *List definition for muli enum settings 2023-11-28 19:09:02 +00:00
vdimir
a6eb2ad930
Fix JoinStep::describeJoinActions 2023-11-28 19:09:01 +00:00
Nikolai Kochetov
4d906d6e3c Cleanup 2023-11-28 18:35:19 +00:00
avogar
4d9a1b50f9 Add information about new _size virtual column in file/s3/url/hdfs/azure table functions 2023-11-28 18:15:07 +00:00
Nikolai Kochetov
84194f10f2 Fixing tests. 2023-11-28 18:04:30 +00:00
Nikolai Kochetov
08a7575984 Re-implement filtering a bit. 2023-11-28 16:17:35 +00:00
Han Fei
e4421e2808 remove wrong code 2023-11-28 16:46:55 +01:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Han Fei
6733438adc Merge branch 'master' into hanfei/statistic 2023-11-28 16:35:00 +01:00
Han Fei
8e84c734bf rename some code 2023-11-28 16:32:47 +01:00
zvonand
c306d21b54 merge master + resolve conflicts 2023-11-28 15:51:21 +01:00
Mikhail Koviazin
daaa575106
Remove unused Strings from MergeTreeData
Compiler does not emit a warning when there is an unused variable of
a type with a non-trivial constructor. std::string is one of them.
Hence, these two variables were overlooked. This commit removes them
2023-11-28 16:33:44 +02:00
Alexander Tokmakov
e40c71a74c
Merge pull request #56533 from ClickHouse/rmt_check_shutdown_flags_in_retry_loops
ReplicatedMergeTree: check shutdown flags in retry loops
2023-11-28 15:24:40 +01:00
Alexander Sapin
2cd9f5f3cc Fix known issue 2023-11-28 15:13:12 +01:00
Alexander Sapin
e6855cef4d Followup 2023-11-28 14:01:31 +01:00
Alexander Sapin
b5404f4a9f Followup 2023-11-28 13:34:05 +01:00
Alexander Sapin
e1aa62d9cd Make alter materialized view non expiremental and deprecate allow_experimental_alter_materialized_view_structure obsolete 2023-11-28 13:17:55 +01:00
Antonio Andelic
636ac3f349 Fix assert 2023-11-28 12:07:05 +00:00
vdimir
af81cb8252
Setting JoinAlgorithm respect specified order 2023-11-28 11:01:35 +00:00
Antonio Andelic
9dd4783675 Merge branch 'master' into fix-readonly-replica 2023-11-28 10:43:29 +00:00
Robert Schulze
d459fa1fc6
Merge remote-tracking branch 'ClickHouse/master' into alias_column_in_indices 2023-11-28 10:15:25 +00:00
Raúl Marín
ff0df49e2b Fix comment in MergeTreeDataPartWide::checkConsistency 2023-11-28 10:09:03 +00:00
Nikolai Kochetov
7e1f625179 Revert system tables to master. 2023-11-28 10:07:36 +00:00
Julia Kartseva
57a4492275
Merge pull request #56490 from jkartseva/fuzz_json
Fuzz JSON table function
2023-11-27 21:47:53 -08:00
Ubuntu
a85cf758b0 Fuzz JSON table function
Create a table function `fuzzJSON`

An example query:
```
CREATE NAMED COLLECTION json_fuzzer AS json_str={};

SELECT *
FROM fuzzJSON(json_fuzzer, json_str = '{"students" : ["Alice", "Bob"]}', random_seed = 666, max_output_length = 128, probability = 0.9)
LIMIT 3

Query id: 7f802052-efb0-41b4-87fa-03b7dd290e9d

┌─json──────────────────────────────────────────────────────────────────────────────────┐
│ {"ade0yX":[9200406572736542991, true, "sm"]}                                          │
│ {"students":["Alice", "eSN3WO#a6NYTBe0$#OWwyIQ"], "cVoP2BuQugQ":17384271928263249403} │
│ {"students":["Alice", "Bob", "T+-k4+PJGkL%XRRaF2BoeN@&A"]}                            │
└───────────────────────────────────────────────────────────────────────────────────────┘
```

Next step:
* Generate invalid string

Fixes #35962

add Object('json')

use named collection
2023-11-28 01:58:24 +00:00
Han Fei
7bd2ea3add Merge branch 'master' into hanfei/statistic 2023-11-27 23:31:23 +01:00
Antonio Andelic
544f6081eb
Merge pull request #57259 from ClickHouse/ignore-column-comments
Ignore comments when comparing column descriptions
2023-11-27 20:43:39 +01:00
Nikolai Kochetov
e1b00b3ee0 Merge branch 'master' into SystemTablesFilterEngine 2023-11-27 18:54:21 +00:00
Antonio Andelic
9077550bb1 Better Readonly metric 2023-11-27 14:25:45 +00:00
Antonio Andelic
9707796869 Ignore comments when comparing column descriptions 2023-11-27 12:25:41 +00:00
vdimir
bbc202a891
Merge pull request #56816 from AVMusorin/improve-settings-rocksdb
Show total_bytes and total_rows in system tables for RocksDB storage
2023-11-27 11:42:15 +01:00
Kruglov Pavel
b10e46b2bc
Merge pull request #57126 from Avogar/size-virtual-column
Add _size virtual column to s3/file/hdfs/url/azureBlobStorage engines
2023-11-27 10:12:18 +01:00
flynn
aa15a4e46d Merge branch 'master' of github.com:ClickHouse/ClickHouse into alias_column_in_indices 2023-11-27 03:52:00 +00:00
flynn
f995ac87c5 review fix 2023-11-27 03:50:34 +00:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
2023-11-26 19:20:19 +01:00
Alexey Milovidov
a9bb76378f
Merge pull request #56724 from canhld94/ch_replicated_column_mismatch
Do not allow tables on different replicas have different aggregate functions in SimpleAggregateFunction column
2023-11-26 17:20:10 +01:00
Alexey Milovidov
1cc33f3430
Merge pull request #56164 from azat/not-byte-identical-message
Add more details to "Data after merge is not byte-identical to data on another replicas"
2023-11-26 04:14:15 +01:00
Alexey Milovidov
304d6375be
Merge pull request #56225 from azat/rocksdb-compact
Allow manual compaction of rocksdb via OPTIMIZE query
2023-11-26 03:59:54 +01:00
Robert Schulze
5b7d2a903d
Merge pull request #57152 from rschu1ze/fine-granular-plan-opt-settings
Fine-granular enablement/disabling of plan-level optimizations
2023-11-25 20:44:23 +01:00
flynn
9f174b1c5e Support use alias column in indices 2023-11-25 15:40:22 +00:00
Vitaly Baranov
2e7f314599
Merge pull request #50209 from ilejn/merge_row_policy
Engine Merge obeys row policy
2023-11-25 10:34:22 +01:00
Sema Checherinda
bd36dce81b
Merge pull request #57129 from ClickHouse/fix_something
Fix something in ReplicatedMergeTree
2023-11-24 20:19:25 +01:00
Alexander Tokmakov
deda8483be
Merge pull request #56536 from ClickHouse/move-partition-inactive-replica
Fix race between REPLACE_RANGE and GET_PART (set actual part name when fetching)
2023-11-24 20:09:24 +01:00
Aleksandr Musorin
c3b617bd7b Move class functions upper 2023-11-24 19:00:00 +01:00
Kseniia Sumarokova
13187bf4f1
Merge pull request #56959 from melvynator/master
Add experimental flag for MaterializedPostgreSQL table engine.
2023-11-24 16:16:42 +01:00
Kseniia Sumarokova
c25c6ec403
Merge pull request #57031 from aiven-sal/aiven-sal/fix_namedcoll2
NamedCollections: make exception message more informative.
2023-11-24 16:15:16 +01:00
alesapin
a3c2896e9d
Merge pull request #56846 from aalexfvk/add_cancellation_point_for_moving_background_operation
Add cancellation hook for moving background operation
2023-11-24 12:12:47 +01:00
Alexey Milovidov
d72bc854d1
Merge pull request #57074 from ClickHouse/less-allocation-in-arenas
Avoid excessive allocation in Arena
2023-11-23 23:02:26 +01:00
Igor Nikonov
0434044a7e
Merge pull request #57054 from ClickHouse/pr-cleanup-narrow-dependency
Parallel replicas: cleanup, narrow dependency
2023-11-23 20:12:37 +01:00
Robert Schulze
77d48e4d55
"optimise" --> "optimize". Sorry, friends from the UK. 2023-11-23 18:41:35 +00:00
Aleksei Filatov
1a03f5f7f4 Merge remote-tracking branch 'upstream/master' into add_cancellation_point_for_moving_background_operation 2023-11-23 16:43:33 +03:00
serxa
663c8cd944 Merge branch 'master' into dashboards-table 2023-11-23 10:29:48 +00:00
serxa
0fc25fbda8 Merge branch 'master' into async-loader-integration 2023-11-23 09:32:43 +00:00
Alexander Tokmakov
a0ce9f46cf fix 2023-11-22 23:15:20 +01:00
Alexey Milovidov
2ba2f48051 Merge branch 'master' of github.com:ClickHouse/ClickHouse into less-allocation-in-arenas 2023-11-22 22:11:59 +01:00
avogar
70f45c11bd Fix 2023-11-22 19:56:42 +00:00
Alexander Tokmakov
ef3e558a65 fix no such key for empty parts 2023-11-22 20:51:10 +01:00
Alexander Tokmakov
d9eb68ba35 fix unexpected outdated parts 2023-11-22 20:29:57 +01:00
Alexander Tokmakov
9f1a9826af fix replace/move partition 2023-11-22 20:04:14 +01:00
Alexander Tokmakov
86f7732c24
Update StorageReplicatedMergeTree.cpp 2023-11-22 19:41:27 +01:00
avogar
031f030102 Remove unused method 2023-11-22 18:27:32 +00:00
Kruglov Pavel
b84e3cf683
Merge branch 'master' into size-virtual-column 2023-11-22 19:25:00 +01:00
avogar
4a86f4a7b9 Fix style changes 2023-11-22 18:24:34 +00:00
avogar
007353a2dd Add _size virtual column to s3/file/hdfs/url/azureBlobStorage engines 2023-11-22 18:12:36 +00:00
Alexander Tokmakov
d277bfb225 setActualPartName for executeReplaceRange 2023-11-22 18:48:25 +01:00
vdimir
ffbe85d3a0
Merge pull request #56668 from ClickHouse/vdimir/analyzer_s3_partition_pruning
Analyzer: filtering by virtual columns for StorageS3
2023-11-22 16:44:44 +01:00
Duc Canh Le
44a3dc1cd8 add new method to check if two types are exactly identical
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
2023-11-22 15:02:34 +00:00
Dmitry Novik
7539928814
Merge pull request #56079 from ZhiguoZh/20231027-combine-filter-avx512
Optimize DB::combineFilters with AVX512_VBMI2 intrinsic
2023-11-22 15:42:33 +01:00
zvonand
5153798aeb Introduced fileCluster table function
Added fileCluster function
Added test and docs
2023-11-22 15:06:04 +01:00
Kruglov Pavel
9fb1acc947
Merge pull request #56810 from Avogar/iceberg-metadata-files
Iceberg metadata files
2023-11-22 14:40:04 +01:00
Alexey Milovidov
9cf7ebeced Minor changes 2023-11-22 12:13:47 +01:00
Ilya Golshtein
c2816ec589 merge_row_policy: make clang-tidy happy 2023-11-21 22:06:18 +00:00
Dmitry Novik
1e0b4f979c
Add extra test 2023-11-21 22:55:55 +01:00
Kseniia Sumarokova
04aa0b10fc
Merge pull request #56990 from ClickHouse/s3queue-fix-incorrect-ref-count-from-exception-in-ctor
S3Queue: fix metadata reference increment
2023-11-21 21:37:23 +01:00
Alexander Tokmakov
990e3ea1c8 Merge branch 'master' into rmt_check_shutdown_flags_in_retry_loops 2023-11-21 18:01:16 +01:00
vdimir
a139ae97eb
Merge pull request #52918 from ClickHouse/vdimir/s3_blob_log
Add system table with blob storage operations log
2023-11-21 17:40:42 +01:00
Igor Nikonov
63a1625b77 Merge remote-tracking branch 'origin/master' into pr-cleanup-narrow-dependency 2023-11-21 16:05:48 +00:00
Igor Nikonov
ce98dfb251 Settings pointer to reference 2023-11-21 16:04:54 +00:00
Igor Nikonov
66f6a6575f Cleanup iteration: settings usage 2023-11-21 13:29:04 +00:00
Igor Nikonov
83c6ac4f6c Remove storage as parameter dependency, only columns are needed 2023-11-21 12:14:12 +00:00
alesapin
227bb30d02
Merge pull request #56986 from kevinmingtarja/implicit-constraint-collapsing-merged-tree
Add implicit constraint for CollapsingMergeTree sign column
2023-11-21 12:43:51 +01:00
kssenii
474f1ae0b6 Better 2023-11-21 11:42:23 +01:00
Aleksandr Musorin
ee0ae17ce2 Added optimize_trivial_approximate_count_query documentatiosn for RocksDB 2023-11-21 11:31:56 +01:00
Salvatore Mesoraca
f96cd4cfb8
Update src/Storages/NamedCollectionsHelpers.cpp
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
2023-11-21 11:14:48 +01:00
Salvatore Mesoraca
970e34f32e
NamedCollections: make exception message more informative.
Fix test 00002_log_and_exception_messages_formatting
2023-11-21 10:21:28 +01:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Amos Bird
ee1cec4d38
Allow RENAME of index columns. 2023-11-21 15:42:54 +08:00
Amos Bird
6581aaf9e8
Prevent incompatible ALTER of projection columns. 2023-11-21 15:42:54 +08:00
kevinyhzou
3adc8fdf78 Fix ci 2023-11-21 11:22:12 +08:00
Kseniia Sumarokova
2beee5d9cb
Merge branch 'master' into s3queue-fix-empty-key-check 2023-11-20 20:17:47 +01:00
jsc0218
3d5265c1d3 Merge remote-tracking branch 'origin/master' into SystemTablesFilterEngine
merge master
2023-11-20 18:10:54 +00:00