Commit Graph

3274 Commits

Author SHA1 Message Date
Sergei Trifonov
273770519b
Merge branch 'master' into master 2023-03-20 18:28:10 +01:00
Alexey Milovidov
4a44dafdee Merge branch 'master' into decimal256-various 2023-03-20 03:30:14 +01:00
Alexey Milovidov
ddf29b69f9
Merge branch 'master' into fuzzer-of-data-formats 2023-03-20 03:26:51 +03:00
Robert Schulze
0e01e912fe
Preparation for libcxx(abi), llvm, clang-tidy 16 2023-03-19 17:25:41 +00:00
Alexey Milovidov
26d17fe560 Merge branch 'master' of github.com:ClickHouse/ClickHouse into decimal256-various 2023-03-17 22:15:16 +01:00
Alexey Milovidov
c11915d35b Merge branch 'fix-bson-2' of github.com:Avogar/ClickHouse into fuzzer-of-data-formats 2023-03-17 22:05:46 +01:00
alexX512
023c0ba46e Fix comments 2023-03-17 20:05:10 +00:00
Alexey Perevyshin
2c6534f5c8
Merge branch 'master' into master 2023-03-16 23:57:36 +04:00
Alexey Milovidov
704cb54afe Merge branch 'master' of github.com:ClickHouse/ClickHouse into fuzzer-of-data-formats 2023-03-16 19:50:33 +01:00
serxa
3ad7194e31 add pool size metrics 2023-03-16 17:31:41 +00:00
robot-clickhouse-ci-1
1223b8959b
Merge pull request #47626 from ClickHouse/rs/follow-up-to-47614
Follow-up to #47614
2023-03-16 00:31:51 +01:00
Alexey Milovidov
92a5b81222 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fuzzer-of-data-formats 2023-03-15 23:59:16 +01:00
pufit
f4d9b4a888
Merge pull request #47086 from pkit/pkit/insert_http_body
use http request body in predefined http handlers
2023-03-15 17:41:05 -04:00
Alexey Perevyshin
1e9ef12f51
Merge branch 'master' into master 2023-03-15 22:21:02 +04:00
Robert Schulze
785ade7c39
Follow-up to #47614 2023-03-15 18:05:53 +00:00
Han Fei
a532503466
Merge branch 'master' into hanfei/optimize-regexp-tree-1 2023-03-15 17:56:01 +01:00
Han Fei
d78a9e03ad refine 2023-03-15 15:38:11 +01:00
Vladimir C
045960f1a1
Merge pull request #47108 from ClickHouse/vdimir/http_wait_end_of_query_settings 2023-03-15 15:31:12 +01:00
Alexey Milovidov
bb6b775884 Merge branch 'master' into fuzzer-of-data-formats 2023-03-15 12:42:00 +01:00
alexX512
d47e10484f Merge branch 'master' of github.com:ClickHouse/ClickHouse 2023-03-15 11:14:34 +00:00
Azat Khuzhin
bcf381c5ae Reimplement interserver mode to avoid replay attacks
Prevous implementation (DBMS_MIN_REVISION_WITH_INTERSERVER_SECRET)
accepts the salt from the client, which make it useless.

Reimplement the protocol to send the salt by the server and use it in
the client instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-03-15 08:28:43 +01:00
alexX512
14a7c5a019 Add stop_reading_on_first_cancel setting 2023-03-15 06:06:55 +00:00
Alexey Milovidov
e952ecae8c Add Decimal256 to a lot of places 2023-03-14 22:10:52 +01:00
Alexey Milovidov
e443c4e682
Merge pull request #47538 from Avogar/proper-parquet-fix
Proper fix for bug in parquet, revert reverted #45878
2023-03-14 22:29:39 +03:00
Alexey Milovidov
f331b9b398 Fix errors and add tests 2023-03-13 23:49:28 +01:00
Alexey Milovidov
14647525f8 Merge branch 'fix-bson-bug' of github.com:Avogar/ClickHouse into fuzzer-of-data-formats 2023-03-13 22:45:00 +01:00
Nikolai Kochetov
669a92bae0 Merge branch 'master' into aggregate-projections-analysis-query-plan 2023-03-13 19:55:49 +01:00
Alexey Milovidov
a411ae10b5
Merge pull request #47486 from ClickHouse/revert_25674
Remove "asynchronous connection drain"
2023-03-13 21:40:39 +03:00
avogar
4213ec609f Proper fix for bug in parquet, revert reverted #45878 2023-03-13 18:22:09 +00:00
Alexey Milovidov
f33b651686 Add fuzzer for data formats 2023-03-13 04:51:50 +01:00
avogar
5a18acde90 Revert #45878 and add a test 2023-03-11 21:15:14 +00:00
Alexander Tokmakov
7b1b238d0b Revert "Merge pull request #25674 from amosbird/distributedreturnconnection"
This reverts commit 5ffd99dfd4, reversing
changes made to 2796aa333f.
2023-03-11 19:09:47 +01:00
Alexander Tokmakov
bd124a24ea Revert "Merge pull request #31965 from azat/connection-drain-pull"
This reverts commit 6cb54b4092, reversing
changes made to 42d938ffa0.
2023-03-11 17:24:38 +01:00
Vladimir C
1b264ad51e
Merge branch 'master' into vdimir/http_wait_end_of_query_settings 2023-03-09 13:37:56 +01:00
Kruglov Pavel
fe973f3d6f
Merge branch 'master' into native-types-conversions 2023-03-09 13:03:25 +01:00
Kruglov Pavel
71b6d6c6ae
Merge pull request #47114 from Avogar/parquet-compression
Improve working with compression methods in Parquet/ORC/Arrow formats
2023-03-09 13:02:18 +01:00
Azat Khuzhin
d7b5e36218
Merge branch 'master' into mv-ignore-errors 2023-03-08 22:13:05 +01:00
Kruglov Pavel
69a1309ade
Merge branch 'master' into native-types-conversions 2023-03-07 20:06:17 +01:00
Antonio Andelic
35c15e6ef8 Merge branch 'master' into custom-key-parallel-replicas 2023-03-07 09:37:38 +00:00
avogar
d8e5fb5195 Use switch, use lz4 by default 2023-03-06 15:12:15 +00:00
Nikita Mikhaylov
099013831a
Added batching for reads and retries for the most heavy function in backups (#47243) 2023-03-05 16:15:03 +01:00
Nikita Mikhaylov
5c4da5aa4a
Use separate thread pool for IO operations for backups (#47174) 2023-03-03 20:05:42 +01:00
Antonio Andelic
737cf8e149 Better 2023-03-03 15:14:49 +00:00
Alexander Tokmakov
c518f3690e Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-03-02 18:24:08 +01:00
vdimir
b782cba809 Add settings http_wait_end_of_query and http_response_buffer_size 2023-03-02 16:03:10 +01:00
Kruglov Pavel
3b034e7c35
Merge branch 'master' into parquet-compression 2023-03-02 15:55:48 +01:00
Antonio Andelic
01cf9c94f4 Merge branch 'master' into custom-key-parallel-replicas 2023-03-02 14:28:42 +00:00
Constantine Peresypkin
fdc6de2a27 use http request body in predefined http handlers
adds a new "virtual" parameter `__request_body`
which can be used in http rules, see below

tremendously helps in producing arbitrary REST interfaces

for example:

```xml
<http_handlers>
  <rule>
    <methods>POST</methods>
    <url>/ingest</url>
    <handler>
      <type>predefined_query_handler</type>
      <query>INSERT INTO events(id, data) SELECT {id:UInt32}, {__request_body:String}</query>
    </handler>
  </rule>
</http_handlers>
```
2023-03-02 12:51:48 +01:00
avogar
5ab5902f38 Allow control compression in Parquet/ORC/Arrow output formats, support more compression for input formats 2023-03-01 21:27:46 +00:00
Alexander Gololobov
f64d08bd5c Enable lightweight delete support by default 2023-03-01 19:35:55 +01:00
Alexander Tokmakov
a97e15e36f Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-03-01 14:26:41 +01:00
pufit
29296cb004
Merge pull request #46550 from ClickHouse/46231-systerm-server-settings
Implement `system.server_settings`
2023-02-28 11:23:15 -05:00
avogar
ab899bf2f3 Allow types conversion in Native input format 2023-02-27 19:28:19 +00:00
Kruglov Pavel
36e65f5f84
Use versions vith dots 2023-02-27 19:00:40 +01:00
Kruglov Pavel
443dedddca
Merge branch 'master' into use-parquet-2 2023-02-27 14:31:43 +01:00
Kruglov Pavel
e9078b402b
Merge pull request #46742 from Avogar/update-settings
Enable input_format_json_ignore_unknown_keys_in_named_tuple by default
2023-02-27 14:30:28 +01:00
Nikolai Kochetov
3392b96cc3
Merge branch 'master' into aggregate-projections-analysis-query-plan 2023-02-27 14:07:04 +01:00
pufit
6a99c1b68a
Apply suggestions from code review
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
2023-02-24 16:17:49 -05:00
Kruglov Pavel
ee754b9582
Merge pull request #46731 from Avogar/fix-local-json-abort
Fix possible clickhouse-local abort on JSONEachRow schema inference
2023-02-24 14:12:20 +01:00
Kruglov Pavel
944c240615
Update history 2023-02-24 14:09:58 +01:00
pufit
2fbe75957d Add default column for system.settings and system.server_settings 2023-02-23 18:24:57 -05:00
Alexander Tokmakov
fe92fd8a61 Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-02-23 22:35:31 +01:00
avogar
b8156f459f Better solution 2023-02-23 18:13:35 +00:00
avogar
54622566df Add setting to change parquet version 2023-02-23 16:14:10 +00:00
Kruglov Pavel
35887911e3
Merge pull request #46284 from Avogar/null-as-default-all-formats
Extend setting `input_format_null_as_default` for more formats
2023-02-23 14:48:22 +01:00
Kseniia Sumarokova
30a8eb0c2f
Merge pull request #46670 from kssenii/hide-disk-setting
Hide arguments of custom disk merge tree setting
2023-02-23 11:42:53 +01:00
Alexey Milovidov
c0bc549e77
Merge pull request #46709 from azat/bg-pool-no-query
BackgroundSchedulePool should not have any query context
2023-02-23 10:24:46 +03:00
pufit
0fcda19d20 Fix description typo 2023-02-22 18:27:23 -05:00
Alexander Tokmakov
e660c0838c fix multipart requests 2023-02-22 17:54:35 +01:00
avogar
e0931dbdbe Enable input_format_json_ignore_unknown_keys_in_named_tuple by default 2023-02-22 16:40:53 +00:00
Azat Khuzhin
2ca47a6eb6 BackgroundSchedulePool should not have any query context
BackgroundSchedulePool is used for some peridic jobs, not from the query
context, i.e. flush of Buffer table.

And for such jobs there cannot be any query context, and more
importantly it will not work correctly since that query_context will
eventually expires.

And this is the reason of this failures [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/46668/015991bc5e20c787851050c2eaa13f0fef3aac00/stateless_tests_flaky_check__asan_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-22 10:50:51 +01:00
Alexander Tokmakov
592af6d652 fix incomplete interst through http 2023-02-22 02:34:03 +01:00
kssenii
1f0ab8d427 Hide disk setting arguemtns 2023-02-21 19:24:01 +01:00
Igor Nikonov
bf05082984
Merge branch 'master' into keeper-retries-by-default 2023-02-21 17:59:55 +01:00
Alexey Milovidov
848286eca7
Merge pull request #40945 from arthurpassos/add_select_final_mt_setting
Add user setting to force select final at query level
2023-02-21 19:58:45 +03:00
Kruglov Pavel
ef0d6becba
Merge branch 'master' into null-as-default-all-formats 2023-02-21 16:52:39 +01:00
Azat Khuzhin
a110e0f022 Allow to ignore errors while pushing to MATERILIZED VIEW
This can be useful in the following scenarious:

- you want to duplicate the data to another table and you don't care
  about the errors
- you want to duplicate system.*_log to another server, you are adding
  materialized view that will push to Distributed table, but you don't
  want to miss original blocks in the local system.*_log
- you want to push some data to a 3d party service, using i.e. URL
  engine.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-21 16:28:38 +01:00
Alexander Gololobov
5354b0d069
Merge pull request #46140 from ClickHouse/split_prewhere_dag
Split PREWHERE DAG into multiple read steps
2023-02-21 14:37:09 +01:00
Alexey Milovidov
67d808f49a Add settings changes history 2023-02-21 00:30:38 +01:00
Alexey Milovidov
4d0ba39e7f Merge branch 'master' into keeper-retries-by-default 2023-02-21 00:27:37 +01:00
Kruglov Pavel
9866ecfe8b
Merge branch 'master' into null-as-default-all-formats 2023-02-20 20:49:30 +01:00
Nikolai Kochetov
ee597b939f Merge branch 'master' into aggregate-projections-analysis-query-plan 2023-02-20 19:46:52 +00:00
Arthur Passos
4f2fcf763b
Merge branch 'master' into add_select_final_mt_setting 2023-02-20 16:22:14 -03:00
Nikolai Kochetov
2a8460d962 Merge branch 'master' into aggregate-projections-analysis-query-plan 2023-02-20 13:54:05 +01:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
pufit
ec49fbec23
Update src/Core/ServerSettings.h
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
2023-02-19 16:49:01 -05:00
Arthur Passos
3d9b23e5f3
Merge branch 'master' into add_select_final_mt_setting 2023-02-19 15:01:41 -03:00
Alexey Milovidov
17992b178a
Merge pull request #45364 from nickitat/aggr_partitions_independently
Add option to aggregate partitions independently
2023-02-19 17:44:18 +03:00
Maksim Kita
6dffe52e68
Merge pull request #46471 from kitaisreal/analyzer-planner-fixes-before-enable-by-default
Analyzer planner fixes before enable by default
2023-02-19 13:49:18 +03:00
Nikita Taranov
f70044f34b Merge branch 'master' into aggr_partitions_independently 2023-02-18 13:19:05 +00:00
pufit
4496db66b2 Implement system.server_settings 2023-02-17 23:46:09 -05:00
Igor Nikonov
486501296c Merge remote-tracking branch 'origin/master' into igor/remove_redundant_distinct 2023-02-17 14:40:02 +00:00
Antonio Andelic
ab51c1d975
Merge branch 'master' into custom-key-parallel-replicas 2023-02-17 15:06:41 +01:00
Maksim Kita
11965e59e2 Analyzer Planner disable by default 2023-02-16 19:46:24 +01:00
Alexander Gololobov
a6061e022e Merge branch 'master' into split_prewhere_dag 2023-02-16 14:43:58 +01:00
youennL-cs
6526c2a8ab
[RFC] Replacing merge tree new engine (#41005)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add keyword 'CLEANUP' when OPTIMIZE

* Cleanup uniquely when it's a replacingMergeTree

* Propagate CLEANUP information and change from 'with_cleanup' to 'cleanup'

* Cleanup data flagged as 'is_deleted'

* Fix merge when optimize and add a test

* Fix OPTIMIZE and INSERT + add tests

* New fix for cleanup at the merge

* Cleanup debug logs

* Add the SETTINGS option 'clean_deleted_rows' that can be 'never' or 'always'

* Fix regression bug; Now REplicatedMergeTree can be called as before without 'is_deleted'

* Add Replicated tests

* Disable tag 'long' for our test and cleanup some white spaces

* Update tests

* Fix tests and remove additional useless whitespace

* Fix replica test

* Style clean && add condition check for is_deleted values

* clean_deleted_rows settings is nom an enum

* Add valid default value to the clean_deleted_rows settings

* Update cleanup checkers to use the enum and fix typos in the test

* Fix submodule contrib/AMQP-CPP pointer

* Add missing messages in test reference and remove a print with non derterministic order

* fix replica test reference

* Fix edge case

* Fix a typo for the spell checker

* Fix reference

* Fix a condition to raise an error if is_deleted differ from 0/1 and cleanup

* Change tests file name and update number

* This should fix the ReplacingMergeTree parameter set

* Fix replicated parameters

* Disable allow_deprecated_syntax_for_merge_tree for our new column

* Fix a test

* Remove non deterministic order print in the test

* Test on replicas

* Remove a condition, when checking optional parameters, that should not be sueful since we disabled the deprected_syntaxe

* Revert "Remove a condition, when checking optional parameters, that should not be useful since we disabled the deprected_syntaxe"

This reverts commit b65d64c05e.

* Fix replica management and limit the number of argument to two maximum, due to the possiblity of deprecated table create/attach failing otherwise

* Test a fix for replicated log information error

* Try to add sync to have consistent results

* Change path of replicas that should cause one issue and add few prints in case it's not that

* Get cleanup info on replicas only if information found

* Fix style issues

* Try to avoid replication error 'cannot select parts...' and and replica read/write field order

* Cleanup according to PR reviews
 and add tests on error raised.

* Update src/Storages/MergeTree/registerStorageMergeTree.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* Select ... FINAL don't show rows with is_deleted = true

* Update and fix SELECT ... FINAL merge parameter

* Remove is_deleted rows only on the version inserted when merge

* Fix (master) updates issues

* Revert changes that should not be commited

* Add changes according to review

* Revert changes that should not be commited - part 2

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-02-16 16:03:16 +03:00
Maksim Kita
70eb41c6a8 Analyzer add single_join_prefer_left_table setting 2023-02-16 12:17:03 +01:00
Maksim Kita
96fb630231 Analyzer Planner enable by default 2023-02-16 12:17:01 +01:00
Alexey Milovidov
77f60311da
Merge branch 'master' into add_select_final_mt_setting 2023-02-15 15:35:37 +03:00
Igor Nikonov
9c25706711 Update settings history 2023-02-14 14:29:25 +00:00