thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-01 20:12:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
93,046 Commits 9,193 Branches 1,782 Tags 2.2 GiB
6e6f77ef8a
Commit Graph

1129 Commits

Author SHA1 Message Date
tavplubix
7742b96497
Remove metadata leftovers on drop database (#30054) 
* remove metadata leftovers on drop database

* Update InterpreterDropQuery.cpp

* Update DatabaseCatalog.cpp
 2021-10-13 13:34:18 +03:00
kssenii
1c8b1b1133 Better 2021-10-12 09:33:07 +00:00
kssenii
e7dbfff5d2 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-12 06:19:15 +00:00
Azat Khuzhin
063f9cffab Allow memory profiler under sanitizers 
Only query profiler cannot work reliably with sanitizers (due to
unwinding from signal handler), but memory profiler should be fine.

Plus sometimes the problem appears only on build with sanitizers, so it
will be useful to have memory related profiling in trace_log.

Also there is a flaky check for stateless tests, that uses build with
ASan, and now trace_log there is empty, which sometimes does not allow
to debug further.
 2021-10-11 10:21:26 +03:00
kssenii
fde6f0507d Merge remote-tracking branch 'origin/retriable-http' into disk-async-read 2021-10-10 20:23:44 +03:00
kssenii
f3f820c6e0 HTTP read buffer retries 2021-10-08 15:41:55 +03:00
kssenii
d6e1d0503d Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-07 16:21:00 +03:00
Nikita Mikhaylov
b31d11478b
Merge pull request #29680 from azat/defines 
Cleanup common defines
 2021-10-04 20:13:29 +03:00
Maksim Kita
8eda58acfb
Merge pull request #29516 from lingtaolf/feature/support-HSTS 
Support HSTS in Clickhouse HTTP server
 2021-10-04 12:49:51 +03:00
Azat Khuzhin
0b83345560 Remove macros for distributed connection timeouts 2021-10-03 14:34:03 +03:00
Azat Khuzhin
f6e3583ffa Remove macros for hedged request timeouts 2021-10-03 14:34:03 +03:00
Azat Khuzhin
0a6885d806 Remove DEFAULT_LIVE_VIEW_HEARTBEAT_INTERVAL_SEC 2021-10-03 14:34:03 +03:00
Azat Khuzhin
dc2cb6d996 Remove DBMS_DEFAULT_DISTRIBUTED_CONNECTIONS_POOL_SIZE 2021-10-03 14:34:03 +03:00
Nikita Mikhaylov
9756b8dc33
Added an ability to execute more merges and mutations than threads, added new settings (#29140) 2021-10-01 00:26:24 +03:00
凌涛
a13d6b0aea Check all requests 2021-09-30 13:40:48 +08:00
Azat Khuzhin
a218c3a28c Enable per-query memory profiler (set to memory_profiler_step=4MiB) globally 
Let's see at performance tests. And yes, they should show the
difference, if any, since they uses clickhouse python driver, which does
not knows about default settings, so new server will have this setting
set, while old server - upstream/master, will not.
 2021-09-27 22:46:35 +03:00
Maksim Kita
d0f53cfc31 Added setting use_skip_indexes 2021-09-27 12:17:00 +03:00
Kruglov Pavel
55553026ee
Merge pull request #29206 from azat/strict-external-queries 
Add ability to disable converting expressions to local filter for external queries
 2021-09-23 13:22:12 +03:00
kssenii
1b9565dfa8 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-09-22 22:08:44 +03:00
Kseniia Sumarokova
17b1dd4bc6
Merge branch 'master' into better-retries-2 2021-09-21 13:55:53 +03:00
Anton Popov
ea4fd19e28
Merge pull request #29087 from CurtizJ/asyn-inserts-follow-up 
Minor enhancements in async inserts
 2021-09-21 13:38:52 +03:00
kssenii
975033e6e3 Fix 2021-09-21 09:44:54 +00:00
Vladimir C
bc7c41efff
Merge pull request #26657 from hexiaoting/fuse_quantile 2021-09-21 12:33:07 +03:00
Vladimir C
9abd8aea6d
Merge pull request #29123 from depressed-pho/csv-custom-null 2021-09-21 11:35:21 +03:00
Azat Khuzhin
9c20519f33 Add ability to disable converting expressions to local filter for external queries 
Sometimes it is undesirable to remove any expressions from WHERE, since
this may lead to reading the whole table, and this is pretty heavy job
for MySQL and PostgreSQL (even if you read only one column).

So external_table_strict_query had been introduced to prohibit this and
fail the query instead.
 2021-09-21 02:41:07 +03:00
Anton Popov
83e45adad6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-20 20:04:50 +03:00
kssenii
f312760dca Better 2021-09-20 19:57:15 +03:00
kssenii
12b90da5de Better retries 2021-09-19 20:50:50 +00:00
PHO
3c4b1ea9c5 New setting: output_format_csv_null_representation 
This is the same as output_format_tsv_null_representation but is for CSV output.
 2021-09-17 17:58:23 +09:00
Anton Popov
99175f7acc minor enhancements in async inserts 2021-09-16 20:55:34 +03:00
Anton Popov
f6191b98e7 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-15 17:47:22 +03:00
Anton Popov
9646edf3dc fix possible race in AsynchronousInsertQueue 2021-09-15 02:54:10 +03:00
Anton Popov
fc17936c12 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-14 23:02:30 +03:00
mergify[bot]
fa903ac255
Merge branch 'master' into describe-with-subcolumns 2021-09-14 14:43:29 +00:00
vdimir
a3304a87a4
Rename optimize_syntax_fuse_aggregate -> optimize_syntax_fuse_functions 2021-09-14 15:27:12 +03:00
vdimir
a1463d5719
Merge remote-tracking branch 'origin/master' into fuse_quantile 2021-09-14 15:23:06 +03:00
vdimir
5a1aeeb044
Use optimize_syntax_fuse_aggregate instead of optimize_fuse_sum_count_avg and optimize_fuse_quantile 2021-09-14 15:21:16 +03:00
Anton Popov
02a7edf902 allow to include subccolumns in describe query 2021-09-12 00:07:33 +03:00
Pavel Kruglov
9c756fc4b2 Fix bug with LowCardinality in short-curcuit function evaluation 2021-09-11 12:56:25 +03:00
Anton Popov
8e5b3b2f6c Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-09 15:12:34 +03:00
Anton Popov
4c388e3d84 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-09-09 14:10:16 +03:00
tavplubix
341a6c51d6
Merging #24866 (#28691) 
* Add StorageSystemISTables.cpp/.h

* Another attempt

* Columns and Views

* Add information schema db and fix information schema 'tables' table

* fix build

* remove copy-paste, add views to system tables

* add test

* fix

* fix_tests

Co-authored-by: Damir Petrov <petrovdamir2235@gmail.com>
Co-authored-by: Damir Petrov <0442a403@verstehen.sas.yp-c.yandex.net>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-09-09 12:37:51 +03:00
Anton Popov
1f41f4541f change timeouts in async insert to milliseconds 2021-09-08 18:30:04 +03:00
Anton Ivashkin
5ff99314b1 Reduce default settings for S3 multipart upload part size 2021-09-07 12:51:00 +03:00
Anton Popov
5e421ab272 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-02 02:42:50 +03:00
alexey-milovidov
77d085f264
Merge pull request #27527 from evillique/log_queries_probability 
Add log_queries_probability setting
 2021-09-01 00:51:23 +03:00
Alexey Milovidov
ca6f6306f3 Merge branch 'master' into async-reads 2021-08-31 02:52:58 +03:00
Kseniia Sumarokova
00bea71610
Merge pull request #25251 from kssenii/disk-over-web-server 
ClickHouse over web server with static files
 2021-08-30 16:18:59 +03:00
Nikolay Degterinsky
2433d88b82 Merge branch 'master' into log_queries_probability 2021-08-30 12:55:33 +03:00
Alexey Milovidov
e4f3d97b52 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-29 21:42:14 +03:00
Alexey Milovidov
79e0433ba7 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-28 01:19:16 +03:00
Anton Popov
9e67943878 add waiting for async inserts 2021-08-28 00:29:10 +03:00
tavplubix
c36fe210b8
Revert "Add test for #13398" 2021-08-27 21:07:10 +03:00
tavplubix
703101fe4d
Merge pull request #27931 from ClickHouse/wait_for_all_replicas_timeouts 
Avoid too long waiting for inactive replicas
 2021-08-27 14:31:36 +03:00
Alexey Milovidov
46732e781f Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-27 01:57:48 +03:00
Kruglov Pavel
5bc332c40c
Merge pull request #28054 from kssenii/mysql 
Add test for #13398
 2021-08-26 16:39:43 +03:00
Anton Popov
6d3274c22c Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-26 03:31:46 +03:00
Alexey Milovidov
3cb9cf7242 Merge branch 'master' of github.com:yandex/ClickHouse into async-reads 2021-08-26 01:55:46 +03:00
Alexey Milovidov
d8250ab3d6 Preparation 2021-08-26 01:14:42 +03:00
Nikolay Degterinsky
2c81a35b4e Add setting 2021-08-25 17:41:21 +03:00
Alexey Milovidov
8f57216180 Progress on development 2021-08-25 00:45:58 +03:00
Alexey Milovidov
4d9ad3725d Merge branch 'master' into async-reads 2021-08-25 00:45:40 +03:00
Azat Khuzhin
37043b74bf Enable optimize_distributed_group_by_sharding_key by default 
Since it requires optimize_skip_unused_shards this should be safe.
 2021-08-24 23:49:50 +03:00
Vitaly Baranov
64f69462c6
Merge pull request #27921 from vitlibar/add-tests-for-field-init 
Allow implicit cast bool to Field
 2021-08-24 17:15:44 +03:00
kssenii
0d5000f917 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into mysql 2021-08-24 07:30:42 +00:00
kssenii
b536653851 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-24 07:29:50 +00:00
kssenii
962f415740 Add test and settings 2021-08-23 22:18:04 +00:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Alexander Tokmakov
cc9c2fd63b make code better 2021-08-23 15:57:50 +03:00
Vitaly Baranov
89166a655a Init SettingFieldBool with true or false, not with 1 or 0. 2021-08-23 09:47:36 +03:00
alexey-milovidov
4922fb791e
Merge pull request #27882 from azat/enable-tcp_keep_alive_timeout 
RFC: Enable tcp_keep_alive_timeout by default
 2021-08-21 22:39:12 +03:00
Kseniia Sumarokova
19afb22b4d
Merge pull request #27932 from amosbird/constkeys 
Add a setting to bring back the old behavior of grouping by constant keys on empty set.
 2021-08-21 08:02:30 +03:00
Kseniia Sumarokova
38a35c1d9e
Merge pull request #27530 from kssenii/col-identifier-as-col-number 
Allow positional arguments for group by, order by, limit by
 2021-08-20 21:08:58 +03:00
kssenii
945e2c4ce5 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-20 18:01:23 +00:00
Amos Bird
5708ed42ec
Add setting. 
empty_result_for_aggregation_by_constant_keys_on_empty_set
 2021-08-20 22:34:32 +08:00
Anton Popov
61239343e3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-20 16:33:30 +03:00
Alexander Tokmakov
59eb3aa9a9 avoid too long waiting for inactive replicas 2021-08-20 15:59:57 +03:00
alexey-milovidov
6be085ed17
Update Settings.h 2021-08-20 09:05:48 +03:00
Azat Khuzhin
9aecb27913 Decrease tcp_keep_alive_timeout default (to avoid overlaps with receive_timeout) 2021-08-20 07:16:17 +03:00
Azat Khuzhin
d610974c4c Set tcp_keep_alive_timeout to 300 by default 
This may fix possible issues with remote queries, when remote server
abnormaly terminates and the initiator does not have proper timeout.

This should not harm anything, since the timeout is pretty huge (300
seconds).
 2021-08-19 09:06:41 +03:00
alexey-milovidov
eb3bbdfab5
Merge pull request #27680 from ClickHouse/decimal-no-trailing-zeros 
Remove trailing zeros from Decimal serialization #15794
 2021-08-18 05:24:13 +03:00
Pavel Kruglov
7c98fc11b9 Fix performance, make only one setting 2021-08-16 14:30:56 +03:00
Alexey Milovidov
8adaef7c8e Make text format for Decimal tuneable 2021-08-16 11:03:23 +03:00
Alexey Milovidov
d184b79bba Progress on async reads. 2021-08-16 03:00:32 +03:00
kssenii
daac6323fa Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-14 13:15:04 +00:00
mergify[bot]
319e5af6eb
Merge branch 'master' into short-circuit 2021-08-13 13:01:46 +00:00
Pavel Kruglov
36bea6abb4 Add setting optimize_short_circuit_function_evaluation 2021-08-13 14:48:32 +03:00
kssenii
37554014aa Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into col-identifier-as-col-number 2021-08-13 05:28:22 +00:00
Nikita Mikhaylov
8c06abee73
Merge pull request #25902 from Avogar/arrow-nested 
Refactor ArrowColumnToCHColumn, support inserting Nested as Array(Struct) in Arrow/ORC/Parquet
 2021-08-12 20:02:01 +03:00
Kseniia Sumarokova
c50294fda8
Update Settings.h 2021-08-10 17:40:06 +03:00
kssenii
503a5edc59 Fix 2021-08-10 14:01:59 +00:00
kssenii
554231c6fc Support positional arguments for GROUP BY, ORDER BY, LIMIT BY 2021-08-10 13:38:42 +00:00
Raúl Marín
d1d47658f5 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-08-10 10:49:05 +02:00
Pavel Kruglov
a95d45a157 Some clean up 2021-08-09 17:57:02 +03:00
Pavel Kruglov
50d4192126 Implement short circuit function evaluation, first attempt 2021-08-09 17:43:10 +03:00
kssenii
d767f6704f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-09 12:14:23 +00:00
Nikita Mikhaylov
73d3f2c60f
Merge pull request #26313 from fastio/control_execution_period_of_clearOldTemporaryDirectories 
Control the execution period of clear old temporary directories by parameter
 2021-08-09 14:29:24 +03:00
alexey-milovidov
5f84564207
Merge pull request #27378 from amosbird/hyperscansanitizer 
Add settings to check hyperscan regexp length.
 2021-08-09 02:48:14 +03:00
alexey-milovidov
677fddaa37
Merge pull request #27104 from azat/enable-distributed_push_down_limit 
Enable distributed_push_down_limit by default
 2021-08-09 02:45:52 +03:00
Alexey Milovidov
e45a09e2f5 Fix wrong PR #27184 2021-08-08 21:09:39 +03:00
mergify[bot]
bd313279e8
Merge branch 'master' into hyperscansanitizer 2021-08-08 17:03:05 +00:00
Azat Khuzhin
0677a6c6cb Enable distributed_push_down_limit by default 2021-08-08 19:30:50 +03:00
alexey-milovidov
5523e5e153
Update Settings.h 2021-08-07 18:48:52 +03:00
Amos Bird
5ef59d4828
Add setting to log formatted query into system.query_log 2021-08-07 14:26:38 +08:00
Amos Bird
aa33a7add1
Add settings to check hyperscan regexp length. 2021-08-07 13:07:41 +08:00
kssenii
073d7fdd5e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-06 19:42:29 +00:00
mergify[bot]
c696817a79
Merge branch 'master' into control_execution_period_of_clearOldTemporaryDirectories 2021-08-06 16:16:22 +00:00
Anton Popov
3a0d4807a5 Merge branch 'async-insert' of git://github.com/abyss7/ClickHouse into merging-20557 2021-08-06 16:20:02 +03:00
Raúl Marín
479f053f2e Merge remote-tracking branch 'blessed/master' into materialization_log 2021-08-05 17:24:10 +02:00
Alexander Kuzmenkov
4bc652699f
Merge pull request #27184 from ClickHouse/aku/window-functions 
mark window functions as ready for general use
 2021-08-05 14:18:28 +03:00
Pavel Kruglov
e4c5d7e3b1 Support inserting nested as Array of structs, add some refactoring 2021-08-05 14:10:27 +03:00
Alexander Kuzmenkov
0730e685b0 Mark window functions ready for general use. 2021-08-04 17:45:32 +03:00
Anton Popov
e36736b50c Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-08-02 22:52:02 +03:00
Raúl Marín
9f32ecca89 Merge remote-tracking branch 'blessed/master' into materialization_log 2021-08-02 16:21:10 +02:00
alesapin
181f93c60a
Merge pull request #24997 from evillique/nlp 
NLP functions
 2021-08-02 10:50:40 +03:00
alexey-milovidov
d2b7e23899
Merge pull request #26961 from Enmk/fix/extractAllGroupsHorizontal_max_matches_per_row 
Updated extractAllGroupsHorizontal - flexible limit on number of matches
 2021-07-31 04:54:24 +03:00
alesapin
02176fb4c7 Add experimental flag 2021-07-30 18:25:51 +03:00
alexey-milovidov
582fa31b18
Update Settings.h 2021-07-30 12:30:36 +03:00
Vasily Nemkov
a1fb16df52 setting regexp_max_matches_per_row instead of 3rd argument 2021-07-30 12:28:21 +03:00
Raúl Marín
6b9ec2a62e WIP 2021-07-27 16:28:28 +02:00
Raúl Marín
efe1938437 WIP: Slow improvements 2021-07-27 16:26:29 +02:00
Raúl Marín
ea5c02a605 WIP 2021-07-27 16:26:27 +02:00
Alexander Tokmakov
63ab38ee09 rename MaterializeMySQL to MaterializedMySQL 2021-07-26 21:17:28 +03:00
hexiaoting
6b73fa5307 Fix style check errors 2021-07-23 10:57:34 +08:00
Maksim Kita
46fd046f11 Setting min_count_to_compile_aggregate_expression fix 2021-07-23 00:22:04 +03:00
hexiaoting
de30962ebd Fuse multi quantile funcs with same arguments to one function. 2021-07-21 14:43:40 +08:00
Anton Popov
f99374cca6 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-07-20 18:20:21 +03:00
Amos Bird
dbfb699690
Asynchronously drain connections. 2021-07-19 21:53:29 +08:00
alexey-milovidov
f481a5a28e
Merge pull request #26380 from ClickHouse/use-hedged-requests-again 
Use hedged requests, again
 2021-07-17 01:16:32 +03:00
alexey-milovidov
4103c22310
Merge pull request #26245 from arenadata/ADQM-322 
Avro string for ClickHouse String
 2021-07-16 12:23:50 +03:00
Alexey Milovidov
2d42b1ebf6 Use hedged requests, again 2021-07-15 19:43:42 +03:00
Peng Jian
4bc07771ee Fix the clearOldTemporaryDirectories is executed too frequently 2021-07-14 19:26:46 +08:00
Alexey Milovidov
9dd0a76220 Add a setting "function_range_max_elements_in_block" and increase it 5 times 2021-07-14 11:31:08 +03:00
Ilya Golshtein
16532658c2 Avro string for ClickHouse string 2021-07-13 20:03:00 +03:00
Anton Popov
5d175bf557 dynamic columns: support distributed tables 2021-07-12 17:54:02 +03:00
mergify[bot]
b7ae4e4ced
Merge branch 'master' into async-insert 2021-07-12 09:21:37 +00:00
Maksim Kita
dc81ae7c6e Compile AggregateFunctionBitwise 2021-07-10 01:51:34 +03:00
Anton Popov
e1afdc1f1e dynamic subcolumns: add mode for DESCRIBE query to show deduced object types 2021-07-09 17:54:53 +03:00
Maksim Kita
577e69898a Compile aggregate functions perf tests fix 2021-07-05 12:17:01 +03:00
alexey-milovidov
fef7c557f6
Merge pull request #24789 from kitaisreal/compile-aggregation-functions 
Compile aggregate functions
 2021-07-05 02:57:49 +03:00
Kseniia Sumarokova
8351f1db99
Merge pull request #20470 from kssenii/pg-ch-replica 
MaterializePostgreSQL table engine and database engine
 2021-07-04 19:34:54 +03:00
Kseniia Sumarokova
9b4d35ef95
Merge pull request #25940 from kssenii/move-to-prewhere-if-final 
Add setting optimize_move_to_prewhere_if_final
 2021-07-04 11:33:24 +03:00
alexey-milovidov
56d695cd12
Update Settings.h 2021-07-03 22:10:22 +03:00
Alexey Milovidov
127455f15f Remove experimental ANTLR parser 2021-07-03 18:48:03 +03:00
kssenii
a9a91a4f15 Setting move to prewhere if final 2021-07-03 11:37:50 +00:00
Maksim Kita
da8c957167 Aggregator added CompiledExpressionCache 2021-07-01 22:56:36 +03:00
Maksim Kita
d24d3ae992 Added second variant of compilation 2021-07-01 22:56:36 +03:00
Maksim Kita
3fe559b31f Compile aggregate functions 2021-07-01 22:56:35 +03:00
kssenii
9d5847a1a8 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-29 21:30:31 +00:00
Ivan Lezhankin
1bb41593d1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-28 22:55:56 +03:00
Maksim Kita
352e1f27ff Update using Map datatyle in system log tables before merge 2021-06-28 14:42:21 +03:00
Maksim Kita
294b937a39
Merge branch 'system-query-log-map-type-merge' into system-querylog-map-continue 2021-06-28 11:27:59 +03:00
kssenii
aa7133c239 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-27 20:08:00 +00:00
kssenii
c3a2fc0584 Review fixes 2021-06-27 19:17:17 +00:00
alexey-milovidov
0a26687115
Merge pull request #23864 from azat/dist-split-batch-and-retry 
Add ability to split distributed batch on failures (i.e. due to memory limits)
 2021-06-27 19:28:26 +03:00
kssenii
3d90f0710c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-26 22:04:47 +00:00
Azat Khuzhin
a0209178cc Add ability to split distributed batch on failures 
Add distributed_directory_monitor_split_batch_on_failure setting (OFF by
default), that will split the batch and send files one by one in case of
retriable errors.

v2: more error codes
 2021-06-23 02:48:47 +03:00
Anton Popov
c2754c34f4 Merge remote-tracking branch 'upstream/master' into HEAD 2021-06-21 12:42:48 +03:00
Kruglov Pavel
a54cbef5b7
Merge pull request #24341 from Avogar/arrow 
Support complex types in Arrow/Parquet/ORC
 2021-06-21 11:05:57 +03:00
Anton Popov
e409ead64a fix tests 2021-06-20 23:18:05 +03:00
Anton Popov
d8b6f15ef4
Merge pull request #23027 from azat/distributed-push-down-limit 
Add ability to push down LIMIT for distributed queries
 2021-06-20 23:08:50 +03:00
kssenii
f33387837e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-06-18 14:27:48 +00:00
mergify[bot]
c723dd7d40
Merge branch 'master' into arrow 2021-06-18 12:18:57 +00:00
Ivan Lezhankin
37365589ed Merge branch 'master' into async-insert 2021-06-17 16:57:45 +03:00
Amos Bird
31815f4a68
Proper documentation of the setting 2021-06-17 14:27:33 +08:00
Amos Bird
c8ea6527cb
Add prefer_global_in_and_join setting 2021-06-17 14:27:29 +08:00
Anton Popov
8bd7cbd925 add compatibility setting for name of columns 2021-06-16 21:10:07 +03:00
Ivan Lezhankin
ba08a580f8 Add test 2021-06-16 17:33:14 +03:00
Ivan Lezhankin
b182d87d9c Add settings for HTTP header limitations 2021-06-15 17:33:46 +03:00
Pavel Kruglov
a4decd0848 Support Map type, fix and add tests 2021-06-15 16:15:27 +03:00
Pavel Kruglov
c8b37977da Fix bugs, support dictionary for Arrow format 2021-06-15 16:15:27 +03:00
kssenii
2cfcc9980c Better 2021-06-13 19:38:31 +00:00
Ivan Lezhankin
2c7cb41a69 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-11 20:04:26 +03:00
Anton Popov
bb6f0dfbb8
Merge pull request #24406 from CurtizJ/rewrite-functions-to-subcolumns 
Optimize some functions to subcolumns
 2021-06-09 11:07:09 +03:00
Azat Khuzhin
18e8f0eb5e Add ability to push down LIMIT for distributed queries 
This way the remote nodes will not need to send all the rows, so this
will decrease network io and also this will make queries w/
optimize_aggregation_in_order=1/LIMIT X and w/o ORDER BY faster since it
initiator will not need to read all the rows, only first X (but note
that for this you need to your data to be sharded correctly or you may
get inaccurate results).

Note, that having lots of processing stages will increase the complexity
of interpreter (it is already not that clean and simple right now).

Although using separate QueryProcessingStage looks pretty natural.

Another option is to make WithMergeableStateAfterAggregation always, but
in this case you will not be able to disable only this optimization,
i.e. if there will be some issue with it.

v2: fix OFFSET
v3: convert 01814_distributed_push_down_limit test to .sh and add retries
v4: add test with OFFSET
v5: add new query stage into the bash completion
v6/tests: use LIMIT O,L syntax over LIMIT L OFFSET O since it is broken in ANTLR parser
          https://clickhouse-test-reports.s3.yandex.net/23027/a18a06399b7aeacba7c50b5d1e981ada5df19745/functional_stateless_tests_(antlr_debug).html#fail1
v7/tests: set use_hedged_requests to 0, to avoid excessive log entries on retries
          https://clickhouse-test-reports.s3.yandex.net/23027/a18a06399b7aeacba7c50b5d1e981ada5df19745/functional_stateless_tests_flaky_check_(address).html#fail1
 2021-06-09 02:29:50 +03:00
Ivan Lezhankin
973cea73b7 Merge branch 'master' into async-insert 2021-06-08 15:10:30 +03:00
alesapin
5736eaf8fd Increase background schedule pool default size 2021-06-08 13:55:14 +03:00
Alexey Milovidov
8d85145725 Remove experimental compression codecs 2021-06-06 02:26:52 +03:00
Alexey Milovidov
17962459f5 Merge branch 'master' into issue-16775 2021-06-06 02:18:28 +03:00
kssenii
e23bc14582 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-03 19:44:55 +00:00
Vladimir
17f0900c31
Merge pull request #19766 from achimbab/pr_sequenceNextNode 2021-06-03 13:16:42 +03:00
alesapin
4a1bf51ad1
Merge branch 'master' into limited_fetches 2021-06-02 23:01:47 +03:00
Ivan Lezhankin
791cbcf0b1 Merge remote-tracking branch 'upstream/master' into async-insert 2021-06-02 18:50:37 +03:00
Ivan Lezhankin
84c23dc060 Merge branch 'master' into async-insert 2021-06-02 18:06:21 +03:00
vdimir
ec09dfe635
Set allow_experimental_funnel_functions = false by default (not used before) 2021-06-02 09:16:31 +03:00
vdimir
1b936c8af6
Remove example usage of allow_experimental_funnel_functions from windowFunnel 2021-06-02 09:16:21 +03:00
kssenii
12ea45e956 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-06-01 17:43:34 +00:00
alesapin
c3321bef30 Merge branch 'master' into limited_fetches 2021-06-01 13:09:02 +03:00
Anton Popov
b24f65e1ad Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-31 23:04:52 +03:00
Alexey Milovidov
3791c1c99d Someone added useful settings to the section with obsolete settings 2021-05-31 22:26:34 +03:00
Alexey Milovidov
4187aa444b Fix typo. Remove "allow_experimental_bigint_types" setting. 2021-05-31 22:21:28 +03:00
Vxider
ac0468d389 Merged with master 2021-05-28 09:14:55 +00:00
Vxider
ad05f4f303 Merged with master 2021-05-28 07:36:19 +00:00
alesapin
42b2383e01 Basic throttler implementation 2021-05-26 23:37:44 +03:00
Vladimir
fc15cad7e0
Pass Settings to aggregate function creator 2021-05-26 14:58:31 +03:00
Vladimir
7224e25156
Revert "Pass Settings to aggregate function creator" 2021-05-26 14:46:42 +03:00
Vladimir
6650c67b01
Merge pull request #22762 from vdimir/create-aggregate-func-settings 2021-05-26 14:32:14 +03:00
Anton Popov
a36c1793d0 Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-24 15:57:43 +03:00
Alexey Milovidov
6f70feed2f Merge branch 'master' into issue-16775 2021-05-24 05:56:57 +03:00
kssenii
a78c9e63e7 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-23 12:07:26 +00:00
Alexey Milovidov
e000b994e5 Add missing whitespace 2021-05-23 03:26:31 +03:00
alexey-milovidov
1ed30f7246
Merge branch 'master' into issue-16775 2021-05-22 17:14:23 +03:00
Anton Popov
13cfedd188 optimize some functions to subcolumns 2021-05-21 21:48:19 +03:00
Nikita Vasilev
9f2bcacb46 Merge remote-tracking branch 'upstream/master' into nikvas0/simple_optimizations 2021-05-20 19:34:11 +03:00
Maksim Kita
25fe0e88f8 Compile expressions added partition by test 2021-05-20 11:16:15 +03:00
Maksim Kita
b0476c1fa2
Merge pull request #24273 from kitaisreal/expression-actions-compile-only-necessary-places 
ExpressionActions compile only necessary places
 2021-05-20 10:53:30 +03:00
Maksim Kita
c50909468f Updated ExpressionActionSettings 2021-05-19 17:32:07 +03:00
alesapin
301dc848f5 Merge branch 'master' into nvartolomei-parts-move 2021-05-19 17:17:01 +03:00
Alexander Kuzmenkov
e9b69bbd70
Merge pull request #23906 from azat/fix-distributed_group_by_no_merge 
distributed_group_by_no_merge fixes
 2021-05-19 16:16:08 +03:00
alesapin
17f229857c Merge branch 'master' into nvartolomei-parts-move 2021-05-17 13:52:48 +03:00
kssenii
8a4711b716 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-16 20:55:18 +00:00
Nikita Vasilev
6102ea643a Merge remote-tracking branch 'upstream/master' into nikvas0/simple_optimizations 2021-05-16 19:51:41 +03:00
Azat Khuzhin
a44ddd2064 Update distributed_group_by_no_merge description 2021-05-15 15:03:13 +03:00
Alexander Kuzmenkov
5ed6a273d3
Merge pull request #24058 from ClickHouse/aku/uniq-two-level 
lower two-level aggregation threshold for uniq test to avoid jitter
 2021-05-14 15:50:48 +03:00
Alexander Kuzmenkov
6f16d92cfe
Update Settings.h 2021-05-14 15:50:30 +03:00
Alexander Kuzmenkov
220f5b512e
try even smaller value 2021-05-13 06:15:44 +03:00
kssenii
407db17e2e Remove setting, use native transaction 2021-05-12 22:33:15 +00:00
kssenii
0955e5f0be Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-12 15:11:15 +00:00
Alexander Kuzmenkov
f6cb9f95df lower two-level aggregation threshold for uniq test to avoid jitter 2021-05-12 16:01:48 +03:00
alesapin
46e136b5c4
Merge branch 'master' into nv/parts-uuid-move-shard 2021-05-11 15:36:40 +03:00
Amos Bird
264cff6415
Projections 
TODO (suggested by Nikolai)

1. Build query plan fro current query (inside storage::read) up to WithMergableState
2. Check, that plan is simple enough: Aggregating - Expression - Filter - ReadFromStorage (or simplier)
3. Check, that filter is the same as filter in projection, and also expression calculates the same aggregation keys as in projection
4. Return WithMergableState if projection applies

3 will be easier to do with ActionsDAG, cause it sees all functions, and dependencies are direct (but it is possible with ExpressionActions also)

Also need to figure out how prewhere works for projections, and
row_filter_policies.

wip
 2021-05-11 18:12:23 +08:00
vdimir
905e54a0af
Merge branch 'master' into create-aggregate-func-settings 2021-05-11 11:18:24 +03:00
Nikita Vasilev
af9f811ab5 Merge remote-tracking branch 'upstream/master' into nikvas0/simple_optimizations 2021-05-10 22:08:58 +03:00
kssenii
5c24f0943c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-05-08 14:54:44 +00:00
Maksim Kita
9daf5b8308 Updated Settings 2021-05-08 17:39:34 +03:00
kssenii
912878ad7f Review fixes 2021-05-07 12:05:34 +00:00
kssenii
8662078b35 Review fixes 2021-05-07 09:55:18 +00:00
hexiaoting
a1d521902c Merge remote-tracking branch 'origin/master' into dev-continue-sundyli 2021-05-07 14:22:53 +08:00
fibersel
cb53bbb7b0 add experimental codecs flag, add integration test for experimental codecs 2021-05-06 14:57:22 +03:00
kssenii
7643d84095 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-30 22:02:51 +00:00
Nikita Vasilev
e565bc47a8 settings off 2021-04-30 12:34:00 +03:00
Nicolae Vartolomei
1fa5871ff7 Fix bad rebase and introduce part_moves_between_shards_enable setting 2021-04-27 14:20:13 +01:00
Nicolae Vartolomei
53d57ffb52 Part movement between shards 
Integrate query deduplication from #17348
 2021-04-27 14:20:12 +01:00
Nikolai Kochetov
29fd51e37d Enable async_socket_for_remote 2021-04-27 13:30:12 +03:00
Nikita Vasilev
0ec2726bf3 Merge remote-tracking branch 'upstream/master' into nikvas0/simple_optimizations 2021-04-26 17:33:32 +03:00
Vladimir
fd777bcd4b
Merge branch 'master' into create-aggregate-func-settings 2021-04-26 11:12:09 +03:00
kssenii
459612c895 Use a separate setting 2021-04-22 22:14:14 +00:00
Ivan Lezhankin
484528ba2c Fix problem with use-after-free inside shared_ptr 2021-04-21 16:19:28 +03:00
Nikolai Kochetov
44112587d4
Merge pull request #23309 from ClickHouse/try-fix-crash-with-unknown-packet 
Fix crash in case of unknown packet
 2021-04-21 08:47:17 +03:00
alexey-milovidov
a311a5c0b5
Merge pull request #23322 from azat/disable-min_bytes_to_use_mmap_io 
Disable min_bytes_to_use_mmap_io by default
 2021-04-20 19:27:47 +03:00
Ivan
10c5ba3022
Merge branch 'master' into async-insert 2021-04-20 17:58:15 +03:00
Nikolai Kochetov
ecb2a96af8 Revert "Enable async_socket_for_remote" 
This reverts commit ff908baa06.
 2021-04-20 17:14:54 +03:00
Nikita Mikhaylov
f55eacee7d
Merge pull request #22988 from excitoon-favorites/s3connectionresetretries 
Retries on HTTP connection drops during reads from S3
 2021-04-20 14:59:21 +03:00
Kseniia Sumarokova
b43656bf89
Merge pull request #22697 from TCeason/feature/add_MaterializeMySQL_sync_bytes_judgment 
Add MySQL read history data bytes judgment
 2021-04-20 10:26:42 +03:00
Azat Khuzhin
6fba6f7ddd Disable min_bytes_to_use_mmap_io by default 
Reading files using mmap() does not have any significant benefits over
plain read() [1].

  [1]: https://gist.github.com/azat/3d6c8d82bdd91e7a38d997fd6bcfd574

And not only it does not have significant benefits, it also has some
issues, due to max_server_memory_usage (default to 90% of available
RAM), since when you read files with mmap() eventually process RSS may
exceed max_server_memory_usage, and in this case any allocation will
fail (with "Memory limit exceeded (total)") error (yes kernel will
unload pages, but likely it will happens after queries will starting to
fail), like in this test [2].

  [2]: https://gist.github.com/azat/4813489828162e6c2ce131963c6a1acb

TL;DR;

Note that there was also an idea to take those mmap()'ed regions in
memory tracking (#23211), but there are some drawbacks (since accounting
mmap() is tricky, first of all you need to account only once per inode
for file and plus kernel can unload some pages and those memory will not
be used by the server anymore).

And as an adddition to #23211 there was #23212, that adds
max_bytes_to_use_mmap_io, but since mmap is not a subject for memory
accounting there is no need in it.
 2021-04-20 00:25:16 +03:00
Ivan Lezhankin
a36596e303 Add settings 2021-04-19 22:16:34 +03:00
Ivan Lezhankin
7a88ca31cc Merge branch 'master' into async-insert 2021-04-19 18:33:43 +03:00
Nikolai Kochetov
ff908baa06 Enable async_socket_for_remote 2021-04-19 17:31:46 +03:00
Nikolai Kochetov
0c23d44b99 Add test. 2021-04-19 17:12:08 +03:00
Alexander Kuzmenkov
2a4bcb6e3f
Merge pull request #23159 from ClickHouse/aku/merge-fusecount 
merging sumCount fusion PR #21337
 2021-04-19 16:47:13 +03:00
Vladimir Chebotarev
9d4295f261 Removed infinite retries. 2021-04-19 11:02:36 +03:00
TCeason
b4bf53dfc9 add some comment and modify a parameter type 2021-04-19 15:34:27 +08:00
TCeason
63403c709c modify settings name to external_xxx and rewrite Storage MySQL max_block_size 2021-04-19 10:51:50 +08:00
TCeason
87aa904440 Modify according to review opinion 2021-04-19 10:51:50 +08:00
TCeason
472c131420 Add MySQL read history data bytes judgment 2021-04-19 10:51:50 +08:00
kssenii
88b50621d4 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-18 14:40:37 +00:00
Alexey Milovidov
21ff5d85d9 Disable hedged requests 2021-04-18 15:13:57 +03:00
vdimir
92e813c806
Merge branch 'master' into create-aggregate-func-settings 2021-04-16 15:48:39 +03:00
Alexey Milovidov
9110a76d00 Reordered settings to avoid confusion 2021-04-16 00:14:37 +03:00
Alexander Kuzmenkov
2489b6af96 cleanup 2021-04-15 19:40:49 +03:00
Alexander Kuzmenkov
3b95b637a5 Merge remote-tracking branch 'origin/master' into HEAD 2021-04-15 18:19:53 +03:00
vdimir
33a9189151
Merge remote-tracking branch 'upstream/master' into create-aggregate-func-settings 2021-04-15 13:41:21 +03:00
Vladimir Chebotarev
10bf56e540 Attempt to add retries to S3 reads. 2021-04-15 09:47:19 +03:00
Ivan Lezhankin
9e5d274467 Merge remote-tracking branch 'upstream/master' into async-insert 2021-04-14 20:16:05 +03:00
alexey-milovidov
1945d01719
Merge pull request #21972 from kssenii/nanodbc 
Switch to nanodbc
 2021-04-13 12:23:57 +03:00
Maksim Kita
e687fc320f
Merge pull request #22997 from ClickHouse/increase-max-uri-size 
Increase `max_uri_size` to 1 MiB by default
 2021-04-13 11:26:23 +03:00
hexiaoting
77c460e8d1 Merge remote-tracking branch 'origin/master' into dev-sumcount 2021-04-13 11:14:14 +08:00
alexey-milovidov
d239c66fd3
Merge pull request #21511 from azat/optimize_skip_unused_shards-improvement 
Rewrite IN in query for remote shards to exclude values that do not belong to the shard
 2021-04-13 01:18:11 +03:00
Alexey Milovidov
e260b2a60a Increase max_uri_size to 1 MiB by default 2021-04-12 21:06:47 +03:00
vdimir
f2c109cc86
Merge branch 'master' into create-aggregate-func-settings 2021-04-12 11:38:20 +03:00
vdimir
f022f0744b
Add setting allow_experimental_funnel_functions and use in windowFunnel 2021-04-12 11:35:07 +03:00
Azat Khuzhin
fbb386dca5 Rewrite IN in query for remote shards to exclude values that does not belongs to shard 
v2: fix optimize_skip_unused_shards_rewrite_in for sharding_key wrapped into function
v3: fix column name for optimize_skip_unused_shards_rewrite_in
v4: fix optimize_skip_unused_shards_rewrite_in with Null
v5:
- squash with Remove query argument for IStreamFactory::createForShard()
- use proper column after function execution (using sharding_key_column_name)
- update the test reference since (X) now is tuple(X)
 2021-04-12 10:37:28 +03:00
kssenii
7a287e6fe9 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into nanodbc 2021-04-11 21:36:08 +00:00
kssenii
c1adece94f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-11 06:00:18 +00:00
kssenii
1c501e7d97 Support replica identity index 2021-04-10 16:10:28 +00:00
Alexey Milovidov
173c0e5c43 More conservative value of background_fetches_pool_size setting 2021-04-09 23:45:14 +03:00
Alexander Kuzmenkov
e44b3822e3
Merge pull request #21850 from fastio/handle_errors_for_kafka_engine 
Handle errors for Kafka engine
 2021-04-09 22:59:40 +03:00
kssenii
8bdcac4f06 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-09 13:12:15 +00:00
Alexander Kuzmenkov
0264124146
Merge pull request #21942 from ucasFL/distributed_depth 
Add settings max_distributed_depth
 2021-04-09 15:52:58 +03:00
kssenii
72d89bc933 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into nanodbc 2021-04-09 11:13:01 +00:00
kssenii
9fa21a6272 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-04-08 12:18:54 +00:00
kssenii
0b39480d51 Better 2021-04-06 20:01:23 +00:00
Nikolai Kochetov
6102652c99 Merge branch 'master' into better-filter-push-down 2021-04-06 13:38:03 +03:00
Nikolai Kochetov
44700d97e9 Add setting to disable query plan optimization. Fix perftest. 2021-04-06 11:25:54 +03:00
feng lv
f6e4e13f89 fix conflict 2021-04-05 14:08:30 +00:00
feng lv
56073db22d max distributed depth 
Add settings max_distributed_depth

fix style

fix

fix

fix

fix test

fix

fix
 2021-04-05 14:00:54 +00:00
kssenii
13d1f17d3e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-04-03 08:00:09 +00:00
kssenii
2de6b99f84 Better names 2021-04-02 16:12:14 +00:00
kssenii
99f0b08ac3 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-04-01 13:48:33 +00:00
Ivan Lezhankin
cc1c439ad8 Merge remote-tracking branch 'upstream/master' into poco-http2 2021-04-01 16:20:15 +03:00
kssenii
2c70183652 Fix settings 2021-04-01 11:41:22 +00:00
Peng Jian
909d5ad2b5 Handle errors for Kafka engine 2021-03-31 17:15:57 +08:00
alexey-milovidov
6ed78827a5
Merge pull request #22326 from ClickHouse/test-mmap-io 
Run performance tests with enabled mmap IO
 2021-03-31 06:33:56 +03:00
alexey-milovidov
0c70b06960
Merge branch 'master' into system-querylog-map 2021-03-31 04:54:30 +03:00
kssenii
ce05087b1b Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into replicas-shards-for-mysql-and-postgres 2021-03-30 17:27:10 +00:00
Amos Bird
69204e1d21
Add prefer_column_name_to_alias settings (#22044) 2021-03-30 16:51:45 +03:00
Ivan Lezhankin
873a98b33b Add test 2021-03-30 16:19:05 +03:00
Alexey Milovidov
85e4cfa731 Run performance tests with enabled mmap IO 2021-03-30 00:04:03 +03:00
tavplubix
3c0f5a57ec
Merge pull request #21535 from ClickHouse/distributed_ddl_improvements 
Distributed DDL improvements
 2021-03-29 22:40:11 +03:00
Ivan Lezhankin
de05803177 Merge branch 'master' into async-insert 2021-03-29 20:59:02 +03:00
kssenii
c17748a0ba Add proper settings 2021-03-27 21:57:35 +00:00
Nikita Vasilev
02090bf59a Merge remote-tracking branch 'upstream/master' into nikvas0/simple_optimizations 2021-03-27 13:03:55 +03:00
tavplubix
48b89a0ace
Merge pull request #21998 from Avogar/hedged-test 
Update tests for hedged requests
 2021-03-23 18:18:49 +03:00
Pavel Kruglov
dbc5018000 Update tests for hedged requests 2021-03-22 22:18:06 +03:00
Alexander Kuzmenkov
2f5dbf57b6
Merge pull request #21839 from kssenii/add-postgres-connection-pool 
Add connection pool for postgres engine
 2021-03-22 19:49:51 +03:00
tavplubix
5dfd0045a6
Merge branch 'master' into distributed_ddl_improvements 2021-03-22 18:13:19 +03:00
kssenii
ed158e2b77 Proper setting 2021-03-20 14:42:03 +00:00
kssenii
619f0f51fd Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-03-20 09:47:05 +00:00
kssenii
3b32c8fe9a Block on empty connection pool 2021-03-19 21:00:45 +00:00
Ivan Lezhankin
4f1fad97ab Merge remote-tracking branch 'upstream/master' into async-insert 2021-03-19 23:41:01 +03:00
Pavel Kruglov
d380454044 Change read_data_timeout from sec to ms, add profile event for hedged requests and better process receiving packets 2021-03-18 20:16:17 +03:00
kssenii
a03e849a8f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-03-16 10:06:10 +00:00
tavplubix
6a265f468e
Merge branch 'master' into distributed_ddl_improvements 2021-03-15 11:24:14 +03:00
Alexey Milovidov
bbd6ac2d3a Merge branch 'master' into add_postgresql_engine_docs 2021-03-14 14:22:56 +03:00
Alexander Tokmakov
83b3e4e0f5 allow to use Cluster from Replicated db 2021-03-09 20:05:24 +03:00
vdimir
36006a7e96
Merge branch 'master' into cross-to-inner-join-rewrite 2021-03-09 09:36:15 +03:00
Alexander Tokmakov
163e27b299 support query and session settings for distributed DDL 2021-03-09 01:57:53 +03:00
Alexander Tokmakov
5070b5b85b detach permanently setting for Replicated db 2021-03-09 00:34:52 +03:00
Alexander Tokmakov
2022b90919 configurable output mode for distributed DDL 2021-03-08 23:35:09 +03:00
Azat Khuzhin
16f4c02d42 Add optimize_skip_unused_shards_limit 
Limit for number of sharding key values, turns off
optimize_skip_unused_shards if the limit is reached
 2021-03-26 06:09:00 +03:00
Slach
5791cf5f2b fix review comments https://github.com/ClickHouse/ClickHouse/pull/21078#discussion_r580413875 and https://github.com/ClickHouse/ClickHouse/pull/21078#discussion_r580410204 
Signed-off-by: Slach <bloodjazman@gmail.com>
 2021-03-08 03:39:24 +03:00
Nikita Vasilev
728c9ccb6d Merge remote-tracking branch 'upstream/master' into nikvas0/simple_optimizations 2021-03-05 19:17:01 +03:00
Ivan Lezhankin
5cfeedb1e0 Add setting http_max_uri_size 2021-03-04 15:21:23 +03:00
Ivan Lezhankin
74214865eb Merge remote-tracking branch 'upstream/master' into async-insert 2021-03-04 14:14:28 +03:00
Ivan Lezhankin
b726845be2 [WIP] 2021-03-04 14:10:21 +03:00
Pavel Kruglov
dc9dd8ccdc Fix typo 2021-03-03 16:16:17 +03:00
hexiaoting
4fe75ad168 fix error 2021-03-03 17:18:53 +08:00
Pavel Kruglov
82a6be44cf Disable changing replica when we have made progress 2021-03-03 01:03:03 +03:00
Pavel Kruglov
153bfbfc28 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-03-02 11:59:32 +03:00
kssenii
a3c7402200 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-03-02 07:07:56 +00:00
hexiaoting
e116346cbb Merge remote-tracking branch 'origin/master' into dev-sumcount 2021-03-02 10:11:50 +08:00
hexiaoting
16333e9f3a Suppport fuse aggregate functions:sum/avg/count to sumCount 2021-03-01 18:04:34 +08:00
Pavel Kruglov
197058dc12 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-02-27 00:36:33 +03:00
Nikolai Kochetov
d91b8a3acb
Merge branch 'master' into filter-push-down 2021-02-26 19:33:27 +03:00
Nikolai Kochetov
d328bfa41f Review fixes. Add setting max_optimizations_to_apply. 2021-02-26 19:29:56 +03:00
kssenii
4cdb55babe Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into pg-ch-replica 2021-02-22 12:39:35 +00:00
kssenii
ace76bb7fa Better, remove debug logs 2021-02-22 12:35:53 +00:00
alexey-milovidov
50639b7dba
Merge pull request #21052 from ClickHouse/file-engine-settings 
Merging #20620
 2021-02-22 13:10:31 +03:00
Alexey Milovidov
46ac365629 Merge branch 'master' into topvisor/file-engine-settings 2021-02-21 23:10:26 +03:00
Alexey Milovidov
0a62f95bbe Merge branch 'master' into distributed_aggregation_memory_efficient_by_default 2021-02-21 23:06:49 +03:00
Kruglov Pavel
815b4ff0d9
Merge branch 'master' into hedged-requests 2021-02-21 17:10:24 +03:00
alexey-milovidov
20a7ccf5b1
Merge pull request #20174 from amosbird/alias 
Normalized function names (lower-case case-insensitive names)
 2021-02-21 16:23:51 +03:00
tavplubix
a697b578bc
Merge pull request #16193 from ClickHouse/database_replicated 
DatabaseReplicated: continuation of #10485
 2021-02-20 19:39:34 +03:00
vdimir
e052a5a05e
Revert "Support old cross to inner join rewrite behaviour" 
This reverts commit 527210b5e4.
 2021-02-19 15:14:24 +03:00
Amos Bird
2c4bc43014
Backward compatible 2021-02-18 20:20:29 +08:00
Alexey Milovidov
0d2d2dc2ed Merge branch 'master' into sundy-li/system-querylog-map 2021-02-18 00:47:54 +03:00
Pavel Kruglov
1b78de2142 Use fibers in HedgedRequests 2021-02-17 20:34:52 +03:00
vdimir
527210b5e4
Support old cross to inner join rewrite behaviour 2021-02-17 12:44:53 +03:00
Alexey Milovidov
fa20016091 Enable distributed_aggregation_memory_efficient by default 2021-02-17 09:53:18 +03:00
M0r64n
75117389ec Add a couple of QOL file engine settings 2021-02-16 18:50:11 +04:00
Alexander Tokmakov
bf6f64a3fb Merge branch 'master' into database_replicated 2021-02-16 01:28:19 +03:00
vdimir
9afa6b5b1b
Add option cross_to_inner_join_rewrite 2021-02-15 22:40:32 +03:00
Kruglov Pavel
598576ce70
Merge branch 'master' into hedged-requests 2021-02-15 16:51:45 +03:00
Alexander Tokmakov
9c7cf9e92e remove some debug code 2021-02-15 13:26:34 +03:00
Alexey Milovidov
4d650a2a56 Adjust config 2021-02-11 16:41:21 +03:00
Amos Bird
fc6c57f8e0
Fix 2021-02-10 12:48:41 +08:00
Amos Bird
565a148da9
Normalize count() variants 2021-02-10 12:48:41 +08:00
Anton Popov
f1beeb7c25
Merge pull request #20099 from azat/remove-not-existing-setting 
Remove never existing insert_in_memory_parts_timeout
 2021-02-08 16:09:03 +03:00
Pavel Kruglov
f946aab759 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-02-06 17:38:56 +03:00
Kruglov Pavel
d94e8624d7
Merge branch 'master' into shard-id 2021-02-06 16:48:17 +03:00
Azat Khuzhin
48b415d3e4 Remove never existing insert_in_memory_parts_timeout 
Fixes: a43cb93be5 ("remove questionable functionality")
Cc: @CurtizJ
 2021-02-05 01:26:40 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID 
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
 2021-02-02 16:53:39 +00:00
Pavel Kruglov
60a92e9a99 Fix build, add comments, update tests 2021-02-02 15:14:31 +03:00
feng lv
0edf65c094 fix test 
fix

update

fix spell
 2021-02-02 09:22:30 +00:00
feng lv
4279c7da41 add setting insert_shard_id 
add test

fix style

fix
 2021-02-02 04:26:59 +00:00
Vitaliy Zakaznikov
3de4690450 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into live_view_periodic_refresh 2021-01-29 19:57:13 -05:00
Pavel Kruglov
25e85d71ee Merge branch 'master' of github.com:ClickHouse/ClickHouse into hedged-requests 2021-01-29 21:08:47 +03:00
sundy-li
7083ef45a6 Merge remote-tracking branch 'origin/master' into system-querylog-map 2021-01-29 09:26:02 +08:00
sundyli
77194e4a5d
Merge branch 'master' into system-querylog-map 2021-01-28 09:59:41 +08:00
Alexey Milovidov
065d22cf2b Merge branch 'master' into disable-checksums-on-read 2021-01-28 03:07:31 +03:00
alexey-milovidov
2028228f4b
Merge pull request #18684 from amosbird/betterwith 
better CTE and with global
 2021-01-27 04:33:17 +03:00
Alexey Milovidov
a6f5f40a65 Merge branch 'master' into disable-checksums-on-read 2021-01-27 04:09:00 +03:00
Amos Bird
3afcccaad2
Fix global with after union refactoring 2021-01-26 10:29:12 +08:00
Alexey Milovidov
9ee5c1535e Allow to disable checksums on read 2021-01-25 23:29:04 +03:00
Alexey Milovidov
cfd47b2980 Merge branch 'master' into issue18494 2021-01-25 22:18:23 +03:00
sundy-li
685210e02b Merge branch 'master' into system-querylog-map 2021-01-23 11:28:22 +08:00
flynn
e75b116466
Rewrite sum(if()) and sumIf to countIf in special cases (#17041) 
Co-authored-by: vdimir <vdimir@yandex-team.ru>
 2021-01-21 12:01:35 +03:00
sundy-li
22c5fa0974 Merge branch 'master' into system-querylog-map 2021-01-20 23:00:22 +08:00
Pavel Kruglov
97b5179e55 Implement HedgedRequests 2021-01-19 22:41:05 +03:00
Alexander Kuzmenkov
d22c04568d
Merge pull request #13405 from excitoon-favorites/s3keepalive 
Connection pools for S3
 2021-01-18 12:52:37 +03:00
alexey-milovidov
78fff6bc39
Merge branch 'master' into multiple-nested 2021-01-15 14:54:27 +03:00
alexey-milovidov
2ca185c9f9
Merge pull request #16995 from sundy-li/alias_column_partition_prune 
Add optimize_respect_aliases
 2021-01-14 11:30:45 +03:00
Alexey Milovidov
c51df126f1 Disable "optimize_move_functions_out_of_any" by default #18973 2021-01-12 19:46:18 +03:00
sundy-li
ad01534bd5 Merge branch 'master' into alias_column_partition_prune 2021-01-12 18:28:30 +08:00
Anton Popov
36ae0e4d35 Merge remote-tracking branch 'upstream/master' into HEAD 2021-01-11 13:51:12 +03:00
Alexey Milovidov
7e9120b34f Merge branch 'master' into chenziliang/feature/wildcard-dynamic-columns 2021-01-10 04:04:47 +03:00
Amos Bird
a3d19fa64d
Correctly override default settings remotely 2021-01-08 12:28:09 +08:00
Amos Bird
0260953a47
better 2021-01-06 17:18:48 +08:00
Vladimir Chebotarev
5cbc25c647 Added global setting s3_max_connections. 2021-01-05 13:34:55 +03:00
sundy-li
cef8bcc011 (feature) Use Map data type for system logs tables 2021-01-05 10:10:19 +00:00
Nikita Vasilev
92e4f1370e CNF 2021-01-04 23:55:32 +03:00
Amos Bird
a157a5b3b3
add max_partitions_to_read setting 2021-01-04 12:40:48 +08:00
Nikita Vasilev
1d5b62cc18 add simple optimization 2021-01-03 13:24:09 +03:00
sundyli
f8815f233c
Merge branch 'master' into alias_column_partition_prune 2021-01-02 11:23:55 +08:00
Alexey Milovidov
4b3ae495d6 Merge branch 'master' into CurtizJ-multiple-nested 2021-01-02 00:25:16 +03:00
alexey-milovidov
a7c350adda
Merge branch 'master' into feature/wildcard-dynamic-columns 2020-12-31 17:47:21 +03:00
Nikita Mikhailov
8e31ce4a5c Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-30 18:22:11 +03:00
Nikita Mikhailov
b94a654715 build fix 2020-12-30 16:55:31 +03:00
Alexey Milovidov
68fac02d35 Change union_default_mode to throw exception 2020-12-29 20:27:00 +03:00
TszkitLo40
f7d7880ad5 fix build 2020-12-29 10:13:27 +08:00
Nikita Mikhailov
c3288c3fbf Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-28 15:09:37 +03:00
alfredlu
7ec8a73cc3 fix build 2020-12-28 10:38:16 +08:00
alfredlu
51cfbe8c2e Add log_comment setting 2020-12-27 18:35:46 +08:00
Anton Popov
11283e3d81 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-25 21:25:59 +03:00
Amos Bird
6fc225e676
Distributed insertion to one random shard (#18294) 
* Distributed insertion to one random shard

* add some tests

* add some documentation

* Respect shards' weights

* fine locking

Co-authored-by: Ivan Lezhankin <ilezhankin@yandex-team.ru>
 2020-12-23 19:04:05 +03:00
Anton Popov
57857dda63 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-23 15:16:43 +03:00
Nikolai Kochetov
af7f5c9518
Merge pull request #17868 from ClickHouse/async-read-from-socket 
Async read from socket
 2020-12-23 12:20:42 +03:00
Alexander Kuzmenkov
51b2329295
Merge branch 'master' into aku/window-prototype 2020-12-23 09:26:11 +03:00
Nikita Mikhailov
dcfbe782c6 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-23 05:20:22 +03:00
Anton Popov
b6ff6300b2 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-22 18:06:21 +03:00
Alexander Kuzmenkov
85c1bc1253 Merge remote-tracking branch 'origin/master' into tmp 2020-12-21 10:46:21 +03:00
Alexey Milovidov
ae9b67fefa Support to parse Arrays in CSV as nested CSV in a String 2020-12-20 13:26:08 +03:00
alexey-milovidov
34e4ace029
Merge pull request #17525 from ClickHouse/null-as-default-default 
Attempt to make NULL as default by default
 2020-12-18 18:00:34 +03:00
Nikolai Kochetov
c7ef57c6fd Add setting async_socket_for_remote 2020-12-18 16:15:03 +03:00
Anton Popov
009efeff40 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-18 16:14:09 +03:00
Alexey Milovidov
a0f7a12c4f Merge branch 'master' into null-as-default-default 2020-12-18 08:09:35 +03:00
Alexander Kuzmenkov
4bfcb2fc16 a setting 2020-12-18 04:25:21 +03:00
Nikita Mikhailov
6a55e018e4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into parallel-parsing-input-format 2020-12-18 02:25:32 +03:00
Alexey Milovidov
e4433157e7 Merge branch 'master' into insert_quorum_parallel-default 2020-12-17 22:42:32 +03:00
Anton Popov
a42b00c9aa Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-17 20:43:23 +03:00
hexiaoting
cb91d64fe7
Add new feature: limit length and offset setting (#17633) 
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
 2020-12-17 17:16:14 +03:00
sundy-li
84807e9791 rename settings 2020-12-17 02:17:59 +00:00
Alexey Milovidov
6a7c73a7bd Merge branch 'master' into insert_quorum_parallel-default 2020-12-17 04:54:17 +03:00
sundy-li
24a190a461 update setting description 2020-12-16 10:46:47 +00:00
sundy-li
5c9f84b2f4 Merge branch 'master' into alias_column_partition_prune 2020-12-16 18:44:00 +08:00
Anton Popov
e51cf03f21 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-15 21:08:01 +03:00
Nikolai Kochetov
6defcbb662
Merge branch 'master' into optimize-data-on-insert 2020-12-15 16:50:42 +03:00
sundy-li
cf087002db Merge branch 'master' into alias_column_partition_prune 2020-12-15 16:37:00 +08:00
Anton Popov
c78861e86e Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-15 01:45:01 +03:00
Anton Popov
afc2d7099b add conversion from array to map 2020-12-15 01:16:04 +03:00
nikitamikhaylov
8ff072c702 better 2020-12-15 00:56:47 +03:00
nikitamikhaylov
57705f5b73 delete and fix strange code 2020-12-15 00:56:47 +03:00
Nikita Mikhaylov
9922324787 it works 2020-12-15 00:56:47 +03:00
alexey-milovidov
363ff79cca
Merge pull request #17934 from Jokser/s3-adaptive-single-part-upload 
Adaptive choose of single/multi part upload in WriteBufferFromS3
 2020-12-14 23:00:43 +03:00
Pavel Kovalenko
346c241c20 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload 
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
 2020-12-14 11:00:30 +03:00
Alexey Milovidov
6341521b10 Remove obsolete settings 2020-12-14 02:40:11 +03:00
alexey-milovidov
bce02de691
Merge pull request #17539 from azat/remerge_sort_lowered_memory_bytes_ratio 
Add remerge_sort_lowered_memory_bytes_ratio setting
 2020-12-13 04:38:50 +03:00
sundyli
36ca21c24a
Merge branch 'master' into alias_column_partition_prune 2020-12-13 00:45:14 +08:00
sundy-li
9407028540 update 2020-12-13 00:42:15 +08:00
Pavel Kovalenko
db7a87089b Adaptive choose of single/multi part upload in WriteBufferFromS3. 2020-12-09 17:09:04 +03:00
Kruglov Pavel
e19eb6f17a
Merge branch 'master' into optimize-data-on-insert 2020-12-08 15:57:46 +03:00
Anton Popov
d7aad3bf79 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-07 19:13:01 +03:00
Anton Popov
60b0cbb1c1
Merge pull request #15939 from Avogar/optimize_final_optimization 
Optimize final optimization
 2020-12-05 02:26:27 +03:00
Pavel Kruglov
905ba78adc Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-12-04 18:56:46 +03:00
Ivan
315ff4f0d9
ANTLR4 Grammar for ClickHouse and new parser (#11298) 2020-12-04 05:15:44 +03:00
Anton Popov
cd1917c7a6
Merge branch 'master' into optimize_final_optimization 2020-12-03 16:52:51 +03:00
Azat Khuzhin
ccd794a676 Add remerge_sort_lowered_memory_bytes_ratio setting 
Sometimes default ratio is not good enough (2), since it depends on lots
of factors:
- size of LIMIT
- max_memory_usage
- ...

So add a separate setting for it.

But note that it make sense to set
remerge_sort_lowered_memory_bytes_ratio only in range (1, 2]

Since <= 1 will just use more CPU, and > 2 will use more RAM and will
unlikely be faster.
 2020-11-29 21:16:42 +03:00
Alexey Milovidov
7d79a3cc35 Make insert_quorum_parallel by default 2020-11-29 18:13:57 +03:00
Alexey Milovidov
20e56da8ac Attempt to make NULL as default by default 2020-11-28 10:46:08 +03:00
Ken Chen
349571a458 Add asterisk_include_materialized_columns and asterisk_include_materialized_columns switches to include materialized columns and alias columns respectively for wildcard query 2020-11-28 00:06:53 +00:00