Commit Graph

2851 Commits

Author SHA1 Message Date
alesapin
9e98cdad5a Accident changes 2020-12-14 16:05:07 +03:00
alesapin
47e3f619e5 Remove redundant code 2020-12-14 16:01:01 +03:00
alesapin
a87fb3fae0 Fix style 2020-12-14 15:52:15 +03:00
alesapin
303c12d2e4 Slightly better 2020-12-14 15:51:14 +03:00
alesapin
6fb23da851 Refactoring 2020-12-14 15:03:49 +03:00
alesapin
70be03aa7d Working compact parts 2020-12-14 14:36:15 +03:00
alesapin
fbcf85cb42 Something working for wide parts 2020-12-14 14:06:02 +03:00
alexey-milovidov
d55c5bec94
Merge pull request #18053 from azat/librdkafka-snappy-UB-fix
Fix possible corruption in librdkafka snappy decompression
2020-12-14 13:59:11 +03:00
Pavel Kovalenko
346c241c20 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
2020-12-14 11:00:30 +03:00
alesapin
12b6a54a24 Merge branch 'master' into try_rewrite_writer 2020-12-14 10:30:01 +03:00
alesapin
fd695c9ff4 Dive into debug 2020-12-14 10:28:42 +03:00
Maksim Kita
1bccd6dff6 Fixed style issues 2020-12-14 00:50:55 +03:00
Azat Khuzhin
2bf226decc Fix typo in KafkaSettings comments 2020-12-14 00:38:10 +03:00
Maksim Kita
69127ca0ed Changed BlocksList to Blocks 2020-12-14 00:21:25 +03:00
Vitaly Baranov
9310437920 Fix access rights required for the merge() table function. 2020-12-13 23:34:53 +03:00
Maksim Kita
3c6df61e5c MemoryStorage read optimization 2020-12-13 22:24:15 +03:00
Alexander Tokmakov
0c80fce7f0 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-12-13 17:56:50 +03:00
sundy-li
041f5c8843 add comments 2020-12-13 19:30:53 +08:00
sundy-li
9ff39da5d2 fixes build 2020-12-13 09:55:56 +08:00
sundyli
36ca21c24a
Merge branch 'master' into alias_column_partition_prune 2020-12-13 00:45:14 +08:00
sundy-li
9407028540 update 2020-12-13 00:42:15 +08:00
Azat Khuzhin
77a1d00dee Add new remote context as interpreter context to the Pipe 2020-12-12 17:43:10 +03:00
Azat Khuzhin
5b3ab48861 More forward declaration for generic headers
The following headers are pretty generic, so use forward declaration as
much as possible:
- Context.h
- Settings.h
- ConnectionTimeouts.h
(Also this shows that some missing some includes -- this has been fixed)

And split ConnectionTimeouts.h into ConnectionTimeoutsContext.h (since
module part cannot be added for it, due to recursive build dependencies
that will be introduced)

Also remove Settings from the RemoteBlockInputStream/RemoteQueryExecutor
and just pass the context, since settings was passed only in speicifc
places, that can allow making a copy of Context (i.e. Copier).

Approx results (How much units will be recompiled after changing file X?):

- ConnectionTimeouts.h
  - mainline: 100

- Context.h:
  - mainline: ~800
  - patched:  415

- Settings.h:
  - mainline: 900-1K
  - patched:  440 (most of them because of the Context.h)
2020-12-12 17:43:10 +03:00
Amos Bird
04128c33e1
disable TSAN for real 2020-12-12 14:17:48 +08:00
Amos Bird
0aa75e6023
Another try of TSan fix 2020-12-12 13:59:34 +08:00
alexey-milovidov
cfeaff6aa7
Update StorageSystemStackTrace.cpp 2020-12-12 13:59:34 +08:00
alexey-milovidov
b513eeee7f
Another try... 2020-12-12 13:59:34 +08:00
Amos Bird
5e71f4e810
Have to resort to NO_SANITIZE_THREAD 2020-12-12 13:59:34 +08:00
Amos Bird
c4306b5788
Another try of TSan fix 2020-12-12 13:59:34 +08:00
Amos Bird
d0044886ab
fix TSan 2020-12-12 13:59:34 +08:00
Amos Bird
4809b5975a
Fix system.stack_trace in daemon 2020-12-12 13:59:33 +08:00
alexey-milovidov
adbe8e1ebb
Merge pull request #17903 from ClickHouse/attach_table_from_path
Implement ATTACH TABLE name FROM 'path/to/data/'
2020-12-12 02:42:53 +03:00
Alexander Kuzmenkov
dcc4b5609f fixup 2020-12-11 21:16:26 +03:00
Alexander Kuzmenkov
d54d96701a fixup 2020-12-11 18:57:26 +03:00
Pavel Kovalenko
59a5e63417 Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload
# Conflicts:
#	src/Disks/S3/DiskS3.cpp
#	src/Disks/S3/DiskS3.h
#	src/Disks/S3/registerDiskS3.cpp
#	src/IO/WriteBufferFromS3.cpp
#	src/IO/WriteBufferFromS3.h
2020-12-11 18:28:41 +03:00
alesapin
a24a76828e Fix 2020-12-11 18:18:51 +03:00
alesapin
4b83707c63 Compileable code for some reason 2020-12-11 18:00:58 +03:00
alesapin
57c30c4164 Garbage 2020-12-11 16:20:19 +03:00
Alexander Tokmakov
58a31e4882 Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-12-11 16:14:03 +03:00
alesapin
7ffd76a605 Fix bug 2020-12-11 11:41:02 +03:00
alesapin
cbba73672e Merge branch 'master' into try_rewrite_writer 2020-12-11 11:09:47 +03:00
Alexander Kuzmenkov
d757f32300 Remove some redundant includes to speed up build 2020-12-11 02:58:11 +03:00
Alexander Kuzmenkov
dea446be2f Remove some redundant includes to speed up build 2020-12-11 02:56:57 +03:00
alexey-milovidov
7f4b5a0d1b
Merge pull request #17854 from weeds085490/dev/add_diag_info_when_circle_in_merge
add diagnostic information when two merge tables try to read each oth…
2020-12-10 22:56:34 +03:00
alexey-milovidov
c35c52be87
Merge pull request #17867 from ucasFL/mergetree-setting
Add settings `min_compress_block_size` and `max_compress_block_size` to MergeTreeSettings
2020-12-10 22:54:05 +03:00
alexey-milovidov
f1ee1ef16a
Update MergeTreeSettings.h 2020-12-10 22:50:54 +03:00
Alexander Kuzmenkov
88e7bc5b60 Merge origin/master into tmp (using imerge) 2020-12-10 22:12:42 +03:00
alesapin
1cb06bd975 Split one method 2020-12-10 19:29:10 +03:00
alesapin
90fa9a2073 Merge branch 'master' into try_rewrite_writer 2020-12-10 18:36:10 +03:00
alesapin
f3d1dc360c Fix style 2020-12-10 12:24:52 +03:00
alesapin
0704b478ee Less constuctors 2020-12-10 12:22:43 +03:00
Vladimir Chebotarev
39d867e56d
Added proper authentication for S3 client (#16856) 2020-12-10 12:19:42 +03:00
alesapin
2787b615b0 Make interface as simple as possible 2020-12-10 11:57:52 +03:00
alesapin
fef65b0cbd
Merge pull request #17919 from ClickHouse/additional_size_check
Add additional columns size check for MergeTree in debug mode
2020-12-09 22:40:06 +03:00
alesapin
7979b3655c Better formating 2020-12-09 21:22:07 +03:00
alesapin
3b88c077ed Mechanical refactoring 2020-12-09 21:19:49 +03:00
alesapin
af73db93e6 Mechanical refactoring 2020-12-09 21:10:09 +03:00
Pavel Kruglov
c8c543ca94 Add throwing exceptions 2020-12-09 18:07:58 +03:00
Nikolai Kochetov
32b38f389e Merge branch 'master' into async-read-from-socket 2020-12-09 17:15:36 +03:00
Nikolai Kochetov
088c128832 Fixing build and tests. 2020-12-09 17:11:20 +03:00
Pavel Kovalenko
db7a87089b Adaptive choose of single/multi part upload in WriteBufferFromS3. 2020-12-09 17:09:04 +03:00
alesapin
b307e545a9 Fix check 2020-12-09 14:46:04 +03:00
alesapin
0f4056fd95 Add additional size check in debug mode 2020-12-09 14:23:37 +03:00
alesapin
7783ddb3af
Revert "Fix index granularity calculation on block borders" 2020-12-09 14:05:48 +03:00
Vasily Nemkov
59fc301344 Fixed test to be less flaky
Also logging expanded list of columns passed from `DEDUPLICATE BY` to actual deduplication routines.
2020-12-08 19:44:34 +03:00
Alexander Tokmakov
04ceaa66f2 implement ATTACH TABLE name FROM 'path/to/data/ 2020-12-08 17:15:23 +03:00
Ivan
0ba712a2d3
Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-12-08 16:45:51 +03:00
Kruglov Pavel
e19eb6f17a
Merge branch 'master' into optimize-data-on-insert 2020-12-08 15:57:46 +03:00
alexey-milovidov
ab10cb4bf9
Merge pull request #17707 from ClickHouse/quota-by-x-forwarded-for
Allow quotas to be keyed by proxy-forwarded IP address; add support for PROXY protocol
2020-12-08 14:48:05 +03:00
Vasily Nemkov
8c5daf0925 Fixed building tests with GCC-10
Also a minor cleanup of the test code.
2020-12-08 13:15:18 +03:00
Nikolai Kochetov
7d77a7ba24
Merge pull request #16952 from Enmk/DateTime64_fixes_comparison
Fixed comparison of DateTime64 with different scales
2020-12-08 12:40:41 +03:00
Anton Popov
06d5b87bc9 fix nested and subcolumns 2020-12-07 22:02:26 +03:00
Anton Popov
d7aad3bf79 Merge remote-tracking branch 'upstream/master' into HEAD 2020-12-07 19:13:01 +03:00
alesapin
b5d2559626
Merge pull request #17709 from kssenii/better-startup
rabbitmq: fix startup with no connection
2020-12-07 19:08:18 +03:00
Alexey Milovidov
1ed329017b Merge branch 'master' into quota-by-x-forwarded-for 2020-12-07 18:22:19 +03:00
Vasily Nemkov
168155eeec Minor: cleanup 2020-12-07 18:07:40 +03:00
Vasily Nemkov
dbdc018ab8 Fixed and refined unite-test
* no more undefined values for attributes in ReplicatedMergeTreeLogEntry
* validation of string serialization format
2020-12-07 17:38:25 +03:00
Anton Popov
66e0add2ba fix nested 2020-12-07 16:35:12 +03:00
feng lv
50b64ba1f8 fix 2020-12-07 13:18:31 +00:00
feng lv
d7b30f74d1 add min-max_compress_block_size to mergetree settings 2020-12-07 12:47:51 +00:00
root
e07374331a add diagnostic information when two merge tables try to read each other's data 2020-12-07 15:23:58 +08:00
Vasily Nemkov
70ea507dae OPTIMIZE DEDUPLICATE BY columns
Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on.

Following syntax variants are now supported:

OPTIMIZE TABLE table DEDUPLICATE; -- the old one
OPTIMIZE TABLE table DEDUPLICATE BY *;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY * EXCEPT (colX, colY);
OPTIMIZE TABLE table DEDUPLICATE BY col1,col2,col3;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex');
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT colX;
OPTIMIZE TABLE table DEDUPLICATE BY COLUMNS('column-matched-by-regex') EXCEPT (colX, colY);

Note that * behaves just like in SELECT: MATERIALIZED, and ALIAS columns are not used for expansion.
Also, it is an error to specify empty list of columns, or write an expression that results in an empty list of columns, or deduplicate by an ALIAS column.
Column transformers other than EXCEPT are not supported.
2020-12-07 09:44:07 +03:00
feng lv
a634e4ed80 fix incorrect initialize MergeTreeWriterSettings 2020-12-06 08:57:02 +00:00
kssenii
8f1ed36897 Fix races 2020-12-06 00:24:49 +00:00
alesapin
27c3301083
Merge pull request #17800 from nvartolomei/nv/waitForAllReplicasToProcessLogEntry-foreign-shard
Update StorageReplicatedMergeTree::waitForAllReplicasToProcessLogEntry to support waiting on foreign shards / tables
2020-12-05 16:15:46 +03:00
alesapin
0eec52b1fd
Merge pull request #17737 from ClickHouse/fix_segfault_in_distributed_out_stream
Fix segfault when 'not enough space'
2020-12-05 16:14:30 +03:00
Anton Popov
d1ef6a897e
Merge pull request #17802 from detailyang/patch-1
hotfix:check in_memory_parts_enable_wal
2020-12-05 03:05:38 +03:00
Anton Popov
60b0cbb1c1
Merge pull request #15939 from Avogar/optimize_final_optimization
Optimize final optimization
2020-12-05 02:26:27 +03:00
Pavel Kruglov
5ae6c6dab9 Fix build error 2020-12-04 20:40:28 +03:00
Pavel Kruglov
4b58528b9e Rename getPos to getRow, change mergeBlock, pass setting instead of context 2020-12-04 19:25:30 +03:00
Pavel Kruglov
905ba78adc Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-12-04 18:56:46 +03:00
detailyang
e7a151fd1e
hotfix:check in_memory_parts_enable_wal 2020-12-04 23:14:27 +08:00
kssenii
448463ebe0 Better 2020-12-04 14:50:42 +00:00
alesapin
42544a9833 Merge branch 'master' into fix_segfault_in_distributed_out_stream 2020-12-04 17:41:10 +03:00
Pavel Kruglov
9dbced0474 Pass setting instead of context 2020-12-04 17:01:59 +03:00
Nicolae Vartolomei
796aee032d Update StorageReplicatedMergeTree::waitForAllReplicasToProcessLogEntry to support waiting on foreign shards / tables
This is not used anywhere yet but needed for an upcoming PR for part movement between shards.
2020-12-04 13:01:12 +00:00
Nikolai Kochetov
f31610a2e1 Remove Wait status. 2020-12-04 13:55:01 +03:00
roverxu
2a6c1463a1 Add metrics for part number in MergeTree 2020-12-04 11:42:58 +08:00
Ivan
315ff4f0d9
ANTLR4 Grammar for ClickHouse and new parser (#11298) 2020-12-04 05:15:44 +03:00
kssenii
f16d6a4128 Better 2020-12-03 14:11:35 +00:00
Anton Popov
cab9855dd1
Update StorageReplicatedMergeTree.cpp 2020-12-03 16:54:05 +03:00
Anton Popov
cd1917c7a6
Merge branch 'master' into optimize_final_optimization 2020-12-03 16:52:51 +03:00
kssenii
ebab21178e Fix, add test 2020-12-02 18:45:30 +00:00
Alexander Kuzmenkov
8ee86e35d2 debug 2020-12-02 21:16:31 +03:00
Azat Khuzhin
5365718f01
Fix optimize_distributed_group_by_sharding_key for query with OFFSET only (#16996)
* Fix optimize_distributed_group_by_sharding_key for query with OFFSET only

* Fix 01244_optimize_distributed_group_by_sharding_key flakiness
2020-12-02 20:11:39 +03:00
Alexander Tokmakov
a0ef3a113a Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-12-02 18:29:02 +03:00
Alexander Tokmakov
bfbf150c67 fix segfault when 'not enough space' 2020-12-02 17:49:43 +03:00
kssenii
8d3a538629 Better startup 2020-12-02 01:17:50 +00:00
Alexey Milovidov
c9aa412151 Allow quotas to be keyed by proxy-forwarded IP address 2020-12-02 00:09:16 +03:00
Vasily Nemkov
87945fb68b Merge branch 'master' into HEAD 2020-11-30 23:25:57 +03:00
Alexey Milovidov
6590fdfdd2 Fix logging 2020-11-30 11:44:01 +03:00
alesapin
25f40db2fb
Merge pull request #17499 from ClickHouse/concurrent_mutation_and_random_kill
Fix kill mutation on concurrent alter queries
2020-11-30 10:51:50 +03:00
Alexey Milovidov
2e6bedce91 Relax "Too many parts" threshold, part 1 2020-11-29 18:08:02 +03:00
Alexey Milovidov
4a66db1bf4 Fix #10976 2020-11-29 16:43:24 +03:00
alexey-milovidov
fabceebbce
Merge pull request #17145 from amosbird/cddt
Fix unmatched type comparison in KeyCondition
2020-11-29 14:29:35 +03:00
alexey-milovidov
f4a61ac3c3
Merge pull request #17527 from ucasFL/spelling
fix spelling errors
2020-11-29 13:45:42 +03:00
feng lv
7e3524caa1 fix spelling errors 2020-11-28 08:17:20 +00:00
alesapin
cc66e4fdd6
Merge pull request #17496 from ClickHouse/fix_mark_cache_size
More accurate mark cache size calculation
2020-11-28 10:42:56 +03:00
alexey-milovidov
c189f6405f
Merge pull request #16767 from azat/optimize_trivial_count_query-fix
Fix optimize_trivial_count_query with partition predicate
2020-11-28 08:43:12 +03:00
Nikita Mikhaylov
6f3db7ff50
Merge pull request #17330 from CurtizJ/move-to-prewhere
Allow to move conditions to prewhere with compact parts
2020-11-28 02:02:42 +03:00
tavplubix
aaaa7ad59f
Merge pull request #16806 from TCeason/add_test_for_materialize_mysql
add network partition integration test for MaterializeMySQL
2020-11-27 20:39:06 +03:00
alesapin
6567796014 Fix kill mutation on concurrent alter queries 2020-11-27 18:46:52 +03:00
alesapin
ba481e6665 Fix cache size calculation 2020-11-27 16:17:10 +03:00
Anton Popov
1a4ed07a98 add some more comments 2020-11-27 15:47:27 +03:00
Nikolai Kochetov
85283b3944
Merge pull request #17397 from ClickHouse/fix-types-for-read-from-join
Fix crash while reading from JOIN table with LowCardinality types
2020-11-27 13:37:05 +03:00
alexey-milovidov
dfae1efbbd
Merge pull request #17070 from fastio/master
Support multiple ZooKeeper clusters
2020-11-27 10:38:01 +03:00
alexey-milovidov
1cd09fa367
Merge pull request #15127 from ucasFL/add-mutation-for-storagememory
Add mutation support for StorageMemory
2020-11-26 22:22:59 +03:00
alexey-milovidov
ee80ee75de
Update StorageMemory.cpp 2020-11-26 22:22:26 +03:00
alesapin
6719fb3a1d
Merge pull request #17416 from ClickHouse/tavplubix-patch-2
Fix backoff in BackgroundJobExecutor
2020-11-26 16:43:15 +03:00
Nikita Mikhaylov
084c75fa6e
Merge pull request #17309 from nikitamikhaylov/merging-sequential-consistency
Merging #16309
2020-11-26 15:26:05 +03:00
Amos Bird
022ba2b0a9
Fix unmatched type comparison in KeyCondition 2020-11-26 16:15:50 +08:00
alesapin
960d077612 Better functions 2020-11-26 10:25:57 +03:00
alexey-milovidov
75a78e6c20
Update BackgroundJobsExecutor.h 2020-11-26 07:09:05 +03:00
Azat Khuzhin
0b47f4a9e9 Fix optimize_trivial_count_query with partition predicate
Consider the following example:

    CREATE TABLE test(p DateTime, k int) ENGINE MergeTree PARTITION BY toDate(p) ORDER BY k;
    INSERT INTO test VALUES ('2020-09-01 00:01:02', 1), ('2020-09-01 20:01:03', 2), ('2020-09-02 00:01:03', 3);

- SELECT count() FROM test WHERE toDate(p) >= '2020-09-01' AND p <= '2020-09-01 00:00:00'
  In this case rpn will be (FUNCTION_IN_RANGE, FUNCTION_UNKNOWN (due to strict), FUNCTION_AND)
  and for optimize_trivial_count_query we cannot use index if there is at least one FUNCTION_UNKNOWN.
  since there is no post processing and return count() based on only the first predicate is wrong.

  Before this patch FUNCTION_UNKNOWN was allowed for optimize_trivial_count_query, and the result was wrong.

And two examples above just to show the difference, the behaviour hadn't been changed with this patch:

- SELECT * FROM test WHERE toDate(p) >= '2020-09-01' AND p <= '2020-09-01 00:00:00'
  In this case will be (FUNCTION_IN_RANGE, FUNCTION_IN_RANGE (due to non-strict), FUNCTION_AND)
  so it will prune everything out and nothing will be read.

- SELECT * FROM test WHERE toDate(p) >= '2020-09-01' AND toUnixTimestamp(p)%5==0
  In this case will be (FUNCTION_IN_RANGE, FUNCTION_UNKNOWN, FUNCTION_AND)
  and all, two, partitions will be scanned, but due to filtering later none of rows will be matched.
2020-11-25 23:09:17 +03:00
tavplubix
6477251ea1
Update BackgroundJobsExecutor.h 2020-11-25 23:06:00 +03:00
nikitamikhaylov
72c7cd6693 replace Context& to Settings& 2020-11-25 16:47:32 +03:00
Nikolai Kochetov
89ee8aefd9 Cast columns to sample structure for StirageJoin::read 2020-11-25 15:41:12 +03:00
Alexander Kuzmenkov
edce1e636e
Merge pull request #16535 from ClickHouse/aku/opentelemetry
OpenTelemetry improvements
2020-11-25 14:10:17 +03:00
alesapin
3e1b2f515b
Merge pull request #17311 from ClickHouse/fix_integration_tests
Fix some flaky tests
2020-11-25 10:09:27 +03:00
alesapin
7e014967db Less broken ugly hack 2020-11-25 00:07:56 +03:00
Alexander Kuzmenkov
1d7dcef4a0 Merge remote-tracking branch 'origin/master' into HEAD 2020-11-24 21:45:10 +03:00
Alexander Kuzmenkov
8de415861f
Merge pull request #17206 from ClickHouse/aku/noescape
Remove escaping from toString(std::string)
2020-11-24 21:23:39 +03:00
alesapin
b28fc3d4f6 Ugly fix for dangling reference 2020-11-24 19:24:36 +03:00
tavplubix
085359c110
Merge pull request #17274 from ClickHouse/fix_ast_formatting_in_logs
Fix AST formatting in log messages
2020-11-24 19:00:56 +03:00
Alexander Kuzmenkov
9b0368ffa5
Merge pull request #16338 from ucasFL/union-distinct
Add UNION DISTINCT and setting `union_default_mode`
2020-11-24 17:30:24 +03:00
Anton Popov
be493ed561 fix nested and subcolumns 2020-11-24 17:14:47 +03:00
Alexander Tokmakov
dce0bf8efc Merge branch 'master' into allow_atomic_database_inside_materialize_mysql 2020-11-24 15:28:31 +03:00
nikitamikhaylov
5be300ff22 Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-sequential-consistency 2020-11-24 14:46:57 +03:00
alesapin
45f09fe293
Merge pull request #17220 from ianton-ru/s3_max_redirects
S3 max redirects
2020-11-24 12:13:10 +03:00
Anton Popov
2fb8717dfd
Merge pull request #17253 from amosbird/globalinfix
global in set index.
2020-11-24 03:06:00 +03:00
Anton Popov
852a08eacb allow to move conditions to prewhere with compact parts 2020-11-23 21:27:59 +03:00
Alexander Kuzmenkov
951a59eff7 Merge remote-tracking branch 'origin/master' into HEAD 2020-11-23 21:00:22 +03:00
alexey-milovidov
084cac5bd5
Update StorageMemory.cpp 2020-11-23 20:45:59 +03:00
Alexander Kuzmenkov
412c2fe079 Merge remote-tracking branch 'origin/master' into HEAD 2020-11-23 19:51:18 +03:00
Anton Popov
b384beb564 Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-23 17:46:51 +03:00
Amos Bird
172b7e9ed1
global in set index. 2020-11-23 22:05:08 +08:00
feng lv
005f345139 Merge branch 'master' of github.com:ClickHouse/ClickHouse into add-mutation-for-storagememory 2020-11-23 13:56:19 +00:00
nikitamikhaylov
68bef22fda Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-sequential-consistency 2020-11-23 16:28:35 +03:00
Vasily Nemkov
c375dc6b14 Merge remote-tracking branch 'upstream/master' into DateTime64_fixes_comparison 2020-11-23 15:49:36 +03:00
Anton Ivashkin
aa208a9d9f Use only 's3_max_redirect' in params instead of all settings 2020-11-23 14:02:17 +03:00
Alexey Milovidov
9969a73d50 Merge branch 'master' into azat-merge-distributed-join-fix 2020-11-23 12:43:15 +03:00
taichong
5c3d4b2245 add network partition integration test for MaterializeMySQL 2020-11-23 15:47:44 +08:00
robot-clickhouse
dc19fba36f Auto version update to [20.13.1.1] [54444] 2020-11-23 10:29:26 +03:00
feng lv
18491d8920 fix 2020-11-23 05:30:36 +00:00
Alexey Milovidov
fbbf439338 Merge branch 'master' into Enmk-DateTime64_fixes_comparison 2020-11-23 05:46:17 +03:00
Alexander Tokmakov
60a5782c75 fix AST formatting in log messages 2020-11-22 20:23:12 +03:00
tavplubix
5cc9cb01cd
Merge pull request #16751 from amosbird/globalcontext
Make global_context consistent.
2020-11-22 18:46:17 +03:00
alesapin
9c8b0da382
Merge pull request #16033 from nvartolomei/nv/parts-uuid
Add unique identifiers IMergeTreeDataPart structure
2020-11-22 16:13:19 +03:00
sundy-li
fe8b11fd5f add replaceAliasColumnsInFilter function 2020-11-22 12:16:27 +08:00
Anton Popov
1f65478b49 implement nested with multiple nesting 2020-11-21 20:27:42 +03:00
sundy-li
f4808df41e add tests for second indexes 2020-11-21 23:04:39 +08:00
sundy-li
128785e187 do not pass columns into indexes 2020-11-21 22:31:31 +08:00
sundy-li
11907eca8f rewrite in excutor 2020-11-21 21:29:51 +08:00
sundy-li
ad99dab3ad add ColumnAliasesVisitor to improve partition prune for mergetree 2020-11-21 20:38:21 +08:00
Nikolai Kochetov
cf678b6092 Merge branch 'master' into order-inputs-in-action-dag 2020-11-21 13:54:50 +03:00
Pavel Kruglov
623dc2df7a Update comment 2020-11-20 17:32:39 +03:00
Pavel Kruglov
ca3fe49a2a Make setting global 2020-11-20 17:29:13 +03:00
Nicolae Vartolomei
7c8bc1c04e Use JSON metadata in WAL 2020-11-20 13:49:17 +00:00
Nicolae Vartolomei
040aba9f85 Add uuid.txt to checksums for parts stored on disk
We are breaking backwards compatibility anyway (but agted by a setting)
2020-11-20 13:49:17 +00:00
Nicolae Vartolomei
94293ca3ce Assign UUIDs to parts only when configured to do so
Avoid breaking backwards compatibility by default for now.
2020-11-20 13:49:17 +00:00
Pavel Kruglov
e5588b94aa Fix style and tests 2020-11-20 13:38:53 +03:00
Amos Bird
1d9d586e20
Make global_context consistent. 2020-11-20 18:23:14 +08:00
Alexander Kuzmenkov
b05ac888bd Merge remote-tracking branch 'origin/master' into HEAD 2020-11-20 12:50:08 +03:00
Pavel Kruglov
625962fb59 Remove debug logging 2020-11-20 03:19:36 +03:00
Pavel Kruglov
3ee4972586 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-11-20 01:23:51 +03:00
Pavel Kruglov
4c30857759 Minor change 2020-11-20 01:22:40 +03:00
Azat Khuzhin
5297417050 Fix Merge(Distributed()) with JOIN 2020-11-20 00:07:30 +03:00
Alexander Kuzmenkov
0d43e4c448 fixup 2020-11-19 20:16:45 +03:00
alesapin
3f01096c86 Less verbose logging when fetch is impossible 2020-11-19 20:03:20 +03:00
alesapin
2623d35f68
Merge pull request #17120 from ClickHouse/fix_granularity_on_block_borders
Fix index granularity calculation on block borders
2020-11-19 18:36:07 +03:00
Nicolae Vartolomei
746f8e45f5 All new parts must have uuids 2020-11-19 13:18:03 +00:00
Nicolae Vartolomei
425dc4b11b Add unique identifiers IMergeTreeDataPart structure
For now uuids are not generated at all, they are present only if the
part is updated manually (as you can see in the integration test).

The only place where they can be seen today by an end user is in
`system.parts` table. I was looking for hiding this column behind an
option but couldn't find an easy way to do that.

Likely this is also required for WAL, but need to think how not to break
compatibility.

Relates to #13574, https://github.com/ClickHouse/ClickHouse/issues/13574

Next 1: In the upcoming PR the plan is to integrate de-duplication based on
these fingerprints in the query pipeline.

Next 2: We'll enable automatic generation of uuids and come up with a
way for conditionally sending uuids when processing distributed queries
only when part movement is in progress.
2020-11-19 13:14:25 +00:00
Alexander Kuzmenkov
bcf15170ed Merge remote-tracking branch 'origin/master' into HEAD 2020-11-19 15:29:51 +03:00
Peng Jian
10cefe4f51 fix code style 2020-11-19 15:44:47 +08:00
Peng Jian
3c86c8b3c9 fix test cases 2020-11-19 15:44:47 +08:00
Peng Jian
091f7065cd fix build 2020-11-19 15:44:47 +08:00
Peng Jian
2df0c3e0b0 Add helper function for extracting zookeeper name and path. 2020-11-19 15:44:47 +08:00
Peng Jian
a0683ce460 Support mulitple ZooKeeper clusters 2020-11-19 15:44:47 +08:00
Alexander Kuzmenkov
f8fd365d15 compile 2020-11-18 20:49:51 +03:00
Alexander Kuzmenkov
1570320e20 fixes for context hierarchy 2020-11-18 20:43:18 +03:00
alesapin
7080f424e2 Fix bug for skip indices and make code more complex 2020-11-18 15:04:13 +03:00
Pavel Kruglov
547ec19fb3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize-data-on-insert 2020-11-18 12:01:59 +03:00
filimonov
258170b325
Update ReplicatedMergeTreeMergeStrategyPicker.cpp 2020-11-18 08:45:44 +01:00
Mikhail Filimonov
234c671e52
After CR fixes 2020-11-18 08:45:44 +01:00
Mikhail Filimonov
97fef77ed1
execute_merges_on_single_replica 2020-11-18 08:45:33 +01:00
Nikolai Kochetov
13eceff55c Remove Converting transform and step. 2020-11-17 20:16:55 +03:00
Pavel Kruglov
53b8ff42c2 Add test and set setting to 0 in some other tests 2020-11-17 19:56:36 +03:00
alesapin
389b883534 Fix index granularity calculation on block borders 2020-11-17 12:37:13 +03:00
alexey-milovidov
b494b18748
Merge pull request #17108 from ClickHouse/ya-make-owners
Add "OWNER" to ya.make files.
2020-11-17 05:19:46 +03:00
Alexey Milovidov
24f4fa6edf Follow Arcadia ya.make rules 2020-11-17 00:16:50 +03:00
Alexey Milovidov
3df04ce0c2 Follow Arcadia ya.make rules 2020-11-16 21:24:58 +03:00
tavplubix
88de1b052c
Merge pull request #16048 from ClickHouse/materialized-view-check-stack-size
Fix possible stack overflow in materialized views.
2020-11-16 12:02:27 +03:00
Alexander Tokmakov
5c6191f798 fix 2020-11-16 00:31:52 +03:00
Anton Popov
de5ead0c40 add setting remove_empty_parts 2020-11-15 05:24:47 +03:00
alexey-milovidov
915d96fee0
Merge pull request #16955 from nikitamikhaylov/fix-typos
Fix typos
2020-11-14 17:17:37 +03:00
alexey-milovidov
8ef4bfc0b5
Merge pull request #16968 from amosbird/blocktypemismatch
Fix missing conversion of unmutated parts
2020-11-14 17:16:51 +03:00
Vasily Nemkov
e8ef167d78 Fixed 01087_table_function_generate for DateTime64 2020-11-14 09:28:52 +03:00
Anton Ivashkin
15b5d3a55a Merge master 2020-11-13 19:32:35 +03:00
Anton Ivashkin
026f7e0a27 Add 's3_max_redirects' setting 2020-11-13 19:31:51 +03:00
Nikolai Kochetov
ed98b48845
Merge pull request #16618 from ClickHouse/actions-dag-f14
Actions dag 2
2020-11-13 18:34:51 +03:00
alesapin
b2a8b6135d
then 2020-11-13 16:24:14 +03:00
alesapin
b8e33bd35b Add comments 2020-11-13 15:42:26 +03:00
Pavel Kruglov
264e9daf6f Fix style 2020-11-13 11:16:30 +03:00
Pavel Kruglov
6a57c0a8cf Move merge in MergeTreeDataWriter 2020-11-13 10:55:56 +03:00
alesapin
8d3858fc22
Merge pull request #16895 from CurtizJ/remove-empty-parts
Remove empty parts after they were pruned by TTL [2]
2020-11-13 10:00:18 +03:00
Amos Bird
fdf979d5ce
Fix missing conversion of unmutated parts 2020-11-13 11:45:50 +08:00
nikitamikhaylov
c0940f2ebd done 2020-11-13 02:27:18 +03:00
Pavel Kruglov
8d5e0784d3 Add setting optimize_on_insert 2020-11-12 23:37:23 +03:00
alesapin
bbd85c3b2d
Merge pull request #16390 from ClickHouse/add_separate_pool_for_fetches
Add separate pool for fetches
2020-11-12 22:42:28 +03:00
Anton Popov
4cecd89e84 fix dropping of empty parts 2020-11-12 20:37:54 +03:00
tavplubix
e0dc004b95
Merge pull request #16885 from amosbird/dd
Disable ATTACH/DETACH TABLE <DICTIONARY>
2020-11-12 16:34:21 +03:00
Anton Popov
0aa35d53c7 Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-12 14:31:58 +03:00
feng lv
b1cc582668 fix conflict 2020-11-12 10:42:53 +00:00
alesapin
2cac5e5d97
Merge pull request #16875 from CurtizJ/refactor-alter-partition
Avoid code duplication in alterPartition
2020-11-12 13:19:59 +03:00
alesapin
a8d466a645 Merge branch 'master' into add_separate_pool_for_fetches 2020-11-12 12:51:02 +03:00
Nikolai Kochetov
46f70dd0de Merge branch 'master' into actions-dag-f14 2020-11-12 11:54:44 +03:00
feng lv
b7e5c6ed4d fix conflict 2020-11-12 02:26:21 +00:00
sundyli
5ef809931e
Merge branch 'master' into storage-rocksdb 2020-11-12 08:26:27 +08:00
tavplubix
058aa8f85e
Merge pull request #16824 from ClickHouse/replace_stringstreams_with_buffers
Replace std::*stringstreams with DB::*Buffers
2020-11-12 01:11:44 +03:00
Anton Popov
0a6d93d929
Update MergeTreeData.h 2020-11-11 21:17:41 +03:00
Nikolai Kochetov
58c4a7b890 Some renames. 2020-11-11 19:52:27 +03:00
Anton Popov
2497598a49 Remove empty parts after they were pruned by TTL 2020-11-11 19:19:43 +03:00
Anton Popov
c01870076d
Update MergeTreeData.h 2020-11-11 19:09:01 +03:00
Amos Bird
134786afd5
Disable ATTACH/DETACH TABLE <DICTIONARY> 2020-11-11 23:04:49 +08:00
Anton Popov
6f0dc08ba7 avoid code duplication in alterPartition 2020-11-11 16:34:07 +03:00
Alexander Tokmakov
7da5e92754 fix 2020-11-11 13:41:12 +03:00
Pavel Kruglov
148dc71d0e Fix StorageMergeTree.h 2020-11-11 13:38:00 +03:00
Pavel Kruglov
287d71b363 Fix build 2020-11-11 13:34:32 +03:00
Nikolai Kochetov
1846bb3cac Merge branch 'master' into actions-dag-f14 2020-11-11 13:08:57 +03:00
sundy-li
579f4c50ed improve rocksdb-cmake build 2020-11-11 17:39:23 +08:00
sundy-li
d8376a2d60 Merge branch 'storage-rocksdb' of github.com:sundy-li/ClickHouse into storage-rocksdb 2020-11-11 13:19:30 +08:00
feng lv
fec4080d42 Merge branch 'master' of github.com:ClickHouse/ClickHouse into union-distinct 2020-11-11 03:10:10 +00:00
sundy-li
c56d1212a2 rocksdb-cmake && fix iteration bug && improve tests 2020-11-11 09:39:09 +08:00
Alexey Milovidov
765c9eaeed Fix improper error handling 2020-11-11 09:08:53 +08:00
Alexey Milovidov
07fbab9645 More normal 2020-11-11 09:08:53 +08:00
Alexey Milovidov
33599ae9b0 Remove unneeded dir 2020-11-11 09:08:53 +08:00
Alexey Milovidov
73e2f1b7ed Improve performance two times 2020-11-11 09:08:53 +08:00
Alexey Milovidov
0001433b82 Better code 2020-11-11 09:08:53 +08:00
Alexey Milovidov
3915d65c2f Fix bad wording 2020-11-11 09:08:53 +08:00
Alexey Milovidov
6376c8db7e Make code slightly better 2020-11-11 09:08:53 +08:00
Alexey Milovidov
742ffdd145 Update ya.make 2020-11-11 09:08:53 +08:00
Alexey Milovidov
630aa9b0c3 Rename Rocksdb to RocksDB 2020-11-11 09:08:53 +08:00
sundy-li
46c01eb7a2 improve unbundle build 2020-11-11 09:08:53 +08:00
sundy-li
fb7066d73d * fix fasttest and cmake && pipline for all_scan
* unique the keys
* add inputstream && outputstream
2020-11-11 09:08:53 +08:00
sundy-li
f95a66b584 add supportsIndexForIn && supportsParallelInsert, support parallel reading 2020-11-11 09:08:52 +08:00
sundy-li
fced1a0e28 fix extract keys 2020-11-11 09:08:52 +08:00
sundy-li
c7617173c4 add sync_file_range for glibc-compatibility 2020-11-11 09:08:52 +08:00
sundy-li
84dab52e45 skip rocksdb test in fasttest 2020-11-11 09:08:52 +08:00
sundy-li
b092ebe40d add primary key for StorageEmbeddedRocksdb 2020-11-11 09:08:52 +08:00
sundy-li
1fa4906dd2 update cmake && remove useless codes 2020-11-11 09:08:52 +08:00
sundy-li
6f98a9e1c8 fix styles 2020-11-11 09:08:52 +08:00
root
e9de5b6ad4 Add StorageEmbeddedRocksdb Engine 2020-11-11 09:08:52 +08:00
Pavel Kruglov
dd06ff4f94 Minor change 2020-11-11 00:02:11 +03:00
Pavel Kruglov
1c35b47073 Add checking select decision in merge 2020-11-10 23:52:26 +03:00
Pavel Kruglov
c648f62629 Merge branch 'master' of github.com:ClickHouse/ClickHouse into optimize_final_optimization 2020-11-10 22:58:21 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
Nikolai Kochetov
c6575c9032 Update ExpressionActions constructur 2020-11-10 19:27:55 +03:00
Nikolai Kochetov
1db8e77371 Add comments. Update ActionsDAG::Index 2020-11-10 17:54:59 +03:00
Pavel Kruglov
9120189d8a Add SelectPartsDecision enum class 2020-11-10 17:42:56 +03:00
Anton Popov
9b354be66f Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-10 16:21:11 +03:00
Nikolai Kochetov
07fe3a6347 Fix build. 2020-11-10 15:14:05 +03:00
Nikolai Kochetov
195c941c4e Merge branch 'master' into storage-read-query-plan 2020-11-10 15:02:22 +03:00
alesapin
2e357516a6 Merge branch 'master' into add_separate_pool_for_fetches 2020-11-10 13:53:08 +03:00
Nikolai Kochetov
046830967c Try fix tests. 2020-11-10 13:26:26 +03:00
Vladimir Chebotarev
059357d51e
ALTER UPDATE/DELETE ... IN PARTITION with partition pruning in ReplicatedMergeTree (#13403)
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-11-10 13:23:46 +03:00
Nikolai Kochetov
363c1e05c0 Try fix tests. 2020-11-10 12:35:05 +03:00
feng lv
9194985d48 Merge branch 'master' of github.com:ClickHouse/ClickHouse into union-distinct 2020-11-10 06:09:39 +00:00
Alexander Tokmakov
5cdfcfb307 remove other stringstreams 2020-11-09 22:12:44 +03:00
Nikolai Kochetov
e41b1ae52b Empty commit. 2020-11-09 19:35:43 +03:00
Alexander Tokmakov
62ff00ee8b use WriteBuffer in formatAST(...) 2020-11-09 19:05:40 +03:00
alesapin
ea7fe94aa9
Merge pull request #15511 from nvartolomei/nv/drop-detached-part-quorum-possible-solution
ALTER TABLE ... DROP|DETACH PART
2020-11-09 18:00:30 +03:00
alexey-milovidov
e47ed55d32
Merge pull request #14977 from achimbab/indexOf_bloomfilter
Fix indexOf() to use BloomFilter
2020-11-09 17:58:42 +03:00
Nikolai Kochetov
99cc9b1ec0 Fix build 2020-11-09 16:20:56 +03:00
Anton Popov
829c7c99ea implement nested with multiple nesting 2020-11-09 15:26:24 +03:00
Nikolai Kochetov
8c4db34f9d Update after merge. 2020-11-09 14:58:11 +03:00
Nikolai Kochetov
6717c7a0af Merge branch 'master' into actions-dag-f14 2020-11-09 14:57:48 +03:00
tavplubix
67099f28ac
Merge pull request #16591 from ClickHouse/aku/create-file
Support `SETTINGS` clause for File engine
2020-11-09 14:15:42 +03:00
alesapin
e347f4b72f Add some comments 2020-11-09 12:14:20 +03:00
alesapin
72046313d4 Merge branch 'master' into nvartolomei-drop-part 2020-11-09 10:02:02 +03:00
alesapin
4e846d2197 Merge branch 'master' into add_separate_pool_for_fetches 2020-11-09 10:00:21 +03:00
alexey-milovidov
f39457bc77
Merge pull request #16788 from azat/fix-use_compact_format_in_distributed_parts_names
Apply use_compact_format_in_distributed_parts_names for each INSERT (with internal_replication)
2020-11-08 23:23:10 +03:00
Alexey Milovidov
8bb10ac3d8 Fix improper error handling 2020-11-08 20:29:16 +03:00
Alexey Milovidov
fa29c2f00d More normal 2020-11-08 20:22:21 +03:00
Alexey Milovidov
e072d18a83 Remove unneeded dir 2020-11-08 20:13:17 +03:00
Alexey Milovidov
1250006e4e Improve performance two times 2020-11-08 20:09:41 +03:00
Alexey Milovidov
2d9f07e601 Better code 2020-11-08 19:24:58 +03:00
alexey-milovidov
0e6ae4aff7
Merge pull request #16253 from amosbird/pf
Prune partition in verbatim way.
2020-11-08 18:58:02 +03:00
Alexey Milovidov
2ea93f4b67 Fix bad wording 2020-11-08 18:52:55 +03:00
Alexey Milovidov
4353a56790 Make code slightly better 2020-11-08 18:45:24 +03:00
Alexey Milovidov
7d0f1d33c6 Update ya.make 2020-11-08 18:43:05 +03:00
Alexey Milovidov
c89a363980 Rename Rocksdb to RocksDB 2020-11-08 18:41:16 +03:00
Alexey Milovidov
e75230dee3 Merge branch 'master' into sundy-li-storage-rocksdb 2020-11-08 17:51:44 +03:00
Alexey Milovidov
3f5839c83c Add one more case 2020-11-08 16:45:57 +03:00
Alexey Milovidov
8e1a2fc9e4 Fix UB and overflow 2020-11-08 16:41:25 +03:00
alexey-milovidov
8264bd900b
Update MergeTreeIndexConditionBloomFilter.cpp 2020-11-08 16:17:03 +03:00
alexey-milovidov
85f7eadc4f
Update MergeTreeIndexConditionBloomFilter.cpp 2020-11-08 16:14:32 +03:00
alexey-milovidov
dfa35a5e6d
Update MergeTreeIndexConditionBloomFilter.cpp 2020-11-08 16:11:18 +03:00
Alexey Milovidov
cc012e2cc7 Merge branch 'master' into achimbab-indexOf_bloomfilter 2020-11-08 14:42:03 +03:00
alexey-milovidov
2d427eb1e8
Merge pull request #16496 from ClickHouse/register-formats
Fix inconsistency in FormatFactory
2020-11-08 14:27:33 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream
Fix "server failed to start" error
2020-11-08 14:27:08 +03:00
Azat Khuzhin
04db0834bf Apply use_compact_format_in_distributed_parts_names for each INSERT (with internal_replication)
Before this patch use_compact_format_in_distributed_parts_names was
applied only from default profile (at server start) for
internal_replication=1, and was ignored on INSERT.
2020-11-08 03:05:52 +03:00
Alexey Milovidov
ba4ae00121 Whitespace 2020-11-08 00:30:40 +03:00
Alexey Milovidov
1ea3afadbc Merge with master 2020-11-08 00:28:39 +03:00
Alexey Milovidov
5314185e25 Merge branch 'master' into azat-optimize_skip_unused_shards-optimization 2020-11-08 00:17:59 +03:00
Alexey Milovidov
dba57529f8 Merge branch 'master' of github.com:yandex/ClickHouse into fix-stringstream 2020-11-07 22:03:38 +03:00
feng lv
1ca7dcb0a8 fix conflict 2020-11-07 11:44:29 +00:00
alexey-milovidov
8dbd04a229
Merge pull request #16727 from azat/distribution_queue-masking
Mask password in data_path in the system.distribution_queue
2020-11-07 13:40:41 +03:00
Alexander Kuzmenkov
79f759fb6c Merge remote-tracking branch 'origin/master' into HEAD 2020-11-07 11:54:11 +03:00
Alexander Kuzmenkov
3c60f6cec2 make a separate settings collection + some cleanup 2020-11-07 11:53:39 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Anton Popov
245c395a68 Merge remote-tracking branch 'upstream/master' into HEAD 2020-11-06 22:00:32 +03:00
Anton Popov
01a04e06ef implement nested with multiple nesting 2020-11-06 21:57:04 +03:00
Azat Khuzhin
c057e86040 Fix path masking in system.distribution_queue for multiple nodes 2020-11-06 21:04:50 +03:00
sundy-li
e55824c3df improve unbundle build 2020-11-07 00:28:11 +08:00
sundy-li
94cb974f42 * fix fasttest and cmake && pipline for all_scan
* unique the keys
* add inputstream && outputstream
2020-11-06 23:47:26 +08:00
sundy-li
c88ede4f97 add supportsIndexForIn && supportsParallelInsert, support parallel reading 2020-11-06 23:47:01 +08:00
sundy-li
8dd9d166dc fix extract keys 2020-11-06 23:47:01 +08:00
sundy-li
88250fec5d add sync_file_range for glibc-compatibility 2020-11-06 23:46:44 +08:00
sundy-li
1f4a63b6c6 skip rocksdb test in fasttest 2020-11-06 23:46:44 +08:00
sundy-li
09bcc16669 add primary key for StorageEmbeddedRocksdb 2020-11-06 23:46:05 +08:00
sundy-li
546d7da12e update cmake && remove useless codes 2020-11-06 23:45:41 +08:00
sundy-li
2f5e903c10 fix styles 2020-11-06 23:45:41 +08:00
root
e474bcca8e Add StorageEmbeddedRocksdb Engine 2020-11-06 23:45:13 +08:00
alexey-milovidov
7fb53b205c
Merge pull request #16637 from azat/mt-read_in_order-spread-fix
Fix spreading for ReadInOrderOptimizer with expression in ORDER BY
2020-11-06 17:36:03 +03:00
Nikolai Kochetov
c10f733587 Merge branch 'master' into storage-read-query-plan 2020-11-06 15:43:46 +03:00
Nikolai Kochetov
9aeb757da4 Merge branch 'master' into actions-dag-f14 2020-11-06 15:04:20 +03:00
Amos Bird
2b0085c106
Pruning is different from counting 2020-11-06 19:58:03 +08:00
Amos Bird
aa436a3cb1
Transform single point 2020-11-06 14:59:55 +08:00
Amos Bird
30bf5e6d26
Prune partition in verbatim way. 2020-11-06 09:56:13 +08:00
Azat Khuzhin
b2e17916e4 Mask password in data_path in the system.distribution_queue 2020-11-06 00:10:25 +03:00
Azat Khuzhin
f23995d290 Remove empty directories for async INSERT at start of Distributed engine
Will be created by DistributedBlockOutputStream on demand.
2020-11-05 23:50:30 +03:00
Azat Khuzhin
59cdc964a1 Do not store reference to BackgroundSchedulePool in DirectoryMonitor (useless) 2020-11-05 23:43:34 +03:00
robot-clickhouse
f7b7854ace Auto version update to [20.12.1.1] [54443] 2020-11-05 21:52:50 +03:00
alexey-milovidov
ee3e289ec2
Merge pull request #15983 from ClickHouse/no_background_pool_no_more
Replace BackgroundProcessingPool with SchedulePool task and ThreadPool.
2020-11-05 21:38:22 +03:00
alexey-milovidov
39cd1c0371
Merge pull request #16700 from ClickHouse/better-arcadia
Improve Arcadia
2020-11-05 15:34:34 +03:00
Anton Popov
bea0984d12
Merge pull request #16642 from kitaisreal/bloom-filter-index-added-big-int-and-uuid-support
BloomFilter index added big integers and UUID support
2020-11-05 15:26:08 +03:00
Alexander Kuzmenkov
2efbcbaa14 Merge remote-tracking branch 'origin/master' into HEAD 2020-11-05 14:28:56 +03:00
Alexander Kuzmenkov
14f31f5117 add support of settings to URL, fix url and file table functions 2020-11-05 14:28:20 +03:00
Alexey Milovidov
b56486510f Improve Arcadia 2020-11-05 12:55:01 +03:00
Alexander Kuzmenkov
faf1f1e0cd Merge remote-tracking branch 'origin/master' into HEAD 2020-11-05 11:09:11 +03:00
Maksim Kita
7dd749fc4c BloomFilter index added big integers and UUID support 2020-11-04 23:41:10 +03:00
Alexey Milovidov
1bcf22d42f Fix 'max_parallel_replicas' without sampling. 2020-11-04 18:59:14 +03:00
Alexey Milovidov
f7c77b4a25 Fix unit tests 2020-11-04 13:14:23 +03:00
alexey-milovidov
4fede5b008
Merge pull request #16438 from azat/errorCode-v2
Add errorCodeToName() and system.errors
2020-11-04 12:40:47 +03:00
alexey-milovidov
0456cb2a09
Merge pull request #16638 from azat/simplify-ReadInOrderOptimizer
[RFC] Do not pass StoragePtr to ReadInOrderOptimizer::getInputOrder()
2020-11-04 11:29:06 +03:00
alesapin
20043e77bb Merge branch 'master' into nvartolomei-drop-part 2020-11-04 10:22:23 +03:00
Azat Khuzhin
2389406c21 Fix spreading for ReadInOrderOptimizer with expression in ORDER BY
This will fix optimize_read_in_order/optimize_aggregation_in_order with
max_threads>0 and expression in ORDER BY
2020-11-04 07:07:26 +03:00
alexey-milovidov
8a3f38f332
Merge branch 'master' into errorCode-v2 2020-11-04 01:40:41 +03:00
alexey-milovidov
df83291603
Merge pull request #16612 from ClickHouse/alter-name-collision
Fix possible name collision in calculation of DEFAULT expression.
2020-11-04 01:34:31 +03:00
Nikolai Kochetov
54a9b80a11 Fix build 2020-11-03 22:30:58 +03:00
Nikolai Kochetov
4984fdc693 Fix build 2020-11-03 22:28:50 +03:00
Nikolai Kochetov
3e012265d3 Fix build 2020-11-03 22:26:55 +03:00
Nikolai Kochetov
7b61f5d641 Refactor more 2020-11-03 22:05:47 +03:00
Azat Khuzhin
e7f15dff55 Do not pass StoragePtr to ReadInOrderOptimizer::getInputOrder()
Looks like this is not required anymore, since #11745
2020-11-03 21:22:46 +03:00
alesapin
faca9d9b91 Check max bytes on disk 2020-11-03 18:41:46 +03:00
Nikolai Kochetov
6767a226fc Merge branch 'master' into actions-dag-f14 2020-11-03 15:21:06 +03:00
Nikolai Kochetov
07a7c46b89 Refactor ExpressionActions [Part 3] 2020-11-03 14:28:28 +03:00
alexey-milovidov
11fc6fd8c9
Merge pull request #16584 from ClickHouse/drop_table_task_better_scheduling
Better scheduling of drop table task
2020-11-03 14:22:51 +03:00
Anton Popov
85ab1e9bc3
Merge pull request #15938 from Avogar/select_final
Select final
2020-11-03 13:49:47 +03:00
alesapin
6e32e17a7d Merge branch 'master' into alter-name-collision 2020-11-03 13:16:34 +03:00
alesapin
da8ac948bf Disallow to drop parts with parallel quorum 2020-11-03 12:24:10 +03:00
alesapin
294489f6ad Merge branch 'master' into nvartolomei-drop-part 2020-11-03 10:19:03 +03:00
alesapin
01898be459
Merge pull request #16606 from ClickHouse/better_serialize_last_index_row
Simplier serialization of last index column
2020-11-03 09:59:50 +03:00
Anton Popov
a3a8e18637
Merge branch 'master' into select_final 2020-11-03 00:00:43 +03:00
Alexander Kuzmenkov
f10a5207f4
Merge pull request #16601 from ClickHouse/ban_aggregate_function_in_primary_key
Restrict to use non-comparable types in primary key
2020-11-02 23:27:46 +03:00
Alexey Milovidov
e6d8ab2270 Fix possible name collision in ALTER 2020-11-02 21:37:23 +03:00
alesapin
dac6705995 DROP PART and clear blocks in a single zookeeper op 2020-11-02 20:30:59 +03:00
Alexander Tokmakov
ac32809b6a fix #16482 2020-11-02 19:40:39 +03:00
alesapin
364607d87d Merge with master 2020-11-02 19:18:18 +03:00
alesapin
67d22b3e57 Merge branch 'master' into nvartolomei-drop-part 2020-11-02 19:09:09 +03:00
Alexander Kuzmenkov
548ac9d57b Merge remote-tracking branch 'origin/master' into HEAD 2020-11-02 17:37:30 +03:00
alesapin
7f8560e0c0
Merge pull request #16538 from amosbird/fp2
Allow fetching duplicate parts to detach
2020-11-02 16:10:42 +03:00
feng lv
4d722bc52c fix conflict 2020-11-02 12:52:12 +00:00
alesapin
fbda80ac89 Simplier storage of last index column 2020-11-02 15:19:37 +03:00
alesapin
360548180c Remove redundant include 2020-11-02 14:42:25 +03:00
alesapin
f3b3025719 Ban non comparable types in primary key 2020-11-02 14:39:27 +03:00
feng lv
1a34abadbc fix
fix

fix
2020-11-02 10:46:58 +00:00
alesapin
8b0d9fbc07 Merge branch 'no_background_pool_no_more' into add_separate_pool_for_fetches 2020-11-02 10:54:09 +03:00
Alexander Kuzmenkov
99ee127620 Support SETTINGS clause for File engine
Accept the usual user settings related to file formats.

Most of the diff are the mechanistic code changes required to allow
providing the required FormatSettings to the format factory. The File
engine then extracts these settings from the `CREATE` query, and specifies
them when creating the format parser.
2020-11-02 10:50:38 +03:00
alesapin
101aa9e931 Merge branch 'master' into no_background_pool_no_more 2020-11-02 10:49:19 +03:00
feng lv
3dce3c6a21 fix 2020-11-02 05:28:37 +00:00
alesapin
ae53092536 Merge branch 'master' into system_fetches_table 2020-10-31 18:45:17 +03:00
tavplubix
db146ee615
Merge pull request #16555 from ClickHouse/yet_another_fix_for_00993
Better logging in replication queue, fix flacky test
2020-10-31 14:38:15 +03:00
Azat Khuzhin
e07473f1ea Update comment for StorageSystemErrors 2020-10-31 11:29:17 +03:00
tavplubix
ce6c44eb9f
Update ReplicatedMergeTreeQueue.cpp 2020-10-30 18:47:58 +03:00
Alexander Tokmakov
cf8202d36e better logging, fix flacky test 2020-10-30 15:41:39 +03:00
alesapin
be183027b3
Merge pull request #16516 from zlobober/remove_g0
Remove -g0 form Arcadia build settings.
2020-10-30 13:58:08 +03:00
alesapin
15ba4dd851 Review fix 2020-10-30 11:52:11 +03:00
alesapin
b34960bffa Merge branch 'master' into system_fetches_table 2020-10-30 11:33:37 +03:00
Azat Khuzhin
3661769e1b Remove some absolute changes for errorCodeToName/system.errors 2020-10-30 09:04:22 +03:00
Amos Bird
18c5abc44e
Allow fetching duplicate parts to detach 2020-10-30 10:55:50 +08:00
Azat Khuzhin
ce9897bfd8 Fix readability-container-size-empty in StorageSystemErrors.cpp 2020-10-30 00:07:40 +03:00
Nikolai Kochetov
1c106691b5
Merge pull request #16423 from amosbird/jbodread
Balanced reading from JBOD
2020-10-29 19:22:45 +03:00
Maxim Akhmedov
3627fabfb9 Remove -g0 form Arcadia build settings. 2020-10-29 17:37:23 +03:00
Anton Popov
e32349ea56 fixes related to Nested 2020-10-29 17:14:23 +03:00
alesapin
0e07afb821 Merge branch 'no_background_pool_no_more' into add_separate_pool_for_fetches 2020-10-29 12:53:55 +03:00
alesapin
40fc512e79 Merge branch 'master' into no_background_pool_no_more 2020-10-29 12:53:34 +03:00
Azat Khuzhin
b42f77a791 Covert error codes list to lookup table and drop generation of them. 2020-10-29 10:55:41 +03:00
Azat Khuzhin
b2e2322895 Add system.errors table
Contains error codes with number of times they have been triggered.

Columns:

-   `name` ([String](../../sql-reference/data-types/string.md)) — name of the error (`errorCodeToName`).
-   `code` ([Int32](../../sql-reference/data-types/int-uint.md)) — code number of the error.
-   `value` ([UInt64](../../sql-reference/data-types/int-uint.md)) - number of times this error has been happened.

**Example**

``` sql
SELECT *
FROM system.errors
WHERE value > 0
ORDER BY code ASC
LIMIT 1

┌─name─────────────┬─code─┬─value─┐
│ CANNOT_OPEN_FILE │   76 │     1 │
└──────────────────┴──────┴───────┘
2020-10-29 10:55:38 +03:00
Nikolai Kochetov
671d2b7f1b
Update MergeTreeReadPool.cpp
Fix comment
2020-10-29 10:04:23 +03:00
alexey-milovidov
0faf2bc7e3
Merge pull request #16332 from ClickHouse/fix-queue-processing-of-very-large-entries
Fix processing of very large entries in queue
2020-10-29 09:09:29 +03:00
alexey-milovidov
bf3acecf6f
Merge pull request #16032 from Avogar/mysql_bug
Don't touch MySQL database if it's unnecessary
2020-10-29 04:14:26 +03:00
Nikolai Kochetov
10bad32fb4 Refactor code a little bit. Add comment. 2020-10-29 00:52:26 +03:00
Amos Bird
f995ef9797
Balanced reading from JBOD 2020-10-29 04:05:07 +08:00
alesapin
57c3935a26
Merge pull request #16426 from kssenii/rabbit-optimize
Optimize rabbitmq engine
2020-10-28 22:24:22 +03:00
alesapin
9830ff1003
Merge pull request #16094 from nvartolomei/nv/wal-metadata-compatibility
RFC: Add metadata section to MergeTreeWriteAheadLog
2020-10-28 15:54:46 +03:00
alesapin
8e8bdeb5d7
Merge pull request #16434 from ClickHouse/fix_fake_race_on_merges_list
Fix fake race condition on system.merges merge_algorithm
2020-10-28 15:51:53 +03:00
alesapin
0931658c2f Merge branch 'master' into add_separate_pool_for_fetches 2020-10-28 14:20:56 +03:00
tavplubix
cd70dc82cd
Merge pull request #16409 from azat/drop-distributed-fix
Fix DROP TABLE for Distributed (racy with INSERT)
2020-10-28 14:10:32 +03:00
alesapin
3c31a5134e Fix bug 2020-10-28 14:03:26 +03:00
tavplubix
4a63f1271b
Merge pull request #16244 from ClickHouse/fix_race_data_parts_exchange_endpoint
Fix race on StorageReplicatedMergeTree::data_parts_exchange_endpoint
2020-10-28 13:25:55 +03:00
alesapin
c10370f98d Merge branch 'master' into fix-queue-processing-of-very-large-entries 2020-10-28 11:19:34 +03:00
alesapin
617e42ddb4
Merge branch 'master' into rabbit-optimize 2020-10-28 10:24:05 +03:00
alesapin
1c9e24b5e2 Merge branch 'master' into system_fetches_table 2020-10-28 10:20:04 +03:00
kssenii
4b59882c30 Remove redundant, add virt col 2020-10-27 20:34:31 +00:00
Alexander Kuzmenkov
6509f1eb39
Merge branch 'master' into minumum 2020-10-27 21:38:23 +03:00
Azat Khuzhin
fc14fde24a Fix DROP TABLE for Distributed (racy with INSERT)
<details>

```

drop() on T1275:

    0  DB::StorageDistributed::drop (this=0x7f9ed34f0000) at ../contrib/libcxx/include/__hash_table:966
    1  0x000000000d557242 in DB::DatabaseOnDisk::dropTable (this=0x7f9fc22706d8, context=..., table_name=...)
       at ../contrib/libcxx/include/new:340
    2  0x000000000d6fcf7c in DB::InterpreterDropQuery::executeToTable (this=this@entry=0x7f9e42560dc0, query=...)
       at ../contrib/libcxx/include/memory:3826
    3  0x000000000d6ff5ee in DB::InterpreterDropQuery::execute (this=0x7f9e42560dc0) at ../src/Interpreters/InterpreterDropQuery.cpp:50
    4  0x000000000daa40c0 in DB::executeQueryImpl (begin=<optimized out>, end=<optimized out>, context=..., internal=<optimized out>,
       stage=DB::QueryProcessingStage::Complete, has_query_tail=false, istr=0x0) at ../src/Interpreters/executeQuery.cpp:420
    5  0x000000000daa59df in DB::executeQuery (query=..., context=..., internal=internal@entry=false, stage=<optimized out>,
       may_have_embedded_data=<optimized out>) at ../contrib/libcxx/include/string:1487
    6  0x000000000e1369e6 in DB::TCPHandler::runImpl (this=this@entry=0x7f9ddf3a9000) at ../src/Server/TCPHandler.cpp:254
    7  0x000000000e1379c9 in DB::TCPHandler::run (this=0x7f9ddf3a9000) at ../src/Server/TCPHandler.cpp:1326
    8  0x000000001086fac7 in Poco::Net::TCPServerConnection::start (this=this@entry=0x7f9ddf3a9000)
       at ../contrib/poco/Net/src/TCPServerConnection.cpp:43
    9  0x000000001086ff2b in Poco::Net::TCPServerDispatcher::run (this=0x7f9e4eba5c00)
       at ../contrib/poco/Net/src/TCPServerDispatcher.cpp:114
    10 0x00000000109dbe8e in Poco::PooledThread::run (this=0x7f9e4a2d2f80) at ../contrib/poco/Foundation/src/ThreadPool.cpp:199
    11 0x00000000109d78f9 in Poco::ThreadImpl::runnableEntry (pThread=<optimized out>)
       at ../contrib/poco/Foundation/include/Poco/SharedPtr.h:401
    12 0x00007f9fc3cccea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
    13 0x00007f9fc3bebeaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

StorageDistributedDirectoryMonitor on T166:

    0  DB::StorageDistributedDirectoryMonitor::StorageDistributedDirectoryMonitor (this=0x7f9ea7ab1400, storage_=..., path_=...,
       pool_=..., monitor_blocker_=..., bg_pool_=...) at ../src/Storages/Distributed/DirectoryMonitor.cpp:81
    1  0x000000000dbf684e in std::__1::make_unique<> () at ../contrib/libcxx/include/memory:3474
    2  DB::StorageDistributed::requireDirectoryMonitor (this=0x7f9ed34f0000, disk=..., name=...)
       at ../src/Storages/StorageDistributed.cpp:682
    3  0x000000000de3d5fa in DB::DistributedBlockOutputStream::writeToShard (this=this@entry=0x7f9ed39c7418, block=..., dir_names=...)
       at ../src/Storages/Distributed/DistributedBlockOutputStream.cpp:634
    4  0x000000000de3e214 in DB::DistributedBlockOutputStream::writeAsyncImpl (this=this@entry=0x7f9ed39c7418, block=...,
       shard_id=shard_id@entry=79) at ../src/Storages/Distributed/DistributedBlockOutputStream.cpp:539
    5  0x000000000de3e47b in DB::DistributedBlockOutputStream::writeSplitAsync (this=this@entry=0x7f9ed39c7418, block=...)
       at ../contrib/libcxx/include/vector:1546
    6  0x000000000de3eab0 in DB::DistributedBlockOutputStream::writeAsync (block=..., this=0x7f9ed39c7418)
       at ../src/Storages/Distributed/DistributedBlockOutputStream.cpp:141
    7  DB::DistributedBlockOutputStream::write (this=0x7f9ed39c7418, block=...)
       at ../src/Storages/Distributed/DistributedBlockOutputStream.cpp:135
    8  0x000000000d73b376 in DB::PushingToViewsBlockOutputStream::write (this=this@entry=0x7f9ea7a8cf58, block=...)
       at ../src/DataStreams/PushingToViewsBlockOutputStream.cpp:157
    9  0x000000000d7853eb in DB::AddingDefaultBlockOutputStream::write (this=0x7f9ed383d118, block=...)
       at ../contrib/libcxx/include/memory:3826
    10 0x000000000d740790 in DB::SquashingBlockOutputStream::write (this=0x7f9ed383de18, block=...)
       at ../contrib/libcxx/include/memory:3826
    11 0x000000000d68c308 in DB::CountingBlockOutputStream::write (this=0x7f9ea7ac6d60, block=...)
       at ../contrib/libcxx/include/memory:3826
    12 0x000000000ddab449 in DB::StorageBuffer::writeBlockToDestination (this=this@entry=0x7f9fbd56a000, block=..., table=...)
       at ../src/Storages/StorageBuffer.cpp:747
    13 0x000000000ddabfa6 in DB::StorageBuffer::flushBuffer (this=this@entry=0x7f9fbd56a000, buffer=...,
       check_thresholds=check_thresholds@entry=true, locked=locked@entry=false, reset_block_structure=reset_block_structure@entry=false)
       at ../src/Storages/StorageBuffer.cpp:661
    14 0x000000000ddac415 in DB::StorageBuffer::flushAllBuffers (reset_blocks_structure=false, check_thresholds=true, this=0x7f9fbd56a000)
        at ../src/Storages/StorageBuffer.cpp:605

shutdown() on T1275:

    0  DB::StorageDistributed::shutdown (this=0x7f9ed34f0000) at ../contrib/libcxx/include/atomic:1612
    1  0x000000000d6fd938 in DB::InterpreterDropQuery::executeToTable (this=this@entry=0x7f98530c79a0, query=...)
       at ../src/Storages/TableLockHolder.h:12
    2  0x000000000d6ff5ee in DB::InterpreterDropQuery::execute (this=0x7f98530c79a0) at ../src/Interpreters/InterpreterDropQuery.cpp:50
    3  0x000000000daa40c0 in DB::executeQueryImpl (begin=<optimized out>, end=<optimized out>, context=..., internal=<optimized out>,
       stage=DB::QueryProcessingStage::Complete, has_query_tail=false, istr=0x0) at ../src/Interpreters/executeQuery.cpp:420
    4  0x000000000daa59df in DB::executeQuery (query=..., context=..., internal=internal@entry=false, stage=<optimized out>,
       may_have_embedded_data=<optimized out>) at ../contrib/libcxx/include/string:1487
    5  0x000000000e1369e6 in DB::TCPHandler::runImpl (this=this@entry=0x7f9ddf3a9000) at ../src/Server/TCPHandler.cpp:254
    6  0x000000000e1379c9 in DB::TCPHandler::run (this=0x7f9ddf3a9000) at ../src/Server/TCPHandler.cpp:1326
    7  0x000000001086fac7 in Poco::Net::TCPServerConnection::start (this=this@entry=0x7f9ddf3a9000)
       at ../contrib/poco/Net/src/TCPServerConnection.cpp:43
    8  0x000000001086ff2b in Poco::Net::TCPServerDispatcher::run (this=0x7f9e4eba5c00)
       at ../contrib/poco/Net/src/TCPServerDispatcher.cpp:114
    9  0x00000000109dbe8e in Poco::PooledThread::run (this=0x7f9e4a2d2f80) at ../contrib/poco/Foundation/src/ThreadPool.cpp:199
    10 0x00000000109d78f9 in Poco::ThreadImpl::runnableEntry (pThread=<optimized out>)
       at ../contrib/poco/Foundation/include/Poco/SharedPtr.h:401
    11 0x00007f9fc3cccea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
    12 0x00007f9fc3bebeaf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

```

</details>
2020-10-27 21:19:36 +03:00
alesapin
3ec5fc7a4c Add explicit 2020-10-27 18:29:06 +03:00
alesapin
60f2d822d7 Fix fake race condition on system.merges merge_algorithm 2020-10-27 18:27:12 +03:00
Alexander Kuzmenkov
60544ad46e
Merge pull request #16373 from ClickHouse/better-rethrow
Better exceptions rethrow
2020-10-27 16:38:32 +03:00
alesapin
6253d1466a Fix comments 2020-10-27 16:09:14 +03:00
alesapin
3d3d246bdc Rename table 2020-10-27 16:07:50 +03:00
alesapin
e40c509392 More clear callback name 2020-10-27 16:00:40 +03:00
alesapin
f35edfb2a5 Remove some unused fields 2020-10-27 15:57:55 +03:00
alesapin
ee4e6caf70 Remove redundant metric increment 2020-10-27 15:52:49 +03:00
alesapin
3824fcb076 Rename file and some tweaks 2020-10-27 15:47:42 +03:00
alesapin
31d63efdac Fix style 2020-10-27 15:26:10 +03:00
alesapin
dca35b1913 Add awesome test for fetch 2020-10-27 15:24:10 +03:00
kssenii
a48bc83763 Fix style 2020-10-27 12:04:07 +00:00
Pavel Kruglov
331545fe0d Fix typo 2020-10-27 14:29:36 +03:00