thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 11:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
72,865 Commits 9,338 Branches 1,783 Tags 2.1 GiB
e9e77b4403
Commit Graph

5385 Commits

Author SHA1 Message Date
Vladimir Chebotarev
d1efe3afa7 Review fixes. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
e99433e094 Better functionality. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
328213f5d9 Fixes. 2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
1146efacff Partitioned writes for S3. 2021-08-23 09:22:13 +03:00
Vitaly Baranov
ff1fad8521 Use setSetting() with implicit conversion true/false -> Field. 2021-08-23 08:57:00 +03:00
kssenii
c5ee335c68 Fix tests 2021-08-22 21:41:09 +03:00
kssenii
14afb3c509 File progress for local fix 2021-08-22 20:33:55 +03:00
Nikita Mikhaylov
80ebc6ccea Done 2021-08-22 15:09:49 +00:00
Braulio Valdivielso
d48f6122e3 clean up implementation a bit 2021-08-22 10:06:30 +01:00
Braulio Valdivielso
e95563fb82 leverage bloom filter index for 'hasAll' 
Fixes #26151. For now, this is a draft implementation. There is some
cleanup that can be done, but the new test is passing
 2021-08-21 20:14:56 +01:00
Nikita Mikhaylov
cfa571cac4
Merge pull request #27895 from nikitamikhaylov/temporary_remove_yaml_fuzzer 
Build fuzzers with clang-tidy
 2021-08-21 16:51:31 +03:00
alexey-milovidov
5ac6a99542
Merge pull request #27927 from ClickHouse/fix_27832 
Сheck cluster name before creating Distributed
 2021-08-21 10:40:23 +03:00
Nikita Mikhaylov
f72457fa98 Lower the size of the binaries 2021-08-21 02:20:23 +00:00
kssenii
945e2c4ce5 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-20 18:01:23 +00:00
Ilya Yatsishin
624cb43f7f
Merge pull request #27885 from azat/kafka-write-assert 
Fix incorrect assertion during writing to StorageKafka
 2021-08-20 18:53:22 +03:00
Alexander Tokmakov
42378b5913 fix 2021-08-20 17:05:53 +03:00
Alexander Tokmakov
59eb3aa9a9 avoid too long waiting for inactive replicas 2021-08-20 15:59:57 +03:00
Alexander Tokmakov
8c6dd18917 check cluster name before creating Distributed 2021-08-20 14:55:04 +03:00
mergify[bot]
66c6a85211
Merge branch 'master' into master 2021-08-20 11:50:30 +00:00
Maksim Kita
5f2b28639f Merge branch 'master' into create-user-defined-lambda-function 2021-08-20 11:53:08 +03:00
Azat Khuzhin
b60e5ac801 Fix incorrect assertion during writing to StorageRabbitMQ 2021-08-20 07:14:20 +03:00
Azat Khuzhin
5947e54c1b Fix incorrect assertion during writing to StorageKafka 
The problem it does not triggered in CI always because buffers was not
destroyed by that time.

Fixes: #26547
 2021-08-19 22:07:20 +03:00
nvartolomei
c09c90125f
Merge branch 'master' into nv/last-queue-update-exception 2021-08-19 10:29:16 +01:00
Nikolai Kochetov
7dcff5c90e
Merge pull request #27742 from amosbird/projection-improvement3 
Improve projection analysis.
 2021-08-19 12:11:20 +03:00
alexey-milovidov
215385e4d0
Merge pull request #27794 from filimonov/kafka_null_messages 
Avoid nullptr dereference during processing of NULL messages in Kafka for some formats
 2021-08-19 11:25:07 +03:00
Nikolai Kochetov
3e44ef437e
Merge pull request #26639 from ClickHouse/fix-bad-cast 
Fix bad cast
 2021-08-19 11:17:26 +03:00
alesapin
0aec151719
Merge pull request #27808 from ClickHouse/fix_replicas_may_diverge 
Fix a couple of bugs that may cause replicas to diverge
 2021-08-19 09:58:07 +03:00
Maksim Kita
01682a86b3 Updated user defined functions implementation 2021-08-19 00:54:55 +03:00
Alexander Tokmakov
0ed046eb7b remove irrelevant comments 2021-08-18 15:33:11 +03:00
alexey-milovidov
bcfab277e7
Update ReadBufferFromKafkaConsumer.cpp 2021-08-18 15:25:54 +03:00
Nikolai Kochetov
a46fe11e2c Try fix test_mutations_with_merge_background_task 2021-08-18 13:30:02 +03:00
Alexander Tokmakov
09ff66da0e fix a couple of bugs that may cause replicas to diverge 2021-08-18 12:50:46 +03:00
Mikhail Filimonov
05d77d2873
Safer processing of NULL messages in Kafka for some formats 2021-08-18 11:26:57 +02:00
alesapin
5cc98c67a4 Fix 2021-08-18 11:56:18 +03:00
mergify[bot]
f11e396151
Merge branch 'master' into nv/last-queue-update-exception 2021-08-18 07:00:50 +00:00
vicgao
c16bd25399 modify column datatype to Nullable(datatype) or Nullable(datatype) to Nullable(datatype) donot allways need mutations. 2021-08-17 21:14:13 +08:00
Nikita Mikhaylov
cc8ad0a4e7
Merge pull request #27733 from Algunenano/i27620_take2 
Refactor arrayJoin check on partition expressions
 2021-08-17 15:57:16 +03:00
vicgao
b5610d9a83 modify column datatype to Nullable(datatype) or Nullable(datatype) to Nullable(datatype) donot allways need mutations. 2021-08-17 20:52:16 +08:00
Nikolai Kochetov
98eb619b4b
Merge pull request #23367 from Avogar/short-circuit 
Implement short circuit function evaluation
 2021-08-17 12:36:07 +03:00
Amos Bird
ea7dc495cf
Better code. 2021-08-17 12:32:49 +08:00
Raúl Marín
90f224807a Style 2021-08-16 17:31:37 +02:00
kssenii
bec34531bc Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-16 14:48:32 +00:00
Amos Bird
90881aab09
Better code style 2021-08-16 22:17:07 +08:00
Raúl Marín
3ec904a953 Only analyze if tuple has arguments 2021-08-16 14:47:41 +02:00
Amos Bird
b162a2b699
Improve projection analysis. 
Remove duplicate index analysis and avoid possible invalid limit checks
during projection analysis.
 2021-08-16 20:11:57 +08:00
Raúl Marín
34099bc196 Refactor arrayJoin check on partition expressions 2021-08-16 12:46:18 +02:00
Alexey Milovidov
d184b79bba Progress on async reads. 2021-08-16 03:00:32 +03:00
alexey-milovidov
4a36366925
Merge pull request #27648 from Algunenano/i27620 
Disable arrayJoin on partition expressions
 2021-08-16 02:53:36 +03:00
Alexey Milovidov
0cd3f25ba4 Merge remote-tracking branch 'origin/master' into async-reads 2021-08-16 02:41:01 +03:00
alexey-milovidov
76b0502482
Merge pull request #27690 from kitaisreal/remove-dense-hash-map-set 
Removed DenseHashMap, DenseHashSet
 2021-08-16 02:39:41 +03:00
Maksim Kita
5f5470c2cd Removed DenseHashMap, DenseHashSet 2021-08-15 13:32:56 +03:00
Alexey Milovidov
020beb885d Merge branch 'master' into async-reads 2021-08-15 12:08:19 +03:00
mergify[bot]
bf426cfb96
Merge branch 'master' into fix_intersecting_parts 2021-08-15 09:07:20 +00:00
alexey-milovidov
d88cf81d71
Merge branch 'master' into fix-bad-cast 2021-08-15 09:06:42 +03:00
mergify[bot]
353a324887
Merge branch 'master' into i27620 2021-08-15 02:32:26 +00:00
alexey-milovidov
ab4415a22b
Update MergeTreeData.cpp 2021-08-15 05:28:38 +03:00
kssenii
daac6323fa Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-14 13:15:04 +00:00
kssenii
200d75646a Fix 2021-08-13 20:58:03 +00:00
Alexander Tokmakov
c534363abe fix intersecting parts 2021-08-13 20:33:10 +03:00
Raúl Marín
73cb7d55ec Disable arrayJoin on partition expressions 2021-08-13 16:18:46 +02:00
mergify[bot]
319e5af6eb
Merge branch 'master' into short-circuit 2021-08-13 13:01:46 +00:00
Nikolai Kochetov
66fdb859f3
Merge pull request #27528 from amosbird/projection-fix12 
Fix projection materialization with missing columns
 2021-08-13 15:20:51 +03:00
Kseniia Sumarokova
0917488cec
Merge pull request #27382 from kssenii/cast-internal 
add Cast internal function
 2021-08-13 13:13:51 +03:00
Nikolai Kochetov
ad00aaa18c
Merge pull request #27575 from kitaisreal/removed-some-data-streams 
Removed some data streams
 2021-08-13 12:59:00 +03:00
Nikolai Kochetov
d207fbc699
Merge pull request #27152 from qoega/fix-access-gtest-in-arcadia 
Fix arcadia src/Access gtest
 2021-08-13 10:51:29 +03:00
alexey-milovidov
f20eae9a45
Merge pull request #27577 from ClickHouse/remove-streams-from-lv 
Remove streams from lv
 2021-08-13 03:01:33 +03:00
Nikolai Kochetov
224e0a547a Fix build. 2021-08-12 16:02:17 +03:00
Nikolai Kochetov
9c066d476e Fix tests. 2021-08-12 13:46:58 +03:00
Nikolai Kochetov
5e5a6ff654 Merge branch 'master' into qoega-fix-access-gtest-in-arcadia 2021-08-12 12:02:15 +03:00
mergify[bot]
80eaea1c51
Merge branch 'master' into short-circuit 2021-08-12 08:38:45 +00:00
Maksim Kita
124a87684f Removed some data streams 2021-08-11 23:39:01 +03:00
kssenii
dcc6f597dc Move templates from FunctionsConvertion 2021-08-11 19:09:51 +00:00
kssenii
7555fc19fd Revert "Move some templates" 
This reverts commit 7f247becca.
 2021-08-11 18:49:25 +00:00
Nikolai Kochetov
beed4c21a5 Use processors in Storage::watch 2021-08-11 20:28:54 +03:00
mergify[bot]
68a52fc065
Merge branch 'master' into projection-fix12 2021-08-11 17:08:04 +00:00
Alexander Kuzmenkov
8f9845722a
Merge pull request #27250 from azat/minmax-nullable-new-format 
Add new index data skipping minmax index format for proper Nullable support
 2021-08-11 13:56:16 +03:00
Nikolai Kochetov
81b85c30f7 Remove streams from LV part 1. 2021-08-11 11:35:10 +03:00
kssenii
7d8b694a9f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal 2021-08-11 07:37:21 +00:00
kssenii
7f247becca Move some templates 2021-08-11 07:32:46 +00:00
Alexey Milovidov
08f8511854 Fix Style 2021-08-11 06:52:28 +03:00
alexey-milovidov
54d8282209
Merge pull request #27248 from azat/mt-mutation-stuck-fix 
Fix mutation stuck on invalid partitions in non-replicated MergeTree
 2021-08-11 06:10:15 +03:00
alexey-milovidov
a0a5c0da32
Update BackgroundJobsExecutor.cpp 2021-08-11 06:08:30 +03:00
Alexey Milovidov
c590a239d5 Merge branch 'master' into fix-bad-cast 2021-08-11 05:48:37 +03:00
Amos Bird
e63c26edb7
Fix projection materialization with missing columns 2021-08-10 21:47:27 +08:00
Nikolai Kochetov
8613cfd4e2
Merge branch 'master' into fix-access-gtest-in-arcadia 2021-08-10 16:13:11 +03:00
Maksim Kita
87fb75b272
Merge pull request #27191 from FArthur-cmd/memory-tracker-26043 
Client with flag for memory tracker
 2021-08-10 14:01:02 +03:00
kssenii
eb4746ac69 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal 2021-08-10 09:38:41 +00:00
Nikolai Kochetov
a1ec7f75c5 Merge branch 'master' into qoega-fix-access-gtest-in-arcadia 2021-08-10 11:31:47 +03:00
Nikolai Kochetov
bb4c11cd27
Merge pull request #27298 from ClickHouse/fix-23515 
Fix bug from #23515.
 2021-08-09 23:25:30 +03:00
Nikolai Kochetov
8cc493a3cd Try fix build. 2021-08-09 18:09:29 +03:00
Pavel Kruglov
70b51133c1 Try to simplify code 2021-08-09 18:01:08 +03:00
Pavel Kruglov
0662df8b76 Fix performance with JIT, add arguments to function isSuitableForShortCircuitArgumentsExecution 2021-08-09 17:54:14 +03:00
Pavel Kruglov
e792fa588f Mark all Functions as sutable or not for executing as short circuit arguments 2021-08-09 17:50:09 +03:00
Maksim Kita
b8b24684b4
Merge pull request #27273 from ClickHouse/remove-streams-from-dicts 
Remove streams from dicts
 2021-08-09 15:58:37 +03:00
Nicolae Vartolomei
3f291b024a
Use plain mutex instead of MultiVersion 2021-08-09 13:58:23 +01:00
kssenii
d767f6704f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-09 12:14:23 +00:00
Nikita Mikhaylov
73d3f2c60f
Merge pull request #26313 from fastio/control_execution_period_of_clearOldTemporaryDirectories 
Control the execution period of clear old temporary directories by parameter
 2021-08-09 14:29:24 +03:00
Nikolai Kochetov
4a00e402ae Fix spelling. 2021-08-09 12:09:09 +03:00
Nikolai Kochetov
372293b85d
Merge branch 'master' into fix-27179 2021-08-09 11:59:22 +03:00
Alexey Milovidov
8b9da6fe26 Merge branch 'master' into async-reads 2021-08-09 03:33:32 +03:00
mergify[bot]
316c254536
Merge branch 'master' into fix-bad-cast 2021-08-08 23:45:19 +00:00
kssenii
7991bb6e83 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal 2021-08-08 21:30:17 +00:00
Azat Khuzhin
038241b6ed Add new index data skipping minmax index format for proper Nullable support 
Note, that it cannot be done w/o new extension, since index does not
have any header.

v2: use IDisk interface for existence check
v3: remove extra file existence check
v4: fix MATERIALIZE INDEX
 2021-08-08 19:30:12 +03:00
Azat Khuzhin
91d7f3daa7 Fix lock-order-inversion while notifying about finished mutations 
TSAN reports [1]:

  WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=36)
    Cycle in lock order graph: M16388 (0x7b7400011d68) => M1030334152907497744 (0x000000000000) => M16388

    Mutex M1030334152907497744 acquired here while holding mutex M16388 in thread T4:
      0 pthread_mutex_lock <null> (clickhouse+0x967d536)
      1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse+0x1b25c7d9)
      2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x1b25c7d9)
      3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x15220cd9)
      4 DB::StorageMergeTree::getIncompleteMutationsStatus(long, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std:>
      5 DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0::operator()() const obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:464:36 (clickhouse+0x1521e2b1)
      6 void std::__1::condition_variable::wait<DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0>(std::__1::unique_lock<std::__1::mutex>&, DB::StorageMergeTree::waitForMutation(lon>
      7 DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:469:29 (clickhouse+0x1521e2b1)
      8 DB::StorageMergeTree::mutate(DB::MutationCommands const&, std::__1::shared_ptr<DB::Context const>) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:496:9 (clickhouse+0x15221738)
      9 DB::InterpreterAlterQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterAlterQuery.cpp:113:16 (clickhouse+0x141182f6)
      10 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:560:32 (clickhouse+0x149152f6)
      11 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:909:30 (clickhous>
      12 DB::TCPHandler::runImpl() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:313:24 (clickhouse+0x153270af)

      Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

    Mutex M16388 acquired here while holding mutex M1030334152907497744 in thread T59:
      0 pthread_mutex_lock <null> (clickhouse+0x967d536)
      1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse+0x1b25c7d9)
      2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x1b25c7d9)
      3 std::__1::unique_lock<std::__1::mutex>::unique_lock(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:119:61 (clickhouse+0x15226c31)
      4 DB::StorageMergeTree::selectPartsToMutate(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&) obj-x86_64-linux->
      5 DB::StorageMergeTree::scheduleDataProcessingJob(DB::IBackgroundJobExecutor&) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:1060:28 (clickhouse+0x15228b10)
      6 DB::BackgroundJobsExecutor::scheduleJob() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:229:17 (clickhouse+0x14f441dc)
      7 DB::IBackgroundJobExecutor::backgroundTaskFunction() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:185:10 (clickhouse+0x14f438ed)

  [1]: https://clickhouse-test-reports.s3.yandex.net/27248/4f3b80ff33c846465983aa2bc9ae9490e1118b15/fuzzer_tsan/report.htmlfail1
 2021-08-08 19:28:29 +03:00
Azat Khuzhin
7964355ecf Fix mutation stuck on invalid partitions in non-replicated MergeTree 
v2: Do not try to process empty mutations
    Found with flaky check [1].

      [1]: https://clickhouse-test-reports.s3.yandex.net/27248/66e8c0833392c20ba8dba3780f2b0d5c18f8194e/functional_stateless_tests_flaky_check_(address).html#fail1
 2021-08-08 19:28:15 +03:00
Azat Khuzhin
5139067631 Guard BackgroundJobsExecutor from thread termination in case of uncaught exception 2021-08-08 19:27:58 +03:00
Azat Khuzhin
702d9955c0 Fix distributed queries with zero shards and aggregation 2021-08-08 19:22:49 +03:00
mergify[bot]
ec331132c1
Merge branch 'master' into fix-bad-cast 2021-08-08 15:20:27 +00:00
Azat Khuzhin
3be3c503aa Fix some comments 2021-08-08 09:58:07 +03:00
alexey-milovidov
a7e26506cf
Merge pull request #26951 from ianton-ru/zero_copy_by_default 
Set allow_remote_fs_zero_copy_replication to true by default
 2021-08-08 04:46:49 +03:00
alexey-milovidov
ba3fbbaab0
Update ReplicatedMergeTreeMergeStrategyPicker.cpp 2021-08-08 04:43:59 +03:00
alexey-milovidov
e6e9a27d8f
Merge pull request #27377 from evillique/bz2 
Add support for bzip2 compression method
 2021-08-08 04:40:13 +03:00
Alexey Milovidov
1ad1e62b47 Fix unit test 2021-08-08 04:02:48 +03:00
Alexey Milovidov
24cef99065 Merge branch 'master' into fix-bad-cast 2021-08-08 04:00:29 +03:00
alexey-milovidov
c5207fc237
Merge pull request #26466 from azat/optimize-dist-select 
Rework SELECT from Distributed optimizations
 2021-08-08 03:59:32 +03:00
mergify[bot]
9869c8bb10
Merge branch 'master' into zero_copy_by_default 2021-08-07 16:53:44 +00:00
alexey-milovidov
24cc073a22
Merge pull request #27328 from ClickHouse/fix_partition_id_validation 
Fix partition id validation
 2021-08-07 18:45:19 +03:00
kssenii
9ca422f0c5 Introduce CAST for internal usage 2021-08-07 09:03:10 +00:00
Nikolay Degterinsky
9a45458faf Add bzip2 file compression 2021-08-07 00:25:52 +00:00
Alexander Tokmakov
5e2bfd5ba1 fix partition id validation 2021-08-06 23:56:51 +03:00
kssenii
073d7fdd5e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-06 19:42:29 +00:00
Nikolai Kochetov
3cbeab8959 Fix incorrect row-level filtering 2021-08-06 21:16:06 +03:00
Nikolai Kochetov
157ecef3c2
Merge pull request #27227 from ClickHouse/do-not-update-settings-for-view 
Revert #24095. User-level settings will affect queries from view.
 2021-08-06 20:57:36 +03:00
mergify[bot]
c696817a79
Merge branch 'master' into control_execution_period_of_clearOldTemporaryDirectories 2021-08-06 16:16:22 +00:00
Nikolai Kochetov
7320411a7f Fix bug from #23515. 2021-08-06 16:39:11 +03:00
Anton Popov
3a0d4807a5 Merge branch 'async-insert' of git://github.com/abyss7/ClickHouse into merging-20557 2021-08-06 16:20:02 +03:00
vdimir
ffab19caad
Fix hasJoin in StorageView.cpp 2021-08-06 13:48:47 +03:00
vdimir
76ca40857a
Better error message for changed json_use_nulls in StorageView, check ast 2021-08-06 12:16:26 +03:00
tavplubix
14ef66e49b
Merge pull request #25895 from ClickHouse/improve_create_or_replace 
Improve CREATE OR REPLACE query
 2021-08-06 11:58:46 +03:00
Nikolai Kochetov
39ff5ab478 Merge branch 'master' into remove-streams-from-dicts 2021-08-06 11:46:27 +03:00
Nikolai Kochetov
13f95f3fdf Streams -> Processors for dicts, part 3. 2021-08-06 11:41:45 +03:00
Alexey Milovidov
cea9690327 Merge branch 'master' into async-reads 2021-08-05 23:58:57 +03:00
alexey-milovidov
1a3d8cebd2
Merge pull request #27197 from azat/nullable-index-fix 
Fix on-disk format breakage for secondary indices over Nullable column
 2021-08-05 21:52:18 +03:00
Artur
d2869d2e2b add test for memory tracker in client 2021-08-05 16:20:24 +00:00
vdimir
a427b19d3a
Check join_use_nulls changed in StorageView::read 2021-08-05 18:38:52 +03:00
Kseniia Sumarokova
5edd9e0513
Merge pull request #27198 from kssenii/postgres-db-schema 
Support schema for postgres database engine
 2021-08-05 18:30:47 +03:00
Kruglov Pavel
59a94bd322
Merge pull request #27183 from ClickHouse/fix_detached_parts_name_partsing 
Fix part name parsing in system.detached_parts
 2021-08-05 18:21:03 +03:00
Nikolai Kochetov
fc12310ee8 Revert #24095. User-level settings will affect queries from view. 2021-08-05 17:17:22 +03:00
Alexander Kuzmenkov
d852207b0e
Merge pull request #25721 from CurtizJ/read-in-order-perf-2 
Reduce memory usage in queries with `ORDER BY primary_key
 2021-08-05 16:13:48 +03:00
Maksim Kita
7fdf3cc263
Merge pull request #27180 from kitaisreal/storage-system-replicas-added-column-replica-is-active 
Storage system replicas added column replica is active
 2021-08-05 12:46:53 +03:00
terrylin
dd539f0a0d improvement of materilize ttl 2021-08-05 15:33:02 +08:00
terrylin
bd3d9a4518 materialize ttl recalculate only (optional) 2021-08-05 14:17:48 +08:00
Azat Khuzhin
dee27fcbb9 Fix on-disk format breakage for secondary indices over Nullable column 
[1] breaks on disk format (and the relevant change in the:

  [1]: https://github.com/ClickHouse/ClickHouse/pull/12455#discussion_r682830812

Too bad that I checked this patchset only for compatibility after
reverting this patch [2] (use case: I've applied it manually, then
revert it, and data skipping indexes over Nullable column had been
broken)

  [2]: https://github.com/ClickHouse/ClickHouse/pull/12455#issuecomment-823423772

But this patchset actually breaks compatibility with older versions of
clickhouse for Nullable data skipping indexes after simple upgrade:

Here is a simple reproducer:

    --
    -- run this with 21.6 or similar (i.e. w/o this patch)
    --

    CREATE TABLE data
    (
        `key` Int,
        `value` Nullable(Int),
        INDEX value_index value TYPE minmax GRANULARITY 1
    )
    ENGINE = MergeTree
    ORDER BY key;

    INSERT INTO data SELECT
        number,
        number
    FROM numbers(10000);

    SELECT * FROM data WHERE value = 20000 SETTINGS force_data_skipping_indices = 'value_index' SETTINGS force_data_skipping_indices = 'value_index', max_rows_to_read=1;

Now upgrade and run the query again:

    SELECT * FROM data WHERE value = 20000 SETTINGS force_data_skipping_indices = 'value_index' SETTINGS force_data_skipping_indices = 'value_index', max_rows_to_read=1;

And it will fail because of on disk format changes:

    $ ll --time-style=+ data/*/data/all_1_1_0/skp*.idx
    -rw-r----- 1 azat azat 36  data/with_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
    -rw-r----- 1 azat azat 37  data/without_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx

    $ md5sum data/*/data/all_1_1_0/skp*.idx
    a19c95c4a14506c65665a1e30ab404bf  data/with_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
    e50e2fcfa873b232196623d56ab26105  data/without_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx

Note, that there is no stable release with this patch included yet, so
no need to backport.

Also note that you may create data skipping indexes over Nullable
column even before [3].

  [3]: https://github.com/ClickHouse/ClickHouse/pull/12455

v2: break cases when granulas has Null in values due to backward
compatibility
 2021-08-05 00:19:38 +03:00
kssenii
f06703a7c9 Support schema for postgres database engine 2021-08-04 20:52:45 +00:00
Azat Khuzhin
7a47327980 Fix used_storages by using log_queries from local context 
I don't have log_queries enabled by default.
 2021-08-04 21:44:19 +03:00
Anton Popov
9586bb7e90 fix pvs 2021-08-04 20:42:22 +03:00
terrylin
6f42ec6b9b fix compile error 2021-08-05 00:16:21 +08:00
Anton Popov
915ef50c2d optimize reading in order of key more 2021-08-04 18:18:03 +03:00
Alexander Tokmakov
23f8b3d07d fix part name parsing in system.detached_parts 2021-08-04 17:42:48 +03:00
terrylin
88b9e7f24f materialize ttl recalculate only optional 2021-08-04 22:16:13 +08:00
Anton Ivashkin
61016da2b1 Set allow_remote_fs_zero_copy_replication to true by default 2021-08-04 17:08:09 +03:00
Maksim Kita
3f48c85722 StorageSystemReplicas added column replica_is_active 2021-08-04 16:19:42 +03:00
Maksim Kita
4dc4854fad
Merge pull request #27168 from excitoon-favorites/safers3readbuffer 
Safer `ReadBufferFromS3` for merges and backports
 2021-08-04 15:01:21 +03:00
Yatsishin Ilya
55e06ea16b fix 2021-08-04 13:06:01 +03:00
Vladimir Chebotarev
c2410920d3 Safer ReadBufferFromS3 for merges and backports. 2021-08-04 09:14:20 +03:00
Alexey Milovidov
fd3b5680e2 Merge branch 'master' into async-reads 2021-08-03 23:13:31 +03:00
mergify[bot]
dc57254982
Merge branch 'master' into improve_create_or_replace 2021-08-03 11:39:07 +00:00
Maksim Kita
d9383270f6
Merge pull request #27054 from MaxWk/feature/add-default-database-to-system-users 
add column default_database to system.users
 2021-08-03 13:37:15 +03:00
Azat Khuzhin
97851bde08 Fix Distributed over Distributed for WithMergeableStateAfterAggregation* stages 
In case if one Distributed has multiple shards, and underlying
Distributed has only one, there can be the case when the query will be
tried to process from Complete to WithMergeableStateAfterAggregation,
which is obviously wrong.
 2021-08-03 10:10:08 +03:00
Kseniia Sumarokova
5d3abc14c7
Merge pull request #22045 from OmarBazaraa/mongo-ssl 
Support SSL connection in MongoDB engine
 2021-08-03 08:29:34 +03:00
tavplubix
58b50bad3a
Merge pull request #27033 from ClickHouse/database_replicated_fixes 
Fix assertions in Replicated database
 2021-08-02 23:33:18 +03:00
Azat Khuzhin
ff12f5102a Avoid running LIMIT BY/DISTINCT step on the initiator for optimize_distributed_group_by_sharding_key 
Before the following queries was running LimitBy/Distinct step on the
initator:

  select distinct sharding_key from dist order by k

While this can be omitted.
 2021-08-02 21:04:30 +03:00
Azat Khuzhin
2fb95d9ee0 Rework SELECT from Distributed query stages optimization 
Before this patch it wasn't possible to optimize simple SELECT * FROM
dist ORDER BY (w/o GROUP BY and DISTINCT) to more optimal stage
(QueryProcessingStage::WithMergeableStateAfterAggregationAndLimit),
since that code was under
allow_nondeterministic_optimize_skip_unused_shards, rework it and make
it possible.

Also now distributed_push_down_limit is respected for
optimize_distributed_group_by_sharding_key.

Next step will be to enable distributed_push_down_limit by default.

v2: fix detection of aggregates
 2021-08-02 21:04:29 +03:00
Azat Khuzhin
bb6d030fb8 Optimize distributed SELECT w/o GROUP BY 2021-08-02 21:04:29 +03:00
OmarBazaraa
71e1c82f87 Refactor 2021-08-02 15:16:39 +00:00
Alexander Tokmakov
42a8bb6872 fix assertions in Replicated database 2021-08-02 16:19:11 +03:00
OmarBazaraa
91928fdf5a Fix 2021-08-02 13:11:49 +00:00
OmarBazaraa
8577938efe Throw exception in case SSL is not enabled 2021-08-02 12:34:50 +00:00
Anton Popov
fc9d72e75d fix performance of short queries with large number of columns 2021-08-02 15:03:55 +03:00
Nikolai Kochetov
6951e8147d
Merge pull request #27020 from amosbird/shardlevelconstness 
Shard-level const column
 2021-08-02 13:45:20 +03:00
Anton Popov
cc3ed12ed6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-02 02:56:00 +03:00
万康
66631ca680 add column default_database 2021-08-01 08:11:01 +08:00
Alexey Milovidov
2a0cba2b9f Merge branch 'master' into async-reads 2021-08-01 01:25:13 +03:00
alexey-milovidov
701e2ffd79
Merge pull request #27002 from azat/mutation-stuck 
Fix possible mutation stuck due to race with DROP_RANGE
 2021-07-31 21:17:46 +03:00
terrylin
09a4f82d0f code style check 2021-07-31 20:36:40 +08:00
Amos Bird
cd302eacc1
Fix projection match 2021-07-31 15:44:16 +08:00
tavplubix
00ac8194dc
Merge pull request #26896 from ClickHouse/small_improvement_in_queue 
Maybe fix extremely rare `intersecting parts`.
 2021-07-30 13:04:20 +03:00
terrylin
c49c2e5e18 fix compile error 2021-07-30 17:15:04 +08:00
mergify[bot]
41273ef5f3
Merge branch 'master' into mutation-stuck 2021-07-30 08:42:45 +00:00
Azat Khuzhin
bc9a4366d8 Fix possible mutation stack due to race with DROP_RANGE 
After #25884 parts that was dropped (DROP_RANGE) will not be removed
from old mutations and this will stuck the mutation.

Interesting, that this mutation may continue after some merge.
 2021-07-30 08:22:10 +03:00
terrylin
55f3e7e65a improvement of materilize ttl 2021-07-30 12:27:15 +08:00
alesapin
21599ffefc Simple validation for partition id before drop partition 2021-07-29 16:11:05 +03:00
alesapin
5799487caa Fix stupid bug 2021-07-29 14:54:36 +03:00
OmarBazaraa
a665cd3308 Fixes 2021-07-29 08:38:39 +00:00
Alexey Milovidov
1149058084 Merge branch 'master' into async-reads 2021-07-28 22:29:51 +03:00
alesapin
71169d7937
Merge pull request #26716 from nvartolomei/nv/part-cleanup-sequence 
Avoid deleting old parts from FS on shutdown for replicated engine
 2021-07-28 18:29:37 +03:00
OmarBazaraa
3a393042a2 Fix 2021-07-28 15:28:30 +00:00
OmarBazaraa
d8fb1cb0e0 Ref. 2021-07-28 14:08:57 +00:00
alesapin
bdc6858801 More correct comparsion 2021-07-28 13:19:34 +03:00
alesapin
26056cfb25 Fix extremely rare race condition on remove of unsuccessful entry from queue 2021-07-28 13:06:22 +03:00
OmarBazaraa
a154ccef7b Fix 2021-07-28 09:38:34 +00:00
Ivan
1417143560 Update StorageMongoDBSocketFactory.cpp 2021-07-28 09:38:34 +00:00
Ivan
0d029c1cbc Update StorageMongoDBSocketFactory.cpp 2021-07-28 09:38:34 +00:00
OmarBazaraa
58ea75915d Fix 2021-07-28 09:38:34 +00:00
Ubuntu
fce9351256 MongoDB SSL Connection 2021-07-28 09:38:34 +00:00
Alexander Tokmakov
fab4529c97 Merge branch 'master' into rename_materialize_mysql 2021-07-28 12:18:01 +03:00
alesapin
1277db4435
Merge pull request #26129 from ClickHouse/system_zookeeper_log 
Add system.zookeeper_log table
 2021-07-28 11:22:29 +03:00
kssenii
c6ee94db2e Fix 2021-07-28 08:11:10 +00:00
Alexey Milovidov
9baf84224a Fix build 2021-07-28 02:54:20 +03:00
Alexey Milovidov
fd5c1561e2 Merge branch 'master' into async-reads 2021-07-27 23:58:32 +03:00
Alexander Tokmakov
fc9ab2cda7 Merge branch 'master' into rename_materialize_mysql 2021-07-27 22:38:40 +03:00
Nicolae Vartolomei
8b07a7f180 Store exception generated when we tried to update the queue last time 
The use case is to alert when queue contains broken entries. Especially
important when ClickHouse breaks backwards compatibility between
versions and log entries written by newer versions aren't parseable by
old versions.

```
Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected 'quorum: ' before: 'merge_type: 2\n'
```
 2021-07-27 15:42:40 +01:00
Kseniia Sumarokova
b5dce17707
Merge pull request #26821 from azat/rocksdb-improvements 
Add system.rocksdb and read rocksdb/rocksdb_TABLE options from config
 2021-07-27 17:31:12 +03:00
tavplubix
168edaed73
Merge pull request #26808 from amosbird/safe2 
Make sure table is readonly when restarting fails.
 2021-07-27 17:21:08 +03:00
Nikolai Kochetov
97bc754dd8
Merge pull request #26758 from ClickHouse/output-streams-to-processors 
Remove some output streams
 2021-07-27 17:19:26 +03:00
alexey-milovidov
d7b2404119
Merge pull request #15408 from ClickHouse/random_timezone_for_stateless_tests 
Use random timezone in stateless and stateful tests
 2021-07-27 16:18:14 +03:00
Nikolai Kochetov
a9effdad18 Merge branch 'master' into output-streams-to-processors 2021-07-27 11:31:53 +03:00
Azat Khuzhin
8ef677b15f Add ability to set any rocksdb option via config.xml 
v2: Cover rocksdb options in ClickHouse config
v3: add missing __init__.py
v4: Rework rocksdb options from config
v5: add column_family_options support
 2021-07-27 08:40:50 +03:00
Alexey Milovidov
5e2ea4e042 Whitespace 2021-07-27 05:20:03 +03:00
Kseniia Sumarokova
ee13f2e6bc
Merge pull request #26795 from kssenii/fixing-sqlite 
Fix sqlite engine attach
 2021-07-27 00:37:31 +03:00
Azat Khuzhin
c7d2a5aad8 Add system.rocksdb table (with various rocksdb internals) 
v2: fix USE_ROCKSDB check (there is #cmakedefine01 so it is either 0/1,
so we should not check with #ifdef)
 2021-07-26 23:13:59 +03:00
Alexander Tokmakov
63ab38ee09 rename MaterializeMySQL to MaterializedMySQL 2021-07-26 21:17:28 +03:00
Nikolai Kochetov
61d8f880cd Rename some files. 2021-07-26 19:48:25 +03:00
Nikolai Kochetov
9b5a816b43 Merge branch 'master' into output-streams-to-processors 2021-07-26 18:03:11 +03:00
Nikolai Kochetov
0eb563dc1b Fix more tests. 2021-07-26 17:47:29 +03:00
Kseniia Sumarokova
5c5cb5d0d6
Update src/Storages/StorageSQLite.h 
Co-authored-by: Vladimir <vdimir@yandex-team.ru>
 2021-07-26 14:26:06 +03:00
Amos Bird
623faf47e4
Make sure table is readonly when restarting fails. 2021-07-26 19:21:14 +08:00
Nikolai Kochetov
fa1c223269 Fix some tests. 2021-07-26 13:08:40 +03:00
mergify[bot]
2929cf3ba8
Merge branch 'master' into system_zookeeper_log 2021-07-26 08:40:55 +00:00
mergify[bot]
044be267d6
Merge branch 'master' into improve_create_or_replace 2021-07-26 08:38:48 +00:00
kssenii
f3129b80f9 Fix sqlite attach 2021-07-26 07:20:41 +00:00
Alexey Milovidov
5c3f5cb35a Experiment with asynchronous readers 2021-07-26 03:34:36 +03:00
Alexey Milovidov
b6fdb395c2 Less verbose RocksDB logs #26252 2021-07-25 23:09:50 +03:00
alexey-milovidov
22fa1efacb
Merge pull request #26777 from ClickHouse/bolonini-read-from-file 
Merging #25960 (Bolonini/read_from_file)
 2021-07-25 01:33:34 +03:00
Alexey Milovidov
2f2dce399a Fix style 2021-07-25 01:26:21 +03:00
Alexey Milovidov
215505b985 Remove unused error code 2021-07-24 19:51:56 +03:00
Alexey Milovidov
774f6d2617 Allow to read file descriptor multiple times in File storage 2021-07-24 19:50:03 +03:00
Alexey Milovidov
456801ccb8 Merge branch 'pipe_reading' of github.com:BoloniniD/ClickHouse into bolonini-read-from-file 2021-07-24 19:24:22 +03:00
alexey-milovidov
d3127d6265
Merge pull request #26731 from vdimir/fix-async-drain-connection-uaf 
Fix use after free in AsyncDrain connection from S3Cluster
 2021-07-24 18:55:03 +03:00
BoloniniD
2fa02061b7 Fix parallel reads 2021-07-24 15:31:35 +03:00
alexey-milovidov
039bb1cc5e
Merge pull request #26720 from azat/part_log-fix-event_time_microseconds 
Fix event_time_microseconds for REMOVE_PART in system.part_log
 2021-07-24 15:09:37 +03:00
Alexey Milovidov
a4b6181920 Fix weirdness 2021-07-24 05:07:37 +03:00
Alexey Milovidov
edfeb0957f Fix strange code 2021-07-24 04:52:18 +03:00
alexey-milovidov
dab9cfb9c9
Merge pull request #26713 from ClickHouse/remove-more-and--more-streams 
Remove more streams.
 2021-07-24 02:24:10 +03:00
BoloniniD
07c57edbfc Merge branch 'master' of github.com:ClickHouse/ClickHouse into pipe_reading 2021-07-23 23:39:38 +03:00
Nikolai Kochetov
9c92f43359 Update storages. 2021-07-23 22:33:59 +03:00
Nikolai Kochetov
2dc5c89b66 Update Storage::write 2021-07-23 17:25:35 +03:00
vdimir
dccc379d39
Fix use after free in AsyncDrain connection from S3Cluster 2021-07-23 10:40:03 +03:00
Vitaly Baranov
19d5a6ab2f
Merge pull request #26714 from vitlibar/new-function-current-profiles 
New functions currentProfiles(), enabledProfiles(), defaultProfiles().
 2021-07-23 09:10:29 +03:00
Azat Khuzhin
00e2083421 Fix event_time_microseconds for REMOVE_PART in system.part_log 2021-07-23 00:59:08 +03:00
Vitaly Baranov
7afcc65060 Add new functions currentProfiles(), enabledProfiles(), defaultProfiles(). 2021-07-22 22:20:53 +03:00
Nikolai Kochetov
3c17a62686
Merge pull request #26590 from ClickHouse/remove-some-more-streams 
Remove some streams.
 2021-07-22 21:28:50 +03:00
Nicolae Vartolomei
f35e6eee19 Avoid deleting old parts from FS on shutdown for replicated engine 
This was introduced in https://github.com/ClickHouse/ClickHouse/pull/8602.
The idea was to avoid data re-appearing in ClickHouse after DROP/DETACH
PARTITION. This problem was only present in MergeTree engine and I don't
understand why we need to do the same in ReplicatedMergeTree.

For ReplicatedMergeTree the state of truth is stored in ZK, deleting
things from filesystem just introduces inconsistencies and this is the
main source for errors like "No active replica has part X or covering
part".

The resulting problem is fixed by
https://github.com/ClickHouse/ClickHouse/pull/25820, but in my opinion
we would better avoid introducing the ZK/FS inconsistency in the first
place.

When does this inconsistency appear? Often the sequence is like this:

0. Write 2 parts to ZK [all_0_0_0, all_1_1_0]
1. A merge gets scheduled
2. New part replaces old parts [new: all_0_1_1, old: all_0_0_0, all_1_1_0]
3. Replica gets shutdown and old parts are removed from filesystem
4. Replica comes back online, metadata about all parts is still stored in ZK for this new replica.
5. Other replica after cleanup thread runs will have only [all_0_1_1] in
   ZK
5. User triggers a DROP_RANGE after a while (drop range is for all_0_1_9999*)
6. Each replica deletes from ZK only [all_0_1_1]. The replica that got
   restarted uses its in-memory state to choose nodes to delete from ZK.
7. Restart the replica again. It will now think that there are 2 parts
   that it lost and needs to fetch them [all_0_0_0, all_1_1_0].

`clearOldPartsAndRemoveFromZK` which is triggered from cleanup thread
runs cleanup sequence correctly, it first removes things from ZK and
then from filesystem. I don't see much benefit of triggering it on
shutdown and would rather have it called only from a single place.

---

This is a very, very edge case situation but it proves that the current
"fix" (https://github.com/ClickHouse/ClickHouse/pull/25820) isn't
complete.

```
create table test(
    v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'one')
order by v
settings old_parts_lifetime = 30;

create table test2(
    v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'two')
order by v
settings old_parts_lifetime = 30;

create table test3(
    v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'three')
order by v
settings old_parts_lifetime = 30;

insert into table test values (1), (2), (3);
insert into table test values (4);

optimize table test final;

detach table test;
detach table test2;

alter table test3 drop partition tuple();

attach table test;
attach table test2;
```

```
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/one/parts
all_0_0_0
all_1_1_0
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/two/parts
all_0_0_0
all_1_1_0
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/three/parts
```

```
detach table test;
attach table test;
```

`test` will now figure out that parts exist only in ZK and will issue `GET_PART`
after first removing parts from ZK.

`test2` will receive fetch for unknown parts and will trigger part checks itself.
Because `test` doesn't have the parts anymore in ZK `test2` will mark them as LostForever.
It will also not insert empty parts, because the partition is empty.

`test` is left with `GET_PART` in the queue and stuck.

```
SELECT
    table,
    type,
    replica_name,
    new_part_name,
    last_exception
FROM system.replication_queue

Query id: 74c5aa00-048d-4bc1-a2ea-6f69501c11a0

Row 1:
──────
table:          test
type:           GET_PART
replica_name:   one
new_part_name:  all_0_0_0
last_exception: Code: 234. DB::Exception: No active replica has part all_0_0_0 or covering part. (NO_REPLICA_HAS_PART) (version 21.9.1.1)

Row 2:
──────
table:          test
type:           GET_PART
replica_name:   one
new_part_name:  all_1_1_0
last_exception: Code: 234. DB::Exception: No active replica has part all_1_1_0 or covering part. (NO_REPLICA_HAS_PART) (version 21.9.1.1)
```
 2021-07-22 17:48:16 +01:00
Nikolai Kochetov
f56a45155f Merge branch 'master' into remove-more-and--more-streams 2021-07-22 19:10:39 +03:00
Nikolai Kochetov
fd754430eb Remove more streams. 2021-07-22 19:05:52 +03:00
Maksim Kita
e08fc94137
Merge pull request #26678 from CurtizJ/minor-bugfix 
Minor bugfix
 2021-07-22 15:54:24 +03:00
Nikolai Kochetov
3ed3f7a9f7 Fix integration tests. 2021-07-22 13:38:22 +03:00
feihengye
6230ad0160
#26640 change kafka engine max consumers from 16 to physical cpu cores (#26642) 
Co-authored-by: jhonye <jhonye@tencent.com>
 2021-07-22 11:54:08 +03:00
Nikolai Kochetov
5ffd99dfd4
Merge pull request #25674 from amosbird/distributedreturnconnection 
Drain connection asynchronously
 2021-07-22 11:04:49 +03:00
Anton Popov
a36c387e4e fix iterating over list 2021-07-21 21:47:20 +03:00
Nikolai Kochetov
f38de35b14 Rename some constants. 2021-07-21 19:13:17 +03:00
Nikolai Kochetov
65d3e713d6 Fix another one test. 2021-07-21 15:16:13 +03:00
alexey-milovidov
fcca3e3e56
Merge pull request #26423 from ClickHouse/fix-header-for-scalar-query-with-empty-result 
Fix incompatible result type for scalar queries with empty result.
 2021-07-21 02:07:04 +03:00
Nikolai Kochetov
179ec05a72 Remove some streams. 2021-07-20 21:18:43 +03:00
Nikita Mikhaylov
348a3abb0b
Merge pull request #26508 from kitaisreal/storage-dictionary-quoted-names-fix 
Storage Dictionary quoted names fix
 2021-07-20 19:35:48 +03:00
BoloniniD
49ee24e824 What if we replace unique_lock with shared? 2021-07-20 19:22:41 +03:00
alexey-milovidov
3bcef76a9d
Merge pull request #26529 from kssenii/fixing-rabbitmq 
Fix rabbitmq shutdown in case rabbitmq setup was not started
 2021-07-20 16:41:25 +03:00
Nikita Mikhaylov
47c1bb3416
Merge pull request #25822 from ClickHouse/fix_no_column_materialized_mysql 
Fix 'Not found column' error in MaterializeMySQL
 2021-07-20 15:03:54 +03:00
Nikita Mikhaylov
4d3f828beb
Merge pull request #26314 from kssenii/fix-hdfs-crash 
Fix hdfs crash
 2021-07-20 15:01:56 +03:00
Maksim Kita
9b53f4f2b3 Storage Dictionary quoted names fix 2021-07-20 11:35:10 +03:00
kssenii
63c05e6e14 Fix rabbitmq shutdown in case rabbitmq setup was not started 2021-07-19 21:06:10 +00:00
Amos Bird
dbfb699690
Asynchronously drain connections. 2021-07-19 21:53:29 +08:00
Anton Popov
fa6857920d fix arcadia 2021-07-19 16:45:35 +03:00
Nikolai Kochetov
7a24e72e76 Merge branch 'master' into fix-header-for-scalar-query-with-empty-result 2021-07-19 15:48:44 +03:00
Nikolai Kochetov
96e20e2641 Fix some tests. 2021-07-19 15:35:55 +03:00
mergify[bot]
39f71f4a08
Merge branch 'master' into fix_no_column_materialized_mysql 2021-07-19 12:27:38 +00:00
Kseniia Sumarokova
b0ddc4fb30
Merge pull request #26364 from kssenii/fixing-date32 
Continuation of #25774
 2021-07-18 21:44:45 +03:00
alexey-milovidov
b52411a715
Merge pull request #12455 from amosbird/npc 
Nullable primary key with correct KeyCondition
 2021-07-18 17:52:20 +03:00
BoloniniD
a9deeb6a78 Trying to fix freeze on union all 2021-07-18 15:55:24 +03:00
Alexey Milovidov
261a220227 Remove some code 2021-07-17 21:06:46 +03:00
alexey-milovidov
7dda0da027
Merge pull request #26461 from azat/integration-tests-fixes-v21.9.1.7477 
Fix flaky test_replicated_mutations (due to lack of threads in pool)
 2021-07-17 19:22:30 +03:00
alexey-milovidov
b16e01507f
Merge pull request #26464 from azat/ubsan-dir-mon-fix 
Fix undefined-behavior in DirectoryMonitor (for exponential back off)
 2021-07-17 18:18:42 +03:00
Alexey Milovidov
b3b39d440d Accomodate Arcadia trash 2021-07-17 18:12:33 +03:00
BoloniniD
19b673fb3b Style fix for error message 2021-07-17 15:30:03 +03:00
Alexey Milovidov
9bef9d4f2d Revert "Auto version update to [21.10.1.1] [54455]" 
This reverts commit f2b1395127.
 2021-07-17 13:17:30 +03:00
alexey-milovidov
ca37548888
Merge pull request #26430 from azat/fix-dist-msg 
Fix "While sending batch" (on Distributed async send)
 2021-07-17 13:01:36 +03:00
Azat Khuzhin
d2967ffa0b Fix undefined-behavior in DirectoryMonitor (for exponential back off) 
UBsan reports [1]:

    ../src/Storages/Distributed/DirectoryMonitor.cpp:435:54: runtime error: 2.30584e+19 is outside the range of representable values of type 'unsigned long'

  [1]: https://clickhouse-test-reports.s3.yandex.net/0/10f3500b3be73c9498d994d189784c7d44ed6793/stress_test_(undefined).html#fail1
 2021-07-17 12:18:05 +03:00
Azat Khuzhin
071dc7c3f5 Improve "Not executing log entry queue-X of type MUTATE_PART" message 2021-07-17 11:50:04 +03:00
BoloniniD
2e05063298 Merge branch 'master' of github.com:ClickHouse/ClickHouse into pipe_reading 2021-07-17 10:30:18 +03:00
robot-clickhouse
f2b1395127 Auto version update to [21.10.1.1] [54455] 2021-07-17 08:45:10 +03:00
Alexey Milovidov
bd924a5b8a Whitespace 2021-07-17 01:59:39 +03:00
alexey-milovidov
bc907bd27c
Merge pull request #26336 from azat/dist-per-table-monitor-settings 
Add ability to set Distributed directory monitor settings via CREATE TABLE
 2021-07-17 01:49:40 +03:00
alexey-milovidov
1701cc429d
Merge pull request #26353 from azat/optimize_distributed_group_by_sharding_key-fix 
Fix optimize_distributed_group_by_sharding_key for multiple columns
 2021-07-17 01:45:10 +03:00
alexey-milovidov
c373b86743
Merge pull request #26371 from CurtizJ/improve-perf-get-columns 
Improve performance of getting list of columns
 2021-07-17 01:34:15 +03:00
alexey-milovidov
149c5a0e9b
Merge pull request #26381 from ClickHouse/separate-step-for-distributed 
Add separate step to read from remote.
 2021-07-17 01:29:43 +03:00
Azat Khuzhin
80e614318c Fix "While sending batch" (on Distributed async send) 2021-07-16 22:27:46 +03:00
Kseniia Sumarokova
e844cec16f
Merge branch 'master' into fix-hdfs-crash 2021-07-16 22:21:30 +03:00
Anton Popov
048e089a58 remove useless code 2021-07-16 17:47:03 +03:00
Anton Popov
03c785931a fix pushdown of limit to reading stage 2021-07-16 17:27:38 +03:00
Anton Popov
c45a9e3bd6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-16 17:11:34 +03:00
mergify[bot]
126a2d7ad1
Merge branch 'master' into dist-per-table-monitor-settings 2021-07-16 13:32:40 +00:00
BoloniniD
fcbfa27f33 Merge branch 'master' of github.com:ClickHouse/ClickHouse into pipe_reading 2021-07-16 16:28:26 +03:00
Vitaly Baranov
427813071d
Merge pull request #26363 from vitlibar/fix-crash-login-as-dropped-user 
Fix possible crash when login as dropped user
 2021-07-16 09:47:35 +03:00
Azat Khuzhin
a3653bd665 Fix overflow in exponential sleep in DirectoryMonitor 
UBsan reports:

    SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../src/Storages/Distributed/DirectoryMonitor.cpp:435:53 in
    ../src/Storages/Distributed/DirectoryMonitor.cpp:435: runtime error: 1.15292e+19 is outside the range of representable values of type 'long'
        0 0x1df0c286 in DB::StorageDistributedDirectoryMonitor::run() obj-x86_64-linux-gnu/../src/Storages/Distributed/DirectoryMonitor.cpp:435:53

It is pretty easy to reproduce by limiting max_server_memory_usage
before staring the test.
 2021-07-16 04:10:47 +03:00
Azat Khuzhin
f3d3ec44a6 Add ability to set Distributed directory monitor settings via CREATE TABLE 2021-07-16 04:10:47 +03:00
Anton Popov
9e3af27f40 better performance of getSampleBlockForColumns 2021-07-15 20:36:48 +03:00
Nikolai Kochetov
f36d14f68f Add separate step to read from remote. 2021-07-15 19:15:16 +03:00
Vitaly Baranov
6c74b44729 Fix possible crash when login as dropped user. 2021-07-15 19:01:34 +03:00
Anton Popov
cfdf8fadcf fix build 2021-07-15 18:31:21 +03:00
BoloniniD
f27e69a31f Merge branch 'master' of github.com:ClickHouse/ClickHouse into pipe_reading 2021-07-15 17:37:37 +03:00
Anton Popov
5f71a6f5bb improve performance of getting columns list 2021-07-15 16:29:01 +03:00
kssenii
3ba2d114d5 Revert "Merge pull request #26352 from ClickHouse/revert-25774-datatype-date32" 
This reverts commit 94a210650f, reversing
changes made to 0893b9ff8e.
 2021-07-15 11:41:52 +00:00
Alexey Milovidov
d984fc4301 Fix flaky integration test 2021-07-15 13:35:37 +03:00
alexey-milovidov
9ba4064c86
Revert "Datatype Date32, support range 1925 to 2283" 2021-07-15 10:53:35 +03:00
Azat Khuzhin
7b209694d5 Fix optimize_distributed_group_by_sharding_key for multiple columns 
Before we incorrectly check that columns from GROUP BY was a subset of
columns from sharding key, while this is not right, consider the
following example:

    select k1, any(k2), sum(v) from remote('127.{1,2}', view(select 1 k1, 2 k2, 3 v), cityHash64(k1, k2)) group by k1

Here the columns from GROUP BY is a subset of columns from sharding key,
but the optimization cannot be applied, since there is no guarantee that
particular shard contains distinct values of k1.

So instead we should check that GROUP BY contains all columns that is
required for calculating sharding key expression, i.e.:

    select k1, k2, sum(v) from remote('127.{1,2}', view(select 1 k1, 2 k2, 3 v), cityHash64(k1, k2)) group by k1, k2
 2021-07-15 09:09:58 +03:00
alexey-milovidov
0893b9ff8e
Merge pull request #26218 from amosbird/projection-improve1 
Aggressive IN index analysis for projections.
 2021-07-15 08:13:45 +03:00
alexey-milovidov
323b60fcea
Merge pull request #26321 from ClickHouse/tavplubix-patch-2 
Fix rare logical race in `cloneReplica(...)`
 2021-07-15 08:00:26 +03:00
kssenii
f20ea87bfb Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into add-sqlite-database-path-check 2021-07-14 20:49:05 +00:00
Kseniia Sumarokova
e42a26a585
Merge pull request #26234 from jasine/master 
convert timestamp and  timestamptz data types to DateTime64 in postgres engine
 2021-07-14 23:42:51 +03:00
kssenii
61652acede Add test 2021-07-14 22:49:16 +03:00
tavplubix
41bb8acbb5
Update StorageReplicatedMergeTree.cpp 2021-07-14 20:05:50 +03:00
Filatenkov Artur
f1702d356e
Merge pull request #26120 from lthaooo/settings-merge_selecting_sleep_ms 
add merge_selecting_sleep_ms setting
 2021-07-14 18:28:47 +03:00
Amos Bird
28b567a01b
Update 2021-07-14 23:09:17 +08:00
alexey-milovidov
55fc7323fd
Update MergeTreeDataSelectExecutor.h 2021-07-14 22:56:52 +08:00
alexey-milovidov
1ef068596c
Update MergeTreeDataSelectExecutor.cpp 2021-07-14 22:56:52 +08:00
Amos Bird
0558ecdc3f
Aggressive IN index analysis for projections. 2021-07-14 22:56:52 +08:00
kssenii
d62cbbb8a5 Unification 2021-07-14 13:16:30 +00:00
Peng Jian
5be892484d fix build 2021-07-14 20:18:32 +08:00
Peng Jian
4bc07771ee Fix the clearOldTemporaryDirectories is executed too frequently 2021-07-14 19:26:46 +08:00
Anton Popov
e2249bf1e5
Merge pull request #26256 from zxc111/issue-822 
Check hash function at table creation
 2021-07-14 14:16:02 +03:00
Zhichang Yu
fe5d17680e Merge remote-tracking branch 'origin/master' into zero_copy_hdfs 2021-07-14 02:12:28 +00:00
jasine
ff5ad15aa5
fix: MaterializedPostgreSQL style 2021-07-14 09:27:51 +08:00
jasine
bd59ca22df
fix: style and MaterializedPostgreSQL test (#3) 
fix: style and MaterializedPostgreSQL test (#3)
 2021-07-14 08:51:14 +08:00
kssenii
b393603b57 Fix build 2021-07-13 22:25:26 +00:00
Nikita Mikhaylov
601d04f55d ignore 2021-07-13 22:16:27 +00:00
kssenii
da09ce3f20 Remove unused 2021-07-13 19:51:02 +00:00
kssenii
a90df59b92 Better 2021-07-13 19:46:06 +00:00
jasine
a0ec8e9878
fix: DateTime64 on MaterializedPostgreSQL (#2) 
* fix: DateTime64 on MaterializedPostgreSQL

* Update StoragePostgreSQL.cpp
 2021-07-14 01:59:33 +08:00
Alexander Tokmakov
f0ed6be269 Merge branch 'master' into system_zookeeper_log 2021-07-13 20:49:37 +03:00
Nikita Mikhaylov
33735d02ab fix 2021-07-13 17:11:40 +00:00
Maksim Kita
3d05f07ece
Merge pull request #25917 from excitoon-favorites/defaultblocksizeprediction 
Using required columns for block size prediction
 2021-07-13 20:05:41 +03:00
Vitaly Baranov
3c395389b0
Merge pull request #24206 from alexelex/alexelex-encryption-1 
Data encryption on VFS level
 2021-07-13 19:50:25 +03:00
Anton Popov
cb042afa9d pushdown limit while reading in order of primary key 2021-07-13 17:24:45 +03:00
mergify[bot]
d5898ecba6
Merge branch 'master' into fix_no_column_materialized_mysql 2021-07-13 14:03:05 +00:00
zxc111
7b2fed2951 use flag to check sample_condition 2021-07-13 21:17:22 +08:00
Anton Popov
19d0f1bbfc Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-13 16:12:08 +03:00
kssenii
9688efde96 Merge branch 'add-sqlite-database-path-check' of github.com:kssenii/ClickHouse into add-sqlite-database-path-check 2021-07-13 12:40:43 +00:00
kssenii
f86817d57c Check inclusion inside user_files 2021-07-13 12:38:37 +00:00
Kseniia Sumarokova
d80bcf0bd3
Update StorageSQLite.cpp 2021-07-13 14:40:27 +03:00
Ivan Lezhankin
ab26aed6f9 Merge remote-tracking branch 'upstream/master' into async-insert 2021-07-13 13:54:09 +03:00
xiedeyantu
df438f4288 Modify code comments 2021-07-13 17:52:24 +08:00
kssenii
48ffd3b81d Add check for sqlite database path 2021-07-13 09:40:46 +00:00
alexey-milovidov
b4741ed2fe
Merge pull request #23140 from amosbird/fixrandomoneshardinsert 
Fix misbehaviors when insert_distributed_one_random_shard = 1
 2021-07-13 11:47:53 +03:00
alexey-milovidov
af7ecb7f7a
Merge pull request #25798 from azat/optimize_skip_unused_shards_rewrite_in-types-fix 
Fix optimize_skip_unused_shards_rewrite_in for non-UInt64 types
 2021-07-13 11:40:59 +03:00
alexey-milovidov
101ee12f18
Merge pull request #26038 from amosbird/projection-fix11 
Fix normal projection with prewhere
 2021-07-13 11:40:08 +03:00
Vitaly Baranov
e086e5e384
Merge branch 'master' into alexelex-encryption-1 2021-07-13 11:28:41 +03:00
jasine
9832c48c33
fix postgres test and array of DateTime64 (#1) 
fix postgres test and array of DateTime64
 2021-07-13 15:28:15 +08:00
Zhichang Yu
b4e6689bf9 fix test_hdfs_zero_copy_replication_move[tiered_copy-2] 2021-07-13 07:20:23 +00:00
Zhichang Yu
5047c758f4 fix per review 2021-07-13 07:20:20 +00:00
Zhichang Yu
fbd5eee8a1 hdfs zero copy 2021-07-13 07:19:12 +00:00
Kseniia Sumarokova
a24d2c1faf
Merge pull request #24194 from g-arslan/add-sqlite-support 
SQLite integration
 2021-07-13 09:28:13 +03:00
alexey-milovidov
4b1e38eed9
Merge branch 'master' into projection-fix11 2021-07-13 04:52:15 +03:00
alexey-milovidov
4183f3164a
Merge branch 'master' into fixrandomoneshardinsert 2021-07-13 04:46:40 +03:00
Alexey Milovidov
29e8b783ba Comments 2021-07-13 03:59:52 +03:00
alexey-milovidov
f5f9e35331
Merge pull request #26241 from evillique/column_size 
Add column sizes for `Log` and `TinyLog` tables
 2021-07-13 03:54:03 +03:00
alexey-milovidov
d28b143c49
Merge pull request #26246 from FArthur-cmd/adding-system-warnings-26039 
Adding system warnings 26039
 2021-07-13 03:44:53 +03:00
alexey-milovidov
5295f7ebb2
Merge pull request #26235 from ClickHouse/check-non-deterministic-functions-in-keys 
Check for non-deterministic functions in keys, including constant expressions
 2021-07-13 02:46:44 +03:00
Nikolay Degterinsky
75be358250 Fix style 2021-07-12 19:47:13 +00:00
Nikolay Degterinsky
8d2da5933c Minor fixes 2021-07-12 19:04:53 +00:00
Kseniia Sumarokova
4a6b4d6c26
Merge pull request #25774 from liuneng1994/datatype-date32 
Datatype Date32, support range 1925 to 2283
 2021-07-12 20:33:39 +03:00
zxc111
e0261866a4 add settings && add test case 2021-07-13 01:15:35 +08:00
Nikolay Degterinsky
8bdd1e93ae Fix 2021-07-12 13:40:22 +00:00
Artur
278099e663 correct code style 2021-07-12 12:21:18 +00:00
Nikolay Degterinsky
ab3ab9fa41 Fix mutable 2021-07-12 11:00:26 +00:00
Nikolay Degterinsky
fb661bc38d Minor fixes 2021-07-12 10:58:53 +00:00
Artur
1e90287e9a Adding tests for system.warnings 2021-07-12 10:57:39 +00:00
Nikolay Degterinsky
7a8361c562 Add columns size for Tiny Log 2021-07-12 10:51:05 +00:00
Nikolay Degterinsky
dc18f68b72 Minor fix 2021-07-12 10:21:17 +00:00
Nikolay Degterinsky
fd957f332e Add column sizes for Log tables 2021-07-12 10:06:24 +00:00
mergify[bot]
b7ae4e4ced
Merge branch 'master' into async-insert 2021-07-12 09:21:37 +00:00
kssenii
aecb344baa Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into g-arslan-add-sqlite-support 2021-07-12 07:55:39 +00:00
Alexey Milovidov
2a2aa69459 Handle non-existent "detached" directories 2021-07-12 10:01:26 +03:00
Alexey Milovidov
ff5c433f10 Check for non-deterministic functions in keys, including constant expressions 2021-07-12 08:58:19 +03:00
alexey-milovidov
8f1d7e67ca
Merge pull request #26003 from ClickHouse/ditch-aio 
Remove AIO for sequential reads
 2021-07-12 03:25:23 +03:00
zxc111
cd97478c06 check sampling condition when create table 2021-07-12 00:38:20 +08:00
neng.liu
4669839fbb add comment and update ya.make 2021-07-11 04:55:42 -07:00
Maksim Kita
61bf623a3c
Merge pull request #26210 from ClickHouse/system-stack-trace-fix 
Fix excessive newline in `system.stack_trace`
 2021-07-11 12:31:16 +03:00
Alexey Milovidov
1ce40d6629 Fix bad code (default function argument) 2021-07-11 04:01:23 +03:00
Alexey Milovidov
002e648183 Merge branch 'system-stack-trace-fix' into ditch-aio 2021-07-11 03:35:26 +03:00
Alexey Milovidov
4df8de76c8 Fix #24124 2021-07-11 03:34:00 +03:00
Alexey Milovidov
81646f8389 Merge branch 'master' into ditch-aio 2021-07-11 03:06:45 +03:00
kssenii
10611ef92a Normal exceptions 2021-07-10 11:00:59 +00:00
kssenii
d250d79e33 Generate ya make 2021-07-10 08:07:55 +00:00
kssenii
78094a82ab Better 2021-07-10 08:05:23 +00:00
kssenii
40281f9283 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into g-arslan-add-sqlite-support 2021-07-10 06:50:46 +00:00
robot-clickhouse
8ae8b26954 Auto version update to [21.9.1.1] [54454] 2021-07-10 08:22:52 +03:00
Alexander Tokmakov
ca45054de5 add enums 2021-07-10 00:18:20 +03:00
kssenii
46e9e21419 Fix build 2021-07-09 15:47:57 +00:00
Artur
8113580237 Add warning in debug mode 2021-07-09 14:40:32 +00:00
kssenii
750f9b82fb Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into g-arslan-add-sqlite-support 2021-07-09 10:28:05 +00:00
kssenii
1bf9a9f29f Add .cmake 2021-07-09 10:27:53 +00:00
BoloniniD
21f1e0e626 Merge branch 'master' of github.com:ClickHouse/ClickHouse into pipe_reading 2021-07-09 12:36:00 +03:00
terrylin
faed6263bb add merge_selecting_sleep_ms setting 2021-07-09 17:29:17 +08:00
Artur
146a45a8b3 Adding system.warnings 2021-07-09 09:27:51 +00:00
BoloniniD
407652457f Added fstat for table fd + more fixes 2021-07-09 12:20:11 +03:00
Anton Popov
9f1ffa777f remove unused code 2021-07-08 15:06:33 +03:00
kssenii
b7f3d16c5f Fix conversion 2021-07-07 20:39:41 +00:00
kssenii
ff56f1787c Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into g-arslan-add-sqlite-support 2021-07-07 15:19:12 +00:00
BoloniniD
0f584c3e76 Fix style and add check for seekability 2021-07-07 14:21:32 +03:00
alesapin
4c85dae572
Merge pull request #25743 from ClickHouse/fix_aggregation_ttl 
Fix bug in execution of TTL GROUP BY
 2021-07-07 10:49:16 +03:00