alexey-milovidov
5ac6a99542
Merge pull request #27927 from ClickHouse/fix_27832
...
Сheck cluster name before creating Distributed
2021-08-21 10:40:23 +03:00
Nikita Mikhaylov
f72457fa98
Lower the size of the binaries
2021-08-21 02:20:23 +00:00
kssenii
945e2c4ce5
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-20 18:01:23 +00:00
Ilya Yatsishin
624cb43f7f
Merge pull request #27885 from azat/kafka-write-assert
...
Fix incorrect assertion during writing to StorageKafka
2021-08-20 18:53:22 +03:00
Alexander Tokmakov
42378b5913
fix
2021-08-20 17:05:53 +03:00
Alexander Tokmakov
59eb3aa9a9
avoid too long waiting for inactive replicas
2021-08-20 15:59:57 +03:00
Alexander Tokmakov
8c6dd18917
check cluster name before creating Distributed
2021-08-20 14:55:04 +03:00
mergify[bot]
66c6a85211
Merge branch 'master' into master
2021-08-20 11:50:30 +00:00
Maksim Kita
5f2b28639f
Merge branch 'master' into create-user-defined-lambda-function
2021-08-20 11:53:08 +03:00
Azat Khuzhin
b60e5ac801
Fix incorrect assertion during writing to StorageRabbitMQ
2021-08-20 07:14:20 +03:00
Azat Khuzhin
5947e54c1b
Fix incorrect assertion during writing to StorageKafka
...
The problem it does not triggered in CI always because buffers was not
destroyed by that time.
Fixes : #26547
2021-08-19 22:07:20 +03:00
nvartolomei
c09c90125f
Merge branch 'master' into nv/last-queue-update-exception
2021-08-19 10:29:16 +01:00
Nikolai Kochetov
7dcff5c90e
Merge pull request #27742 from amosbird/projection-improvement3
...
Improve projection analysis.
2021-08-19 12:11:20 +03:00
alexey-milovidov
215385e4d0
Merge pull request #27794 from filimonov/kafka_null_messages
...
Avoid nullptr dereference during processing of NULL messages in Kafka for some formats
2021-08-19 11:25:07 +03:00
Nikolai Kochetov
3e44ef437e
Merge pull request #26639 from ClickHouse/fix-bad-cast
...
Fix bad cast
2021-08-19 11:17:26 +03:00
alesapin
0aec151719
Merge pull request #27808 from ClickHouse/fix_replicas_may_diverge
...
Fix a couple of bugs that may cause replicas to diverge
2021-08-19 09:58:07 +03:00
Maksim Kita
01682a86b3
Updated user defined functions implementation
2021-08-19 00:54:55 +03:00
Alexander Tokmakov
0ed046eb7b
remove irrelevant comments
2021-08-18 15:33:11 +03:00
alexey-milovidov
bcfab277e7
Update ReadBufferFromKafkaConsumer.cpp
2021-08-18 15:25:54 +03:00
Nikolai Kochetov
a46fe11e2c
Try fix test_mutations_with_merge_background_task
2021-08-18 13:30:02 +03:00
Alexander Tokmakov
09ff66da0e
fix a couple of bugs that may cause replicas to diverge
2021-08-18 12:50:46 +03:00
Mikhail Filimonov
05d77d2873
Safer processing of NULL messages in Kafka for some formats
2021-08-18 11:26:57 +02:00
alesapin
5cc98c67a4
Fix
2021-08-18 11:56:18 +03:00
mergify[bot]
f11e396151
Merge branch 'master' into nv/last-queue-update-exception
2021-08-18 07:00:50 +00:00
vicgao
c16bd25399
modify column datatype to Nullable(datatype) or Nullable(datatype) to Nullable(datatype) donot allways need mutations.
2021-08-17 21:14:13 +08:00
Nikita Mikhaylov
cc8ad0a4e7
Merge pull request #27733 from Algunenano/i27620_take2
...
Refactor arrayJoin check on partition expressions
2021-08-17 15:57:16 +03:00
vicgao
b5610d9a83
modify column datatype to Nullable(datatype) or Nullable(datatype) to Nullable(datatype) donot allways need mutations.
2021-08-17 20:52:16 +08:00
Nikolai Kochetov
98eb619b4b
Merge pull request #23367 from Avogar/short-circuit
...
Implement short circuit function evaluation
2021-08-17 12:36:07 +03:00
Amos Bird
ea7dc495cf
Better code.
2021-08-17 12:32:49 +08:00
Raúl Marín
90f224807a
Style
2021-08-16 17:31:37 +02:00
kssenii
bec34531bc
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-16 14:48:32 +00:00
Amos Bird
90881aab09
Better code style
2021-08-16 22:17:07 +08:00
Raúl Marín
3ec904a953
Only analyze if tuple has arguments
2021-08-16 14:47:41 +02:00
Amos Bird
b162a2b699
Improve projection analysis.
...
Remove duplicate index analysis and avoid possible invalid limit checks
during projection analysis.
2021-08-16 20:11:57 +08:00
Raúl Marín
34099bc196
Refactor arrayJoin check on partition expressions
2021-08-16 12:46:18 +02:00
Alexey Milovidov
d184b79bba
Progress on async reads.
2021-08-16 03:00:32 +03:00
alexey-milovidov
4a36366925
Merge pull request #27648 from Algunenano/i27620
...
Disable arrayJoin on partition expressions
2021-08-16 02:53:36 +03:00
Alexey Milovidov
0cd3f25ba4
Merge remote-tracking branch 'origin/master' into async-reads
2021-08-16 02:41:01 +03:00
alexey-milovidov
76b0502482
Merge pull request #27690 from kitaisreal/remove-dense-hash-map-set
...
Removed DenseHashMap, DenseHashSet
2021-08-16 02:39:41 +03:00
Maksim Kita
5f5470c2cd
Removed DenseHashMap, DenseHashSet
2021-08-15 13:32:56 +03:00
Alexey Milovidov
020beb885d
Merge branch 'master' into async-reads
2021-08-15 12:08:19 +03:00
mergify[bot]
bf426cfb96
Merge branch 'master' into fix_intersecting_parts
2021-08-15 09:07:20 +00:00
alexey-milovidov
d88cf81d71
Merge branch 'master' into fix-bad-cast
2021-08-15 09:06:42 +03:00
mergify[bot]
353a324887
Merge branch 'master' into i27620
2021-08-15 02:32:26 +00:00
alexey-milovidov
ab4415a22b
Update MergeTreeData.cpp
2021-08-15 05:28:38 +03:00
kssenii
daac6323fa
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-14 13:15:04 +00:00
kssenii
200d75646a
Fix
2021-08-13 20:58:03 +00:00
Alexander Tokmakov
c534363abe
fix intersecting parts
2021-08-13 20:33:10 +03:00
Raúl Marín
73cb7d55ec
Disable arrayJoin on partition expressions
2021-08-13 16:18:46 +02:00
mergify[bot]
319e5af6eb
Merge branch 'master' into short-circuit
2021-08-13 13:01:46 +00:00
Nikolai Kochetov
66fdb859f3
Merge pull request #27528 from amosbird/projection-fix12
...
Fix projection materialization with missing columns
2021-08-13 15:20:51 +03:00
Kseniia Sumarokova
0917488cec
Merge pull request #27382 from kssenii/cast-internal
...
add Cast internal function
2021-08-13 13:13:51 +03:00
Nikolai Kochetov
ad00aaa18c
Merge pull request #27575 from kitaisreal/removed-some-data-streams
...
Removed some data streams
2021-08-13 12:59:00 +03:00
Nikolai Kochetov
d207fbc699
Merge pull request #27152 from qoega/fix-access-gtest-in-arcadia
...
Fix arcadia src/Access gtest
2021-08-13 10:51:29 +03:00
alexey-milovidov
f20eae9a45
Merge pull request #27577 from ClickHouse/remove-streams-from-lv
...
Remove streams from lv
2021-08-13 03:01:33 +03:00
Nikolai Kochetov
224e0a547a
Fix build.
2021-08-12 16:02:17 +03:00
Nikolai Kochetov
9c066d476e
Fix tests.
2021-08-12 13:46:58 +03:00
Nikolai Kochetov
5e5a6ff654
Merge branch 'master' into qoega-fix-access-gtest-in-arcadia
2021-08-12 12:02:15 +03:00
mergify[bot]
80eaea1c51
Merge branch 'master' into short-circuit
2021-08-12 08:38:45 +00:00
Maksim Kita
124a87684f
Removed some data streams
2021-08-11 23:39:01 +03:00
kssenii
dcc6f597dc
Move templates from FunctionsConvertion
2021-08-11 19:09:51 +00:00
kssenii
7555fc19fd
Revert "Move some templates"
...
This reverts commit 7f247becca
.
2021-08-11 18:49:25 +00:00
Nikolai Kochetov
beed4c21a5
Use processors in Storage::watch
2021-08-11 20:28:54 +03:00
mergify[bot]
68a52fc065
Merge branch 'master' into projection-fix12
2021-08-11 17:08:04 +00:00
Alexander Kuzmenkov
8f9845722a
Merge pull request #27250 from azat/minmax-nullable-new-format
...
Add new index data skipping minmax index format for proper Nullable support
2021-08-11 13:56:16 +03:00
Nikolai Kochetov
81b85c30f7
Remove streams from LV part 1.
2021-08-11 11:35:10 +03:00
kssenii
7d8b694a9f
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal
2021-08-11 07:37:21 +00:00
kssenii
7f247becca
Move some templates
2021-08-11 07:32:46 +00:00
Alexey Milovidov
08f8511854
Fix Style
2021-08-11 06:52:28 +03:00
alexey-milovidov
54d8282209
Merge pull request #27248 from azat/mt-mutation-stuck-fix
...
Fix mutation stuck on invalid partitions in non-replicated MergeTree
2021-08-11 06:10:15 +03:00
alexey-milovidov
a0a5c0da32
Update BackgroundJobsExecutor.cpp
2021-08-11 06:08:30 +03:00
Alexey Milovidov
c590a239d5
Merge branch 'master' into fix-bad-cast
2021-08-11 05:48:37 +03:00
Amos Bird
e63c26edb7
Fix projection materialization with missing columns
2021-08-10 21:47:27 +08:00
Nikolai Kochetov
8613cfd4e2
Merge branch 'master' into fix-access-gtest-in-arcadia
2021-08-10 16:13:11 +03:00
Maksim Kita
87fb75b272
Merge pull request #27191 from FArthur-cmd/memory-tracker-26043
...
Client with flag for memory tracker
2021-08-10 14:01:02 +03:00
kssenii
eb4746ac69
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal
2021-08-10 09:38:41 +00:00
Nikolai Kochetov
a1ec7f75c5
Merge branch 'master' into qoega-fix-access-gtest-in-arcadia
2021-08-10 11:31:47 +03:00
Nikolai Kochetov
bb4c11cd27
Merge pull request #27298 from ClickHouse/fix-23515
...
Fix bug from #23515 .
2021-08-09 23:25:30 +03:00
Nikolai Kochetov
8cc493a3cd
Try fix build.
2021-08-09 18:09:29 +03:00
Pavel Kruglov
70b51133c1
Try to simplify code
2021-08-09 18:01:08 +03:00
Pavel Kruglov
0662df8b76
Fix performance with JIT, add arguments to function isSuitableForShortCircuitArgumentsExecution
2021-08-09 17:54:14 +03:00
Pavel Kruglov
e792fa588f
Mark all Functions as sutable or not for executing as short circuit arguments
2021-08-09 17:50:09 +03:00
Maksim Kita
b8b24684b4
Merge pull request #27273 from ClickHouse/remove-streams-from-dicts
...
Remove streams from dicts
2021-08-09 15:58:37 +03:00
Nicolae Vartolomei
3f291b024a
Use plain mutex instead of MultiVersion
2021-08-09 13:58:23 +01:00
kssenii
d767f6704f
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-09 12:14:23 +00:00
Nikita Mikhaylov
73d3f2c60f
Merge pull request #26313 from fastio/control_execution_period_of_clearOldTemporaryDirectories
...
Control the execution period of clear old temporary directories by parameter
2021-08-09 14:29:24 +03:00
Nikolai Kochetov
4a00e402ae
Fix spelling.
2021-08-09 12:09:09 +03:00
Nikolai Kochetov
372293b85d
Merge branch 'master' into fix-27179
2021-08-09 11:59:22 +03:00
Alexey Milovidov
8b9da6fe26
Merge branch 'master' into async-reads
2021-08-09 03:33:32 +03:00
mergify[bot]
316c254536
Merge branch 'master' into fix-bad-cast
2021-08-08 23:45:19 +00:00
kssenii
7991bb6e83
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into cast-internal
2021-08-08 21:30:17 +00:00
Azat Khuzhin
038241b6ed
Add new index data skipping minmax index format for proper Nullable support
...
Note, that it cannot be done w/o new extension, since index does not
have any header.
v2: use IDisk interface for existence check
v3: remove extra file existence check
v4: fix MATERIALIZE INDEX
2021-08-08 19:30:12 +03:00
Azat Khuzhin
91d7f3daa7
Fix lock-order-inversion while notifying about finished mutations
...
TSAN reports [1]:
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=36)
Cycle in lock order graph: M16388 (0x7b7400011d68) => M1030334152907497744 (0x000000000000) => M16388
Mutex M1030334152907497744 acquired here while holding mutex M16388 in thread T4:
0 pthread_mutex_lock <null> (clickhouse+0x967d536)
1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse+0x1b25c7d9)
2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x1b25c7d9)
3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x15220cd9)
4 DB::StorageMergeTree::getIncompleteMutationsStatus(long, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std:>
5 DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0::operator()() const obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:464:36 (clickhouse+0x1521e2b1)
6 void std::__1::condition_variable::wait<DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::$_0>(std::__1::unique_lock<std::__1::mutex>&, DB::StorageMergeTree::waitForMutation(lon>
7 DB::StorageMergeTree::waitForMutation(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:469:29 (clickhouse+0x1521e2b1)
8 DB::StorageMergeTree::mutate(DB::MutationCommands const&, std::__1::shared_ptr<DB::Context const>) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:496:9 (clickhouse+0x15221738)
9 DB::InterpreterAlterQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterAlterQuery.cpp:113:16 (clickhouse+0x141182f6)
10 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:560:32 (clickhouse+0x149152f6)
11 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:909:30 (clickhous>
12 DB::TCPHandler::runImpl() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:313:24 (clickhouse+0x153270af)
Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message
Mutex M16388 acquired here while holding mutex M1030334152907497744 in thread T59:
0 pthread_mutex_lock <null> (clickhouse+0x967d536)
1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/__threading_support:405:10 (clickhouse+0x1b25c7d9)
2 std::__1::mutex::lock() obj-x86_64-linux-gnu/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x1b25c7d9)
3 std::__1::unique_lock<std::__1::mutex>::unique_lock(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:119:61 (clickhouse+0x15226c31)
4 DB::StorageMergeTree::selectPartsToMutate(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&) obj-x86_64-linux->
5 DB::StorageMergeTree::scheduleDataProcessingJob(DB::IBackgroundJobExecutor&) obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:1060:28 (clickhouse+0x15228b10)
6 DB::BackgroundJobsExecutor::scheduleJob() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:229:17 (clickhouse+0x14f441dc)
7 DB::IBackgroundJobExecutor::backgroundTaskFunction() obj-x86_64-linux-gnu/../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:185:10 (clickhouse+0x14f438ed)
[1]: https://clickhouse-test-reports.s3.yandex.net/27248/4f3b80ff33c846465983aa2bc9ae9490e1118b15/fuzzer_tsan/report.htmlfail1
2021-08-08 19:28:29 +03:00
Azat Khuzhin
7964355ecf
Fix mutation stuck on invalid partitions in non-replicated MergeTree
...
v2: Do not try to process empty mutations
Found with flaky check [1].
[1]: https://clickhouse-test-reports.s3.yandex.net/27248/66e8c0833392c20ba8dba3780f2b0d5c18f8194e/functional_stateless_tests_flaky_check_(address).html#fail1
2021-08-08 19:28:15 +03:00
Azat Khuzhin
5139067631
Guard BackgroundJobsExecutor from thread termination in case of uncaught exception
2021-08-08 19:27:58 +03:00
Azat Khuzhin
702d9955c0
Fix distributed queries with zero shards and aggregation
2021-08-08 19:22:49 +03:00
mergify[bot]
ec331132c1
Merge branch 'master' into fix-bad-cast
2021-08-08 15:20:27 +00:00
Azat Khuzhin
3be3c503aa
Fix some comments
2021-08-08 09:58:07 +03:00
alexey-milovidov
a7e26506cf
Merge pull request #26951 from ianton-ru/zero_copy_by_default
...
Set allow_remote_fs_zero_copy_replication to true by default
2021-08-08 04:46:49 +03:00
alexey-milovidov
ba3fbbaab0
Update ReplicatedMergeTreeMergeStrategyPicker.cpp
2021-08-08 04:43:59 +03:00
alexey-milovidov
e6e9a27d8f
Merge pull request #27377 from evillique/bz2
...
Add support for bzip2 compression method
2021-08-08 04:40:13 +03:00
Alexey Milovidov
1ad1e62b47
Fix unit test
2021-08-08 04:02:48 +03:00
Alexey Milovidov
24cef99065
Merge branch 'master' into fix-bad-cast
2021-08-08 04:00:29 +03:00
alexey-milovidov
c5207fc237
Merge pull request #26466 from azat/optimize-dist-select
...
Rework SELECT from Distributed optimizations
2021-08-08 03:59:32 +03:00
mergify[bot]
9869c8bb10
Merge branch 'master' into zero_copy_by_default
2021-08-07 16:53:44 +00:00
alexey-milovidov
24cc073a22
Merge pull request #27328 from ClickHouse/fix_partition_id_validation
...
Fix partition id validation
2021-08-07 18:45:19 +03:00
kssenii
9ca422f0c5
Introduce CAST for internal usage
2021-08-07 09:03:10 +00:00
Nikolay Degterinsky
9a45458faf
Add bzip2 file compression
2021-08-07 00:25:52 +00:00
Alexander Tokmakov
5e2bfd5ba1
fix partition id validation
2021-08-06 23:56:51 +03:00
kssenii
073d7fdd5e
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server
2021-08-06 19:42:29 +00:00
Nikolai Kochetov
3cbeab8959
Fix incorrect row-level filtering
2021-08-06 21:16:06 +03:00
Nikolai Kochetov
157ecef3c2
Merge pull request #27227 from ClickHouse/do-not-update-settings-for-view
...
Revert #24095 . User-level settings will affect queries from view.
2021-08-06 20:57:36 +03:00
mergify[bot]
c696817a79
Merge branch 'master' into control_execution_period_of_clearOldTemporaryDirectories
2021-08-06 16:16:22 +00:00
Nikolai Kochetov
7320411a7f
Fix bug from #23515 .
2021-08-06 16:39:11 +03:00
vdimir
ffab19caad
Fix hasJoin in StorageView.cpp
2021-08-06 13:48:47 +03:00
vdimir
76ca40857a
Better error message for changed json_use_nulls in StorageView, check ast
2021-08-06 12:16:26 +03:00
tavplubix
14ef66e49b
Merge pull request #25895 from ClickHouse/improve_create_or_replace
...
Improve CREATE OR REPLACE query
2021-08-06 11:58:46 +03:00
Nikolai Kochetov
39ff5ab478
Merge branch 'master' into remove-streams-from-dicts
2021-08-06 11:46:27 +03:00
Nikolai Kochetov
13f95f3fdf
Streams -> Processors for dicts, part 3.
2021-08-06 11:41:45 +03:00
Alexey Milovidov
cea9690327
Merge branch 'master' into async-reads
2021-08-05 23:58:57 +03:00
alexey-milovidov
1a3d8cebd2
Merge pull request #27197 from azat/nullable-index-fix
...
Fix on-disk format breakage for secondary indices over Nullable column
2021-08-05 21:52:18 +03:00
Artur
d2869d2e2b
add test for memory tracker in client
2021-08-05 16:20:24 +00:00
vdimir
a427b19d3a
Check join_use_nulls changed in StorageView::read
2021-08-05 18:38:52 +03:00
Kseniia Sumarokova
5edd9e0513
Merge pull request #27198 from kssenii/postgres-db-schema
...
Support schema for postgres database engine
2021-08-05 18:30:47 +03:00
Kruglov Pavel
59a94bd322
Merge pull request #27183 from ClickHouse/fix_detached_parts_name_partsing
...
Fix part name parsing in system.detached_parts
2021-08-05 18:21:03 +03:00
Nikolai Kochetov
fc12310ee8
Revert #24095 . User-level settings will affect queries from view.
2021-08-05 17:17:22 +03:00
Alexander Kuzmenkov
d852207b0e
Merge pull request #25721 from CurtizJ/read-in-order-perf-2
...
Reduce memory usage in queries with `ORDER BY primary_key
2021-08-05 16:13:48 +03:00
Maksim Kita
7fdf3cc263
Merge pull request #27180 from kitaisreal/storage-system-replicas-added-column-replica-is-active
...
Storage system replicas added column replica is active
2021-08-05 12:46:53 +03:00
terrylin
dd539f0a0d
improvement of materilize ttl
2021-08-05 15:33:02 +08:00
terrylin
bd3d9a4518
materialize ttl recalculate only (optional)
2021-08-05 14:17:48 +08:00
Azat Khuzhin
dee27fcbb9
Fix on-disk format breakage for secondary indices over Nullable column
...
[1] breaks on disk format (and the relevant change in the:
[1]: https://github.com/ClickHouse/ClickHouse/pull/12455#discussion_r682830812
Too bad that I checked this patchset only for compatibility after
reverting this patch [2] (use case: I've applied it manually, then
revert it, and data skipping indexes over Nullable column had been
broken)
[2]: https://github.com/ClickHouse/ClickHouse/pull/12455#issuecomment-823423772
But this patchset actually breaks compatibility with older versions of
clickhouse for Nullable data skipping indexes after simple upgrade:
Here is a simple reproducer:
--
-- run this with 21.6 or similar (i.e. w/o this patch)
--
CREATE TABLE data
(
`key` Int,
`value` Nullable(Int),
INDEX value_index value TYPE minmax GRANULARITY 1
)
ENGINE = MergeTree
ORDER BY key;
INSERT INTO data SELECT
number,
number
FROM numbers(10000);
SELECT * FROM data WHERE value = 20000 SETTINGS force_data_skipping_indices = 'value_index' SETTINGS force_data_skipping_indices = 'value_index', max_rows_to_read=1;
Now upgrade and run the query again:
SELECT * FROM data WHERE value = 20000 SETTINGS force_data_skipping_indices = 'value_index' SETTINGS force_data_skipping_indices = 'value_index', max_rows_to_read=1;
And it will fail because of on disk format changes:
$ ll --time-style=+ data/*/data/all_1_1_0/skp*.idx
-rw-r----- 1 azat azat 36 data/with_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
-rw-r----- 1 azat azat 37 data/without_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
$ md5sum data/*/data/all_1_1_0/skp*.idx
a19c95c4a14506c65665a1e30ab404bf data/with_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
e50e2fcfa873b232196623d56ab26105 data/without_nullable_patch/data/all_1_1_0/skp_idx_value_index.idx
Note, that there is no stable release with this patch included yet, so
no need to backport.
Also note that you may create data skipping indexes over Nullable
column even before [3].
[3]: https://github.com/ClickHouse/ClickHouse/pull/12455
v2: break cases when granulas has Null in values due to backward
compatibility
2021-08-05 00:19:38 +03:00
kssenii
f06703a7c9
Support schema for postgres database engine
2021-08-04 20:52:45 +00:00
Azat Khuzhin
7a47327980
Fix used_storages by using log_queries from local context
...
I don't have log_queries enabled by default.
2021-08-04 21:44:19 +03:00
Anton Popov
9586bb7e90
fix pvs
2021-08-04 20:42:22 +03:00
terrylin
6f42ec6b9b
fix compile error
2021-08-05 00:16:21 +08:00
Anton Popov
915ef50c2d
optimize reading in order of key more
2021-08-04 18:18:03 +03:00
Alexander Tokmakov
23f8b3d07d
fix part name parsing in system.detached_parts
2021-08-04 17:42:48 +03:00
terrylin
88b9e7f24f
materialize ttl recalculate only optional
2021-08-04 22:16:13 +08:00
Anton Ivashkin
61016da2b1
Set allow_remote_fs_zero_copy_replication to true by default
2021-08-04 17:08:09 +03:00
Maksim Kita
3f48c85722
StorageSystemReplicas added column replica_is_active
2021-08-04 16:19:42 +03:00
Maksim Kita
4dc4854fad
Merge pull request #27168 from excitoon-favorites/safers3readbuffer
...
Safer `ReadBufferFromS3` for merges and backports
2021-08-04 15:01:21 +03:00
Yatsishin Ilya
55e06ea16b
fix
2021-08-04 13:06:01 +03:00
Vladimir Chebotarev
c2410920d3
Safer ReadBufferFromS3
for merges and backports.
2021-08-04 09:14:20 +03:00
Alexey Milovidov
fd3b5680e2
Merge branch 'master' into async-reads
2021-08-03 23:13:31 +03:00
mergify[bot]
dc57254982
Merge branch 'master' into improve_create_or_replace
2021-08-03 11:39:07 +00:00
Maksim Kita
d9383270f6
Merge pull request #27054 from MaxWk/feature/add-default-database-to-system-users
...
add column default_database to system.users
2021-08-03 13:37:15 +03:00
Azat Khuzhin
97851bde08
Fix Distributed over Distributed for WithMergeableStateAfterAggregation* stages
...
In case if one Distributed has multiple shards, and underlying
Distributed has only one, there can be the case when the query will be
tried to process from Complete to WithMergeableStateAfterAggregation,
which is obviously wrong.
2021-08-03 10:10:08 +03:00
Kseniia Sumarokova
5d3abc14c7
Merge pull request #22045 from OmarBazaraa/mongo-ssl
...
Support SSL connection in MongoDB engine
2021-08-03 08:29:34 +03:00
tavplubix
58b50bad3a
Merge pull request #27033 from ClickHouse/database_replicated_fixes
...
Fix assertions in Replicated database
2021-08-02 23:33:18 +03:00
Azat Khuzhin
ff12f5102a
Avoid running LIMIT BY/DISTINCT step on the initiator for optimize_distributed_group_by_sharding_key
...
Before the following queries was running LimitBy/Distinct step on the
initator:
select distinct sharding_key from dist order by k
While this can be omitted.
2021-08-02 21:04:30 +03:00
Azat Khuzhin
2fb95d9ee0
Rework SELECT from Distributed query stages optimization
...
Before this patch it wasn't possible to optimize simple SELECT * FROM
dist ORDER BY (w/o GROUP BY and DISTINCT) to more optimal stage
(QueryProcessingStage::WithMergeableStateAfterAggregationAndLimit),
since that code was under
allow_nondeterministic_optimize_skip_unused_shards, rework it and make
it possible.
Also now distributed_push_down_limit is respected for
optimize_distributed_group_by_sharding_key.
Next step will be to enable distributed_push_down_limit by default.
v2: fix detection of aggregates
2021-08-02 21:04:29 +03:00
Azat Khuzhin
bb6d030fb8
Optimize distributed SELECT w/o GROUP BY
2021-08-02 21:04:29 +03:00
OmarBazaraa
71e1c82f87
Refactor
2021-08-02 15:16:39 +00:00
Alexander Tokmakov
42a8bb6872
fix assertions in Replicated database
2021-08-02 16:19:11 +03:00
OmarBazaraa
91928fdf5a
Fix
2021-08-02 13:11:49 +00:00
OmarBazaraa
8577938efe
Throw exception in case SSL is not enabled
2021-08-02 12:34:50 +00:00
Anton Popov
fc9d72e75d
fix performance of short queries with large number of columns
2021-08-02 15:03:55 +03:00
Nikolai Kochetov
6951e8147d
Merge pull request #27020 from amosbird/shardlevelconstness
...
Shard-level const column
2021-08-02 13:45:20 +03:00
Anton Popov
cc3ed12ed6
Merge remote-tracking branch 'upstream/master' into HEAD
2021-08-02 02:56:00 +03:00
万康
66631ca680
add column default_database
2021-08-01 08:11:01 +08:00
Alexey Milovidov
2a0cba2b9f
Merge branch 'master' into async-reads
2021-08-01 01:25:13 +03:00
alexey-milovidov
701e2ffd79
Merge pull request #27002 from azat/mutation-stuck
...
Fix possible mutation stuck due to race with DROP_RANGE
2021-07-31 21:17:46 +03:00
terrylin
09a4f82d0f
code style check
2021-07-31 20:36:40 +08:00
Amos Bird
cd302eacc1
Fix projection match
2021-07-31 15:44:16 +08:00
tavplubix
00ac8194dc
Merge pull request #26896 from ClickHouse/small_improvement_in_queue
...
Maybe fix extremely rare `intersecting parts`.
2021-07-30 13:04:20 +03:00
terrylin
c49c2e5e18
fix compile error
2021-07-30 17:15:04 +08:00
mergify[bot]
41273ef5f3
Merge branch 'master' into mutation-stuck
2021-07-30 08:42:45 +00:00
Azat Khuzhin
bc9a4366d8
Fix possible mutation stack due to race with DROP_RANGE
...
After #25884 parts that was dropped (DROP_RANGE) will not be removed
from old mutations and this will stuck the mutation.
Interesting, that this mutation may continue after some merge.
2021-07-30 08:22:10 +03:00
terrylin
55f3e7e65a
improvement of materilize ttl
2021-07-30 12:27:15 +08:00
alesapin
21599ffefc
Simple validation for partition id before drop partition
2021-07-29 16:11:05 +03:00
alesapin
5799487caa
Fix stupid bug
2021-07-29 14:54:36 +03:00
OmarBazaraa
a665cd3308
Fixes
2021-07-29 08:38:39 +00:00
Alexey Milovidov
1149058084
Merge branch 'master' into async-reads
2021-07-28 22:29:51 +03:00
alesapin
71169d7937
Merge pull request #26716 from nvartolomei/nv/part-cleanup-sequence
...
Avoid deleting old parts from FS on shutdown for replicated engine
2021-07-28 18:29:37 +03:00
OmarBazaraa
3a393042a2
Fix
2021-07-28 15:28:30 +00:00
OmarBazaraa
d8fb1cb0e0
Ref.
2021-07-28 14:08:57 +00:00
alesapin
bdc6858801
More correct comparsion
2021-07-28 13:19:34 +03:00
alesapin
26056cfb25
Fix extremely rare race condition on remove of unsuccessful entry from queue
2021-07-28 13:06:22 +03:00
OmarBazaraa
a154ccef7b
Fix
2021-07-28 09:38:34 +00:00
Ivan
1417143560
Update StorageMongoDBSocketFactory.cpp
2021-07-28 09:38:34 +00:00
Ivan
0d029c1cbc
Update StorageMongoDBSocketFactory.cpp
2021-07-28 09:38:34 +00:00
OmarBazaraa
58ea75915d
Fix
2021-07-28 09:38:34 +00:00
Ubuntu
fce9351256
MongoDB SSL Connection
2021-07-28 09:38:34 +00:00
Alexander Tokmakov
fab4529c97
Merge branch 'master' into rename_materialize_mysql
2021-07-28 12:18:01 +03:00
alesapin
1277db4435
Merge pull request #26129 from ClickHouse/system_zookeeper_log
...
Add system.zookeeper_log table
2021-07-28 11:22:29 +03:00
kssenii
c6ee94db2e
Fix
2021-07-28 08:11:10 +00:00
Alexey Milovidov
9baf84224a
Fix build
2021-07-28 02:54:20 +03:00
Alexey Milovidov
fd5c1561e2
Merge branch 'master' into async-reads
2021-07-27 23:58:32 +03:00
Alexander Tokmakov
fc9ab2cda7
Merge branch 'master' into rename_materialize_mysql
2021-07-27 22:38:40 +03:00
Nicolae Vartolomei
8b07a7f180
Store exception generated when we tried to update the queue last time
...
The use case is to alert when queue contains broken entries. Especially
important when ClickHouse breaks backwards compatibility between
versions and log entries written by newer versions aren't parseable by
old versions.
```
Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected 'quorum: ' before: 'merge_type: 2\n'
```
2021-07-27 15:42:40 +01:00
Kseniia Sumarokova
b5dce17707
Merge pull request #26821 from azat/rocksdb-improvements
...
Add system.rocksdb and read rocksdb/rocksdb_TABLE options from config
2021-07-27 17:31:12 +03:00
tavplubix
168edaed73
Merge pull request #26808 from amosbird/safe2
...
Make sure table is readonly when restarting fails.
2021-07-27 17:21:08 +03:00
Nikolai Kochetov
97bc754dd8
Merge pull request #26758 from ClickHouse/output-streams-to-processors
...
Remove some output streams
2021-07-27 17:19:26 +03:00
alexey-milovidov
d7b2404119
Merge pull request #15408 from ClickHouse/random_timezone_for_stateless_tests
...
Use random timezone in stateless and stateful tests
2021-07-27 16:18:14 +03:00
Nikolai Kochetov
a9effdad18
Merge branch 'master' into output-streams-to-processors
2021-07-27 11:31:53 +03:00
Azat Khuzhin
8ef677b15f
Add ability to set any rocksdb option via config.xml
...
v2: Cover rocksdb options in ClickHouse config
v3: add missing __init__.py
v4: Rework rocksdb options from config
v5: add column_family_options support
2021-07-27 08:40:50 +03:00
Alexey Milovidov
5e2ea4e042
Whitespace
2021-07-27 05:20:03 +03:00
Kseniia Sumarokova
ee13f2e6bc
Merge pull request #26795 from kssenii/fixing-sqlite
...
Fix sqlite engine attach
2021-07-27 00:37:31 +03:00
Azat Khuzhin
c7d2a5aad8
Add system.rocksdb table (with various rocksdb internals)
...
v2: fix USE_ROCKSDB check (there is #cmakedefine01 so it is either 0/1,
so we should not check with #ifdef)
2021-07-26 23:13:59 +03:00
Alexander Tokmakov
63ab38ee09
rename MaterializeMySQL to MaterializedMySQL
2021-07-26 21:17:28 +03:00
Nikolai Kochetov
61d8f880cd
Rename some files.
2021-07-26 19:48:25 +03:00
Nikolai Kochetov
9b5a816b43
Merge branch 'master' into output-streams-to-processors
2021-07-26 18:03:11 +03:00
Nikolai Kochetov
0eb563dc1b
Fix more tests.
2021-07-26 17:47:29 +03:00
Kseniia Sumarokova
5c5cb5d0d6
Update src/Storages/StorageSQLite.h
...
Co-authored-by: Vladimir <vdimir@yandex-team.ru>
2021-07-26 14:26:06 +03:00
Amos Bird
623faf47e4
Make sure table is readonly when restarting fails.
2021-07-26 19:21:14 +08:00
Nikolai Kochetov
fa1c223269
Fix some tests.
2021-07-26 13:08:40 +03:00
mergify[bot]
2929cf3ba8
Merge branch 'master' into system_zookeeper_log
2021-07-26 08:40:55 +00:00
mergify[bot]
044be267d6
Merge branch 'master' into improve_create_or_replace
2021-07-26 08:38:48 +00:00
kssenii
f3129b80f9
Fix sqlite attach
2021-07-26 07:20:41 +00:00
Alexey Milovidov
5c3f5cb35a
Experiment with asynchronous readers
2021-07-26 03:34:36 +03:00
Alexey Milovidov
b6fdb395c2
Less verbose RocksDB logs #26252
2021-07-25 23:09:50 +03:00
alexey-milovidov
22fa1efacb
Merge pull request #26777 from ClickHouse/bolonini-read-from-file
...
Merging #25960 (Bolonini/read_from_file)
2021-07-25 01:33:34 +03:00
Alexey Milovidov
2f2dce399a
Fix style
2021-07-25 01:26:21 +03:00
Alexey Milovidov
215505b985
Remove unused error code
2021-07-24 19:51:56 +03:00
Alexey Milovidov
774f6d2617
Allow to read file descriptor multiple times in File storage
2021-07-24 19:50:03 +03:00
Alexey Milovidov
456801ccb8
Merge branch 'pipe_reading' of github.com:BoloniniD/ClickHouse into bolonini-read-from-file
2021-07-24 19:24:22 +03:00
alexey-milovidov
d3127d6265
Merge pull request #26731 from vdimir/fix-async-drain-connection-uaf
...
Fix use after free in AsyncDrain connection from S3Cluster
2021-07-24 18:55:03 +03:00
BoloniniD
2fa02061b7
Fix parallel reads
2021-07-24 15:31:35 +03:00
alexey-milovidov
039bb1cc5e
Merge pull request #26720 from azat/part_log-fix-event_time_microseconds
...
Fix event_time_microseconds for REMOVE_PART in system.part_log
2021-07-24 15:09:37 +03:00
Alexey Milovidov
a4b6181920
Fix weirdness
2021-07-24 05:07:37 +03:00
Alexey Milovidov
edfeb0957f
Fix strange code
2021-07-24 04:52:18 +03:00
alexey-milovidov
dab9cfb9c9
Merge pull request #26713 from ClickHouse/remove-more-and--more-streams
...
Remove more streams.
2021-07-24 02:24:10 +03:00
BoloniniD
07c57edbfc
Merge branch 'master' of github.com:ClickHouse/ClickHouse into pipe_reading
2021-07-23 23:39:38 +03:00
Nikolai Kochetov
9c92f43359
Update storages.
2021-07-23 22:33:59 +03:00
Nikolai Kochetov
2dc5c89b66
Update Storage::write
2021-07-23 17:25:35 +03:00
vdimir
dccc379d39
Fix use after free in AsyncDrain connection from S3Cluster
2021-07-23 10:40:03 +03:00
Vitaly Baranov
19d5a6ab2f
Merge pull request #26714 from vitlibar/new-function-current-profiles
...
New functions currentProfiles(), enabledProfiles(), defaultProfiles().
2021-07-23 09:10:29 +03:00
Azat Khuzhin
00e2083421
Fix event_time_microseconds for REMOVE_PART in system.part_log
2021-07-23 00:59:08 +03:00
Vitaly Baranov
7afcc65060
Add new functions currentProfiles(), enabledProfiles(), defaultProfiles().
2021-07-22 22:20:53 +03:00
Nikolai Kochetov
3c17a62686
Merge pull request #26590 from ClickHouse/remove-some-more-streams
...
Remove some streams.
2021-07-22 21:28:50 +03:00
Nicolae Vartolomei
f35e6eee19
Avoid deleting old parts from FS on shutdown for replicated engine
...
This was introduced in https://github.com/ClickHouse/ClickHouse/pull/8602 .
The idea was to avoid data re-appearing in ClickHouse after DROP/DETACH
PARTITION. This problem was only present in MergeTree engine and I don't
understand why we need to do the same in ReplicatedMergeTree.
For ReplicatedMergeTree the state of truth is stored in ZK, deleting
things from filesystem just introduces inconsistencies and this is the
main source for errors like "No active replica has part X or covering
part".
The resulting problem is fixed by
https://github.com/ClickHouse/ClickHouse/pull/25820 , but in my opinion
we would better avoid introducing the ZK/FS inconsistency in the first
place.
When does this inconsistency appear? Often the sequence is like this:
0. Write 2 parts to ZK [all_0_0_0, all_1_1_0]
1. A merge gets scheduled
2. New part replaces old parts [new: all_0_1_1, old: all_0_0_0, all_1_1_0]
3. Replica gets shutdown and old parts are removed from filesystem
4. Replica comes back online, metadata about all parts is still stored in ZK for this new replica.
5. Other replica after cleanup thread runs will have only [all_0_1_1] in
ZK
5. User triggers a DROP_RANGE after a while (drop range is for all_0_1_9999*)
6. Each replica deletes from ZK only [all_0_1_1]. The replica that got
restarted uses its in-memory state to choose nodes to delete from ZK.
7. Restart the replica again. It will now think that there are 2 parts
that it lost and needs to fetch them [all_0_0_0, all_1_1_0].
`clearOldPartsAndRemoveFromZK` which is triggered from cleanup thread
runs cleanup sequence correctly, it first removes things from ZK and
then from filesystem. I don't see much benefit of triggering it on
shutdown and would rather have it called only from a single place.
---
This is a very, very edge case situation but it proves that the current
"fix" (https://github.com/ClickHouse/ClickHouse/pull/25820 ) isn't
complete.
```
create table test(
v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'one')
order by v
settings old_parts_lifetime = 30;
create table test2(
v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'two')
order by v
settings old_parts_lifetime = 30;
create table test3(
v UInt64
)
engine=ReplicatedMergeTree('/clickhouse/test', 'three')
order by v
settings old_parts_lifetime = 30;
insert into table test values (1), (2), (3);
insert into table test values (4);
optimize table test final;
detach table test;
detach table test2;
alter table test3 drop partition tuple();
attach table test;
attach table test2;
```
```
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/one/parts
all_0_0_0
all_1_1_0
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/two/parts
all_0_0_0
all_1_1_0
(CONNECTED [localhost:9181]) /> ls /clickhouse/test/replicas/three/parts
```
```
detach table test;
attach table test;
```
`test` will now figure out that parts exist only in ZK and will issue `GET_PART`
after first removing parts from ZK.
`test2` will receive fetch for unknown parts and will trigger part checks itself.
Because `test` doesn't have the parts anymore in ZK `test2` will mark them as LostForever.
It will also not insert empty parts, because the partition is empty.
`test` is left with `GET_PART` in the queue and stuck.
```
SELECT
table,
type,
replica_name,
new_part_name,
last_exception
FROM system.replication_queue
Query id: 74c5aa00-048d-4bc1-a2ea-6f69501c11a0
Row 1:
──────
table: test
type: GET_PART
replica_name: one
new_part_name: all_0_0_0
last_exception: Code: 234. DB::Exception: No active replica has part all_0_0_0 or covering part. (NO_REPLICA_HAS_PART) (version 21.9.1.1)
Row 2:
──────
table: test
type: GET_PART
replica_name: one
new_part_name: all_1_1_0
last_exception: Code: 234. DB::Exception: No active replica has part all_1_1_0 or covering part. (NO_REPLICA_HAS_PART) (version 21.9.1.1)
```
2021-07-22 17:48:16 +01:00
Nikolai Kochetov
f56a45155f
Merge branch 'master' into remove-more-and--more-streams
2021-07-22 19:10:39 +03:00
Nikolai Kochetov
fd754430eb
Remove more streams.
2021-07-22 19:05:52 +03:00
Maksim Kita
e08fc94137
Merge pull request #26678 from CurtizJ/minor-bugfix
...
Minor bugfix
2021-07-22 15:54:24 +03:00
Nikolai Kochetov
3ed3f7a9f7
Fix integration tests.
2021-07-22 13:38:22 +03:00
feihengye
6230ad0160
#26640 change kafka engine max consumers from 16 to physical cpu cores ( #26642 )
...
Co-authored-by: jhonye <jhonye@tencent.com>
2021-07-22 11:54:08 +03:00
Nikolai Kochetov
5ffd99dfd4
Merge pull request #25674 from amosbird/distributedreturnconnection
...
Drain connection asynchronously
2021-07-22 11:04:49 +03:00
Anton Popov
a36c387e4e
fix iterating over list
2021-07-21 21:47:20 +03:00
Nikolai Kochetov
f38de35b14
Rename some constants.
2021-07-21 19:13:17 +03:00
Nikolai Kochetov
65d3e713d6
Fix another one test.
2021-07-21 15:16:13 +03:00
alexey-milovidov
fcca3e3e56
Merge pull request #26423 from ClickHouse/fix-header-for-scalar-query-with-empty-result
...
Fix incompatible result type for scalar queries with empty result.
2021-07-21 02:07:04 +03:00
Nikolai Kochetov
179ec05a72
Remove some streams.
2021-07-20 21:18:43 +03:00
Nikita Mikhaylov
348a3abb0b
Merge pull request #26508 from kitaisreal/storage-dictionary-quoted-names-fix
...
Storage Dictionary quoted names fix
2021-07-20 19:35:48 +03:00
BoloniniD
49ee24e824
What if we replace unique_lock with shared?
2021-07-20 19:22:41 +03:00
alexey-milovidov
3bcef76a9d
Merge pull request #26529 from kssenii/fixing-rabbitmq
...
Fix rabbitmq shutdown in case rabbitmq setup was not started
2021-07-20 16:41:25 +03:00
Nikita Mikhaylov
47c1bb3416
Merge pull request #25822 from ClickHouse/fix_no_column_materialized_mysql
...
Fix 'Not found column' error in MaterializeMySQL
2021-07-20 15:03:54 +03:00
Nikita Mikhaylov
4d3f828beb
Merge pull request #26314 from kssenii/fix-hdfs-crash
...
Fix hdfs crash
2021-07-20 15:01:56 +03:00
Maksim Kita
9b53f4f2b3
Storage Dictionary quoted names fix
2021-07-20 11:35:10 +03:00
kssenii
63c05e6e14
Fix rabbitmq shutdown in case rabbitmq setup was not started
2021-07-19 21:06:10 +00:00
Amos Bird
dbfb699690
Asynchronously drain connections.
2021-07-19 21:53:29 +08:00