thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-24 00:22:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
74,628 Commits 9,167 Branches 1,781 Tags 2.1 GiB
e66735e21e
Commit Graph

18778 Commits

Author SHA1 Message Date
Nikita Mikhaylov
6c2b43eece
Merge pull request #30092 from nicelulu/modify_comments 
Modify ConnectionPoolWithFailover get_priority ROUND_ROBIN comments
 2021-10-13 14:24:39 +03:00
Nikolai Kochetov
bb9fa38369 Fix header in library bridge 2021-10-13 14:24:12 +03:00
Anton Popov
108aafcd6d
Merge pull request #30074 from azat/move-partition-race-fix 
Fix race between MOVE PARTITION and merges/mutations for MergeTree
 2021-10-13 14:15:23 +03:00
zhangxiao871
923c76fc99 Modify comments 2021-10-13 18:38:24 +08:00
tavplubix
7742b96497
Remove metadata leftovers on drop database (#30054) 
* remove metadata leftovers on drop database

* Update InterpreterDropQuery.cpp

* Update DatabaseCatalog.cpp
 2021-10-13 13:34:18 +03:00
tavplubix
a69b0d9818
Follow-up to #29901 (#30003) 
* more informative logs

* Update ZooKeeper.cpp

* fix
 2021-10-13 13:21:07 +03:00
Nikolai Kochetov
a5fa5c7ea3 Move formats to Impl 2021-10-13 13:01:08 +03:00
Vitaly Baranov
a140f4127e Fix ExternalLoader without additional mutex. 2021-10-13 10:02:42 +03:00
Vitaly Orlov
1623bfa1a2
review fixes 2021-10-13 09:55:35 +03:00
Maksim Kita
cd08a078d1 SQL user defined functions fix alias 2021-10-13 00:16:01 +03:00
Azat Khuzhin
e9ce859b02 Fix race between MOVE PARTITION and merges/mutations for MergeTree 
From time to time 00975_move_partition_merge_tree test got failed, the
reason is that there is merge in progress that writes data to the source
table again:

<details>

    2021.10.12 13:20:40.243839 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> executeQuery: (from [::1]:38060) (comment: 00975_move_partition_merge_tree.sql) ALTER TABLE test_move_partition_src MOVE PARTITION 1 TO TABLE test_move_partition_dest;
    2021.10.12 13:20:40.244482 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_2_2_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_21_21_0
    ...
    2021.10.12 13:20:40.373487 [ 378 ] {} <Trace> test_t1tzb2.test_move_partition_src (467d7145-b47e-444e-867d-7145b47ea44e) (MergerMutator): Merged 6 parts: from 1_2_2_0 to 1_11_11_0
    ...
    2021.10.12 13:20:40.379750 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_15_15_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_28_28_0

</details>

And also remove cleaning of mutations since this will cause deadlock
after doing MOVE PARTITION under
currently_processing_in_background_mutex.

CI: https://clickhouse-test-reports.s3.yandex.net/0/a59c6b1c8eb47ebf77189a491ee0c3980b38e91c/functional_stateless_tests_
Fixes: 00975_move_partition_merge_tree
 2021-10-13 00:14:24 +03:00
Vitaly Orlov
4d2b34a7f0
Review fixes 2021-10-13 00:08:00 +03:00
Nikolay Degterinsky
b6ace277de
Merge pull request #30017 from ucasfl/dot 
Remove redundant dot in exception message
 2021-10-12 20:44:42 +03:00
Maksim Kita
a59c6b1c8e
Merge pull request #29928 from azat/dict-fix-lock-order-inversion 
Fix lock-order-inversion between periodic dictionary reload and config reload
 2021-10-12 19:46:28 +03:00
Kseniia Sumarokova
bb15b57753
Merge pull request #30016 from ucasfl/sample-by-tuple 
Fix  crash of sample by tuple()
 2021-10-12 19:46:21 +03:00
Nikolai Kochetov
f2d8cc547f Fix Kafka source. 2021-10-12 18:09:28 +03:00
Vitaly Orlov
65c6605c2c
Add JSONExtractKeys function 2021-10-12 18:00:25 +03:00
Nikita Mikhaylov
7c67d764c0
Merge pull request #29934 from ClickHouse/remove-bayes-ab 
Remove function bayesAB
 2021-10-12 18:00:25 +03:00
alexey-milovidov
9a1f930b2f
Merge pull request #30010 from ClickHouse/ppc64le 
Add cross-build for PowerPC 64le
 2021-10-12 16:17:41 +03:00
Nikolai Kochetov
c36bb9206e
Merge pull request #29879 from amosbird/queryidholderfix 
Fix potential leak of query_id_holder
 2021-10-12 15:42:02 +03:00
Nikolai Kochetov
077aba4a97
Merge pull request #29520 from amosbird/projection-improve4 
Get rid of naming limitation of projections.
 2021-10-12 14:25:29 +03:00
Dmitry Novik
8be70bc417 Update profile event forwarding 2021-10-12 14:15:07 +03:00
Dmitry Novik
abb74d2a5b Make Counters::Snapshot immutable 2021-10-12 13:46:32 +03:00
Dmitry Novik
857dd52867 Remove redundant enumerator 2021-10-12 13:46:32 +03:00
Dmitry Novik
a33610aa32 Make read of Counters snapshot non-atomic 2021-10-12 13:46:32 +03:00
Vitaly Baranov
4a4d913cfd
Merge pull request #29954 from vitlibar/fix-flaky-test_grpc_protocol 
gRPC: Fix releasing query ID and session ID at the end of query processing
 2021-10-12 13:32:27 +03:00
Nikita Mikhaylov
5bf64c62c2 Delete test 2021-10-12 10:19:21 +00:00
Nikita Mikhaylov
10487f75b2
Merge pull request #28364 from ClickHouse/profile-events-wip 
Forward profile events to client
 2021-10-12 13:16:01 +03:00
Nikolai Kochetov
93b3fcc195 Fix rabbitmq 2021-10-12 13:05:20 +03:00
Kseniia Sumarokova
b2f8cd07e6
Update normalizeString.cpp 2021-10-12 12:54:02 +03:00
Vasily Nemkov
91afa20a73 Unit tests for dictionary with comment 2021-10-12 12:48:12 +03:00
Vladimir C
969999ff10
Merge pull request #29544 from Algunenano/join_deadlock 2021-10-12 12:34:04 +03:00
Maksim Kita
ec898f1a94
Merge pull request #28633 from darkkeks/normalize-utf8-nfc 
Add normalizeUTF8 function with NFC normalization
 2021-10-12 12:25:07 +03:00
Vladimir C
11e37e7640
Merge pull request #28746 from amosbird/allowalias 2021-10-12 12:17:58 +03:00
Maksim Kita
91a48e9651
Merge pull request #29896 from kitaisreal/system-data-skipping-indices-added-size 
System table data skipping indices added size
 2021-10-12 12:16:57 +03:00
Maksim Kita
c15b67c182 Fix naming 2021-10-12 11:42:24 +03:00
alesapin
5ac0d36392
Merge pull request #30005 from ClickHouse/more_strict_check_for_intersecting_parts 
More strict check for intersecting parts
 2021-10-12 11:38:09 +03:00
alesapin
5a0f22fbce Fix resolve host 2021-10-12 11:27:18 +03:00
alesapin
1a778d5c37
Merge pull request #30008 from ClickHouse/tavplubix-patch-3 
Try remove excessive logging
 2021-10-12 10:51:47 +03:00
Nikolai Kochetov
88b1807434 Fix special build. 2021-10-12 10:33:45 +03:00
feng lv
ee1fd495ea remove redundant dot in exception message 2021-10-12 05:30:35 +00:00
feng lv
b5d69d599e fix sample by tuple() 
add tests

fix

fix

fix
 2021-10-12 05:19:32 +00:00
Kseniia Sumarokova
198adc7ecd
Merge pull request #29837 from havardk/gtidupdate 
MaterializedMySQL: Update GTID set at end of transaction
 2021-10-12 08:01:18 +03:00
alexey-milovidov
84e16f9c8e
Merge pull request #29979 from azat/memory-profiler-sanitizers 
Allow memory profiler under sanitizers (ASan/UBsan/MSan only)
 2021-10-12 03:12:19 +03:00
Alexey Milovidov
d4e496c314 Add support for PowerPC build 2021-10-12 01:47:57 +03:00
tavplubix
83a9a8d4dc
Update KeeperStateMachine.cpp 2021-10-11 23:23:56 +03:00
Nikolai Kochetov
1e1d5d7fea Fix style. 2021-10-11 22:21:04 +03:00
alexey-milovidov
f1b550cdaf
Merge pull request #29974 from ClickHouse/minimized-toolchain 
Experiment with minimized toolchain (2)
 2021-10-11 22:04:01 +03:00
Alexander Tokmakov
706e2b6b88 more strict check for intersecting parts 2021-10-11 21:42:46 +03:00
Nikolai Kochetov
2c92caac17 Fix clickhouse local. 2021-10-11 21:20:35 +03:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
alesapin
d301e40651 Cache DNS in Poco handler for S3 if no proxy specified 2021-10-11 18:31:09 +03:00
Dmitry Novik
bfdd34c13d code cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
7c3192735a Reset profile events stream in Connection::sendQuery 2021-10-11 17:39:24 +03:00
Dmitry Novik
9590c97a19 Create ProfileEventsQueue in GRPCServer 2021-10-11 17:39:24 +03:00
Dmitry Novik
3a0764634e Update InternalProfileEventsQueue usage 2021-10-11 17:39:24 +03:00
Dmitry Novik
c1f3e7e0bb Fix build 2021-10-11 17:39:24 +03:00
Dmitry Novik
0bdabf46f2 Send ProfileEvents only to supported clients 2021-10-11 17:39:24 +03:00
Dmitry Novik
9f9af28b5e Output memory usage with progress 2021-10-11 17:39:24 +03:00
Dmitry Novik
73df6190df Cleanup code 2021-10-11 17:39:24 +03:00
Dmitry Novik
1d2e2d7305 cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
7e3caf96be Fix cores approximation 2021-10-11 17:39:24 +03:00
Dmitry Novik
4c6b3c40f2 Calculate approximate cores number used 2021-10-11 17:39:24 +03:00
Dmitry Novik
15ac65aa33 Add thread usage info on client side 2021-10-11 17:39:24 +03:00
Dmitry Novik
9071a7151e Fix communication & race conditions 2021-10-11 17:39:23 +03:00
Dmitry Novik
356723427d WIP on ProfileEvents forwarding 2021-10-11 17:39:23 +03:00
Dmitry Novik
74cdaba7fa WIP on profile events forwarding 2021-10-11 17:29:50 +03:00
Dmitry Novik
803b8623c1 Fix TCPHandler::sendProfileEvents 2021-10-11 17:29:50 +03:00
Dmitry Novik
4c5a774570 Add comment 2021-10-11 17:29:50 +03:00
Dmitry Novik
e9b1e05461 Send profile events from all threads of current group 2021-10-11 17:29:50 +03:00
Dmitry Novik
362bcb2f66 Introduce ProfileEvents packet 2021-10-11 17:29:50 +03:00
Haavard Kvaalen
ab4b229574 Make sure we update position on commit 
Make sure we update GTID set on QueryEvents with "COMMIT"
or "XA COMMIT".  Without this we could have to redo the last transaction
if e.g. ClickHouse was restarted.  Note that this did not affect normal
transactions on InnoDB, since they are terminated with a XID_EVENT.
 2021-10-11 16:20:51 +02:00
Dmitry Novik
a651c1bb67
Merge pull request #29820 from azat/MINSIGSTKSZ-fix 
Fix compilation with glibc 2.34 (MINSIGSTKSZ defined as sysconf(_SC_SIGSTKSZ))
 2021-10-11 17:06:29 +03:00
Amos Bird
635783fb66
Only do TreeOptimizer for initial queries 2021-10-11 20:47:35 +08:00
Amos Bird
f4269ce417
Allow optimize_arithmetic_operations_in_aggregate_functions when alias is used. 2021-10-11 20:47:34 +08:00
alexey-milovidov
3b8d7a5030
Merge pull request #29952 from MikaelUrankar/freebsd-aarch64 
Add support for FreeBSD aarch64
 2021-10-11 14:43:23 +03:00
Alexey Milovidov
2370a8d6da Fix PVS-Studio 2021-10-11 14:28:46 +03:00
Amos Bird
72bccaa501
Fix path name 2021-10-11 19:12:08 +08:00
Maksim Kita
b0d887a0fe Added tests 2021-10-11 14:00:10 +03:00
Amos Bird
83717b7c3b
Get rid of naming limitation of projections. 2021-10-11 18:32:17 +08:00
tavplubix
a9a7fd94cd
Merge pull request #29929 from azat/db-fix-lock-order-inversion 
Fix lock-order-inversion between DROP TABLE for DatabaseMemory and LiveView
 2021-10-11 13:23:58 +03:00
Kseniia Sumarokova
20e8074b5e
Merge pull request #29912 from taiyang-li/remove_unused_var 
Remove unused variable in s3cluster
 2021-10-11 12:45:04 +03:00
Maksim Kita
ce0c41e1ad Fixed tests 2021-10-11 11:39:50 +03:00
Maksim Kita
2d069acc22 System table data skipping indices added size 2021-10-11 11:39:50 +03:00
Vasily Nemkov
2c55807b2a Create dictionary with comment 2021-10-11 11:26:59 +03:00
Amos Bird
ebfb013ea1
Fix potential leak of query_id_holder 2021-10-11 15:56:00 +08:00
Azat Khuzhin
063f9cffab Allow memory profiler under sanitizers 
Only query profiler cannot work reliably with sanitizers (due to
unwinding from signal handler), but memory profiler should be fine.

Plus sometimes the problem appears only on build with sanitizers, so it
will be useful to have memory related profiling in trace_log.

Also there is a flaky check for stateless tests, that uses build with
ASan, and now trace_log there is empty, which sometimes does not allow
to debug further.
 2021-10-11 10:21:26 +03:00
Vladimir C
dc4bb920d3
Merge branch 'master' into join_deadlock 2021-10-11 10:18:17 +03:00
Vladimir C
d2b8d293b4
Merge pull request #29976 from hexiaoting/dev_quantile_fuse2 2021-10-11 10:06:40 +03:00
Vladimir C
6c7fbf0b88
Merge pull request #29956 from azat/fix-tuple-filter 2021-10-11 10:02:47 +03:00
Alexey Milovidov
284e547bc3 Fix error 2021-10-11 09:01:23 +03:00
hexiaoting
aabe52e3e9 Fix bug for quantile fusion 2021-10-11 10:39:55 +08:00
alexey-milovidov
0adbcd58c9
Merge pull request #29930 from azat/buffer-fix-data-race 
Fix data-race between flush() and startup() in StorageBuffer
 2021-10-11 01:11:39 +03:00
mergify[bot]
c07a49ba89
Merge branch 'master' into remove-bayes-ab 2021-10-10 21:10:57 +00:00
Azat Khuzhin
f37dac03f4 Fix lock-order-inversion between DROP TABLE for DatabaseMemory and LiveView 
CI stress founds [1], TSan report:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=509)
  Cycle in lock order graph: M71436201 (0x7b5800a91308) => M237700395169415472 (0x000000000000) => M71436201

  Mutex M237700395169415472 acquired here while holding mutex M71436201 in thread T723:
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x154b0db9)
    4 DB::DatabaseWithOwnTablesBase::tryGetTable() const obj-x86_64-linux-gnu/../src/Databases/DatabasesCommon.cpp:37:21 (clickhouse+0x154b0db9)
    5 DB::DatabaseMemory::tryGetTableUUID() const obj-x86_64-linux-gnu/../src/Databases/DatabaseMemory.cpp:95:22 (clickhouse+0x15466bb5)
    6 DB::Context::resolveStorageID() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2672:90 (clickhouse+0x155e6aa1)
    7 DB::JoinedTables::getLeftTableStorage() obj-x86_64-linux-gnu/../src/Interpreters/JoinedTables.cpp:200:29 (clickhouse+0x15eee962)
    8 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:321:33 (clickhouse+0x15b792be)
    9 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:160:7 (clickhouse+0x15b78160)
    10 DB::StorageLiveView::collectMergeableBlocks() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:113:28 (clickhouse+0x16aae192)
    11 DB::StorageLiveView::getNewBlocks() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:384:33 (clickhouse+0x16ab393a)
    12 DB::StorageLiveView::refresh(bool) obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:528:13 (clickhouse+0x16ab6395)
    13 DB::StorageLiveView::read() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:545:9 (clickhouse+0x16ab6395)
    21 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:950:30 (clickhouse+0x16101441)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M71436201 acquired here while holding mutex M237700395169415472 in thread T723:
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x16ab5c29)
    4 DB::StorageLiveView::drop() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:477:21 (clickhouse+0x16ab5c29)
    5 DB::DatabaseMemory::dropTable() obj-x86_64-linux-gnu/../src/Databases/DatabaseMemory.cpp:44:16 (clickhouse+0x15465cd6)
    8 DB::InterpreterDropQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:62:16 (clickhouse+0x15afa679)

  [1]: https://clickhouse-test-reports.s3.yandex.net/29856/42ca2b4bb241827edf69bbd6938d6b19c31935f1/stress_test_(thread).html#fail1

v2: fix attachTableUnlocked() (as pointed by @tavplubix)
v3: use SCOPE_EXIT() and so avoid holding a lock during removing from fs
 2021-10-10 23:51:36 +03:00
Azat Khuzhin
6c33eaee32 Fix filtering by tuple (some conditions was lost during analyzing) 
Fixes: #29281
Fixes: test_cluster_copier/test_three_nodes.py::test
 2021-10-10 22:37:52 +03:00
Nikolai Kochetov
a95c28ec4b
Merge pull request #29898 from ClickHouse/remove-native-stream 
Remove some more streams.
 2021-10-10 21:01:16 +03:00
Vitaly Baranov
1dda596689 Fix releasing query ID and session ID at the end of query processing. 2021-10-10 17:34:45 +03:00
mikael
689c9cdbb9 Add uc_mcontext for FreeBSD aarch64 2021-10-10 14:52:17 +02:00
feng lv
1229a8b528 fix 2021-10-10 04:12:24 +00:00