thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
75,155 Commits 9,328 Branches 1,783 Tags 2.1 GiB
ec9d794246
Commit Graph

18982 Commits

Author SHA1 Message Date
Nikita Mikhaylov
e9bf496d42 Added an ability to generate data in build time 2021-10-15 14:35:50 +00:00
Maksim Kita
a744097fb6 Fixed build 2021-10-15 15:34:13 +03:00
Anton Popov
5b6b8cf564 fix clang-tidy 2021-10-15 15:14:44 +03:00
avogar
df81d3f74a Fix build in fast test 2021-10-15 14:52:59 +03:00
Nikita Mikhaylov
d1138a8a25
Merge pull request #29691 from azat/fix-merge-mutate-memory-tracking 
Fix memory tracking for merges and mutations
 2021-10-15 13:53:52 +03:00
Maksim Kita
65aec857ec FunctionsJSON updated 2021-10-15 13:12:59 +03:00
Nikolai Kochetov
ad8a344b46 Move TTL streams and algo 2021-10-15 13:11:57 +03:00
Maksim Kita
7beaeb8b13
Merge branch 'master' into indexcache 2021-10-15 12:59:09 +03:00
Maksim Kita
27fdeef4b6
Merge pull request #29650 from kitaisreal/add-default-database-visitor-support-dict-get 
AddDefaultDatabaseVisitor support dictGet
 2021-10-15 12:28:58 +03:00
Maksim Kita
8d9f593a61
Merge pull request #30217 from azat/fix-clang-tidy-json 
Fix clang-tidy warnings in FunctionsJSON code
 2021-10-15 12:27:24 +03:00
liang.huang
f82d79854c
Merge branch 'ClickHouse:master' into master 2021-10-15 16:11:15 +08:00
lhuang09287750
766e08567c complete date32 comparison with date/datetime/string and give a test file 2021-10-15 08:09:37 +00:00
Azat Khuzhin
07adc847db Fix clang-tidy warnings in FunctionsJSON code 2021-10-15 10:11:03 +03:00
Azat Khuzhin
2a7a1d8df5 Avoid losing any allocations context from merges/mutations 2021-10-15 01:43:28 +03:00
Azat Khuzhin
8a209a78d7 Set query_id for mutations/merges 
This will allow to distinguish allocations in trace_log.
 2021-10-15 01:43:28 +03:00
Azat Khuzhin
3477ab52a7 Add StorageID::getShortName() 2021-10-15 01:43:28 +03:00
Azat Khuzhin
8cc45bea7b Avoid accounting memory from another mutation/merge 
Before this path it was possible that one merge/mutation may account
memory from another, due to ThreadStatus::untracked_memory.

And this cause flakiness of 01200_mutations_memory_consumption.
 2021-10-15 01:43:28 +03:00
Azat Khuzhin
fd38cbb0df Fix memory tracking for merges and mutations (by destroying earlier) 
It fixes only some tiny allocations, and so it should not affect any
huge mutations/merges.

And plus, this should not be a real fix, since peak_memory_usage is
obtained before even destrying this objects, and destroing objects will
unlikely update peak memory usage (although it is possible).

v0: do this in dtors
v2: do this earlier
 2021-10-15 01:43:27 +03:00
Azat Khuzhin
e4b39ca99c Remove unused Block member from MutateTask 
v0: Fix excessive memory usage in MutateTask
    This is relevant only for 01200_mutations_memory_consumption test, since
    this is just a block.
v2: just remove unused Block member, since other part had been fixed in #29768
 2021-10-15 01:43:27 +03:00
Azat Khuzhin
a063097fdf Fix INSERT SELECT incorrectly fills MATERIALIZED column based of Nullable column 
Required columns of the default expression should not be converted to NULL,
since this map value to default and MATERIALIZED values will not work.

Consider the following structure:
- A Nullable(Int64)
- X Int64 materialized coalesce(A, -1)

With recursive_null_as_default=true you will get:

    _CAST(coalesce(A, -1), 'Int64') AS X, NULL AS A

And this will ignore default expression.

Fixes: #23524 (Cc: @kssenii)
Fixes: #29729 (Cc: @tavplubix)
Backport: 21.7+
 2021-10-15 01:36:09 +03:00
alexey-milovidov
bc1662b9fe
Merge pull request #29946 from azat/log-fix-data-race 
Fix data-race between LogSink::writeMarks() and LogSource in StorageLog
 2021-10-15 01:30:58 +03:00
alexey-milovidov
8d694cbcdc
Merge pull request #29959 from azat/log-data-race-file-checker-fix 
Fix possible data-race between FileChecker and StorageLog/StorageStripeLog
 2021-10-15 01:30:23 +03:00
Maksim Kita
5b5ba5fbc8
Merge pull request #30179 from kitaisreal/clickhouse-local-support-sql-user-defined-functions 
Support SQL user defined functions for clickhouse-local
 2021-10-14 23:14:18 +03:00
Maksim Kita
888c619472
Merge pull request #30145 from kitaisreal/functions-json-avoid-copying-object-element-during-iteration 
FunctionsJSON avoid copying object element during iteration
 2021-10-14 23:11:16 +03:00
Maksim Kita
5568461acf FunctionsJSON move to cpp file 2021-10-14 23:10:53 +03:00
Maksim Kita
913a8f3640 Fixed style check 2021-10-14 22:15:21 +03:00
Maksim Kita
83879415b4 Fixed tests 2021-10-14 22:12:06 +03:00
Nikolai Kochetov
6b251ddd97
Merge pull request #30193 from ClickHouse/try-to-fix-01288_shard_max_network_bandwidth 
Smaller smoothing window in throttler.
 2021-10-14 22:10:37 +03:00
avogar
2da8180613 Add space after comma 2021-10-14 21:39:09 +03:00
avogar
8729201208 Remove redundant move 2021-10-14 21:36:57 +03:00
avogar
89c1a04ef4 Fix comments 2021-10-14 21:35:56 +03:00
Nikolai Kochetov
aa56fd1592 Fix style. 2021-10-14 21:29:02 +03:00
Nikolai Kochetov
e9cab3aaad Smaller smoothing window in throttler. 2021-10-14 21:20:19 +03:00
Dmitry Novik
2f6d771d94 Remove trailing whitespace 2021-10-14 18:15:30 +03:00
mergify[bot]
92dd528c78
Merge branch 'master' into remove_trash 2021-10-14 14:14:07 +00:00
Anton Popov
476d7a411f allow to remove SAMPLE BY expression 2021-10-14 17:05:20 +03:00
avogar
f754881e1f Fix output String data into Text CapnProto type 2021-10-14 16:43:23 +03:00
Kruglov Pavel
9ec6930c15 Better exception handling 2021-10-14 16:43:23 +03:00
Kruglov Pavel
95790b8a1c Update CapnProtoUtils.cpp 2021-10-14 16:43:23 +03:00
avogar
5d16dc7f9a Try to fix tests, update capnp lib to eliminate problem with UB sanitizer 2021-10-14 16:43:23 +03:00
Kruglov Pavel
9ddcdbba39 Add INCORRECT_DATA error code 2021-10-14 16:43:23 +03:00
avogar
f88a2ad653 Handle exception when cannot extract value from struct, add test for it 2021-10-14 16:43:23 +03:00
avogar
ed8818a773 Fix style, better check in enum comparison 2021-10-14 16:43:22 +03:00
Kruglov Pavel
1cd938fbba Fix typo 2021-10-14 16:43:22 +03:00
avogar
ce22f534c4 Add CapnProto output format, refactor CapnProto input format 2021-10-14 16:43:22 +03:00
Maksim Kita
04e07ed86d FunctionsJSON avoid copying object element during iteration 2021-10-14 16:37:52 +03:00
avogar
a1a4df2501 Fix handling exception 'unrecognised option' in clickhouse-local and client 2021-10-14 16:34:05 +03:00
Maksim Kita
c1f04452de AddDefaultDatabaseVisitor support dictGet 2021-10-14 16:31:32 +03:00
tavplubix
746964af88
Merge pull request #30084 from vitlibar/fix-external-loader-without-extra-mutex 
Fix ExternalLoader without additional mutex.
 2021-10-14 16:18:17 +03:00
Maksim Kita
fbaa9e7be4 Support SQL user defined functions for clickhouse-local 2021-10-14 16:15:39 +03:00
Maksim Kita
4ed28d8e52
Merge pull request #29981 from kitaisreal/added-function-tokens 
Added function tokens
 2021-10-14 15:36:06 +03:00
Anton Popov
4d020c96e0 support nullable arguments in function initializeAggregation 2021-10-14 15:09:06 +03:00
Anton Popov
6c1da023f7
Merge pull request #30142 from kitaisreal/polygon-dictionary-support-read 
Polygon dictionary support read
 2021-10-14 15:03:10 +03:00
Anton Popov
c2d376d0d6
Merge branch 'master' into better-interfaces 2021-10-14 14:33:36 +03:00
Anton Popov
5802037f1e
Merge pull request #29916 from ucasfl/format-alter 
Format improvement of AlterQuery
 2021-10-14 14:30:08 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Maksim Kita
8ed4e73705
Merge pull request #30000 from ClickHouse/profile-snapshot 
Make read of Counters snapshot non-atomic
 2021-10-14 13:22:44 +03:00
alesapin
90ff7f05fd Start keeper asynchronously if has connection to other nodes 2021-10-14 13:21:41 +03:00
Maksim Kita
562c8984a4 Added documentation 2021-10-14 13:15:45 +03:00
Maksim Kita
6c6dd7a0c8
Merge pull request #30056 from orloffv/orloffv.JSONExtractKeys 
Add JSONExtractKeys function
 2021-10-14 12:50:19 +03:00
Nikolai Kochetov
3d3e143c29
Merge pull request #30001 from ClickHouse/remove-streams-from-formats 
Remove streams from formats.
 2021-10-14 12:39:07 +03:00
Maksim Kita
f589084a94
Merge pull request #29899 from Enmk/Governance/dictionary_comment 
Governance/dictionary comment
 2021-10-14 12:35:10 +03:00
Maksim Kita
8d544a55b7
Merge pull request #30075 from kitaisreal/sql-user-defined-functions-fix-alias 
SQL user defined functions fix alias
 2021-10-14 11:15:32 +03:00
Anton Popov
92413aed68 better interfaces for IDataType and ISerialization 2021-10-14 05:36:49 +03:00
Maksim Kita
659a46a972 WriteBufferToRabbitMQProduced fixed queue usage 2021-10-14 00:33:18 +03:00
Maksim Kita
35f2d914dc Fixed build 2021-10-14 00:33:18 +03:00
Maksim Kita
3d2e1a24d9 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
201523f4e6 Fixed build 2021-10-14 00:33:18 +03:00
Maksim Kita
c6eece5a03 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
46ba649821 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
04047f76c7 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
d55561422f Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
aae409b321 Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
afdc2fe50d Fixed tests 2021-10-14 00:33:18 +03:00
Maksim Kita
c9b6c2661b Refactor ConcurrentBoundedQueue 2021-10-14 00:33:18 +03:00
Azat Khuzhin
eba6410fd5 Fix data-race between LogSink::writeMarks() and LogSource::readData() in StorageLog 
CI founds [1], TSan report:

    WARNING: ThreadSanitizer: data race (pid=497)
      Write of size 8 at 0x7b1c00e4e8d8 by thread T922:
        4 DB::LogSink::writeMarks() obj-x86_64-linux-gnu/../src/Storages/StorageLog.cpp:462:72 (clickhouse+0x1665cdaa)
        5 DB::LogSink::consume() obj-x86_64-linux-gnu/../src/Storages/StorageLog.cpp:324:5 (clickhouse+0x1665c216)

      Previous read of size 8 at 0x7b1c00e4e8d8 by thread T711:
        1 DB::LogSource::readData()::$_0::operator()(bool) const::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Storages/StorageLog.cpp:188:26 (clickhouse+0x16661baf)
        9 DB::LogSource::readData() obj-x86_64-linux-gnu/../src/Storages/StorageLog.cpp:204:20 (clickhouse+0x1665bbcb)

      [1]: https://clickhouse-test-reports.s3.yandex.net/29930/4bc90d1dd7dbd4b8a9b6920d00ca24e8b160358e/stress_test_(thread).html#fail1

Fixes: #29470
v2: rework locking, but this produce deadlock for 01499_log_deadlock test
v3: introduce per-file marks mutex
 2021-10-14 00:17:45 +03:00
Azat Khuzhin
e4db862d85 Fix possible data-race between FileChecker and StorageLog/StorageStripeLog 2021-10-14 00:16:58 +03:00
Maksim Kita
58728fe78f Fixed tests 2021-10-14 00:08:43 +03:00
Maksim Kita
66f2163b60 Added function tokens 2021-10-14 00:08:15 +03:00
mergify[bot]
c3a0abc430
Merge branch 'master' into profile-snapshot 2021-10-13 18:44:57 +00:00
Nikolai Kochetov
2957971ee3 Remove some last streams. 2021-10-13 21:22:02 +03:00
Nikolai Kochetov
b24b458df7
Update HTTPDictionarySource.cpp 2021-10-13 20:13:30 +03:00
Amos Bird
8851cb8459
Use upstream replxx 2021-10-14 00:52:53 +08:00
Maksim Kita
d1a184c500 Fixed style check 2021-10-13 19:37:16 +03:00
Maksim Kita
5802213722 PolygonDictionary support read method 2021-10-13 19:31:59 +03:00
alesapin
1801ea4955 Merge branch 'master' into remove_trash 2021-10-13 18:10:31 +03:00
alesapin
f1ccb88f3f
Merge pull request #29999 from ClickHouse/use_dns_cache_for_s3_in_most_cases 
Cache DNS in Poco handler for S3 if no proxy specified
 2021-10-13 18:09:52 +03:00
Dmitry Novik
006a9a7c7d Print more info about memory utilization 2021-10-13 16:26:54 +03:00
Nikita Mikhaylov
57a623aba0
Merge branch 'master' into codegen_select_fuzzer_v1 2021-10-13 16:01:40 +03:00
alesapin
d2dfbb5ab6 Remove trash from MergeTreeReadPool 2021-10-13 15:25:44 +03:00
Nikolai Kochetov
49ac602ee6 Fix GRPC 2021-10-13 15:06:56 +03:00
Pavel Cheremushkin
562138c9fa codegen_fuzzer: removing errors on warnings from protobuf-generated code in more gentle way 2021-10-13 15:01:21 +03:00
Nikita Mikhaylov
6c2b43eece
Merge pull request #30092 from nicelulu/modify_comments 
Modify ConnectionPoolWithFailover get_priority ROUND_ROBIN comments
 2021-10-13 14:24:39 +03:00
Nikolai Kochetov
bb9fa38369 Fix header in library bridge 2021-10-13 14:24:12 +03:00
Anton Popov
108aafcd6d
Merge pull request #30074 from azat/move-partition-race-fix 
Fix race between MOVE PARTITION and merges/mutations for MergeTree
 2021-10-13 14:15:23 +03:00
zhangxiao871
923c76fc99 Modify comments 2021-10-13 18:38:24 +08:00
tavplubix
7742b96497
Remove metadata leftovers on drop database (#30054) 
* remove metadata leftovers on drop database

* Update InterpreterDropQuery.cpp

* Update DatabaseCatalog.cpp
 2021-10-13 13:34:18 +03:00
tavplubix
a69b0d9818
Follow-up to #29901 (#30003) 
* more informative logs

* Update ZooKeeper.cpp

* fix
 2021-10-13 13:21:07 +03:00
Nikolai Kochetov
a5fa5c7ea3 Move formats to Impl 2021-10-13 13:01:08 +03:00
Vitaly Baranov
a140f4127e Fix ExternalLoader without additional mutex. 2021-10-13 10:02:42 +03:00
Vitaly Orlov
1623bfa1a2
review fixes 2021-10-13 09:55:35 +03:00
Amos Bird
23602f4607
Speed up part loading for JBOD 2021-10-13 12:33:54 +08:00
Alexey Milovidov
0ad20e6613 Fix test 2021-10-13 03:14:44 +03:00
Alexey Milovidov
9939e7c1eb Merge branch 'master' into clickhouse-obfuscator-2 2021-10-13 01:16:26 +03:00
Maksim Kita
cd08a078d1 SQL user defined functions fix alias 2021-10-13 00:16:01 +03:00
Azat Khuzhin
e9ce859b02 Fix race between MOVE PARTITION and merges/mutations for MergeTree 
From time to time 00975_move_partition_merge_tree test got failed, the
reason is that there is merge in progress that writes data to the source
table again:

<details>

    2021.10.12 13:20:40.243839 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> executeQuery: (from [::1]:38060) (comment: 00975_move_partition_merge_tree.sql) ALTER TABLE test_move_partition_src MOVE PARTITION 1 TO TABLE test_move_partition_dest;
    2021.10.12 13:20:40.244482 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_2_2_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_21_21_0
    ...
    2021.10.12 13:20:40.373487 [ 378 ] {} <Trace> test_t1tzb2.test_move_partition_src (467d7145-b47e-444e-867d-7145b47ea44e) (MergerMutator): Merged 6 parts: from 1_2_2_0 to 1_11_11_0
    ...
    2021.10.12 13:20:40.379750 [ 26955 ] {2dbfea50-639e-49ac-9e82-933a00ea04a3} <Debug> test_t1tzb2.test_move_partition_dest (cab3b005-d54b-4cdc-8ab3-b005d54becdc): Cloning part /var/lib/clickhouse/store/467/467d7145-b47e-444e-867d-7145b47ea44e/1_15_15_0/ to /var/lib/clickhouse/store/cab/cab3b005-d54b-4cdc-8ab3-b005d54becdc/tmp_move_from_1_28_28_0

</details>

And also remove cleaning of mutations since this will cause deadlock
after doing MOVE PARTITION under
currently_processing_in_background_mutex.

CI: https://clickhouse-test-reports.s3.yandex.net/0/a59c6b1c8eb47ebf77189a491ee0c3980b38e91c/functional_stateless_tests_
Fixes: 00975_move_partition_merge_tree
 2021-10-13 00:14:24 +03:00
Vitaly Orlov
4d2b34a7f0
Review fixes 2021-10-13 00:08:00 +03:00
Dmitry Novik
f1bbc7f9b6 Fix hardware utilization info printing in client 2021-10-12 23:17:15 +03:00
Nikolay Degterinsky
b6ace277de
Merge pull request #30017 from ucasfl/dot 
Remove redundant dot in exception message
 2021-10-12 20:44:42 +03:00
Maksim Kita
a59c6b1c8e
Merge pull request #29928 from azat/dict-fix-lock-order-inversion 
Fix lock-order-inversion between periodic dictionary reload and config reload
 2021-10-12 19:46:28 +03:00
Kseniia Sumarokova
bb15b57753
Merge pull request #30016 from ucasfl/sample-by-tuple 
Fix  crash of sample by tuple()
 2021-10-12 19:46:21 +03:00
Nikolai Kochetov
f2d8cc547f Fix Kafka source. 2021-10-12 18:09:28 +03:00
Vitaly Orlov
65c6605c2c
Add JSONExtractKeys function 2021-10-12 18:00:25 +03:00
Nikita Mikhaylov
7c67d764c0
Merge pull request #29934 from ClickHouse/remove-bayes-ab 
Remove function bayesAB
 2021-10-12 18:00:25 +03:00
alexey-milovidov
9a1f930b2f
Merge pull request #30010 from ClickHouse/ppc64le 
Add cross-build for PowerPC 64le
 2021-10-12 16:17:41 +03:00
Nikolai Kochetov
c36bb9206e
Merge pull request #29879 from amosbird/queryidholderfix 
Fix potential leak of query_id_holder
 2021-10-12 15:42:02 +03:00
Nikolai Kochetov
077aba4a97
Merge pull request #29520 from amosbird/projection-improve4 
Get rid of naming limitation of projections.
 2021-10-12 14:25:29 +03:00
Dmitry Novik
8be70bc417 Update profile event forwarding 2021-10-12 14:15:07 +03:00
Dmitry Novik
abb74d2a5b Make Counters::Snapshot immutable 2021-10-12 13:46:32 +03:00
Dmitry Novik
857dd52867 Remove redundant enumerator 2021-10-12 13:46:32 +03:00
Dmitry Novik
a33610aa32 Make read of Counters snapshot non-atomic 2021-10-12 13:46:32 +03:00
Vitaly Baranov
4a4d913cfd
Merge pull request #29954 from vitlibar/fix-flaky-test_grpc_protocol 
gRPC: Fix releasing query ID and session ID at the end of query processing
 2021-10-12 13:32:27 +03:00
Nikita Mikhaylov
5bf64c62c2 Delete test 2021-10-12 10:19:21 +00:00
Nikita Mikhaylov
10487f75b2
Merge pull request #28364 from ClickHouse/profile-events-wip 
Forward profile events to client
 2021-10-12 13:16:01 +03:00
Nikolai Kochetov
93b3fcc195 Fix rabbitmq 2021-10-12 13:05:20 +03:00
Kseniia Sumarokova
b2f8cd07e6
Update normalizeString.cpp 2021-10-12 12:54:02 +03:00
Vasily Nemkov
91afa20a73 Unit tests for dictionary with comment 2021-10-12 12:48:12 +03:00
Vladimir C
969999ff10
Merge pull request #29544 from Algunenano/join_deadlock 2021-10-12 12:34:04 +03:00
Maksim Kita
ec898f1a94
Merge pull request #28633 from darkkeks/normalize-utf8-nfc 
Add normalizeUTF8 function with NFC normalization
 2021-10-12 12:25:07 +03:00
Vladimir C
11e37e7640
Merge pull request #28746 from amosbird/allowalias 2021-10-12 12:17:58 +03:00
Maksim Kita
91a48e9651
Merge pull request #29896 from kitaisreal/system-data-skipping-indices-added-size 
System table data skipping indices added size
 2021-10-12 12:16:57 +03:00
Maksim Kita
c15b67c182 Fix naming 2021-10-12 11:42:24 +03:00
alesapin
5ac0d36392
Merge pull request #30005 from ClickHouse/more_strict_check_for_intersecting_parts 
More strict check for intersecting parts
 2021-10-12 11:38:09 +03:00
alesapin
5a0f22fbce Fix resolve host 2021-10-12 11:27:18 +03:00
alesapin
1a778d5c37
Merge pull request #30008 from ClickHouse/tavplubix-patch-3 
Try remove excessive logging
 2021-10-12 10:51:47 +03:00
Nikolai Kochetov
88b1807434 Fix special build. 2021-10-12 10:33:45 +03:00
feng lv
ee1fd495ea remove redundant dot in exception message 2021-10-12 05:30:35 +00:00
feng lv
b5d69d599e fix sample by tuple() 
add tests

fix

fix

fix
 2021-10-12 05:19:32 +00:00
Kseniia Sumarokova
198adc7ecd
Merge pull request #29837 from havardk/gtidupdate 
MaterializedMySQL: Update GTID set at end of transaction
 2021-10-12 08:01:18 +03:00
Pavel Cheremushkin
9ed3361298 removing code generated files, since protobuf generation is now done in CMake 2021-10-12 03:53:43 +03:00
Pavel Cheremushkin
7b9358a597 adding codegen fuzzer + code generation script. new contrib added: libprotobuf-mutator 2021-10-12 03:21:31 +03:00
alexey-milovidov
84e16f9c8e
Merge pull request #29979 from azat/memory-profiler-sanitizers 
Allow memory profiler under sanitizers (ASan/UBsan/MSan only)
 2021-10-12 03:12:19 +03:00
Alexey Milovidov
d4e496c314 Add support for PowerPC build 2021-10-12 01:47:57 +03:00
tavplubix
83a9a8d4dc
Update KeeperStateMachine.cpp 2021-10-11 23:23:56 +03:00
Nikolai Kochetov
1e1d5d7fea Fix style. 2021-10-11 22:21:04 +03:00
alexey-milovidov
f1b550cdaf
Merge pull request #29974 from ClickHouse/minimized-toolchain 
Experiment with minimized toolchain (2)
 2021-10-11 22:04:01 +03:00
Alexander Tokmakov
706e2b6b88 more strict check for intersecting parts 2021-10-11 21:42:46 +03:00
Nikolai Kochetov
2c92caac17 Fix clickhouse local. 2021-10-11 21:20:35 +03:00
Nikolai Kochetov
ec18340351 Remove streams from formats. 2021-10-11 19:11:50 +03:00
alesapin
d301e40651 Cache DNS in Poco handler for S3 if no proxy specified 2021-10-11 18:31:09 +03:00
Dmitry Novik
bfdd34c13d code cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
7c3192735a Reset profile events stream in Connection::sendQuery 2021-10-11 17:39:24 +03:00
Dmitry Novik
9590c97a19 Create ProfileEventsQueue in GRPCServer 2021-10-11 17:39:24 +03:00
Dmitry Novik
3a0764634e Update InternalProfileEventsQueue usage 2021-10-11 17:39:24 +03:00
Dmitry Novik
c1f3e7e0bb Fix build 2021-10-11 17:39:24 +03:00
Dmitry Novik
0bdabf46f2 Send ProfileEvents only to supported clients 2021-10-11 17:39:24 +03:00
Dmitry Novik
9f9af28b5e Output memory usage with progress 2021-10-11 17:39:24 +03:00
Dmitry Novik
73df6190df Cleanup code 2021-10-11 17:39:24 +03:00
Dmitry Novik
1d2e2d7305 cleanup 2021-10-11 17:39:24 +03:00
Dmitry Novik
7e3caf96be Fix cores approximation 2021-10-11 17:39:24 +03:00
Dmitry Novik
4c6b3c40f2 Calculate approximate cores number used 2021-10-11 17:39:24 +03:00
Dmitry Novik
15ac65aa33 Add thread usage info on client side 2021-10-11 17:39:24 +03:00
Dmitry Novik
9071a7151e Fix communication & race conditions 2021-10-11 17:39:23 +03:00
Dmitry Novik
356723427d WIP on ProfileEvents forwarding 2021-10-11 17:39:23 +03:00
Dmitry Novik
74cdaba7fa WIP on profile events forwarding 2021-10-11 17:29:50 +03:00
Dmitry Novik
803b8623c1 Fix TCPHandler::sendProfileEvents 2021-10-11 17:29:50 +03:00
Dmitry Novik
4c5a774570 Add comment 2021-10-11 17:29:50 +03:00
Dmitry Novik
e9b1e05461 Send profile events from all threads of current group 2021-10-11 17:29:50 +03:00
Dmitry Novik
362bcb2f66 Introduce ProfileEvents packet 2021-10-11 17:29:50 +03:00
Haavard Kvaalen
ab4b229574 Make sure we update position on commit 
Make sure we update GTID set on QueryEvents with "COMMIT"
or "XA COMMIT".  Without this we could have to redo the last transaction
if e.g. ClickHouse was restarted.  Note that this did not affect normal
transactions on InnoDB, since they are terminated with a XID_EVENT.
 2021-10-11 16:20:51 +02:00
Dmitry Novik
a651c1bb67
Merge pull request #29820 from azat/MINSIGSTKSZ-fix 
Fix compilation with glibc 2.34 (MINSIGSTKSZ defined as sysconf(_SC_SIGSTKSZ))
 2021-10-11 17:06:29 +03:00
Amos Bird
635783fb66
Only do TreeOptimizer for initial queries 2021-10-11 20:47:35 +08:00
Amos Bird
f4269ce417
Allow optimize_arithmetic_operations_in_aggregate_functions when alias is used. 2021-10-11 20:47:34 +08:00
alexey-milovidov
3b8d7a5030
Merge pull request #29952 from MikaelUrankar/freebsd-aarch64 
Add support for FreeBSD aarch64
 2021-10-11 14:43:23 +03:00
Alexey Milovidov
2370a8d6da Fix PVS-Studio 2021-10-11 14:28:46 +03:00
Amos Bird
72bccaa501
Fix path name 2021-10-11 19:12:08 +08:00
Maksim Kita
b0d887a0fe Added tests 2021-10-11 14:00:10 +03:00
Amos Bird
83717b7c3b
Get rid of naming limitation of projections. 2021-10-11 18:32:17 +08:00
tavplubix
a9a7fd94cd
Merge pull request #29929 from azat/db-fix-lock-order-inversion 
Fix lock-order-inversion between DROP TABLE for DatabaseMemory and LiveView
 2021-10-11 13:23:58 +03:00
Kseniia Sumarokova
20e8074b5e
Merge pull request #29912 from taiyang-li/remove_unused_var 
Remove unused variable in s3cluster
 2021-10-11 12:45:04 +03:00
Maksim Kita
ce0c41e1ad Fixed tests 2021-10-11 11:39:50 +03:00
Maksim Kita
2d069acc22 System table data skipping indices added size 2021-10-11 11:39:50 +03:00
Vasily Nemkov
2c55807b2a Create dictionary with comment 2021-10-11 11:26:59 +03:00
Amos Bird
ebfb013ea1
Fix potential leak of query_id_holder 2021-10-11 15:56:00 +08:00
Azat Khuzhin
063f9cffab Allow memory profiler under sanitizers 
Only query profiler cannot work reliably with sanitizers (due to
unwinding from signal handler), but memory profiler should be fine.

Plus sometimes the problem appears only on build with sanitizers, so it
will be useful to have memory related profiling in trace_log.

Also there is a flaky check for stateless tests, that uses build with
ASan, and now trace_log there is empty, which sometimes does not allow
to debug further.
 2021-10-11 10:21:26 +03:00
Vladimir C
dc4bb920d3
Merge branch 'master' into join_deadlock 2021-10-11 10:18:17 +03:00
Vladimir C
d2b8d293b4
Merge pull request #29976 from hexiaoting/dev_quantile_fuse2 2021-10-11 10:06:40 +03:00
Vladimir C
6c7fbf0b88
Merge pull request #29956 from azat/fix-tuple-filter 2021-10-11 10:02:47 +03:00
Alexey Milovidov
284e547bc3 Fix error 2021-10-11 09:01:23 +03:00
hexiaoting
aabe52e3e9 Fix bug for quantile fusion 2021-10-11 10:39:55 +08:00
alexey-milovidov
0adbcd58c9
Merge pull request #29930 from azat/buffer-fix-data-race 
Fix data-race between flush() and startup() in StorageBuffer
 2021-10-11 01:11:39 +03:00
mergify[bot]
c07a49ba89
Merge branch 'master' into remove-bayes-ab 2021-10-10 21:10:57 +00:00
Azat Khuzhin
f37dac03f4 Fix lock-order-inversion between DROP TABLE for DatabaseMemory and LiveView 
CI stress founds [1], TSan report:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=509)
  Cycle in lock order graph: M71436201 (0x7b5800a91308) => M237700395169415472 (0x000000000000) => M71436201

  Mutex M237700395169415472 acquired here while holding mutex M71436201 in thread T723:
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x154b0db9)
    4 DB::DatabaseWithOwnTablesBase::tryGetTable() const obj-x86_64-linux-gnu/../src/Databases/DatabasesCommon.cpp:37:21 (clickhouse+0x154b0db9)
    5 DB::DatabaseMemory::tryGetTableUUID() const obj-x86_64-linux-gnu/../src/Databases/DatabaseMemory.cpp:95:22 (clickhouse+0x15466bb5)
    6 DB::Context::resolveStorageID() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2672:90 (clickhouse+0x155e6aa1)
    7 DB::JoinedTables::getLeftTableStorage() obj-x86_64-linux-gnu/../src/Interpreters/JoinedTables.cpp:200:29 (clickhouse+0x15eee962)
    8 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:321:33 (clickhouse+0x15b792be)
    9 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:160:7 (clickhouse+0x15b78160)
    10 DB::StorageLiveView::collectMergeableBlocks() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:113:28 (clickhouse+0x16aae192)
    11 DB::StorageLiveView::getNewBlocks() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:384:33 (clickhouse+0x16ab393a)
    12 DB::StorageLiveView::refresh(bool) obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:528:13 (clickhouse+0x16ab6395)
    13 DB::StorageLiveView::read() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:545:9 (clickhouse+0x16ab6395)
    21 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:950:30 (clickhouse+0x16101441)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M71436201 acquired here while holding mutex M237700395169415472 in thread T723:
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x16ab5c29)
    4 DB::StorageLiveView::drop() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:477:21 (clickhouse+0x16ab5c29)
    5 DB::DatabaseMemory::dropTable() obj-x86_64-linux-gnu/../src/Databases/DatabaseMemory.cpp:44:16 (clickhouse+0x15465cd6)
    8 DB::InterpreterDropQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:62:16 (clickhouse+0x15afa679)

  [1]: https://clickhouse-test-reports.s3.yandex.net/29856/42ca2b4bb241827edf69bbd6938d6b19c31935f1/stress_test_(thread).html#fail1

v2: fix attachTableUnlocked() (as pointed by @tavplubix)
v3: use SCOPE_EXIT() and so avoid holding a lock during removing from fs
 2021-10-10 23:51:36 +03:00
Azat Khuzhin
6c33eaee32 Fix filtering by tuple (some conditions was lost during analyzing) 
Fixes: #29281
Fixes: test_cluster_copier/test_three_nodes.py::test
 2021-10-10 22:37:52 +03:00
Nikolai Kochetov
a95c28ec4b
Merge pull request #29898 from ClickHouse/remove-native-stream 
Remove some more streams.
 2021-10-10 21:01:16 +03:00
Vitaly Baranov
1dda596689 Fix releasing query ID and session ID at the end of query processing. 2021-10-10 17:34:45 +03:00
mikael
689c9cdbb9 Add uc_mcontext for FreeBSD aarch64 2021-10-10 14:52:17 +02:00
feng lv
1229a8b528 fix 2021-10-10 04:12:24 +00:00
Alexey Milovidov
103b3c91ba Remove 'printf' function usage. 2021-10-10 04:21:07 +03:00
Alexey Milovidov
2a20bf4909 Remove function bayesAB #26233 2021-10-10 04:16:32 +03:00