thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-11 17:02:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
84,583 Commits 9,299 Branches 1,782 Tags 2.1 GiB
457fa0d1ba
Commit Graph

5679 Commits

Author SHA1 Message Date
Nikolai Kochetov
38fb50f736
Merge pull request #33958 from Algunenano/mv_cacheable_scalars 
Scalar cache improvements
 2022-02-09 16:46:53 +03:00
Maksim Kita
9f29c977de
Merge pull request #34234 from azat/agg-leak 
Fix memory leak in AggregatingInOrderTransform
 2022-02-09 12:42:04 +01:00
alesapin
10c3e6e546 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-09 14:14:58 +03:00
mergify[bot]
fd5caa3196
Merge branch 'master' into short-circuit-bug 2022-02-09 11:04:19 +00:00
Kseniia Sumarokova
86956ca08b
Merge pull request #34392 from kssenii/http-buffer-skip-not-found-url-for-globs 
Allow to skip not found urls for globs
 2022-02-09 09:32:09 +01:00
Azat Khuzhin
4fa2ae76bc Fix memory leak in AggregatingInOrderTransform 
Reproducer:

    # NOTE: we need clickhouse from 33957 since right now LSan is broken due to getauxval().
    $ url=https://s3.amazonaws.com/clickhouse-builds/33957/e04b862673644d313712607a0078f5d1c48b5377/package_asan/clickhouse
    $ wget $url -o clickhouse-asan
    $ chmod +x clickhouse-asan
    $ ./clickhouse-asan server &

    $ ./clickhouse-asan client
    :) create table data (key Int, value String) engine=MergeTree() order by key
    :) insert into data select number%5, toString(number) from numbers(10e6)

    # usually it is enough one query, benchmark is just for stability of the results
    # note, that if the exception was not happen from AggregatingInOrderTransform then add --continue_on_errors and wait
    $ ./clickhouse-asan benchmark --query 'select key, uniqCombined64(value), groupArray(value) from data group by key' --optimize_aggregation_in_order=1 --memory_tracker_fault_probability=0.01, max_untracked_memory='2Mi'

LSan report:

    ==24595==ERROR: LeakSanitizer: detected memory leaks

    Direct leak of 3932160 byte(s) in 6 object(s) allocated from:
        0 0xcadba93 in realloc ()
        1 0xcc108d9 in Allocator<false, false>::realloc() obj-x86_64-linux-gnu/../src/Common/Allocator.h:134:30
        2 0xde19eae in void DB::PODArrayBase<>::realloc<DB::Arena*&>(unsigned long, DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h:161:25
        3 0xde5f039 in void DB::PODArrayBase<>::reserveForNextSize<DB::Arena*&>(DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h
        4 0xde5f039 in void DB::PODArray<>::push_back<>(DB::GroupArrayNodeString*&, DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h:432:19
        5 0xde5f039 in DB::GroupArrayGeneralImpl<>::add() const obj-x86_64-linux-gnu/../src/AggregateFunctions/AggregateFunctionGroupArray.h:465:31
        6 0xde5f039 in DB::IAggregateFunctionHelper<>::addBatchSinglePlaceFromInterval() const obj-x86_64-linux-gnu/../src/AggregateFunctions/IAggregateFunction.h:481:53
        7 0x299df134 in DB::Aggregator::executeOnIntervalWithoutKeyImpl() obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:869:31
        8 0x2ca75f7d in DB::AggregatingInOrderTransform::consume() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingInOrderTransform.cpp:124:13

    ...

    SUMMARY: AddressSanitizer: 4523184 byte(s) leaked in 12 allocation(s).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-09 09:23:56 +03:00
Yakov Olkhovskiy
b9777be271 insertion with ephemerals requires columns list, insertion without columns list omits ephemeral 2022-02-09 01:11:29 +00:00
Anton Popov
587d7399ba support dynamic subcolumns for Memory engine 2022-02-09 03:18:53 +03:00
alesapin
02a93cb852 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-08 19:42:27 +03:00
avogar
08c009ab50 Fix possible error 'Cannot convert column Function to mask' 2022-02-08 18:23:43 +03:00
Nikolai Kochetov
0f7c0c72bd
Merge pull request #34305 from amosbird/projection-fix27 
Fix various issues when projection is enabled by default
 2022-02-08 17:19:56 +03:00
Kruglov Pavel
720310fcf1
Merge branch 'master' into http-buffer-skip-not-found-url-for-globs 2022-02-08 16:56:22 +03:00
Yatsishin Ilya
3e2e422ddd refactoring 2022-02-08 12:21:16 +00:00
Yatsishin Ilya
bcf4303286 Merge remote-tracking branch 'origin' into default-table-engine 2022-02-08 10:35:29 +00:00
Yakov Olkhovskiy
97daa84f6f revert unnecesarry change 2022-02-08 00:14:15 +00:00
Yakov Olkhovskiy
4d5fb56c29 added EPHEMERAL default for column (preliminary) 2022-02-07 23:21:10 +00:00
kssenii
eba3011ada Fix 2022-02-07 20:40:47 +01:00
Nicolae Vartolomei
1cdb50cf13 Disable optimize_trivial_count when deduplication for part movement feature is enabled 
Fixes #34089
 2022-02-07 18:26:49 +00:00
kssenii
2e58733750 Merge master 2022-02-07 15:23:10 +01:00
avogar
975ba9146f Fix tests naming, remove code duplication 2022-02-07 14:12:19 +03:00
mergify[bot]
dd947f964c
Merge branch 'master' into mv_cacheable_scalars 2022-02-07 10:07:26 +00:00
alesapin
ba28c2c013 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-07 12:44:56 +03:00
Yatsishin Ilya
e0803064e6 Merge remote-tracking branch 'origin' into default-table-engine 2022-02-07 08:45:32 +00:00
Amos Bird
a6f0b01e6a
Fix order by after aggregation 2022-02-07 00:42:11 +08:00
Amos Bird
52aabf98fe
Revise and add more comments 2022-02-06 16:53:54 +08:00
Amos Bird
1ab773cc90
Fix aggregation_in_order with normal projection 2022-02-06 16:46:12 +08:00
Amos Bird
7674bc986e
Disable projection when there is JOIN or SAMPLE 2022-02-06 16:46:09 +08:00
Maksim Kita
bc6328e573 Fixed tests 2022-02-04 20:56:49 +00:00
alesapin
c028269e6f Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 18:02:25 +03:00
Maksim Kita
629c8ae915
Merge pull request #34284 from CurtizJ/async-insert-in-client 
Minor fixes for #34267
 2022-02-04 13:13:47 +01:00
alesapin
2ed45b2a98 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-04 11:23:46 +03:00
Alexander Tokmakov
3e700e854d cancel merges before acquiring lock for truncate 2022-02-03 23:44:53 +03:00
Anton Popov
9b844c6b42
Merge pull request #32748 from CurtizJ/read-in-order-fixed-prefix 
Support `optimize_read_in_order` if prefix of sorting key is already sorted
 2022-02-03 18:17:08 +03:00
alesapin
2a9bc7cba8 Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-03 16:20:06 +03:00
alesapin
8e33427e19 Fix check for existing table 2022-02-03 15:17:55 +03:00
Anton Popov
0755cfa584 minor fixes 2022-02-03 15:04:13 +03:00
alexey-milovidov
a6d1fcadcb
Merge pull request #34267 from CurtizJ/async-insert-in-client 
Support async inserts in clickhouse-client for queries with inlined data
 2022-02-03 05:14:47 +03:00
Maksim Kita
02c6cfba6a
Merge pull request #34225 from vitlibar/fix-checking-grants-for-show-grants 
Fix checking grants for SHOW GRANTS
 2022-02-02 22:51:51 +01:00
Yatsishin Ilya
cce0452d19 add more tests, fix style, fix MATERIALIZED VIEW POPULATE without ENGINE 2022-02-02 18:07:03 +00:00
Anton Popov
5bb1b3ce77 support async inserts in clickhouse-client for queries with inlined data 2022-02-02 20:59:37 +03:00
Vitaly Baranov
30557aebfb Add helper class to cache the result of checking access. 2022-02-02 22:22:41 +07:00
Yatsishin Ilya
8c822755cb fix AS 2022-02-02 12:33:29 +00:00
Nikolay Degterinsky
29c15f8c65
Merge pull request #33723 from FrankChen021/trace_id_order 
Parse and store opentelemetry trace-id in big-endian order
 2022-02-02 14:59:46 +03:00
tavplubix
7304fd654a
Merge pull request #34224 from CurtizJ/fix-query-metric 
Fix metric `Query`
 2022-02-02 13:39:07 +03:00
Yatsishin Ilya
759574d6f0 improve 2022-02-01 16:59:41 +00:00
Anton Popov
a4aa1f37ce remove unused code 2022-02-01 17:44:23 +03:00
Yatsishin Ilya
2a698b8da3 Merge remote-tracking branch 'origin/master' into default-table-engine 2022-02-01 13:23:52 +00:00
Vitaly Baranov
1f81e43f1e Fix checking grants for SHOW GRANTS. 2022-02-01 19:55:24 +07:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
Anton Popov
c28255850a fix metric Query 2022-02-01 15:06:49 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Ilya Yatsishin
9c2ef98d23
Apply suggestions from code review 
Co-authored-by: tavplubix <tavplubix@gmail.com>
 2022-02-01 10:03:50 +03:00
Amos Bird
ec7d367814
DiskLocal checker 
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
 2022-02-01 05:55:27 +08:00
Yatsishin Ilya
676fc5b20f Initial commit 2022-01-31 17:10:59 +00:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
Maksim Kita
9c91a50050 Sort block refactoring 2022-01-29 15:50:55 +00:00
Maksim Kita
f3453024ff
Merge pull request #34060 from amosbird/optimizetupleorderby 
Make ORDER BY tuple almost as fast as ORDER BY columns
 2022-01-29 15:58:09 +01:00
Amos Bird
04cff632db
Revise 2022-01-29 20:55:52 +08:00
alexey-milovidov
6535b75322
Merge pull request #34001 from azat/memory-tracker-fix 
Fix memory accounting for queries that uses < max_untracker_memory
 2022-01-29 00:59:53 +03:00
Amos Bird
2b3ce910c6
Fix collation check 2022-01-29 04:49:58 +08:00
Federico Rodriguez
c0a5bd187a ExecuteScalarSubqueriesVisitor missing static const 
Added string_view
 2022-01-28 15:00:04 -05:00
Amos Bird
09a3ab79b7
Fix all const 2022-01-29 01:37:39 +08:00
Anton Popov
fa883f2ac3 fix tests 2022-01-28 17:23:51 +03:00
Raúl Marín
6c79e147d0 Add a few extra comments around uses_view_source and why it's propagated 2022-01-28 14:47:49 +01:00
Azat Khuzhin
162f96f8e1 Get back detachQueryIfNotDetached() into ExternalLoader 
ExternalLoader from ThreadPool (async loading) is done from the server
context, not from the query context, and the context will already go
away, so we should detachQueryIfNotDetached() to avoid trigger assertion
in ThreadStatus.

CI: https://s3.amazonaws.com/clickhouse-test-reports/34001/8cace291d17fa9553a98b2a1e8bf15b30fe5a1bd/stateless_tests__debug__actions__[3/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Azat Khuzhin
b0c862c297 Fix memory accounting for queries that uses < max_untracker_memory 
MemoryTracker starts accounting memory directly only after per-thread
allocation exceeded max_untracker_memory (or memory_profiler_step).

But even memory under this limit should be accounted too, and there is
code to do this in ThreadStatus dtor, however due to
PullingAsyncPipelineExecutor detached the query from thread group that
memory was not accounted.

So remove CurrentThread::detachQueryIfNotDetached() from threads that
uses ThreadFromGlobalPool since it has ThreadStatus, and the query will
be detached using CurrentThread::defaultThreadDeleter.

Note, that before this patch memory accounting works for HTTP queries
due to it had been accounted from ParallelFormattingOutputFormat, but
not for TCP.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Raúl Marín
64799867a1 Add a comment about IInterpreterUnionOrSelectQuery::usesViewSource() 2022-01-28 12:39:56 +01:00
Nikolai Kochetov
94999e86c2
Merge pull request #33392 from azat/_shard_num 
Replace old _shard_num implementation with shardNum() function
 2022-01-28 11:02:19 +03:00
alexey-milovidov
63d8d75001
Merge pull request #33957 from azat/fix-getauxval 
Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work)
 2022-01-27 23:05:23 +03:00
tavplubix
8c9266b24c
Merge pull request #33978 from amosbird/separatesystemlog 
Separate base parts out of SystemLog
 2022-01-27 20:07:55 +03:00
Amos Bird
faee95b897
Make ORDER BY tuple almost as fast as ORDER BY columns 
We have special optimizations for multiple column ORDER BY: https://github.com/ClickHouse/ClickHouse/pull/10831 . It's beneficial to also apply to tuple columns.

Before:

select * from numbers(300000000) order by (1 - number , number + 1 , number) limit 10;
2.613 sec.

After:

select * from numbers(300000000) order by (1 - number , number + 1 , number) limit 10;
0.755 sec

No tuple:

select * from numbers(300000000) order by 1 - number , number + 1 , number limit 10;
0.755 sec
 2022-01-27 21:42:08 +08:00
Azat Khuzhin
af9910b518 Fix leak of demangle result (for opentelementry) 
LSan found [1]:

    Direct leak of 5170176 byte(s) in 5049 object(s) allocated from:
        0 0xc598edd in malloc (/usr/bin/clickhouse+0xc598edd)
        1 0x39679739 in (anonymous namespace)::itanium_demangle::initializeOutputStream(char*, unsigned long*, (anonymous namespace)::itanium_demangle::OutputStream&, unsigned long) obj-x86_64-linux-gnu/../contrib/libcxxabi/src/demangle/Utility.h:178:31
        2 0x39679739 in __cxa_demangle obj-x86_64-linux-gnu/../contrib/libcxxabi/src/cxa_demangle.cpp:351:13
        3 0x28f6f3ed in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:662:44

  [1]:
https://s3.amazonaws.com/clickhouse-test-reports/33957/08f4f45fd9da923ae3e3fdd8a527c297d35247eb/stress_test__address__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-27 13:46:37 +03:00
kssenii
c91b86e220 Better 2022-01-26 21:56:08 +03:00
Raúl Marín
045c92e2b9 Fix readability issues 2022-01-26 17:36:45 +01:00
Raúl Marín
78d2caee7e Can only pre-save scalars if they are cacheable 2022-01-26 17:36:45 +01:00
Raúl Marín
e77fc9e9be Comment 2022-01-26 17:36:45 +01:00
Raúl Marín
444a8e2519 Minimize changes, improve scalar subquery for MVs 2022-01-26 17:36:45 +01:00
Raúl Marín
4b5ab80e3b Better scalar cache handling 
- Fixes global CTE scalar cache.
- Adds MVs back (views dependent on the source are cached locally and others globally
 2022-01-26 17:36:45 +01:00
tavplubix
9e91a9dfd1
Merge pull request #33429 from zzsmdfj/MaterializeMySQL_support_set_and_other_dataType 
MaterializeMySQL support SET YEAR TIME GEOMETRY dataType
 2022-01-26 11:29:46 +03:00
Maksim Kita
bf768a94e8
Merge pull request #33971 from Avogar/fix-ipv6 
Fix parsing IPv6 from query parameter and fix IPv6 to string conversion
 2022-01-25 22:45:11 +01:00
Vitaly Baranov
cc38fe3fb6 Fix checking access for the SYSTEM command. 2022-01-25 18:54:59 +07:00
Amos Bird
385caa3e0a
Separate base parts out of SystemLog 2022-01-25 18:48:43 +08:00
avogar
ab49472155 Add comment 2022-01-25 10:10:04 +03:00
avogar
ad01d68f36 Fix parsing IPv6 from query parameter and fix IPv6 to string conversion 2022-01-25 10:09:17 +03:00
Ilya Yatsishin
35c03b317e
Merge pull request #33639 from Algunenano/async_metrics_handle_failures 
AsynchronousMetrics: Ignore inaccessible sensors
 2022-01-24 14:04:07 +03:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806 
Fix logical error on RESTORE REPLICA
 2022-01-24 13:28:37 +03:00
Maksim Kita
7fffe5846f
Merge pull request #33924 from kitaisreal/sql-user-defined-functions-invalid-lambda-crash-fix 
SQLUserDefinedFunctions invalid lambda additional cases
 2022-01-23 22:06:49 +01:00
Maksim Kita
243a280936 Updated tests 2022-01-23 17:02:43 +00:00
alexey-milovidov
d7a63dfda6
Merge pull request #33677 from ucasfl/fix-sumif 
Fix sumIf rewrite
 2022-01-22 17:12:13 +03:00
Maksim Kita
1bc3f086b1
Merge pull request #33889 from kitaisreal/sql-user-defined-functions-invalid-lambda-crash-fix 
SQLUserDefinedFunctions invalid lambda additional fixes
 2022-01-22 12:09:42 +01:00
alexey-milovidov
eb6849f7c7
Merge pull request #33842 from azat/cmake-contrib-fixes 
More cmake external modules cleanups
 2022-01-22 10:34:54 +03:00
mergify[bot]
7734f9d629
Merge branch 'master' into fix-sumif 2022-01-22 06:08:47 +00:00
Maksim Kita
3a252cd3a0 Updated tests 2022-01-21 20:49:42 +00:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Maksim Kita
330987a64b
Merge pull request #33868 from kitaisreal/sql-user-defined-functions-invalid-lambda-crash-fix 
SQLUserDefinedFunctions invalid lambda crash fix
 2022-01-21 16:23:12 +01:00
Maksim Kita
3b24c39294 SQLUserDefinedFunctions invalid lambda crash fix 2022-01-21 14:33:57 +00:00
Nikolai Kochetov
012453e212
Merge pull request #33588 from alexeypavlenko/fix_nested_default 
Fix default value generation for columns with dot
 2022-01-21 17:19:42 +03:00
Kruglov Pavel
daba57a933
Merge pull request #33737 from Avogar/bool-type 
Some improvements and fixes for Bool data type
 2022-01-21 15:24:58 +03:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
Azat Khuzhin
dc7affd813 contrib/libmetrohash: add ALIAS library 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-21 10:11:23 +03:00
zzsmdfj
411f43ec4d to MaterializeMySQL_support_set_and_other_dataType 2022-01-21 12:24:12 +08:00
zzsmdfj
e380e59b16 MaterializeMySQL support SET YEAR TIME GEOMETRY dataType 2022-01-21 12:24:11 +08:00
Maksim Kita
e2fe8dc3a7 Fix build with ENABLE_EXAMPLES 2022-01-20 19:29:21 +00:00
tavplubix
41ee114abb
Merge pull request #33819 from bigo-sg/optimize_explain_create_function 
Support explain create function query
 2022-01-20 19:07:43 +03:00
Frank Chen
c1f9d6cd6f Fix traceparent propagation out of ClickHouse via URL engine 2022-01-20 23:12:23 +08:00
taiyang-li
8886ac61e7 remove useless code 2022-01-20 19:33:19 +08:00
avogar
ba54a5ff80 Some improvements and fixes for Bool data type 2022-01-20 14:16:11 +03:00
taiyang-li
e4f5444073 support explain create function query 2022-01-20 16:23:52 +08:00
Azat Khuzhin
48b5c098dd Remove unbundled farmhash support 2022-01-20 10:01:12 +03:00
Azat Khuzhin
8ede97925e Remove unbundled sparsehash support 2022-01-20 10:01:11 +03:00
Frank Chen
c9f9fd2a38 Merge branch 'master' into trace_id_order 2022-01-20 13:07:50 +08:00
Nikita Mikhaylov
e3de3889e3
Revert "Revert "Ignore parse failure of opentelemetry header (#32116)" (#33594)" (#33595) 2022-01-19 20:53:39 +03:00
Amos Bird
62441f0a0f
Fix mutation when table contains projections (#33679) 2022-01-19 15:27:11 +03:00
alexey-milovidov
d222cb9771
Merge pull request #33695 from amosbird/buildimprove 
Some build improvement
 2022-01-19 03:19:22 +03:00
Dmitry Novik
4e612d0e1a
Merge branch 'master' into memory-overcommit 2022-01-18 23:33:29 +03:00
Dmitry Novik
d7b4a32938 fix build 2022-01-18 20:26:12 +00:00
alesapin
e5ec1aa446
Merge pull request #33642 from ClickHouse/add_integration_test_for_zero_duplication 
Fix bug with background moves in zero copy replication
 2022-01-18 18:32:16 +03:00
Kruglov Pavel
2295a07066
Merge pull request #33534 from azat/fwd-decl 
RFC: Split headers, move SystemLog into module, more forward declarations
 2022-01-18 17:22:49 +03:00
Dmitry Novik
c0970b75ee
Merge branch 'master' into memory-overcommit 2022-01-18 15:30:24 +03:00
Dmitry Novik
83c663e2d6 Cleanup after code review 2022-01-18 12:21:59 +00:00
Maksim Kita
30dab61f97
Merge pull request #33526 from kitaisreal/dictionary-rename-fix 
Dictionary rename fix
 2022-01-18 13:11:34 +01:00
frank chen
da0608f365 Parse trace-id in big-endian order 
Signed-off-by: frank chen <frank.chen021@outlook.com>
 2022-01-18 13:24:56 +08:00
Nikolai Kochetov
96ba5b14c5 Update addMissingDefaults.cpp 2022-01-17 22:35:12 +02:00
Nikolai Kochetov
65519527ef Update addMissingDefaults.cpp 
Fix style
 2022-01-17 22:35:11 +02:00
alexeypavlenko
690590f7c3 Fix default value generation for columns with dot 2022-01-17 22:35:11 +02:00
Amos Bird
6d62060e16
Build improvement 2022-01-17 22:36:27 +08:00
alesapin
4caef03e6a Add ephemeral lock in zookeeper 2022-01-17 14:52:51 +03:00
feng lv
1cdadbd250 Fix sumIf rewrite 2022-01-16 06:25:11 +00:00
Alexey Milovidov
4ab7dfc741 Fix #8340 2022-01-16 08:12:55 +03:00
Maksim Kita
a0ad7a1014 Update IExternalLoadable interface 2022-01-16 00:06:10 +00:00
Raúl Marín
ac1665cdcb AsynchronousMetrics: Ignore inaccessible sensors 2022-01-14 15:52:08 +01:00
Azat Khuzhin
4dcb332be9 Add ability to rewrite only JOIN tables in AddDefaultDatabaseVisitor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 11:18:52 +03:00
Azat Khuzhin
249bdce453 Fix AddDefaultDatabaseVisitor for temporary tables 
This will fix rewriting of distributed queries with GLOBAL IN

v2: get external tables only for non-global context

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 11:18:52 +03:00
Azat Khuzhin
0a9b1ee803 Remove RestoreQualifiedNamesMatcher::Data::rename (always true) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 11:18:52 +03:00
Nikita Mikhaylov
340ec0fd20
Revert "Ignore parse failure of opentelemetry header (#32116)" (#33594) 
This reverts commit 4e5f9c8ff7.
 2022-01-13 16:29:18 +04:00
Frank Chen
4e5f9c8ff7
Ignore parse failure of opentelemetry header (#32116) 2022-01-13 16:16:10 +04:00
tavplubix
1075f604ba
Merge pull request #33577 from DevTeamBK/Issue167 
skip MySQLCreateRewritten unit testsuite when enable_mysql is off
 2022-01-13 14:28:48 +03:00
tavplubix
3b7f8a4ae9
Merge pull request #33293 from zhanghuajieHIT/hangup_in_drop_query_log_sync 
fix hang up with command 'drop table system.query_log sync'
 2022-01-13 12:28:31 +03:00
alexey-milovidov
6c71a7c40f
Merge pull request #33518 from hexiaoting/dev-limit-offset 
Fix bug for select from VIEW with format and limit settings
 2022-01-12 23:39:31 +03:00
Suzy Wang
ad11f3f05d skip MySQLCreateRewritten unit testsuite when enable_mysql is off 2022-01-12 11:05:56 -08:00
zhanghuajie
fc097173ef fix hang up with command 'drop table system.query_log sync' 2022-01-12 20:23:22 +08:00
Azat Khuzhin
248b879fd5 Split TraceCollector between Common and Interpreters 
This is required to leave TraceCollector::collect() inside main for
MemoryTrackier.
 2022-01-11 22:30:55 +03:00
hexiaoting
5c847264f7 Fix bug for select from VIEW with format and limit settings 2022-01-11 20:19:41 +08:00
Maksim Kita
8aa930b52f
Merge pull request #33294 from ClickHouse/query-kind-concurent_restriction 
Add IAST::QueryKind enum and use it in query limiter
 2022-01-11 11:23:18 +03:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
c1dea66907 Move TraceCollector into Interpreters 
Since now it relies on SystemLog that is in Interpreters, and it cannot
be moved into Common, since it has lots of dependencies.
 2022-01-10 22:35:42 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog 
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
 2022-01-10 22:01:41 +03:00
Azat Khuzhin
30e0df1e8c Forward declaration of IStorage in InterpreterWatchQuery 2022-01-10 22:01:41 +03:00
Azat Khuzhin
1637c41d42 Remove leftovers of old _shard_num via identifier implementation 2022-01-10 21:21:24 +03:00
Azat Khuzhin
97acf190ed Replace old _shard_num implementation with shardNum() function 
_shard_num via constant identifier (from #7624) has too much issues,
take a look at #16947 for example.

shardNum() function (from #27020) should works better.

This changes the behaviour of _shard_num slightly, now it returns nested
shard number in case of nested distributed tables (Distributed over
Distributed), but this should be minor.

v2: Rewrite _shard_num to shardNum() in TreeRewriter
 2022-01-10 21:21:24 +03:00
Mikhail f. Shiryaev
d35ad19135
Fix review points 2022-01-10 18:23:17 +01:00
Mikhail f. Shiryaev
a8c83dce14
Use map for QueryKindAmounts 2022-01-10 13:49:53 +01:00
Anton Popov
46b9279d81
Merge pull request #33300 from CurtizJ/support-prewhere-storage-merge 
Support `optimize_move_to_prewhere` with storage `Merge`
 2022-01-10 15:15:15 +03:00
Kseniia Sumarokova
e6ea7c361d
Merge pull request #32836 from stigsb/explain-table-override 
Add `EXPLAIN TABLE OVERRIDE` for pre-validating overrides.
 2022-01-10 09:58:16 +03:00
Alexey Milovidov
3787bf72c2 Fix potential exponential growth of untuple function 2022-01-07 09:15:10 +03:00
Azat Khuzhin
5ed7440381 Fix possible use-after-free for INSERT into MV with concurrent DROP 
ASan founds [1]:

    ==553==
    ERROR: AddressSanitizer: heap-use-after-free on address 0x61e004694080 at pc 0x000029150af2 bp 0x7f70b3f8ef10 sp 0x7f70b3f8ef08
    READ of size 8 at 0x61e004694080 thread T477 (QueryPipelineEx)
        0 0x29150af1 in DB::MergeTreeDataWriter::writeTempPart() >
        1 0x293b8e43 in DB::MergeTreeSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeSink.cpp:27:65
        2 0x29dac73b in DB::SinkToStorage::onConsume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5
        3 0x29c72dd2 in DB::ExceptionKeepingTransform::work()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Processors/Transforms/ExceptionKeepingTransform.cpp:151:51

    0x61e004694080 is located 2048 bytes inside of 2480-byte region [0x61e004693880,0x61e004694230)
    freed by thread T199 (BgSchPool) here:
        ...
        4 0x26220f20 in DB::DatabaseCatalog::TableMarkedAsDropped::~TableMarkedAsDropped() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.h:248:12
        5 0x26220f20 in DB::DatabaseCatalog::dropTableDataTask() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:908:1

      [1]: https://s3.amazonaws.com/clickhouse-test-reports/33201/4f04d6af61eabf4899eb8188150dc862aaab80fc/stress_test__address__actions_.html

There was a fix in #32572, but it was not complete (yes it reduced the
race window a lot, but not completely), since the inner table still can
go away after the INSERT chain was built, to fix this obtain the
reference earlier.

Follow-up for: #32572 (cc @tavplubix)
 2022-01-04 13:34:01 +03:00
Maksim Kita
a7d0f3e637 UserDefinedExecutableFunction fix exception 2022-01-02 16:21:52 +03:00
alexey-milovidov
3737d83d3e
Merge pull request #33333 from ClickHouse/add-access-control-to-system-commands 
Add missing access control to some system commands
 2021-12-31 14:36:59 +03:00
alexey-milovidov
6941b072f4
Merge pull request #33330 from cmsxbc/support-self-closing-tag-for-cluster-secure 
support `<secure/>` in cluster configuration.
 2021-12-31 14:36:19 +03:00
Kseniia Sumarokova
a703bcb0c5
Merge pull request #33098 from bigo-sg/lrucache 
Add alternative LRUCache version
 2021-12-30 23:10:28 +03:00
Alexey Milovidov
d547fddf22 Add missing access control to some system commands 2021-12-30 21:37:21 +03:00
cmsxbc
c28fcd5a2f
support <secure/> for cluster 2021-12-30 23:51:05 +08:00
Maksim Kita
0593f1498a
Merge pull request #33285 from zbtzbtzbt/opt_hashTable 
rm redundant judge in hashmap iter operation
 2021-12-30 16:34:08 +03:00
vdimir
ae7e5691b8
Get rid of IAST::getQueryKindString 2021-12-30 12:55:50 +03:00
Stig Bakken
6fbb9f5e87 Implement EXPLAIN TABLE OVERRIDE for pre-validating overrides. 
Example:

```sql
EXPLAIN TABLE OVERRIDE mysql('127.0.0.1:3306', 'db', 'table', 'user', 'pw')
    PARTITION BY tuple(toYYYYMM(created), id % 8)
```

Validations done:

 * check that storage overrides do not reference unknown or nullable columns
 * check that default specifier is not modified for columns
 2021-12-30 09:02:27 +01:00
Anton Popov
abcc48c1f6 Merge remote-tracking branch 'upstream/master' into support-prewhere-storage-merge 2021-12-29 21:07:35 +03:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference 
Automatic schema inference for input formats
 2021-12-29 21:03:48 +03:00
Anton Popov
7c6f7f6732 support 'optimize_move_to_prewhere' with storage 'Merge' 2021-12-29 20:49:10 +03:00
Mikhail f. Shiryaev
a882e64644
Use IAST::QueryKind instead of strings in QueryKindAmount 2021-12-29 17:32:59 +01:00
Mikhail f. Shiryaev
272ea7fc5b
Merge pull request #32609 from cmsxbc/query-kind-concurent_restriction 
add settings: max_concurrent_select_queries and max_concurrent_insert_queries
 2021-12-29 15:23:45 +01:00
Nikolai Kochetov
c715204e18
Merge pull request #32271 from Algunenano/kill_scalar_github 
Be able to KILL scalar queries
 2021-12-29 16:00:59 +03:00
mergify[bot]
5070784282
Merge branch 'master' into memory-overcommit 2021-12-29 10:35:32 +00:00
Kruglov Pavel
dd7f61b337
Merge branch 'master' into schema-inference 2021-12-29 12:59:05 +03:00
zhoubintao
8690cf082f remove redundant judge to make heavy map iter operation faster 2021-12-29 17:48:39 +08:00
Maksim Kita
a5256e5b6e
Merge pull request #30957 from kitaisreal/executable-refactoring 
Executable refactoring
 2021-12-29 12:43:41 +03:00
Maksim Kita
c3c8af747d
Merge pull request #33261 from kitaisreal/dictionaries-fix-comment 
Dictionaries fix comment
 2021-12-29 12:43:24 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
lgbo-ustc
59cbd76880 Add LRUResourceCache 
1. add LRUResourceCache for managing resource cache in lru policy
2. rollback LRUCache to the original version
3. add remove() in LRUCache
4. add unit tests for LRUResourceCache and LRUCache
 2021-12-29 15:25:33 +08:00
Kseniia Sumarokova
9b63fa6949
Merge pull request #33239 from CurtizJ/async-insert-query-log 
Add asynchronous inserts to query log
 2021-12-29 09:34:08 +03:00
Anton Popov
d25560d37a fix queries without table name 2021-12-29 00:14:22 +03:00
Maksim Kita
9ef359ce2c Dictionaries fix comment 2021-12-28 23:50:48 +03:00
Maksim Kita
d1db3c9f42 Updated ShellCommandSource 2021-12-28 22:55:31 +03:00
Maksim Kita
5590cfa3aa Updated executable function integration tests 2021-12-28 22:55:30 +03:00
Maksim Kita
2ffd83ee83 Fix shell command source start 2021-12-28 22:55:30 +03:00
Maksim Kita
eb4e400c54 Executable refactoring 2021-12-28 22:55:30 +03:00
Anton Popov
a6e90a046d add more info to query log in async inserts 2021-12-28 17:53:25 +03:00
Vladimir C
bb6fc853e3
Merge pull request #31442 from vdimir/cluster-discovery 2021-12-28 10:53:30 +03:00
Anton Popov
0e2d0d70ba add asynchronous inserts to query log 2021-12-27 23:16:53 +03:00
vdimir
5f82190cd3
Log exception on cluster discovery initial update 2021-12-27 18:24:25 +03:00
vdimir
01a9199098
Move stop_flag to clusters_to_update in ClusterDiscovery 2021-12-27 16:10:36 +03:00
vdimir
5bd620fba1
Reset backoff when cluster discovery is up to date 2021-12-27 16:10:35 +03:00
vdimir
3934e99dce
update ClusterDiscovery::ConcurrentFlags::set 2021-12-27 16:09:50 +03:00
vdimir
a76e93209d
Backoff for ClusterDiscovery::runMainThread 2021-12-27 16:09:44 +03:00
vdimir
be7efb14c6
force updating cluster discovery periodically 2021-12-27 15:26:12 +03:00
vdimir
3d3fae126e
check if clusters updated successfully on current iteration of discovery 2021-12-27 15:26:12 +03:00
vdimir
36cd9ecb14
log level debug in cluster discovery 2021-12-27 15:26:11 +03:00
vdimir
72e63b4f12
restart runMainThread in cluster discovery 2021-12-27 15:26:11 +03:00
vdimir
4f8a9cc539
cluster discovery: versioning for format of data stored in zk 2021-12-27 15:26:11 +03:00
vdimir
d3b1058c40
cluster discovery configuration, support shards 2021-12-27 15:26:11 +03:00