thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
102,350 Commits 9,326 Branches 1,784 Tags 2.1 GiB
006af1b6a4
Commit Graph

34066 Commits

Author SHA1 Message Date
Alexander Tokmakov
2924f3ba7e
Merge branch 'master' into drop-fix 2022-11-22 17:05:12 +03:00
Sergei Trifonov
59f422f10c
Merge branch 'master' into storage-system-moves 2022-11-22 14:46:00 +01:00
Kruglov Pavel
98d6b96c82
Merge pull request #42033 from mark-polokhov/BSONEachRow 
Add BSONEachRow input/output format
 2022-11-22 14:45:21 +01:00
Azat Khuzhin
e7d6b9ccca Avoid possible DROP hung due to attached web disk 
While dropping the table server scans all disk for existence of data to
remove, but in case of unavailable web disk it will fail, and also not
instantly but after 10 retries (by default, after #42800), that can take
~30 seconds.

But scanning read only disks does not makes any sense, so reorder
conditions to avoid this.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 14:44:56 +01:00
Maksim Kita
1e1492e480 Fixed build 2022-11-22 14:35:44 +01:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob 
fix s3 support question mark wildcard
 2022-11-22 14:26:56 +01:00
Kseniia Sumarokova
392963012a
Merge pull request #43404 from azat/part.log.uuid 
Add table_uuid to system.parts
 2022-11-22 14:25:34 +01:00
Alexander Tokmakov
91b2cdb94a
Merge branch 'master' into tavplubix-patch-5 2022-11-22 16:17:55 +03:00
Kruglov Pavel
62bfb5547c
Merge pull request #43368 from Avogar/fix-if-nothing 
Fix possible `Cannot create non-empty column with type Nothing` in functions if/multiIf
 2022-11-22 14:11:13 +01:00
Sergei Trifonov
3177ed050f
Merge pull request #43307 from ClickHouse/cpu-progress-protocol-fix 
fix CPU load indication on client
 2022-11-22 14:09:55 +01:00
Alexander Gololobov
31bffdd735
Merge pull request #43460 from ClickHouse/smaller_buffer_for_small_files 
Use smaller buffer for small files
 2022-11-22 14:00:22 +01:00
Alexander Tokmakov
4f28334b5c fix mess with signed sizes 2022-11-22 13:57:05 +01:00
Anton Popov
2572202a16
Merge pull request #43461 from CurtizJ/merging-42064 
Merging #42064
 2022-11-22 13:55:01 +01:00
Dmitry Novik
e67bda65a0
Merge pull request #43453 from ClickHouse/fix-hard-limit-in-test 
Do not reuse jemalloc memory in test_global_overcommit
 2022-11-22 13:24:34 +01:00
Alexander Tokmakov
cab33c02af
Merge branch 'master' into revert_SingleValueDataString 2022-11-22 14:27:58 +03:00
Nikolai Kochetov
9e84a351c7 Fixing special build. Disable setting by default. 2022-11-22 11:24:24 +00:00
zzsmdfj
21f2cffa4d fix date_diff wrong result build by darwin 2022-11-22 18:58:37 +08:00
Maksim Kita
04f597c29c Fixed style check 2022-11-22 11:14:55 +01:00
Maksim Kita
0fe17c2c7b Fixed code review issues 2022-11-22 11:14:55 +01:00
Maksim Kita
cdc2fb0b59 Analyzer small fixes 2022-11-22 11:14:55 +01:00
Antonio Andelic
843401c3a3
Merge pull request #43323 from ClickHouse/fix-describe-delta-lake-hudi 
Fix `DESCRIBE` for `deltaLake` and `hudi` table functions
 2022-11-22 09:09:39 +01:00
Anton Popov
0479d8d9f5
Merge pull request #43387 from ClickHouse/fix_default_value_used_in_row_level_filter 
Fix default value used in row level filter
 2022-11-22 01:46:45 +01:00
Dmitry Novik
06ec01aa6e
Merge branch 'master' into explain-query-tree-run 2022-11-21 23:51:22 +01:00
Vitaly Baranov
6c2b53f0ae
Merge pull request #42940 from filimonov/exceptions_with_masker 
Add SensitiveDataMasker to exceptions messages
 2022-11-21 23:20:15 +01:00
Alexander Gololobov
5afd2a1add Pass file size to better choose buffer size 2022-11-21 22:36:45 +01:00
Alexander Gololobov
86a0dd010f Use read_hint and file_size for choosing buffer size 2022-11-21 22:36:28 +01:00
kssenii
eb23e268ee Merge remote-tracking branch 'upstream/master' into named-collections-access-fix 2022-11-21 21:26:18 +01:00
Sergei Trifonov
94a7614c45
Merge branch 'master' into storage-system-moves 2022-11-21 20:10:12 +01:00
Alexander Tokmakov
a7efdd7f4e
Update StorageReplicatedMergeTree.cpp 2022-11-21 21:52:48 +03:00
Nikolai Kochetov
c305afd77a Fix race. 2022-11-21 18:26:31 +00:00
Anton Popov
5532e3db95 Merge branch 'filesystem-function' of https://github.com/ucasfl/ClickHouse into merging-42064 2022-11-21 18:23:11 +00:00
Kruglov Pavel
581e57be9f
Merge pull request #43236 from xiedeyantu/fix_hdfscluster_connection_fail 
fix skip_unavailable_shards does not work using hdfsCluster table function
 2022-11-21 19:22:16 +01:00
serxa
faef2e9b00 fix moves list entry creation 2022-11-21 17:08:54 +00:00
chen
96543074fe
Update S3Common.cpp 2022-11-22 01:03:31 +08:00
serxa
c425da06b2 fix column name 2022-11-21 16:16:53 +00:00
Dmitry Novik
b17dc24a94 Do not reuse jemalloc memory in test 2022-11-21 16:10:47 +00:00
Nikolay Degterinsky
65b5523bdb
Merge pull request #43350 from evillique/fix-parser-5 
Fix function parameters parsing
 2022-11-21 16:23:02 +01:00
Dmitry Novik
239119c499 Set run_passes to 1 by default 2022-11-21 15:05:52 +00:00
Alexey Milovidov
4ab6896ece Remove noise from logs about NetLink in Docker 2022-11-21 15:16:27 +01:00
Kruglov Pavel
406c50277b
Merge pull request #42779 from Avogar/add-format-func-doc 
Add documentation for 'format' table function
 2022-11-21 15:16:26 +01:00
Raúl Marín
35e9e169a1
Merge branch 'master' into revert_SingleValueDataString 2022-11-21 15:11:13 +01:00
Alexey Milovidov
56f8c0b84b Fix typo 2022-11-21 15:10:15 +01:00
Alexey Milovidov
bb5756bb11
Merge pull request #43403 from ClickHouse/table-function-add-comment 
Add a comment
 2022-11-21 15:06:20 +01:00
Kseniia Sumarokova
b9b1d22d72
Update ClientBase.cpp 2022-11-21 15:04:37 +01:00
Kseniia Sumarokova
a59dac6337
Merge pull request #43143 from azat/disks/s3-check-fix 
Add server UUID for disks access checks (read/read-by-offset/write/delete) to avoid possible races
 2022-11-21 15:03:46 +01:00
xiedeyantu
3a3650d370 add comments 2022-11-21 21:46:15 +08:00
kssenii
10a258242d Fix progress from stdin 2022-11-21 14:11:51 +01:00
Kruglov Pavel
975a59af7a
Fix build 2022-11-21 13:51:17 +01:00
Raúl Marín
a52bdca989 Detect the other corrupted state too 2022-11-21 13:33:16 +01:00
Azat Khuzhin
d0ce500f89 Add table_uuid to system.parts 
Can be useful if the table constantly recreated, i.e. in some tests.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-21 12:11:51 +01:00
alesapin
b78b602ea1
Merge branch 'master' into Enable_compress_marks_and_primary_key 2022-11-21 12:05:04 +01:00
kssenii
e9dd4fdd9a Merge remote-tracking branch 'upstream/master' into named-collections-access-fix 2022-11-21 11:31:01 +01:00
Nikolai Kochetov
be1a8054c7
Merge branch 'master' into read-from-mt-in-io-pool 2022-11-21 11:27:41 +01:00
Nikolai Kochetov
4a1774c5ac Fixing build. 2022-11-21 10:27:09 +00:00
Azat Khuzhin
6f0e042914 Fix sumMap() for Nullable(Decimal()) 
Fixes: #43395
Follow-up for: #20970 (cc @kitaisreal)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-21 09:43:21 +01:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader 
Keeper support manually assigning leader
 2022-11-21 09:30:00 +01:00
Antonio Andelic
9a8d82f0fc Merge branch 'master' into fix-describe-delta-lake-hudi 2022-11-21 08:23:33 +00:00
Antonio Andelic
e78761b66c
Merge branch 'master' into keeper-prometheus 2022-11-21 09:10:10 +01:00
Antonio Andelic
af647baeb2
Merge pull request #43315 from ClickHouse/fix-merge-sample-by 
Fix `SAMPLE BY` with Merge tables
 2022-11-21 09:09:46 +01:00
Nikolay Degterinsky
f3c16ec8ed
Merge pull request #41874 from evillique/better-set-params 
Don't require manual string serialization to set query parameters
 2022-11-21 04:09:16 +01:00
Igor Nikonov
7f8d07e4b7 Fix: limit hint for DISTINCT ignores LIMIT BY 
+ test
 2022-11-20 23:06:57 +00:00
Azat Khuzhin
da8fe9c398 Fix possible heap-use-after-free in local if history file cannot be created 
ASAN report:

    Code: 586. DB::ErrnoException: Cannot create file: /src/.clickhouse_history, errno: 2, strerror: No such file or directory. (CANNOT_CREATE_FILE)
    =================================================================
    ==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x6240000208f0 at pc 0x000030d22ade bp 0x7ffff2ff3f70 sp 0x7ffff2ff3f68
    READ of size 8 at 0x6240000208f0 thread T2
        #0 0x30d22add in DB::ProcessList::insert() build_docker/../src/Interpreters/ProcessList.cpp:89:36
        #1 0x31411018 in DB::executeQueryImpl() build_docker/../src/Interpreters/executeQuery.cpp:516:60
        #2 0x3140e1ab in DB::executeQuery() build_docker/../src/Interpreters/executeQuery.cpp:1083:30
        #3 0x3364391e in DB::LocalConnection::sendQuery() build_docker/../src/Client/LocalConnection.cpp:119:21
        #4 0x3367bab0 in DB::Suggest::fetch() build_docker/../src/Client/Suggest.cpp:141:16
        #5 0x336820eb in void DB::Suggest::load<DB::LocalConnection>()::'lambda'()::operator()() const build_docker/../src/Client/Suggest.cpp:118:17

    0x6240000208f0 is located 2032 bytes inside of 7056-byte region [0x624000020100,0x624000021c90)
    freed by thread T0 here:

        #0 0xe381ef2 in operator delete(void*, unsigned long) (/wrk/clickhouse-asan+0xe381ef2) (BuildId: 6ea6d1a5d2d5a164f60f0fd8230936305bc8d9d0)
        #1 0x335509fe in DB::ClientBase::~ClientBase() build_docker/../src/Client/ClientBase.cpp:293:25
        #2 0x1f809bd5 in mainEntryClickHouseLocal(int, char**) build_docker/../programs/local/LocalServer.cpp:804:5
        #3 0xe3856ad in main build_docker/../programs/main.cpp:482:12
        #4 0x7ffff7dc0082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16

    previously allocated by thread T0 here:
        #0 0xe38128d in operator new(unsigned long) (/wrk/clickhouse-asan+0xe38128d) (BuildId: 6ea6d1a5d2d5a164f60f0fd8230936305bc8d9d0)
        #1 0x2f34a7f3 in std::__1::__unique_if<DB::ContextSharedPart>::__unique_single std::__1::make_unique[abi:v15003]<DB::ContextSharedPart>() build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:714:28
        #2 0x2f34a7f3 in DB::Context::createShared() build_docker/../src/Interpreters/Context.cpp:603:32
        #3 0x1f7f901d in DB::LocalServer::processConfig() build_docker/../programs/local/LocalServer.cpp:535:22
        #4 0x1f7f4d92 in DB::LocalServer::main() build_docker/../programs/local/LocalServer.cpp:419:5
        #5 0x3af24ffe in Poco::Util::Application::run() build_docker/../contrib/poco/Util/src/Application.cpp:334:8
        #6 0x1f809bca in mainEntryClickHouseLocal(int, char**) build_docker/../programs/local/LocalServer.cpp:803:20
        #7 0xe3856ad in main build_docker/../programs/main.cpp:482:12
        #8 0x7ffff7dc0082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16

    Thread T2 created by T0 here:
        #0 0xe32fedc in pthread_create (/wrk/clickhouse-asan+0xe32fedc) (BuildId: 6ea6d1a5d2d5a164f60f0fd8230936305bc8d9d0)
        #1 0x336806df in std::__1::__libcpp_thread_create[abi:v15003](unsigned long*, void* (*)(void*), void*) build_docker/../contrib/libcxx/include/__threading_support:376:10
        #2 0x336806df in std::__1:🧵:thread<void DB::Suggest::load<DB::LocalConnection>()::'lambda'(), void>() build_docker/../contrib/libcxx/include/thread:311:16
        #3 0x3367ff5b in void DB::Suggest::load<DB::LocalConnection>(std::__1::shared_ptr<DB::Context const>, DB::ConnectionParameters const&, int) build_docker/../src/Client/Suggest.cpp:110:22
        #4 0x3357fee9 in DB::ClientBase::runInteractive() build_docker/../src/Client/ClientBase.cpp:2066:22
        #5 0x1f7f5264 in DB::LocalServer::main() build_docker/../programs/local/LocalServer.cpp
        #6 0x3af24ffe in Poco::Util::Application::run() build_docker/../contrib/poco/Util/src/Application.cpp:334:8
        #7 0x1f809bca in mainEntryClickHouseLocal(int, char**) build_docker/../programs/local/LocalServer.cpp:803:20
        #8 0xe3856ad in main build_docker/../programs/main.cpp:482:12
        #9 0x7ffff7dc0082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 22:51:27 +01:00
Azat Khuzhin
c393af2812 Fix is_read_only property of local disks 
But it wasn't a problem before, since before it was not possible to have
readonly==true.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:35:41 +01:00
Azat Khuzhin
e6e223adc2 Check if local disk is readonly before access checks 2022-11-20 16:35:32 +01:00
Azat Khuzhin
cb60576221 Change DiskLocal::setup() signature (it always return true) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:31:37 +01:00
Azat Khuzhin
38c009214a Change the name pattern for memory disks 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
11be9b9ad1 Create disk directory before access check for local disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
dddcca5cc1 Fix deadlock in DiskRestartProxy on disk restart 
stacktrace:
    contrib/libcxx/src/condition_variable.cpp:47::std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)
    contrib/libcxx/src/shared_mutex.cpp:65::std::__1::shared_timed_mutex::lock_shared()
    src/Disks/DiskRestartProxy.cpp:229::DB::DiskRestartProxy::writeFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long, DB::WriteMode, DB::WriteSettings const&)
    src/Disks/IDisk.cpp:0::DB::IDisk::checkAccessImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
    contrib/libcxx/include/string:1499::DB::IDisk::checkAccess()
    src/Disks/IDisk.cpp:0::DB::IDisk::startup(std::__1::shared_ptr<DB::Context const>, bool)
    src/Disks/DiskRestartProxy.cpp:375::DB::DiskRestartProxy::restart(std::__1::shared_ptr<DB::Context const>)
    contrib/libcxx/include/__memory/shared_ptr.h:701::DB::InterpreterSystemQuery::restartDisk(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)
    src/Interpreters/InterpreterSystemQuery.cpp:508::DB::InterpreterSystemQuery::execute()
    src/Interpreters/executeQuery.cpp:0::DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*)
    src/Interpreters/executeQuery.cpp:1083::DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    src/Server/TCPHandler.cpp:0::DB::TCPHandler::runImpl()
    src/Server/TCPHandler.cpp:1904::DB::TCPHandler::run()
    contrib/poco/Net/src/TCPServerConnection.cpp:57::Poco::Net::TCPServerConnection::start()
    contrib/libcxx/include/__memory/unique_ptr.h:48::Poco::Net::TCPServerDispatcher::run()
    contrib/poco/Foundation/src/ThreadPool.cpp:213::Poco::PooledThread::run()
    contrib/poco/Foundation/include/Poco/SharedPtr.h:156::Poco::ThreadImpl::runnableEntry(void*)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
44f23c2568 Make disks checks only for clickhouse-server 
This will fix clickhouse-disks

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
2efd29f49d Implement access (read/read-by-offset/write/delete) check for all disks 
Previously we had such access (read/write/delete) checks only for S3 and
Azure disks (read/read-by-offset/write/delete), this patch adds check
for all disks.

Also I've added the disk name into IDisk interface, since it is required
for the error message in IDisk::checkAccess(), but I have to add
DiskEncrypted::encrypted_name due DiskEncrypted inherits from
DiskDecorator not from IDisk, and so does not have ability to set disk
name (DiskEncrypted could pass the disk name to the DiskDecorator, but
it is not used anywere, and besides this will require to duplicate the
name for each user of DiskDecorator).

Also from nwo on, skip_access_check will make sense for every disk, plus
now startup() called for each disk (before it was missed for some of
them).

And I've added skip_access_check as as a member for DiskRestartProxy,
since it calls startup() on restart().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
324b1a7658 Add server UUID for the S3 disks checks to avoid possible races 
Otherwise, if you are lucky enough, race condition is possible, and you
can get some errors because one server already removed the file while
another was trying to read it.

But it was possible only for:
- batch deletes check for "s3" disk
- and all checks for "s3_plain" disk, since this disk does not uses
  random names

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:11:45 +01:00
Kseniia Sumarokova
f0dbfbb0f4
Merge pull request #42800 from azat/disks/web-fix 
Do not suppress exceptions in web disk (and fix retries for requests from web disk)
 2022-11-20 16:07:45 +01:00
Kseniia Sumarokova
c12cfab953
Merge pull request #43343 from azat/disks/write-once 
Allow to "drop tables" from s3_plain disk (so as from web disk)
 2022-11-20 16:04:50 +01:00
Antonio Andelic
77eddd702a
Merge branch 'master' into fix-merge-sample-by 2022-11-20 14:38:53 +01:00
Alexey Milovidov
b3a06175e7 Add a comment 2022-11-20 07:47:20 +01:00
Alexander Tokmakov
854657d372 fix another issue 2022-11-19 23:14:53 +01:00
Alexander Tokmakov
09de30a51a fix overflow, add more tests 2022-11-19 18:58:40 +01:00
Alexander Tokmakov
9464c32bb8 Merge branch 'master' into revert_SingleValueDataString 2022-11-19 18:00:04 +01:00
taofengliu
66f101e451 Merge remote-tracking branch 'upstream/master' into group_by_all 2022-11-19 22:36:31 +08:00
Nikita Taranov
0b4e643c27
Add check to CompressionCodecDelta (#43255) 
* Add check to CompressionCodecDelta

* Apply suggestions from code review

* Update src/Compression/CompressionCodecDelta.cpp
 2022-11-19 14:16:14 +01:00
taofengliu
a529943d9f Merge remote-tracking branch 'upstream/master' into group_by_all 2022-11-19 21:11:23 +08:00
kssenii
3e1ec98436 Merge remote-tracking branch 'origin/named-collections-access-fix' into named-collections-access-fix 2022-11-19 13:22:09 +01:00
kssenii
ae25a90f09 Fix show access command 2022-11-19 13:21:35 +01:00
Azat Khuzhin
4f6703c972 Add is_read_only/is_write_once/is_remote/is_broken to system.disks 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:11:39 +01:00
Azat Khuzhin
c029549859 Allow to drop tables from s3_plain disk (so as from web disk) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:10:27 +01:00
Azat Khuzhin
e2726e03cc Override DiskDecorator::isReadOnly() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:10:27 +01:00
zhongyuankai
a3a3261745
Merge branch 'master' into optimize_ttl 2022-11-19 16:32:26 +08:00
zhongyuankai
ce9b4866ea
Update MergeTreeDataMergerMutator.cpp 2022-11-19 16:31:47 +08:00
Alexey Milovidov
bb0219b2ef
Merge branch 'master' into named-collections-access-fix 2022-11-19 00:11:09 +01:00
Alexander Gololobov
f004eea413 Add columns required fro defaults calculation 2022-11-18 23:42:45 +01:00
alesapin
c7d648174b
Merge pull request #43328 from ClickHouse/better-format-version-check 
Check content of `format_version` file in `MergeTreeData`
 2022-11-18 22:34:37 +01:00
Kruglov Pavel
82d6f5357c
Merge pull request #43156 from azat/dist/insert-apply-timeouts 
Apply connection timeouts settings for Distributed async INSERT from the query
 2022-11-18 21:27:57 +01:00
Nikolai Kochetov
5da1d893fd
Merge branch 'master' into read-from-mt-in-io-pool 2022-11-18 21:10:45 +01:00
Azat Khuzhin
ccde92f855 Fix incorrect UserTimeMicroseconds/SystemTimeMicroseconds accounting 
After #40732 it became possible that getrusage() (from detachQuery(),
from buildPushingToViewsChain()) will be called for incorrect thread,
and so when the difference will be calculated it will be simply garbage.

But actually the root of this problem is #25714, after which it became
possible to have multiple ThreadStatus for one thread, and this is very
tricky (sigh).

Here are some other thoughts about it:
- Make ThreadStatus nested - decided that complexity does not worth it,
  at least only for this case
- Move some members into ThreadGroupStatus - will break per-thread
  statistics (and hence query_thread_log, BTW does somebody uses it?)
- Move some members into a separate structure

But decided to fix the issue w/o any refactoring, to make easy for
backport.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-18 21:10:43 +01:00
Azat Khuzhin
2b936a59e4 Add sanity checks RUsageCounters::incrementProfileEvents() 
Can be triggered with the 02380_insert_mv_race test now in
debug/sanitizers build.
 2022-11-18 21:10:43 +01:00
Nikolai Kochetov
03c35e146b Refactor a bit. 2022-11-18 20:09:20 +00:00
serxa
8e65b799c2 fix test + add one more column 2022-11-18 20:04:11 +00:00
kssenii
98f5cc10bb Merge remote-tracking branch 'upstream/master' into named-collections-access-fix 2022-11-18 19:51:28 +01:00
Alexander Tokmakov
35f677d7d8 make read(...) compatible with buggy versions 2022-11-18 19:48:24 +01:00
Alexander Tokmakov
415eaff8d5 revert incompatible changes 2022-11-18 19:44:47 +01:00
Alexander Tokmakov
6a3b57c27d revert changes in AggregateFunctionMinMaxAny.h 2022-11-18 19:44:36 +01:00
serxa
a05a1b0133 Merge branch 'storage-system-moves' of github.com:ClickHouse/ClickHouse into storage-system-moves 2022-11-18 15:35:00 +00:00
Sergei Trifonov
77100896e3
Merge branch 'master' into storage-system-moves 2022-11-18 16:33:43 +01:00
Sergei Trifonov
35d37bc198
Merge branch 'master' into cpu-progress-protocol-fix 2022-11-18 16:30:40 +01:00
Sergei Trifonov
d05223e70b
Merge pull request #43335 from ClickHouse/revert-43306-revert-43014-disk-s3-throttler 
Revert "Revert "S3 request per second rate throttling""
 2022-11-18 16:22:30 +01:00
Antonio Andelic
25697fd87d
Merge pull request #43289 from ClickHouse/fix-another-race-rmt 
Fix race on write in `ReplicatedMergeTree`
 2022-11-18 16:09:50 +01:00
zzsmdfj
e09092c834 Merge branch 'issue_15357_MaterializeMySQL_support_drop_mulit_table' of github.com:zzsmdfj/ClickHouse into issue_15357_MaterializeMySQL_support_drop_mulit_table 2022-11-18 23:03:48 +08:00
zzsmdfj
47385d259c to MaterializeMySQL_support_drop_mulit_table-fix code-style 2022-11-18 23:03:24 +08:00
avogar
47959d6a70 Allow arguments with type Nothing in functions if/multiIf 2022-11-18 14:56:06 +00:00
alesapin
0f3eb51aed Disable compressed marks by default 2022-11-18 15:54:48 +01:00
Vitaly Baranov
a348332eab
Merge pull request #43227 from vitlibar/improve-masking-sensitive-info 
Improve masking sensitive info
 2022-11-18 15:37:50 +01:00
zzsmdfj
9239f50a87
Merge branch 'master' into issue_15357_MaterializeMySQL_support_drop_mulit_table 2022-11-18 21:33:31 +08:00
zzsmdfj
6a998917b0 to MaterializeMySQL_support_drop_mulit_table 2022-11-18 21:29:13 +08:00
zzsmdfj
91f1df6fc7 to MaterializeMySQL_support_drop_mulit_table 2022-11-18 17:50:44 +08:00
Suzy Wang
8dc99e8d64
Merge branch 'master' into s390x_ip_coding 2022-11-17 23:35:06 -05:00
xiedeyantu
c258d3ac8b fix s3 support question mark wildcard 2022-11-18 12:11:22 +08:00
Nikolay Degterinsky
e7e21adf17
Merge pull request #43207 from freedomDR/improve_unhex 
unhex support FixedString arguments
 2022-11-18 03:09:58 +01:00
Nikolay Degterinsky
a5707fcaec
Merge branch 'master' into better-set-params 2022-11-18 02:39:01 +01:00
Nikolay Degterinsky
a97afbb78b
Merge pull request #43282 from evillique/fix-identifier-cloning 
Fix cloning of ASTIdentifier
 2022-11-18 02:38:20 +01:00
Nikolay Degterinsky
b5872b8a05 Fix function parameters parsing 2022-11-18 01:16:28 +00:00
Anton Popov
9569113169
Merge pull request #43269 from CurtizJ/fix-fuzzer-bad-cast 
Fix race in `IColumn::dumpStructure`
 2022-11-18 01:31:28 +01:00
Alexey Milovidov
2c292e3866
Merge branch 'master' into ldap_fix_search_limit 2022-11-17 22:20:53 +01:00
Mikhail f. Shiryaev
f942ab1bd2
Update version to 22.12.1.1 2022-11-17 22:15:12 +01:00
Arthur Passos
414fd07bba add docs 2022-11-17 17:28:51 -03:00
Arthur Passos
dd37ca7767 add docs 2022-11-17 17:25:27 -03:00
Arthur Passos
12d3f799a5 small change 2022-11-17 17:18:54 -03:00
Arthur Passos
fcc032a31e handle both zero based and non zero based arrow offsets 2022-11-17 17:15:24 -03:00
Nikolai Kochetov
679591cf0a Try fix race. 2022-11-17 19:51:47 +00:00
Alexey Milovidov
0d211ed198
Merge pull request #43316 from ClickHouse/disable-delta-hudi-readonly 
Disable `deltaLake` and `hudi` table functions in readonly mode
 2022-11-17 19:27:43 +01:00
Azat Khuzhin
aa77beabc9 Fix possible "No column to rollback" logical error during INSERT into Buffer 
In case of concurrent INSERT and SELECT, and memory pressure it is
possible to get this error, if you are very lucky.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-17 19:12:58 +01:00
Vitaly Baranov
86e486d051 Remove unused argument hilite in formatWithSecretHidden(). 2022-11-17 18:16:33 +01:00
Vitaly Baranov
050df6ac7f Move InDepthNodeVisitor.h back to src/Interpreters. 2022-11-17 18:16:32 +01:00
Vitaly Baranov
6e0762f1fa Move function findFunctionSecretArguments() to ASTFunction.cpp 2022-11-17 18:16:28 +01:00
Azat Khuzhin
e67fc54fc9 StorageBuffer: log exception before rollback 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-17 18:10:11 +01:00
Sergei Trifonov
f2f0676bcc
Revert "Revert "S3 request per second rate throttling"" 2022-11-17 17:35:04 +01:00
Vitaly Baranov
2b9ed89721 Rename formatWithHiddenSecrets() -> formatWithSecretsHidden(). 2022-11-17 17:30:37 +01:00
serxa
1920c6267d typo 2022-11-17 16:20:21 +00:00
Alexey Milovidov
ef1278f5c4
Merge pull request #43313 from ClickHouse/pmod 
`pmod`: compatibility with Spark, better documentation
 2022-11-17 17:13:45 +01:00
serxa
c0d98aa749 do not reorder, ignore 4 readings, fix period 2022-11-17 15:53:47 +00:00
alesapin
29a08f64af Merge branch 'master' into Enable_compress_marks_and_primary_key 2022-11-17 15:53:37 +01:00
Antonio Andelic
d324e8cc3e Write to first non-readonly disk 2022-11-17 14:52:35 +00:00
Antonio Andelic
d41af8b741 Check and compare content of format_version file 2022-11-17 13:41:36 +00:00
Alexander Tokmakov
5d959ab6a0
Merge pull request #43287 from filimonov/patch-7 
Allow autoremoval of old parts if detach_not_byte_identical_parts enabled
 2022-11-17 16:11:02 +03:00
kssenii
5323c51176 Restrict access to system.named_collections 2022-11-17 13:40:03 +01:00
Nikita Taranov
7beb58b0cf
Optimize merge of uniqExact without_key (#43072) 
* impl for uniqExact

* rm unused (read|write)Text methods

* fix style

* small fixes

* impl for variadic uniqExact

* refactor

* fix style

* more agressive inlining

* disable if max_threads=1

* small improvements

* review fixes

* Revert "rm unused (read|write)Text methods"

This reverts commit a7e7480584.

* encapsulate is_able_to_parallelize_merge in Data

* encapsulate is_exact & argument_is_tuple in Data
 2022-11-17 13:19:02 +01:00
Antonio Andelic
95c1bb977d
Merge branch 'master' into fix-merge-sample-by 2022-11-17 13:16:51 +01:00
alesapin
06f395d946 Add comments 2022-11-17 13:10:14 +01:00
Vladimir C
a29c5b9e3a
Merge pull request #43232 from ClickHouse/vdimir/max_map_null_totals_bug 2022-11-17 13:10:05 +01:00
Sema Checherinda
8e21d2c777
Merge pull request #43285 from CheSema/ubsan-max-ast-depth 
check limits for an AST in select parser fuzzer
 2022-11-17 12:59:02 +01:00
Kruglov Pavel
a406557263
Merge branch 'master' into dist/insert-apply-timeouts 2022-11-17 12:51:47 +01:00
Kruglov Pavel
1b68f605a2
Merge pull request #42761 from AlfVII/fix-slow-json-extract-with-low-cardinality 
Fixed slowness in JSONExtract with LowCardinality(String) tuples
 2022-11-17 12:49:18 +01:00
Antonio Andelic
d5848d53cc Fix DESCRIBE for deltaLake and hudi table functions 2022-11-17 11:46:17 +00:00
alesapin
a79d83549f Merge branch 'master' into fix_parts_to_do_detection 2022-11-17 12:30:05 +01:00
Sema Checherinda
1ad48d091d
Merge pull request #43249 from Algunenano/i43247 
Fix ubsan in AggregateFunctionMinMaxAny::read with high sizes
 2022-11-17 12:21:05 +01:00
Nikita Taranov
11b535d443
impl (#43283) 2022-11-17 11:17:27 +01:00
Maksim Kita
b4bfc09e2d
Merge branch 'master' into vdimir/analyzer_alias_nullptr 2022-11-17 12:46:25 +03:00
Antonio Andelic
41b557f926 Disable DeltaLake and hudi table functions in readonly mode 2022-11-17 09:16:52 +00:00
Antonio Andelic
9af9ef3916 Fix SAMPLE BY with Merge tables 2022-11-17 08:56:51 +00:00
Alexey Milovidov
10d7ce9815 pmod: compatibility with Spark, better documentation 2022-11-17 05:41:03 +01:00
Anton Popov
c67f5e2600
Update gtest_column_dump_structure.cpp 2022-11-17 04:40:09 +01:00
avogar
69d4381a3e Add comments 2022-11-16 22:42:50 +00:00
avogar
e98ac04678 Fix bad cast from LowCardinality column when using short circuit function execution 2022-11-16 22:34:24 +00:00
Maksim Kita
ef7d867966
Merge pull request #43119 from kitaisreal/analyzer-aggregation-totals-rollup-crash-fix 
Analyzer aggregation totals crash fix
 2022-11-17 01:32:37 +03:00
serxa
67fef9a1f1 fix CPU load indication on client by changing order of Progress/ProfileEvents packets in server 2022-11-16 19:52:35 +00:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling" 2022-11-16 22:33:48 +03:00
Sergei Trifonov
159743edd6
Merge pull request #43014 from ClickHouse/disk-s3-throttler 
S3 request per second rate throttling
 2022-11-16 18:51:06 +01:00
Arthur Passos
ed080b8ba5 fix style 2022-11-16 13:22:23 -03:00
Anton Popov
bc19d35009 fix clang-tidy 2022-11-16 15:34:36 +00:00
flynn
9f8dcfca34
Merge branch 'master' into filesystem-function 2022-11-16 23:28:10 +08:00
Arthur Passos
e1236340b5 Flatten list type arrow chunks on parsing 2022-11-16 12:27:01 -03:00
Vitaly Baranov
b280b68333 Fix style. 2022-11-16 15:57:50 +01:00
Vasily Nemkov
0cfbf30b89
Merge branch 'master' into ldap_fix_search_limit 2022-11-16 18:13:19 +04:00
Nikolai Kochetov
17b75d3f5c Fix metrics and progress. 2022-11-16 14:10:56 +00:00
Antonio Andelic
11aba7597e Check if initialization is done on write 2022-11-16 13:33:22 +00:00
Smita Kulkarni
040f2b62d9 Enable compress marks and primary key 
Implementation:
* Set compress_marks and compress_primary_key to true by default.
* Updated getIndexExtensionFromFilesystem to only check for compressed file extension , else return default. When both are present, this function was returning uncompressed file extension giving rise to error as file got removed.
Testing:
* Updated tests where bytes size or compressed data is checked to check with compress_marks and compress_primary_key set to false.
 2022-11-16 14:23:58 +01:00
filimonov
dcb76f6bbf
Allow autoremoval of old parts if detach_not_byte_identical_parts enabled 
Allow autoremoval of old & detached parts if detach_not_byte_identical_parts enabled.

See also #28708 #37975
 2022-11-16 14:16:42 +01:00
Raúl Marín
78d4a4b5c2 Add a detailed comment about the change 2022-11-16 13:59:37 +01:00
Raúl Marín
4e7ccd1c06 Merge remote-tracking branch 'blessed/master' into revert_SingleValueDataString 2022-11-16 12:59:47 +01:00
Sema Checherinda
b13a8d478b check limits for an AST in select parser fuzzer 2022-11-16 11:58:30 +00:00
Vladimir C
2d265523bb
Update src/Analyzer/Passes/QueryAnalysisPass.cpp 2022-11-16 12:45:54 +01:00
Vladimir C
981e6e2c24
Update src/Analyzer/Passes/QueryAnalysisPass.cpp 2022-11-16 12:44:54 +01:00
alesapin
2012f4bd36 Fix use-after-free 2022-11-16 11:50:51 +01:00
Nikolay Degterinsky
584d894f48
Merge pull request #43265 from evillique/fix-parser-4 
Fix parsing of IS NULL operator
 2022-11-16 11:38:39 +01:00
Nikolay Degterinsky
122508d844 Fix cloning of ASTIdentifier 2022-11-16 10:22:28 +00:00
Maksim Kita
64b2a4d1bf
Merge pull request #43110 from bigo-sg/function_factorial 
Add function factorial
 2022-11-16 12:49:47 +03:00
Alexey Milovidov
1ec21e5890
Merge branch 'master' into group_by_all 2022-11-16 06:14:04 +01:00
Alexey Milovidov
923f2dd562
Merge branch 'master' into ldap_fix_search_limit 2022-11-16 06:13:07 +01:00
Alexey Milovidov
b4165f95a9
Merge branch 'master' into build/poco-leftovers 2022-11-16 06:09:58 +01:00
Alexey Milovidov
04687c0d69
Merge pull request #43206 from ClickHouse/miscellaneous-changes-parallel-replicas 
Miscellaneous changes
 2022-11-16 05:30:03 +01:00
Nikolay Degterinsky
b3d02a7bd0
Merge branch 'master' into better-set-params 2022-11-16 04:45:46 +01:00
李扬
6393d11dab
fix doc 2022-11-16 09:34:45 +08:00
JackyWoo
f8d71f60a2
Merge branch 'master' into keeper_manually_assign_leader 2022-11-16 09:34:40 +08:00
Anton Popov
11c28abbed fix race in IColumn::dumpStructure 2022-11-16 00:57:22 +00:00
Vitaly Baranov
4a25fcc056 Fix procesing '' as a database name in remote() 2022-11-16 01:35:11 +01:00
Vitaly Baranov
ce81166c7e Fix style. 2022-11-16 01:35:11 +01:00
Vitaly Baranov
d365570b27 Hide passwords in SHOW CREATE TABLE and system.tables. 2022-11-16 01:35:11 +01:00
Vitaly Baranov
8d72f75556 Make the password wiping an option of IAST::format(). 2022-11-16 01:35:06 +01:00
Nikolai Kochetov
1b1d23c100 Clumsy attempt to fix progress. 2022-11-15 21:23:18 +00:00
Nikolay Degterinsky
82877d0824
Merge pull request #43077 from evillique/query_parameters_inserts 
Add support for interactive parameters in INSERT VALUES queries
 2022-11-15 22:13:25 +01:00
Alexander Tokmakov
69b067b81c
Merge pull request #43244 from ClickHouse/fix-replicated-merge-tree-shared-id 
Don't throw if shared ID already created in `StorageReplicatedMergeTree`
 2022-11-15 23:31:51 +03:00
Nikolay Degterinsky
4778b5c13f Fix IS (NOT)NULL operator 2022-11-15 20:23:49 +00:00
Nikolai Kochetov
cf1afbfb84 Temporarily enable allow_asynchronous_read_from_io_pool_for_merge_tree by default to see thest results. 2022-11-15 17:47:26 +00:00
Nikolai Kochetov
29098045d8 Try read from MergeTree in background pool. 2022-11-15 17:38:37 +00:00
Vitaly Baranov
22abe9a4fb
Merge pull request #42950 from azat/disks/s3_plain/wide-part-fix 
Fixes for s3_plain disk that will allow to attach Wide parts
 2022-11-15 17:56:16 +01:00
Suzy Wang
97d76771a4 update format 2022-11-15 08:53:39 -08:00
Raúl Marín
d49b65cf1e Fix capacity check 2022-11-15 16:31:12 +01:00
Antonio Andelic
6beab87610
Merge pull request #43080 from ucasfl/hudi-udtf 
Add table function hudi and deltaLake
 2022-11-15 15:42:25 +01:00
Raúl Marín
143b67d0be Fix ubsan in AggregateFunctionMinMaxAny::read with high sizes 2022-11-15 15:40:06 +01:00
Sema Checherinda
7e73b187cc
Merge pull request #43159 from Algunenano/over-read 
Fix several buffer over-reads
 2022-11-15 15:04:24 +01:00
alesapin
a12f2d14dc Simpler check 2022-11-15 14:46:38 +01:00
avogar
2af60f34eb Restrict document size in parallel parsing, allow to read ObjectId/JS code into String column 2022-11-15 13:35:17 +00:00
Anton Popov
8c66c759d4
Merge pull request #43133 from amosbird/issue_43111 
Fix Nullable keys in hyperrectangle
 2022-11-15 14:30:13 +01:00
avogar
4d993e653a Fix build and style 2022-11-15 13:06:24 +00:00
alesapin
4ec11d0f58 Remove redundant branch 2022-11-15 13:58:53 +01:00
vdimir
d7c882951f
Fix nullptr dereference in collectScopeValidIdentifiersForTypoCorrection 2022-11-15 12:36:28 +00:00
Antonio Andelic
0e5e58bed9 Remove exception if shared ID already created 2022-11-15 12:18:24 +00:00
Antonio Andelic
96838f5be7 Merge branch 'master' into keeper-prometheus 2022-11-15 12:00:02 +00:00
Anton Popov
a901c22454
Merge pull request #43189 from ClickHouse/proj_later_update_missing_columns_bug 
Fix for missing columns bug with projections an ALTER UPDATE
 2022-11-15 12:48:22 +01:00
Kseniia Sumarokova
9fc96ab7fd
Merge branch 'master' into disks/web-fix 2022-11-15 12:16:32 +01:00
Kseniia Sumarokova
59cf5def67
Merge branch 'master' into disk-s3-throttler 2022-11-15 12:13:37 +01:00
Vladimir C
0017416069
Update src/Common/FieldVisitorSum.cpp 2022-11-15 11:55:54 +01:00
xiedeyantu
ec6698395e fix skip_unavailable_shards does not work using hdfsCluster table function 2022-11-15 13:25:15 +08:00
freedomDR
5753be7b99 remove useless head file 2022-11-15 12:33:14 +08:00
freedomDR
7fdb2f400e unhex support FixedString 2022-11-15 12:33:14 +08:00
taiyang-li
7178d80c51 Merge branch 'function_factorial' of https://github.com/bigo-sg/ClickHouse into function_factorial 2022-11-15 09:46:44 +08:00
taiyang-li
5eba20e119 modify return type from Int64 to UInt64 2022-11-15 09:46:24 +08:00
李扬
10449247fd
Merge branch 'master' into function_factorial 2022-11-15 09:34:43 +08:00
Alexey Milovidov
2c7377be92
Merge pull request #43129 from Algunenano/skip_hash_in_sanitizer 
Skip hash logging on sanitizer builds
 2022-11-15 02:16:27 +01:00
zhongyuankai
35690c1972 Merge branch 'master' into optimize_ttl 2022-11-15 08:49:37 +08:00
zhongyuankai
16f697f61d fix test 2022-11-15 08:47:51 +08:00
Alexey Milovidov
6e01b2b2a1
Merge pull request #43203 from ClickHouse/fix-typo-2 
Fix typo
 2022-11-15 01:39:34 +01:00
Alexey Milovidov
3cab3ce484
Merge pull request #43198 from FArthur-cmd/MySQL_clickhouse_connect 
add columns to information schema
 2022-11-15 01:11:29 +01:00
Nikolay Degterinsky
d48d4cf87e
Merge branch 'master' into query_parameters_inserts 2022-11-15 01:05:40 +01:00
Nikolay Degterinsky
7cb34ee459
Merge pull request #43150 from Tiaonmmn/patch-7 
Update CompressionCodecDeflateQpl.cpp
 2022-11-15 00:59:43 +01:00
alesapin
6a50a32cfc Fix parts_to_do 2022-11-15 00:53:20 +01:00
李扬
1de5bb2392
Add function canonicalRand (#43124) 
* add function canonicalRand

* add perf test

* revert rand.xml
 2022-11-15 00:27:19 +01:00
李扬
ef45889ecb
Add function pmod which return non-negative result based on modulo (#42755) 
* add function positive_modulo

* add ducument

* fix type deduction of positive_modulo

* add function positive_modulo

* add ducument

* fix type deduction of positive_modulo

* add notice

* fix typo

* fix typo

* fix bug

* fix ub error

* fix ub error
 2022-11-15 00:15:58 +01:00
alesapin
f53ada2bbd Revert "Fix bug in replication queue which can lead to premature mutation finish" 
This reverts commit 3b5424811c.
 2022-11-14 22:59:09 +01:00
filimonov
8369998286
Style 2022-11-14 22:06:22 +01:00
vdimir
28fec44ffc
Fix merging null values in AggregateFunctionSumMap 2022-11-14 20:56:36 +00:00
alesapin
3b5424811c Fix bug in replication queue which can lead to premature mutation finish 2022-11-14 21:53:16 +01:00
Vitaly Baranov
21e7d843da
Merge pull request #43097 from vitlibar/correct-error-message-after-dropping-current-user 
Make error message after dropping current user more correct.
 2022-11-14 21:13:58 +01:00
Vitaly Baranov
b8286b836d
Merge pull request #43104 from vitlibar/fix-backup-of-lazy-db 
Fix race for backup of tables in Lazy databases
 2022-11-14 21:11:50 +01:00
avogar
842d25c358 Minor improvements, better docs 2022-11-14 20:05:01 +00:00
Kruglov Pavel
386b69acae
Merge pull request #43131 from xiedeyantu/fix_s3cluster_connection_fail 
fix skip_unavailable_shards does not work using s3Cluster table function
 2022-11-14 20:48:00 +01:00
Mikhail Filimonov
19d39d7881 Add SensitiveDataMasker to exceptions messages 2022-11-14 19:06:17 +01:00
Vitaly Baranov
8e99f5fea3 Move maskSensitiveInfoInQueryForLogging() to src/Parsers/ 2022-11-14 18:55:19 +01:00
Vitaly Baranov
e18c97faf7 Remove dependencies maskSensitiveInfo() from Context. 2022-11-14 18:55:19 +01:00
Kruglov Pavel
831abbaff9
Fix build 2022-11-14 17:29:30 +01:00
zvonand
0b780346cd fix typos 2022-11-14 19:26:15 +03:00