thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-20 14:42:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
101,582 Commits 9,150 Branches 1,778 Tags 2 GiB
31bffdd735
Commit Graph

101582 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexander Gololobov
31bffdd735
Merge pull request #43460 from ClickHouse/smaller_buffer_for_small_files 
Use smaller buffer for small files
 2022-11-22 14:00:22 +01:00
Anton Popov
2572202a16
Merge pull request #43461 from CurtizJ/merging-42064 
Merging #42064
 2022-11-22 13:55:01 +01:00
Mikhail f. Shiryaev
8ce5d90c91
Merge pull request #43467 from ClickHouse/use-bots-tokens 
Use all parameters with prefixes from ssm
 2022-11-22 13:46:27 +01:00
Mikhail f. Shiryaev
7048f36fc7
Save unnecessary API requests 2022-11-22 13:37:08 +01:00
Dmitry Novik
e67bda65a0
Merge pull request #43453 from ClickHouse/fix-hard-limit-in-test 
Do not reuse jemalloc memory in test_global_overcommit
 2022-11-22 13:24:34 +01:00
Mikhail f. Shiryaev
fb95ca43d8
Use all parameters with prefixes from ssm 2022-11-22 12:43:17 +01:00
Alexander Tokmakov
7417d8b7c0
Merge pull request #43038 from Algunenano/revert_SingleValueDataString 
Fix regression on SingleValueDataString
 2022-11-22 14:28:30 +03:00
Alexander Tokmakov
cab33c02af
Merge branch 'master' into revert_SingleValueDataString 2022-11-22 14:27:58 +03:00
Antonio Andelic
843401c3a3
Merge pull request #43323 from ClickHouse/fix-describe-delta-lake-hudi 
Fix `DESCRIBE` for `deltaLake` and `hudi` table functions
 2022-11-22 09:09:39 +01:00
Anton Popov
4502fc0e81 fix flaky tests 2022-11-22 01:03:21 +00:00
Anton Popov
0479d8d9f5
Merge pull request #43387 from ClickHouse/fix_default_value_used_in_row_level_filter 
Fix default value used in row level filter
 2022-11-22 01:46:45 +01:00
Vitaly Baranov
6c2b53f0ae
Merge pull request #42940 from filimonov/exceptions_with_masker 
Add SensitiveDataMasker to exceptions messages
 2022-11-21 23:20:15 +01:00
Alexander Gololobov
5afd2a1add Pass file size to better choose buffer size 2022-11-21 22:36:45 +01:00
Alexander Gololobov
86a0dd010f Use read_hint and file_size for choosing buffer size 2022-11-21 22:36:28 +01:00
Anton Popov
5532e3db95 Merge branch 'filesystem-function' of https://github.com/ucasfl/ClickHouse into merging-42064 2022-11-21 18:23:11 +00:00
Kruglov Pavel
581e57be9f
Merge pull request #43236 from xiedeyantu/fix_hdfscluster_connection_fail 
fix skip_unavailable_shards does not work using hdfsCluster table function
 2022-11-21 19:22:16 +01:00
Igor Nikonov
47ce029cbf
Merge pull request #43326 from ClickHouse/igor/test_read_only_fast_fix 
Fix: make test_read_only_table more stable
 2022-11-21 18:25:14 +01:00
Dmitry Novik
b17dc24a94 Do not reuse jemalloc memory in test 2022-11-21 16:10:47 +00:00
Nikolay Degterinsky
65b5523bdb
Merge pull request #43350 from evillique/fix-parser-5 
Fix function parameters parsing
 2022-11-21 16:23:02 +01:00
Kruglov Pavel
406c50277b
Merge pull request #42779 from Avogar/add-format-func-doc 
Add documentation for 'format' table function
 2022-11-21 15:16:26 +01:00
Raúl Marín
35e9e169a1
Merge branch 'master' into revert_SingleValueDataString 2022-11-21 15:11:13 +01:00
Alexey Milovidov
bb5756bb11
Merge pull request #43403 from ClickHouse/table-function-add-comment 
Add a comment
 2022-11-21 15:06:20 +01:00
Kseniia Sumarokova
a59dac6337
Merge pull request #43143 from azat/disks/s3-check-fix 
Add server UUID for disks access checks (read/read-by-offset/write/delete) to avoid possible races
 2022-11-21 15:03:46 +01:00
Raúl Marín
a52bdca989 Detect the other corrupted state too 2022-11-21 13:33:16 +01:00
Alexey Milovidov
d0560fda00
Merge pull request #43393 from ClickHouse/auto/v22.8.9.24-lts 
Update version_date.tsv and changelogs after v22.8.9.24-lts
 2022-11-21 13:09:16 +01:00
Alexander Tokmakov
fffb711097 add tests 2022-11-21 12:54:21 +01:00
alesapin
f4ef20b5e4
Merge pull request #43295 from ClickHouse/cancel-lambda-api-url 
Cancel lambda api url
 2022-11-21 12:03:37 +01:00
Han Fei
874a1a9d83
Merge pull request #43408 from azat/tests/fix-01926_order_by_desc_limit 
Fix flaky 01926_order_by_desc_limit
 2022-11-21 11:58:33 +01:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader 
Keeper support manually assigning leader
 2022-11-21 09:30:00 +01:00
Antonio Andelic
9a8d82f0fc Merge branch 'master' into fix-describe-delta-lake-hudi 2022-11-21 08:23:33 +00:00
Antonio Andelic
af647baeb2
Merge pull request #43315 from ClickHouse/fix-merge-sample-by 
Fix `SAMPLE BY` with Merge tables
 2022-11-21 09:09:46 +01:00
Nikolay Degterinsky
f3c16ec8ed
Merge pull request #41874 from evillique/better-set-params 
Don't require manual string serialization to set query parameters
 2022-11-21 04:09:16 +01:00
Azat Khuzhin
1831aa45d9 Fix flaky 01926_order_by_desc_limit 
CI captures [1] too long logs flush:

    $ clickhouse-local --file query_log.tsv.gz --input-format TabSeparatedWithNamesAndTypes -q "select event_time, query from table where Settings['log_comment'] = '01926_order_by_desc_limit.sql' and type = 'QueryFinish' and (lower(query) LIKE lower('SELECT s FROM order_by_desc ORDER BY u%') or query like '%query_log%') format Vertical"
    Row 1:
    ──────
    event_time: 2022-11-20 12:46:42
    query:      SELECT s FROM order_by_desc ORDER BY u DESC LIMIT 10 FORMAT Null
    SETTINGS max_memory_usage = '400M';

    Row 2:
    ──────
    event_time: 2022-11-20 12:46:47
    query:      SELECT s FROM order_by_desc ORDER BY u LIMIT 10 FORMAT Null
    SETTINGS max_memory_usage = '400M';

    Row 3:
    ──────
    event_time: 2022-11-20 12:46:54
    query:      SELECT read_rows < 110000 FROM system.query_log
    WHERE type = 'QueryFinish' AND current_database = currentDatabase()
    AND event_time > now() - INTERVAL 10 SECOND
    AND lower(query) LIKE lower('SELECT s FROM order_by_desc ORDER BY u%');

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/43143/c393af2812cb56a60a99d48c15c0443d2da98a8b/stateless_tests__tsan__[2/5].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 21:03:22 +01:00
Azat Khuzhin
c393af2812 Fix is_read_only property of local disks 
But it wasn't a problem before, since before it was not possible to have
readonly==true.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:35:41 +01:00
Azat Khuzhin
e6e223adc2 Check if local disk is readonly before access checks 2022-11-20 16:35:32 +01:00
Azat Khuzhin
cb60576221 Change DiskLocal::setup() signature (it always return true) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:31:37 +01:00
Azat Khuzhin
38c009214a Change the name pattern for memory disks 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
3e42ffd372 tests: fix hdfs disks (chicken and an egg problem with current cluster.py) 
Right now cluster.py first creates HDFS and then creates clickhouse in
one API call, so you cannot interract and add missing mkdirs for the
clickhouse, fix this by using root dir where it is possible.
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
0678fba3d1 tests: fix test_disk_types (by adding mkdir for HDFS) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
11be9b9ad1 Create disk directory before access check for local disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
dddcca5cc1 Fix deadlock in DiskRestartProxy on disk restart 
stacktrace:
    contrib/libcxx/src/condition_variable.cpp:47::std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)
    contrib/libcxx/src/shared_mutex.cpp:65::std::__1::shared_timed_mutex::lock_shared()
    src/Disks/DiskRestartProxy.cpp:229::DB::DiskRestartProxy::writeFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long, DB::WriteMode, DB::WriteSettings const&)
    src/Disks/IDisk.cpp:0::DB::IDisk::checkAccessImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
    contrib/libcxx/include/string:1499::DB::IDisk::checkAccess()
    src/Disks/IDisk.cpp:0::DB::IDisk::startup(std::__1::shared_ptr<DB::Context const>, bool)
    src/Disks/DiskRestartProxy.cpp:375::DB::DiskRestartProxy::restart(std::__1::shared_ptr<DB::Context const>)
    contrib/libcxx/include/__memory/shared_ptr.h:701::DB::InterpreterSystemQuery::restartDisk(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)
    src/Interpreters/InterpreterSystemQuery.cpp:508::DB::InterpreterSystemQuery::execute()
    src/Interpreters/executeQuery.cpp:0::DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*)
    src/Interpreters/executeQuery.cpp:1083::DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    src/Server/TCPHandler.cpp:0::DB::TCPHandler::runImpl()
    src/Server/TCPHandler.cpp:1904::DB::TCPHandler::run()
    contrib/poco/Net/src/TCPServerConnection.cpp:57::Poco::Net::TCPServerConnection::start()
    contrib/libcxx/include/__memory/unique_ptr.h:48::Poco::Net::TCPServerDispatcher::run()
    contrib/poco/Foundation/src/ThreadPool.cpp:213::Poco::PooledThread::run()
    contrib/poco/Foundation/include/Poco/SharedPtr.h:156::Poco::ThreadImpl::runnableEntry(void*)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
44f23c2568 Make disks checks only for clickhouse-server 
This will fix clickhouse-disks

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
2efd29f49d Implement access (read/read-by-offset/write/delete) check for all disks 
Previously we had such access (read/write/delete) checks only for S3 and
Azure disks (read/read-by-offset/write/delete), this patch adds check
for all disks.

Also I've added the disk name into IDisk interface, since it is required
for the error message in IDisk::checkAccess(), but I have to add
DiskEncrypted::encrypted_name due DiskEncrypted inherits from
DiskDecorator not from IDisk, and so does not have ability to set disk
name (DiskEncrypted could pass the disk name to the DiskDecorator, but
it is not used anywere, and besides this will require to duplicate the
name for each user of DiskDecorator).

Also from nwo on, skip_access_check will make sense for every disk, plus
now startup() called for each disk (before it was missed for some of
them).

And I've added skip_access_check as as a member for DiskRestartProxy,
since it calls startup() on restart().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
324b1a7658 Add server UUID for the S3 disks checks to avoid possible races 
Otherwise, if you are lucky enough, race condition is possible, and you
can get some errors because one server already removed the file while
another was trying to read it.

But it was possible only for:
- batch deletes check for "s3" disk
- and all checks for "s3_plain" disk, since this disk does not uses
  random names

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:11:45 +01:00
Kseniia Sumarokova
f0dbfbb0f4
Merge pull request #42800 from azat/disks/web-fix 
Do not suppress exceptions in web disk (and fix retries for requests from web disk)
 2022-11-20 16:07:45 +01:00
Kseniia Sumarokova
c12cfab953
Merge pull request #43343 from azat/disks/write-once 
Allow to "drop tables" from s3_plain disk (so as from web disk)
 2022-11-20 16:04:50 +01:00
Antonio Andelic
77eddd702a
Merge branch 'master' into fix-merge-sample-by 2022-11-20 14:38:53 +01:00
Alexey Milovidov
b3a06175e7 Add a comment 2022-11-20 07:47:20 +01:00
Alexander Tokmakov
854657d372 fix another issue 2022-11-19 23:14:53 +01:00
Alexander Tokmakov
a00f7d142d fix flaky test 2022-11-19 21:02:48 +01:00