thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-25 09:02:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
101,828 Commits 9,170 Branches 1,781 Tags 2.1 GiB
d57a8b502d
Commit Graph

101828 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Azat Khuzhin
3e42ffd372 tests: fix hdfs disks (chicken and an egg problem with current cluster.py) 
Right now cluster.py first creates HDFS and then creates clickhouse in
one API call, so you cannot interract and add missing mkdirs for the
clickhouse, fix this by using root dir where it is possible.
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
0678fba3d1 tests: fix test_disk_types (by adding mkdir for HDFS) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
11be9b9ad1 Create disk directory before access check for local disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
dddcca5cc1 Fix deadlock in DiskRestartProxy on disk restart 
stacktrace:
    contrib/libcxx/src/condition_variable.cpp:47::std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&)
    contrib/libcxx/src/shared_mutex.cpp:65::std::__1::shared_timed_mutex::lock_shared()
    src/Disks/DiskRestartProxy.cpp:229::DB::DiskRestartProxy::writeFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, unsigned long, DB::WriteMode, DB::WriteSettings const&)
    src/Disks/IDisk.cpp:0::DB::IDisk::checkAccessImpl(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
    contrib/libcxx/include/string:1499::DB::IDisk::checkAccess()
    src/Disks/IDisk.cpp:0::DB::IDisk::startup(std::__1::shared_ptr<DB::Context const>, bool)
    src/Disks/DiskRestartProxy.cpp:375::DB::DiskRestartProxy::restart(std::__1::shared_ptr<DB::Context const>)
    contrib/libcxx/include/__memory/shared_ptr.h:701::DB::InterpreterSystemQuery::restartDisk(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)
    src/Interpreters/InterpreterSystemQuery.cpp:508::DB::InterpreterSystemQuery::execute()
    src/Interpreters/executeQuery.cpp:0::DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*)
    src/Interpreters/executeQuery.cpp:1083::DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    src/Server/TCPHandler.cpp:0::DB::TCPHandler::runImpl()
    src/Server/TCPHandler.cpp:1904::DB::TCPHandler::run()
    contrib/poco/Net/src/TCPServerConnection.cpp:57::Poco::Net::TCPServerConnection::start()
    contrib/libcxx/include/__memory/unique_ptr.h:48::Poco::Net::TCPServerDispatcher::run()
    contrib/poco/Foundation/src/ThreadPool.cpp:213::Poco::PooledThread::run()
    contrib/poco/Foundation/include/Poco/SharedPtr.h:156::Poco::ThreadImpl::runnableEntry(void*)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
44f23c2568 Make disks checks only for clickhouse-server 
This will fix clickhouse-disks

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
2efd29f49d Implement access (read/read-by-offset/write/delete) check for all disks 
Previously we had such access (read/write/delete) checks only for S3 and
Azure disks (read/read-by-offset/write/delete), this patch adds check
for all disks.

Also I've added the disk name into IDisk interface, since it is required
for the error message in IDisk::checkAccess(), but I have to add
DiskEncrypted::encrypted_name due DiskEncrypted inherits from
DiskDecorator not from IDisk, and so does not have ability to set disk
name (DiskEncrypted could pass the disk name to the DiskDecorator, but
it is not used anywere, and besides this will require to duplicate the
name for each user of DiskDecorator).

Also from nwo on, skip_access_check will make sense for every disk, plus
now startup() called for each disk (before it was missed for some of
them).

And I've added skip_access_check as as a member for DiskRestartProxy,
since it calls startup() on restart().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
324b1a7658 Add server UUID for the S3 disks checks to avoid possible races 
Otherwise, if you are lucky enough, race condition is possible, and you
can get some errors because one server already removed the file while
another was trying to read it.

But it was possible only for:
- batch deletes check for "s3" disk
- and all checks for "s3_plain" disk, since this disk does not uses
  random names

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:11:45 +01:00
Kseniia Sumarokova
f0dbfbb0f4
Merge pull request #42800 from azat/disks/web-fix 
Do not suppress exceptions in web disk (and fix retries for requests from web disk)
 2022-11-20 16:07:45 +01:00
Kseniia Sumarokova
c12cfab953
Merge pull request #43343 from azat/disks/write-once 
Allow to "drop tables" from s3_plain disk (so as from web disk)
 2022-11-20 16:04:50 +01:00
Antonio Andelic
77eddd702a
Merge branch 'master' into fix-merge-sample-by 2022-11-20 14:38:53 +01:00
Alexey Milovidov
b3a06175e7 Add a comment 2022-11-20 07:47:20 +01:00
Alexander Tokmakov
854657d372 fix another issue 2022-11-19 23:14:53 +01:00
Alexander Tokmakov
a00f7d142d fix flaky test 2022-11-19 21:02:48 +01:00
Alexander Tokmakov
09de30a51a fix overflow, add more tests 2022-11-19 18:58:40 +01:00
Alexander Tokmakov
9464c32bb8 Merge branch 'master' into revert_SingleValueDataString 2022-11-19 18:00:04 +01:00
taofengliu
66f101e451 Merge remote-tracking branch 'upstream/master' into group_by_all 2022-11-19 22:36:31 +08:00
Nikita Taranov
0b4e643c27
Add check to CompressionCodecDelta (#43255) 
* Add check to CompressionCodecDelta

* Apply suggestions from code review

* Update src/Compression/CompressionCodecDelta.cpp
 2022-11-19 14:16:14 +01:00
taofengliu
a529943d9f Merge remote-tracking branch 'upstream/master' into group_by_all 2022-11-19 21:11:23 +08:00
alesapin
93a4950dae
Merge pull request #43222 from zhongyuankai/optimize_ttl 
Priority is given to deleting completely expired Parts
 2022-11-19 13:51:51 +01:00
alesapin
3d497cb9d4
Merge pull request #43373 from ClickHouse/trying_to_split_checks 
Splitting checks in CI more
 2022-11-19 13:42:15 +01:00
kssenii
3e1ec98436 Merge remote-tracking branch 'origin/named-collections-access-fix' into named-collections-access-fix 2022-11-19 13:22:09 +01:00
kssenii
ae25a90f09 Fix show access command 2022-11-19 13:21:35 +01:00
robot-clickhouse
4ee5d8bd43 Update version_date.tsv and changelogs after v22.8.9.24-lts 2022-11-19 11:51:14 +00:00
Azat Khuzhin
4f6703c972 Add is_read_only/is_write_once/is_remote/is_broken to system.disks 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:11:39 +01:00
Azat Khuzhin
c029549859 Allow to drop tables from s3_plain disk (so as from web disk) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:10:27 +01:00
Azat Khuzhin
e2726e03cc Override DiskDecorator::isReadOnly() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:10:27 +01:00
zhongyuankai
a3a3261745
Merge branch 'master' into optimize_ttl 2022-11-19 16:32:26 +08:00
zhongyuankai
ce9b4866ea
Update MergeTreeDataMergerMutator.cpp 2022-11-19 16:31:47 +08:00
zzsmdfj
e177927efb to 15357_MaterializeMySQL_support_drop_mulit_table-fix code-style 2022-11-19 12:05:27 +08:00
Alexey Milovidov
bb0219b2ef
Merge branch 'master' into named-collections-access-fix 2022-11-19 00:11:09 +01:00
Alexander Gololobov
f004eea413 Add columns required fro defaults calculation 2022-11-18 23:42:45 +01:00
Alexander Gololobov
4af8ef381b Test with default value used in row level policy 2022-11-18 23:36:20 +01:00
Alexey Milovidov
b4664cd525
Merge pull request #43365 from ClickHouse/update-security-on-tag 
Update SECURITY.md on new stable tags
 2022-11-18 22:39:17 +01:00
Alexey Milovidov
ec1c30d63e
Merge pull request #43375 from vitaliyf/docs-zookeeper-tips 
docs: Clarified ZooKeeper tips comments.
 2022-11-18 22:36:36 +01:00
alesapin
c7d648174b
Merge pull request #43328 from ClickHouse/better-format-version-check 
Check content of `format_version` file in `MergeTreeData`
 2022-11-18 22:34:37 +01:00
Kruglov Pavel
82d6f5357c
Merge pull request #43156 from azat/dist/insert-apply-timeouts 
Apply connection timeouts settings for Distributed async INSERT from the query
 2022-11-18 21:27:57 +01:00
Azat Khuzhin
ccde92f855 Fix incorrect UserTimeMicroseconds/SystemTimeMicroseconds accounting 
After #40732 it became possible that getrusage() (from detachQuery(),
from buildPushingToViewsChain()) will be called for incorrect thread,
and so when the difference will be calculated it will be simply garbage.

But actually the root of this problem is #25714, after which it became
possible to have multiple ThreadStatus for one thread, and this is very
tricky (sigh).

Here are some other thoughts about it:
- Make ThreadStatus nested - decided that complexity does not worth it,
  at least only for this case
- Move some members into ThreadGroupStatus - will break per-thread
  statistics (and hence query_thread_log, BTW does somebody uses it?)
- Move some members into a separate structure

But decided to fix the issue w/o any refactoring, to make easy for
backport.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-18 21:10:43 +01:00
Azat Khuzhin
2b936a59e4 Add sanity checks RUsageCounters::incrementProfileEvents() 
Can be triggered with the 02380_insert_mv_race test now in
debug/sanitizers build.
 2022-11-18 21:10:43 +01:00
serxa
8e65b799c2 fix test + add one more column 2022-11-18 20:04:11 +00:00
kssenii
98f5cc10bb Merge remote-tracking branch 'upstream/master' into named-collections-access-fix 2022-11-18 19:51:28 +01:00
Alexander Tokmakov
35f677d7d8 make read(...) compatible with buggy versions 2022-11-18 19:48:24 +01:00
Alexander Tokmakov
415eaff8d5 revert incompatible changes 2022-11-18 19:44:47 +01:00
Alexander Tokmakov
6a3b57c27d revert changes in AggregateFunctionMinMaxAny.h 2022-11-18 19:44:36 +01:00
alesapin
f12309ac8e Fix finish check 2022-11-18 19:21:37 +01:00
Vitaliy
9e38b14fc1
docs: Clarified ZooKeeper tips comments. 2022-11-18 12:08:07 -05:00
alesapin
2dbf07f58a Splitting checks 2022-11-18 17:38:17 +01:00
serxa
a05a1b0133 Merge branch 'storage-system-moves' of github.com:ClickHouse/ClickHouse into storage-system-moves 2022-11-18 15:35:00 +00:00
Sergei Trifonov
77100896e3
Merge branch 'master' into storage-system-moves 2022-11-18 16:33:43 +01:00
Sergei Trifonov
35d37bc198
Merge branch 'master' into cpu-progress-protocol-fix 2022-11-18 16:30:40 +01:00
Sergei Trifonov
d05223e70b
Merge pull request #43335 from ClickHouse/revert-43306-revert-43014-disk-s3-throttler 
Revert "Revert "S3 request per second rate throttling""
 2022-11-18 16:22:30 +01:00