thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-03 04:52:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
93,602 Commits 9,181 Branches 1,782 Tags 2.2 GiB
04fd72cdef
Commit Graph

8780 Commits

Author SHA1 Message Date
kssenii
9f1d6814fc Fix 2022-07-15 18:01:54 +02:00
alesapin
5cf4bd50de
Merge pull request #39214 from azat/s3-fix-fetching-memory-part 
Fix fetch of in-memory part with allow_remote_fs_zero_copy_replication
 2022-07-15 17:52:05 +02:00
Alexander Tokmakov
5ef7f5f494
Merge pull request #39229 from azat/fix-noexcept-scope 
[RFC] Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception)
 2022-07-15 18:14:56 +03:00
Vitaly Baranov
c7cef91d4d Two ways to provide access to system.users: "GRANT SHOW USERS ON *.*" and "GRANT SELECT ON system.users" 2022-07-15 15:44:29 +02:00
Vitaly Baranov
fbb2e14d54 Add new table function viewIfPermitted(). 2022-07-15 15:44:29 +02:00
Anton Popov
76eec62469
Merge branch 'master' into randomize-more-settings 2022-07-15 14:12:49 +02:00
Kruglov Pavel
a944a92d4c
Merge pull request #39224 from Avogar/string-view-by-value 
Pass const std::string_view by value, not by reference
 2022-07-15 13:05:56 +02:00
rnbondarenko
a7fe1cf5f0
removed default queue arguments 
declare queue without default arguments x-max-length and x-overflow.
 2022-07-15 13:46:52 +03:00
Alexander Tokmakov
11ea455632
Merge pull request #39245 from ClickHouse/tavplubix-patch-1 
Remove incorrect assertion
 2022-07-15 13:26:14 +03:00
Sergei Trifonov
62f975da60
Merge pull request #39095 from ClickHouse/remote-io-bandwidth-limits 
add max_remote_{read,write}_network_bandwidth_for_server settings
 2022-07-14 23:14:53 +02:00
Alexander Tokmakov
435d98738d
Update ReplicatedMergeTreeQueue.cpp 2022-07-14 22:41:01 +03:00
avogar
9291d33080 Pass const std::string_view & by value, not by reference 2022-07-14 16:11:57 +00:00
Yakov Olkhovskiy
1f09303d54
Merge pull request #39181 from azat/part_log-merge_algorithm 
Add merge_algorithm to system.part_log
 2022-07-14 11:43:04 -04:00
Sergei Trifonov
b2317ff710 renaming to be consistent 2022-07-14 17:33:22 +02:00
Anton Popov
d4cb2184c2
Merge branch 'master' into randomize-more-settings 2022-07-14 17:01:07 +02:00
Nikolai Kochetov
b24eeedf53
Merge branch 'master' into nullable-partition-fix1 2022-07-14 16:41:38 +02:00
Yakov Olkhovskiy
9ecf19c883
Merge pull request #39212 from azat/stack_trace-filter-v2 
Optimize accesses to system.stack_trace (filter by name before sending signal)
 2022-07-14 10:33:26 -04:00
Azat Khuzhin
b20407fab9 Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception) 
Current implementation of NOEXCEPT_SCOPE will not work, you cannot
rethrow exception outside the catch block, this will simply terminate
(via std::terminate) the program.
In other words NOEXCEPT_SCOPE macro will simply call std::terminate on
exception and will lost original exception.

But if NOEXCEPT_SCOPE will accept the code that should be runned w/o
exceptions, then it can catch exception and log it, rewrite it in this
way.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 17:16:18 +03:00
Anton Popov
51533adee8 fix build 2022-07-14 11:33:23 +00:00
Sergei Trifonov
8a693469f0 Merge branch 'master' into remote-io-bandwidth-limits 2022-07-14 12:38:35 +02:00
Kseniia Sumarokova
530dac6487
Merge pull request #37213 from aaapetrenko/master 
add TableFunctionMongoDB and tests
 2022-07-14 12:09:07 +02:00
Azat Khuzhin
531e3b13ce Optimize accesses to system.stack_trace (filter by name before sending signal) 
Because of failed rebase/push this part of the patch was lost, so
resending separatelly (sigh).

Follow-up for: #39177 (cc @yakov-olkhovskiy)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 11:49:34 +03:00
Azat Khuzhin
3e77b80b88 Fix fetch of in-memory part with allow_remote_fs_zero_copy_replication 
CI founds the following error during trying to fetch in-memory part [1]:

    2022.07.13 08:06:54.231033 [ 141093 ] {} <Error> InterserverIOHTTPHandler: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/disks/s3/store/886/88635b40-e4e3-4fe6-a0a0-1b6755463358/all_0_0_0/data.bin, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

<details>

<summary>stacktrace</summary>

    2022.07.13 08:06:54.101825 [ 189342 ] {8dbd11b3-f38a-4d5d-9ded-148987adb71d} <Debug> executeQuery: (from [::1]:54570) (comment: 01643_replicated_merge_tree_fsync_smoke.sql) select 'memory in_memory_parts_insert_sync'; (stage: Complete)
    2022.07.13 08:06:54.131309 [ 691 ] {} <Debug> test_26u6kx.rep_fsync_r2 (39c3823c-22e5-4c05-9dec-cdffd8872c40): Fetching part all_0_0_0 from /clickhouse/tables/test_26u6kx/rep_fsync/replicas/r1
    2022.07.13 08:06:54.231033 [ 141093 ] {} <Error> InterserverIOHTTPHandler: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/disks/s3/store/886/88635b40-e4e3-4fe6-a0a0-1b6755463358/all_0_0_0/data.bin, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):
    0. DB::Exception::Exception() @ 0xba0191a in /usr/bin/clickhouse
    1. DB::throwFromErrnoWithPath() @ 0xba029ca in /usr/bin/clickhouse
    2. DB::OpenedFile::open() const @ 0x156e7fb0 in /usr/bin/clickhouse
    3. DB::OpenedFile::getFD() const @ 0x156e8003 in /usr/bin/clickhouse
    4. DB::ReadBufferFromFilePReadWithDescriptorsCache::ReadBufferFromFilePReadWithDescriptorsCache() @ 0x156e5f23 in /usr/bin/clickhouse
    5. ? @ 0x156e53f0 in /usr/bin/clickhouse
    6. DB::createReadBufferFromFileBase() @ 0x156e52b5 in /usr/bin/clickhouse
    7. DB::DiskLocal::readFile() const @ 0x15e45ea8 in /usr/bin/clickhouse
    8. DB::MetadataStorageFromDisk::readFileToString() const @ 0x15e6ab8b in /usr/bin/clickhouse
    9. DB::MetadataStorageFromDisk::readMetadataUnlocked() const @ 0x15e6cdeb in /usr/bin/clickhouse
    10. DB::MetadataStorageFromDisk::getSerializedMetadata() const @ 0x15e6cfc4 in /usr/bin/clickhouse
    11. DB::DiskObjectStorage::getSerializedMetadata() const @ 0x15e19e2e in /usr/bin/clickhouse
    12. DB::DiskDecorator::getSerializedMetadata() const @ 0x15e54ed1 in /usr/bin/clickhouse
    13. DB::DiskDecorator::getSerializedMetadata() const @ 0x15e54ed1 in /usr/bin/clickhouse
    14. DB::DataPartsExchange::Service::sendPartFromDiskRemoteMeta() @ 0x1700bb9e in /usr/bin/clickhouse
    15. DB::DataPartsExchange::Service::processQuery(DB::HTMLForm const&, DB::ReadBuffer&, DB::WriteBuffer&, DB::HTTPServerResponse&) @ 0x1700a649 in /usr/bin/clickhouse
    16. DB::InterserverIOHTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::InterserverIOHTTPHandler::Output&) @ 0x17433c53 in /usr/bin/clickhouse
    17. DB::InterserverIOHTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x174344f1 in /usr/bin/clickhouse
    18. DB::HTTPServerConnection::run() @ 0x1768714d in /usr/bin/clickhouse
    19. Poco::Net::TCPServerConnection::start() @ 0x1a398093 in /usr/bin/clickhouse
    20. Poco::Net::TCPServerDispatcher::run() @ 0x1a399411 in /usr/bin/clickhouse
    21. Poco::PooledThread::run() @ 0x1a54b7bb in /usr/bin/clickhouse
    22. Poco::ThreadImpl::runnableEntry(void*) @ 0x1a548ec0 in /usr/bin/clickhouse
    23. ? @ 0x7fdf1c204609 in ?
    24. clone @ 0x7fdf1c129133 in ?
     (version 22.7.1.1781 (official build))

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/8b6e31cc615ca52c80724b6e5097777cb9514f07/stateless_tests__release__s3_storage__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 11:43:35 +03:00
Yakov Olkhovskiy
d6a09acb38
Merge pull request #39177 from azat/stack_trace-filter 
Optimize accesses to system.stack_trace
 2022-07-14 01:47:14 -04:00
Anton Popov
089009c996 fix reading from StorageLog with mmap 2022-07-13 20:36:18 +00:00
Sergei Trifonov
15ab3bc99f use context->getWriteSettings() 2022-07-13 19:48:57 +02:00
Sergei Trifonov
af4a5a5f3a fix conflicts 2022-07-13 19:44:29 +02:00
Sergei Trifonov
f854507729 merge master and resolve conflicts 2022-07-13 19:07:28 +02:00
Kseniia Sumarokova
71aba5b4c6
Merge pull request #38860 from kssenii/object-storages-refactor 
Refactoring of code around object storages, added LocalObjectStorage (extracted this diff from PR #36171)
 2022-07-13 16:51:56 +02:00
Azat Khuzhin
3720a3843c Add merge_algorithm to system.part_log 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 17:16:25 +03:00
Anton Popov
54a0985c42 randomize one more setting 2022-07-13 12:50:12 +00:00
Azat Khuzhin
8122c1b07f Optimize accesses to system.stack_trace 
- filter by thread_id
- filter by thread_name
- filter by requested columns (query_id/trace, thread_name)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 15:17:09 +03:00
kssenii
83a09fd961 Merge remote-tracking branch 'petrenko/master' into petrenko-master 2022-07-13 11:28:52 +02:00
kssenii
ae324622d3 Add one more test 2022-07-13 11:27:43 +02:00
Amos Bird
0795cdd36a
Slightly better interface of waitForMutation 2022-07-13 12:34:12 +08:00
Dmitry Novik
5f65b45269
Merge branch 'master' into group-by-use-nulls 2022-07-12 22:36:04 +02:00
kssenii
6f29eada80 Merge master 2022-07-12 19:28:18 +02:00
Kseniia Sumarokova
0fb35b773f
Update StorageMongoDB.cpp 2022-07-12 18:37:19 +02:00
Kseniia Sumarokova
5468a2fde9
Update StorageMongoDB.cpp 2022-07-12 17:20:20 +02:00
Kseniia Sumarokova
869da0d923
Update StorageMongoDB.cpp 2022-07-12 17:19:49 +02:00
kssenii
f46d641440 Fix style check and black check 
t:wq# Date:      Tue Jul 12 14:02:09 2022 +0200
 2022-07-12 16:29:11 +02:00
Nikolai Kochetov
8efbe6d44d
Merge pull request #39119 from ClickHouse/removed-projection-and-old-part 
Properly remove projection from part in case it was removed from table metadata.
 2022-07-12 16:24:30 +02:00
Robert Schulze
69de9ee0e8
Merge pull request #39108 from ClickHouse/prefix_cmake_commands 
Prefix overridden add_executable() command with "clickhouse_"
 2022-07-12 15:05:48 +02:00
Alexander Tokmakov
c99fe6e6c8
Merge branch 'master' into removed-projection-and-old-part 2022-07-12 14:00:55 +03:00
Alexander Tokmakov
b35ad0dbdc
Merge pull request #39091 from ClickHouse/tavplubix-patch-1 
Follow-up to #39036
 2022-07-12 13:47:37 +03:00
Nikolai Kochetov
945b5d4b3f
Merge pull request #38993 from azat/fix-fsync_part_directory-on-fetch 
Fix fsync_part_directory for fetches
 2022-07-12 11:48:42 +02:00
Nikolai Kochetov
cd4211298e Fix typo. 2022-07-12 09:37:00 +00:00
Nikolai Kochetov
03d9301425 Fix log. 2022-07-12 08:31:09 +00:00
Nikolai Kochetov
631be9219b Properly remove projection from part in case it was removed from table metadata. 2022-07-12 08:28:22 +00:00
aaapetrenko
aabf0ed67f refactor Storage 2022-07-11 22:00:57 +02:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_" 
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
 2022-07-11 19:36:18 +02:00
Sergei Trifonov
6ed777d2a8 add azure support 2022-07-11 19:19:57 +02:00
Sergei Trifonov
add0d1698e respect new limits in HDFS 2022-07-11 17:30:23 +02:00
Sergei Trifonov
43779ec280 add max_remote_{read,write}_network_bandwidth_for_server settings 2022-07-11 14:59:39 +02:00
Alexander Tokmakov
e9ddbc247f
Merge pull request #39016 from Algunenano/clearOldLogs_zkerror 
clearOldLogs: Don't report KEEPER_EXCEPTION on concurrent deletes
 2022-07-11 15:16:18 +03:00
Alexander Tokmakov
048d27189e
Update MergeTreeData.cpp 2022-07-11 15:07:31 +03:00
kssenii
4c88527c8e Review fixes 2022-07-09 19:35:07 +02:00
kssenii
bf1c99daab Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
Alexander Tokmakov
51130d723c
Update MergeTreeData.cpp 2022-07-08 20:17:29 +03:00
Raúl Marín
a965438eea clearOldLogs: Don't report errors on concurrent deletes 2022-07-08 13:57:53 +02:00
Azat Khuzhin
b2f7f9c0f0 Fix fsync_part_directory for fetches 
Before it produces the following error:

    2022.07.07 23:40:03.131044 [ 44869 ] {} <Error> default.rep_data: auto DB::StorageReplicatedMergeTree::processQueueEntry(ReplicatedMergeTreeQueue::SelectedEntryPtr)::(anonymous class)::operator()(DB::StorageReplicatedMergeTree::LogEntryPtr &) const: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/tmp-fetch_36_25458_25458_0, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

    0. Poco::Exception::Exception() @ 0x1764634c in /usr/bin/clickhouse
    1. DB::Exception::Exception() @ 0xb42caba in /usr/bin/clickhouse
    2. DB::throwFromErrnoWithPath() @ 0xb42d79b in /usr/bin/clickhouse
    3. DB::LocalDirectorySyncGuard::LocalDirectorySyncGuard() @ 0x1404bbd4 in /usr/bin/clickhouse
    4. DB::DiskLocal::getDirectorySyncGuard() const @ 0x140466a4 in /usr/bin/clickhouse
    5. DB::DataPartsExchange::Fetcher::downloadPartToDisk() @ 0x15122af1 in /usr/bin/clickhouse
    6. DB::DataPartsExchange::Fetcher::fetchPart() @ 0x1511de39 in /usr/bin/clickhouse

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-08 13:48:51 +03:00
lgbo-ustc
e01b6cf8a5 update codes 2022-07-08 09:25:45 +08:00
Alexander Tokmakov
b0f8970d07
Merge pull request #38961 from Algunenano/reduce_zk_eexists 
Stop reporting Zookeeper "Node exists" exceptions in system.errors when they are expected
 2022-07-07 23:16:46 +03:00
Dmitry Novik
d1df66687b
Merge branch 'master' into group-by-use-nulls 2022-07-07 20:54:38 +02:00
Raúl Marín
9b0c7714e5 Style 2022-07-07 17:15:23 +02:00
Kseniia Sumarokova
48220f64d6
Merge pull request #38214 from bigo-sg/storagehive_alter 
support alter on `StorageHive` table
 2022-07-07 17:06:43 +02:00
Raúl Marín
c3935000a8 Avoid reporting ZK EEXIST errors on checkPartChecksumsAndCommit 2022-07-07 16:10:05 +02:00
Raúl Marín
33ffb71453 Avoid 'Node exists' errors when creating Ephemeral Locks 2022-07-07 16:10:04 +02:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
alesapin
1bd97fa6fa
Merge pull request #38851 from Algunenano/rmarin_read_only_on_drop 
Deactivate mutations_finalizing_task during shutdown
 2022-07-07 12:55:50 +02:00
Amos Bird
19bd3d8d9f
Nullable partition pruning fix 2022-07-07 16:02:38 +08:00
lgbo-ustc
337f379bc3 update codes 2022-07-07 09:45:49 +08:00
lgbo-ustc
cda081dc57 update codes 2022-07-07 09:41:52 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9 
Backup Improvements 9
 2022-07-07 02:24:47 +02:00
Vitaly Baranov
7f84cf3968 Fix style. 2022-07-06 16:36:59 +02:00
Maksim Kita
b94489d52c
Merge pull request #38859 from kitaisreal/merge-tree-merge-disable-batch-optimization 
MergeTree merge disable batch optimization
 2022-07-06 15:59:40 +02:00
Vladimir C
3c7fdb604d
Merge pull request #35363 from vdimir/kv-join 2022-07-06 15:57:30 +02:00
Raúl Marín
86d1c319f4 Remove unnecessary log 2022-07-06 12:23:45 +02:00
lgbo-ustc
aeb4f9c6be fixed: crash caused by IHiveFile be shared among threads 2022-07-06 16:56:17 +08:00
lgbo-ustc
946a9b8c63 update codes 2022-07-06 09:26:02 +08:00
Kseniia Sumarokova
a26643a7a6
Merge pull request #38363 from kssenii/pg-add-auto-close 
Support `auto_close` option for postgres engine connection
 2022-07-06 01:10:05 +02:00
Vitaly Baranov
f9204315b5 Store columns.txt in backups for the Memory table engine too. 2022-07-05 19:03:20 +02:00
Kseniia Sumarokova
3e803006a2
Merge pull request #37171 from tchepavel/nats-integration 
Nats Integration
 2022-07-05 17:47:02 +02:00
Maksim Kita
bdc21737d5 MergeTree merge disable batch optimization 2022-07-05 16:15:00 +02:00
Vitaly Baranov
43d35eec1b Write unfinished mutations to backup. 2022-07-05 14:51:09 +02:00
Raúl Marín
9e17ae365b Deactivate mutations_finalizing_task during shutdown 2022-07-05 14:28:20 +02:00
vdimir
17903117d2
better 2022-07-05 10:46:08 +00:00
kssenii
7a4be3b748 Fix build, disable for freebsd 2022-07-05 12:10:49 +02:00
Kruglov Pavel
a6a9824261
Merge pull request #38362 from Avogar/better-exception-messages 
Better exception messages on wrong table engines/functions argument types
 2022-07-05 11:21:44 +02:00
lgbo-ustc
76813ee3c0 update codes 2022-07-05 11:49:55 +08:00
Kseniia Sumarokova
6c3d648f49
Merge branch 'master' into nats-integration 2022-07-04 21:06:07 +02:00
Kseniia Sumarokova
0cd228cab3
Merge pull request #38365 from kssenii/better-rabbit-adddress-setting 
Fix RabbitMQ configuration via connection string setting
 2022-07-04 20:53:43 +02:00
vdimir
c0cb588e45
Fix build 2022-07-04 17:28:34 +00:00
vdimir
43afda408b
do not call setStorageJoin when setting isn't set 2022-07-04 17:28:15 +00:00
vdimir
65ac97ce89
direct join: handle nullable, add tests 2022-07-04 17:28:14 +00:00
vdimir
8359753a85
Move DirectJoin to separate file 2022-07-04 17:28:11 +00:00
vdimir
5ed0911ff4
Fix rocksdb::getSlicedKeys 2022-07-04 17:28:10 +00:00
vdimir
085e70c7a3
wip left kvjoin 2022-07-04 17:28:10 +00:00
vdimir
980000d04c
Change getByKeys, add StorageEmbeddedRocksDB::getByKeysImpl 2022-07-04 17:28:09 +00:00
vdimir
744c692be3
wip key value join 2022-07-04 17:28:09 +00:00