thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 03:12:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
82,934 Commits 9,326 Branches 1,783 Tags 2.1 GiB
f099048dd1
Commit Graph

6867 Commits

Author SHA1 Message Date
Alexander Tokmakov
897e94c16c make restarting thread less bad 2022-02-03 23:29:24 +03:00
Anton Popov
9b844c6b42
Merge pull request #32748 from CurtizJ/read-in-order-fixed-prefix 
Support `optimize_read_in_order` if prefix of sorting key is already sorted
 2022-02-03 18:17:08 +03:00
Nikolai Kochetov
a2a9c10617
Merge pull request #34221 from ClickHouse/revert-34201-revert-34189-less-logging-for-remote_fs_execute_merges_on_single_replica_time_threshold 
Revert "Revert "Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue""
 2022-02-03 13:09:44 +03:00
Nikolai Kochetov
ea044fc6b2 Merge branch 'master' into revert-34212-revert-33291-add-pool-to-s3-write-buffer 2022-02-02 19:40:41 +03:00
Nikolai Kochetov
1117e6095d
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-02-02 18:34:05 +03:00
Azat Khuzhin
15993cb13b Add missing fmt::runtime() in MergeTreeBackgroundExecutor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-02 11:37:53 +03:00
alesapin
47d538b52f
Merge pull request #34232 from ClickHouse/cancelled-merging-parts-error-message 
Change severity of the "Cancelled merging parts" message in logs
 2022-02-02 10:29:07 +03:00
Sergei Trifonov
68bc456830
Merge pull request #34223 from azat/bump-fmt 
Bump fmtlib from 7.0.0 to 8.1.1
 2022-02-02 00:03:25 +03:00
Nikolai Kochetov
dd81eff301 Fix tests. 2022-02-01 15:28:22 +00:00
mergify[bot]
d8bea598b2
Merge branch 'master' into fix-substr-zk-metadata 2022-02-01 14:10:12 +00:00
Azat Khuzhin
5dfafd68a7 ReplicatedMergeTreeQueue: Fix fmt:: and reduce copy-paste of logging and out reason 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Azat Khuzhin
5be76bc969 Use proper fmt-like logging 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Azat Khuzhin
3b3635c6d5 Fix formatting error in logging messages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:04 +03:00
Azat Khuzhin
743096a883 Use proper fmt:: like Exception ctor in DataPartsExchange 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Azat Khuzhin
bedf208cbd Use fmt::runtime() for LOG_* for non constexpr 
Here is oneliner:

    $ gg 'LOG_\(DEBUG\|TRACE\|INFO\|TEST\|WARNING\|ERROR\|FATAL\)([^,]*, [a-zA-Z]' -- :*.cpp :*.h | cut -d: -f1 | sort -u | xargs -r sed -E -i 's#(LOG_[A-Z]*)\(([^,]*), ([A-Za-z][^,)]*)#\1(\2, fmt::runtime(\3)#'

Note, that I tried to do this with coccinelle (tool for semantic
patchin), but it cannot parse C++:

    $ cat fmt.cocci
    @@
    expression log;
    expression var;
    @@

    -LOG_DEBUG(log, var)
    +LOG_DEBUG(log, fmt::runtime(var))

I've also tried to use some macros/templates magic to do this implicitly
in logger_useful.h, but I failed to do so, and apparently it is not
possible for now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

v2: manual fixes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-01 14:30:03 +03:00
Nikolai Kochetov
a6cc61bd14
Revert "Revert "Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue"" 2022-02-01 14:17:46 +03:00
Nikolai Kochetov
2a6eb593be
Revert "Revert "Add pool to WriteBufferFromS3"" 2022-02-01 13:36:51 +03:00
alexey-milovidov
06477c2a7e Update ReplicatedMergeTreeSink.cpp 2022-02-01 09:22:40 +01:00
Igor Nikonov
f4c0b64420 Clean up: insert_deduplication_token setting for INSERT statement 
+ reduce number of allocations on replication merge tree path
+ bash test: move insert block settings into variable

Issue: ClickHouse#7461
 2022-02-01 09:22:33 +01:00
alexey-milovidov
99392b5ca7
Merge pull request #13544 from amosbird/mdha 
Multi-Disk auto-recovery.
 2022-02-01 06:13:26 +03:00
Alexey Milovidov
7dbf0dede5 Change severity of the "Cancelled merging parts" message in logs 2022-02-01 05:55:07 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3" 2022-02-01 05:49:40 +03:00
mergify[bot]
e229487817
Merge branch 'master' into mdha 2022-02-01 01:22:16 +00:00
Dmitry Novik
3a0ed0f8f5
Merge pull request #33324 from azat/buffer-different-structure 
Simplify different block structure (i.e. after ALTER) support for Buffer
 2022-02-01 02:19:49 +03:00
alexey-milovidov
15e4fe5c78
Revert "Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue" 2022-02-01 01:51:39 +03:00
Amos Bird
ec7d367814
DiskLocal checker 
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
 2022-02-01 05:55:27 +08:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines 
Fix benign race condition for storage HDFS, S3, URL
 2022-01-31 22:41:54 +03:00
alesapin
75d73d2785
Merge pull request #34139 from ClickHouse/fix_buf_s3_low_cardinality 
Fix bug with bounded S3 reads and LowCardinality
 2022-01-31 22:41:14 +03:00
Nikolai Kochetov
348d72266a
Merge pull request #34189 from ClickHouse/less-logging-for-remote_fs_execute_merges_on_single_replica_time_threshold 
Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue
 2022-01-31 21:39:04 +03:00
Nikolai Kochetov
a207cdf28f Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue. 2022-01-31 17:53:28 +00:00
Nikolai Kochetov
321fa4a9e8
Merge pull request #33291 from ClickHouse/add-pool-to-s3-write-buffer 
Add pool to WriteBufferFromS3
 2022-01-31 19:37:40 +03:00
Maksim Kita
8513f20cfd
Merge pull request #34145 from kitaisreal/bitset-sort-performance-check 
pdqsort performance check
 2022-01-31 12:35:13 +01:00
alesapin
5230c59f05 Smaller scope 2022-01-31 14:03:06 +03:00
alesapin
1adc7538fe Fix benign race condition for storage HDFS, S3, URL 2022-01-31 13:50:28 +03:00
tavplubix
d19e24f530
Merge pull request #34096 from ClickHouse/fix_race_merge_selecting_task 
Fix race between mergeSelectingTask and queue reinitialization
 2022-01-31 12:16:29 +03:00
Kruglov Pavel
a9d0beb7ae
Fix data race in StorageFile (#34113) 
* Fix data race in StorageFile

* Update StorageFile.h

* Fix
 2022-01-31 11:58:40 +03:00
alesapin
55c7936257 Fix incorrect range for index 2022-01-31 11:11:32 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
alesapin
4f1b902342 Fix compact parts as well 2022-01-30 22:36:19 +03:00
alesapin
4bedcc19b5 Better invariants 2022-01-30 20:40:09 +03:00
alesapin
c237c03c50 Fix 2022-01-30 18:39:26 +03:00
alesapin
bf918892ac More clear code with less getMark calls 2022-01-30 18:21:05 +03:00
alesapin
cb45a348f1 Merge branch 'master' into fix_buf_s3_low_cardinality 2022-01-30 17:30:55 +03:00
alesapin
3f3e90c8ba Remove redundant code 2022-01-29 23:55:45 +03:00
alexey-milovidov
2b43bad923
Update MergeTreeReaderStream.cpp 2022-01-29 19:28:39 +03:00
alesapin
7ada8227cf Fix bug with bounded S3 reads and LowCardinality 2022-01-29 18:28:40 +03:00
Anton Popov
96a506c6fa fix inserts to distributed tables in case of change of native protocol 2022-01-29 03:23:25 +03:00
alexey-milovidov
6535b75322
Merge pull request #34001 from azat/memory-tracker-fix 
Fix memory accounting for queries that uses < max_untracker_memory
 2022-01-29 00:59:53 +03:00
Azat Khuzhin
1519985c98 Fix possible "Can't attach query to the thread, it is already attached" 
After detachQueryIfNotDetached() had been removed it is not enough to
use attachTo() for ThreadPool (scheduleOrThrowOnError()) since the query
may be already attached, if the thread doing multiple jobs, so
CurrentThread::attachToIfDetached() should be used instead.

This should fix all the places from the failures on CI [1]:

    $ fgrep DB::CurrentThread::attachTo -A1 ~/Downloads/47.txt  | fgrep -v attachTo | cut -d' ' -f5,6 | sort | uniq -c
         92 --
          2 /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/deque:1393: DB::ParallelParsingInputFormat::parserThreadFunction(std::__1::shared_ptr<DB::ThreadGroupStatus>,
          4 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeData.cpp:1595: void
         87 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:993: void

  [1]: https://github.com/ClickHouse/ClickHouse/runs/4954466034?check_suite_focus=true

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Azat Khuzhin
b0c862c297 Fix memory accounting for queries that uses < max_untracker_memory 
MemoryTracker starts accounting memory directly only after per-thread
allocation exceeded max_untracker_memory (or memory_profiler_step).

But even memory under this limit should be accounted too, and there is
code to do this in ThreadStatus dtor, however due to
PullingAsyncPipelineExecutor detached the query from thread group that
memory was not accounted.

So remove CurrentThread::detachQueryIfNotDetached() from threads that
uses ThreadFromGlobalPool since it has ThreadStatus, and the query will
be detached using CurrentThread::defaultThreadDeleter.

Note, that before this patch memory accounting works for HTTP queries
due to it had been accounted from ParallelFormattingOutputFormat, but
not for TCP.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Nikolai Kochetov
1c9f026178 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-28 16:01:42 +03:00
Alexander Tokmakov
b3ddc601a5 fix race between mergeSelectingTask and queue reinitialization 2022-01-28 15:50:58 +03:00
liyang830
a69711ccff ExpressionList parse tuple and ttl 2022-01-28 19:07:59 +08:00
alexey-milovidov
f6684dbc62
Merge pull request #32304 from devcrafter/deduplication_token_7461 
insert_deduplication_token setting for INSERT statement
 2022-01-28 13:03:55 +03:00
Kruglov Pavel
efa8775ea6
Merge pull request #33960 from Avogar/autodetect-format 
Detect format and schema from stdin in clickhouse-local
 2022-01-28 12:02:15 +03:00
Azat Khuzhin
42ddf4d2b8 Fix negative StorageBufferBytes by proper accounting of empty blocks 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 11:10:00 +03:00
Azat Khuzhin
d77790182d Tiny cleanup in StorageBuffer::flushBuffer() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 11:10:00 +03:00
Azat Khuzhin
9948525816 Simplify different block sturcture (i.e. after ALTER) support for Buffer 
v2: fix empty block in case of flush

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 11:09:56 +03:00
Nikolai Kochetov
94999e86c2
Merge pull request #33392 from azat/_shard_num 
Replace old _shard_num implementation with shardNum() function
 2022-01-28 11:02:19 +03:00
alexey-milovidov
63d8d75001
Merge pull request #33957 from azat/fix-getauxval 
Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work)
 2022-01-27 23:05:23 +03:00
mergify[bot]
ad146a67ff
Merge branch 'master' into _shard_num 2022-01-27 13:11:42 +00:00
liyang830
c2bdeb349c Merge branch 'fix-substr-zk-metadata' of github.com:sunny19930321/ClickHouse into fix-substr-zk-metadata 2022-01-27 16:35:02 +08:00
liyang830
eca0453564 fix local metadata differ zk metadata 2022-01-27 16:33:40 +08:00
Nikolai Kochetov
9b2998c639 Review fixes. 2022-01-26 18:08:01 +00:00
Nikolai Kochetov
a8171269a1 Review fixes. 2022-01-26 17:55:24 +00:00
Raúl Marín
5a59d976dd CurrentlyExecuting: Require mutex usage explicitly 2022-01-26 18:44:35 +01:00
Azat Khuzhin
08f4f45fd9 Fix Context leak (recursive context reference) in StorageSQLite 
CI founds after LSan had been fixed [1]:

    01889_sqlite_read_write:                                                [ FAIL ] 8.32 sec. - return code: 1

    =================================================================
    ==20649==ERROR: LeakSanitizer: detected memory leaks

    Indirect leak of 1968 byte(s) in 1 object(s) allocated from:
        0 0xc5c1ffd in operator new(unsigned long) (/usr/bin/clickhouse+0xc5c1ffd)
        1 0x25e32d0d in std::__1::__unique_if<DB::StorageInMemoryMetadata>::__unique_single std::__1::make_unique<DB::StorageInMemoryMetadata, DB::StorageInMemoryMetadata const&>(DB::StorageInMemoryMetadata c>
        2 0x25e32d0d in DB::IStorage::setInMemoryMetadata(DB::StorageInMemoryMetadata const&) obj-x86_64-linux-gnu/../src/Storages/IStorage.h:194:22
        3 0x29bdee98 in DB::StorageSQLite::StorageSQLite(DB::StorageID const&, std::__1::shared_ptr<sqlite3>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std:>
        4 0x25ee61d6 in std::__1::shared_ptr<DB::StorageSQLite> shared_ptr_helper<DB::StorageSQLite>::create<DB::StorageID, std::__1::shared_ptr<sqlite3> const&, std::__1::basic_string<char, std::__1::char_tr>
        5 0x25ee61d6 in DB::TableFunctionSQLite::executeImpl(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1:>

    SUMMARY: AddressSanitizer: 171256 byte(s) leaked in 130 allocation(s).

  [1]: https://github.com/ClickHouse/ClickHouse/runs/4929706698?check_suite_focus=true

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-26 12:56:27 +03:00
Nikolai Kochetov
fcc29dbd15 Try to fix integration tests. 2022-01-25 15:26:36 +00:00
Kruglov Pavel
7873b4475f
Merge branch 'master' into autodetect-format 2022-01-25 10:56:52 +03:00
Kruglov Pavel
9f12f4af13
Merge pull request #33302 from Avogar/formats-with-suffixes 
Allow to create new files on insert for File/S3/HDFS engines
 2022-01-25 10:56:15 +03:00
avogar
a6740d2f9a Detect format and schema for stdin in clickhouse-local 2022-01-25 10:25:37 +03:00
Nikolai Kochetov
52b65c3454 Fix style 2022-01-24 16:21:27 +00:00
Nikolai Kochetov
506ee8c024 Refactor some code. 2022-01-24 15:55:29 +00:00
Nikolai Kochetov
4afa75e80b Fix deduplication. 2022-01-24 14:43:36 +00:00
Nikolai Kochetov
2882e46654 Remove debug logging. 2022-01-24 11:48:00 +00:00
tavplubix
da9a38655b
Merge pull request #33847 from ClickHouse/fix_33806 
Fix logical error on RESTORE REPLICA
 2022-01-24 13:28:37 +03:00
tavplubix
766786b570
Update StorageReplicatedMergeTree.h 2022-01-24 12:16:13 +03:00
tavplubix
34baca1298
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
 2022-01-24 12:12:08 +03:00
tavplubix
2f566d2af9
Update src/Storages/StorageReplicatedMergeTree.cpp 
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
 2022-01-24 12:12:03 +03:00
Nikolai Kochetov
f74cf1e152 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:55:42 +03:00
Nikolai Kochetov
a024ddd1d4 Merge branch 'add-pool-to-s3-write-buffer' of github.com:ClickHouse/ClickHouse into add-pool-to-s3-write-buffer 2022-01-21 18:42:24 +00:00
Nikolai Kochetov
b3cbb63487 Merge branch 'master' into add-pool-to-s3-write-buffer 2022-01-21 21:41:54 +03:00
alexey-milovidov
43f2e2c178
Merge pull request #33849 from CurtizJ/fix-sparse-columns 
Fix sparse columns
 2022-01-21 21:13:02 +03:00
Nikolai Kochetov
ae1b669fca Even more async writes. 2022-01-21 15:55:19 +00:00
Kruglov Pavel
daba57a933
Merge pull request #33737 from Avogar/bool-type 
Some improvements and fixes for Bool data type
 2022-01-21 15:24:58 +03:00
Alexander Tokmakov
8921725d71 fix logical error on RESTORE REPLICA 2022-01-21 14:32:19 +03:00
taiyang-li
22dc786dd8 optimize log level 2022-01-21 12:16:49 +08:00
Anton Popov
c2206c0b96 better storing serialization infos 2022-01-21 03:29:30 +03:00
Anton Popov
f78fb247a7 fix column ttl 2022-01-21 02:22:16 +03:00
Anton Popov
6c4c9bc04a
Merge pull request #33234 from CurtizJ/fix-flush-in-memory-parts 
Fix flushing of in-memory parts
 2022-01-21 02:18:03 +03:00
Anton Popov
558eb7ea73 fix sparse columns 2022-01-21 00:06:32 +03:00
Maksim Kita
56a54af27b
Merge pull request #33833 from amosbird/keyconditionfix 
Fix KeyCondition with no common types
 2022-01-20 20:31:27 +01:00
李扬
9f6f84259d
Fix bug of check table when creating data part with wide format and projection (#33774) 2022-01-20 18:05:17 +03:00
Amos Bird
e62f4e50a6
Fix KeyCondition with no common types 2022-01-20 21:56:12 +08:00
Kruglov Pavel
7bfb1231b9
Merge branch 'master' into formats-with-suffixes 2022-01-20 14:47:17 +03:00
avogar
ba54a5ff80 Some improvements and fixes for Bool data type 2022-01-20 14:16:11 +03:00
alexey-milovidov
28a9d569cc
Merge pull request #33690 from azat/remove-unbundled-support 
Remove unbundled support
 2022-01-20 12:29:07 +03:00
Kseniia Sumarokova
064c3dd654
Merge pull request #33724 from liuneng1994/fix_ignore_hdfs_error 
Throw exception when storage hdfs list directory failed
 2022-01-20 11:59:09 +03:00
Azat Khuzhin
b23053d1f6 Remove USE_XXHASH (reduntant) 2022-01-20 10:02:02 +03:00
Azat Khuzhin
e0e81b340d Fix w/o ODBC build 2022-01-20 10:02:02 +03:00