Anton Popov
587d7399ba
support dynamic subcolumns for Memory engine
2022-02-09 03:18:53 +03:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
alexey-milovidov
99392b5ca7
Merge pull request #13544 from amosbird/mdha
...
Multi-Disk auto-recovery.
2022-02-01 06:13:26 +03:00
alexey-milovidov
095d9bfa43
Revert "Add pool to WriteBufferFromS3"
2022-02-01 05:49:40 +03:00
mergify[bot]
e229487817
Merge branch 'master' into mdha
2022-02-01 01:22:16 +00:00
Dmitry Novik
3a0ed0f8f5
Merge pull request #33324 from azat/buffer-different-structure
...
Simplify different block structure (i.e. after ALTER) support for Buffer
2022-02-01 02:19:49 +03:00
alexey-milovidov
15e4fe5c78
Revert "Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue"
2022-02-01 01:51:39 +03:00
Amos Bird
ec7d367814
DiskLocal checker
...
Add DiskLocal checker so that ReplicatedMergeTree can recover data when some of its disks are broken.
2022-02-01 05:55:27 +08:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines
...
Fix benign race condition for storage HDFS, S3, URL
2022-01-31 22:41:54 +03:00
alesapin
75d73d2785
Merge pull request #34139 from ClickHouse/fix_buf_s3_low_cardinality
...
Fix bug with bounded S3 reads and LowCardinality
2022-01-31 22:41:14 +03:00
Nikolai Kochetov
348d72266a
Merge pull request #34189 from ClickHouse/less-logging-for-remote_fs_execute_merges_on_single_replica_time_threshold
...
Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue
2022-01-31 21:39:04 +03:00
Nikolai Kochetov
a207cdf28f
Additionally check remote_fs_execute_merges_on_single_replica_time_threshold inside ReplicatedMergeTreeQueue.
2022-01-31 17:53:28 +00:00
Nikolai Kochetov
321fa4a9e8
Merge pull request #33291 from ClickHouse/add-pool-to-s3-write-buffer
...
Add pool to WriteBufferFromS3
2022-01-31 19:37:40 +03:00
Maksim Kita
8513f20cfd
Merge pull request #34145 from kitaisreal/bitset-sort-performance-check
...
pdqsort performance check
2022-01-31 12:35:13 +01:00
alesapin
5230c59f05
Smaller scope
2022-01-31 14:03:06 +03:00
alesapin
1adc7538fe
Fix benign race condition for storage HDFS, S3, URL
2022-01-31 13:50:28 +03:00
tavplubix
d19e24f530
Merge pull request #34096 from ClickHouse/fix_race_merge_selecting_task
...
Fix race between mergeSelectingTask and queue reinitialization
2022-01-31 12:16:29 +03:00
Kruglov Pavel
a9d0beb7ae
Fix data race in StorageFile ( #34113 )
...
* Fix data race in StorageFile
* Update StorageFile.h
* Fix
2022-01-31 11:58:40 +03:00
alesapin
55c7936257
Fix incorrect range for index
2022-01-31 11:11:32 +03:00
Maksim Kita
5ef83deaa6
Update sort to pdqsort
2022-01-30 19:49:48 +00:00
alesapin
4f1b902342
Fix compact parts as well
2022-01-30 22:36:19 +03:00
alesapin
4bedcc19b5
Better invariants
2022-01-30 20:40:09 +03:00
alesapin
c237c03c50
Fix
2022-01-30 18:39:26 +03:00
alesapin
bf918892ac
More clear code with less getMark calls
2022-01-30 18:21:05 +03:00
alesapin
cb45a348f1
Merge branch 'master' into fix_buf_s3_low_cardinality
2022-01-30 17:30:55 +03:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
alesapin
3f3e90c8ba
Remove redundant code
2022-01-29 23:55:45 +03:00
alexey-milovidov
2b43bad923
Update MergeTreeReaderStream.cpp
2022-01-29 19:28:39 +03:00
alesapin
7ada8227cf
Fix bug with bounded S3 reads and LowCardinality
2022-01-29 18:28:40 +03:00
alexey-milovidov
6535b75322
Merge pull request #34001 from azat/memory-tracker-fix
...
Fix memory accounting for queries that uses < max_untracker_memory
2022-01-29 00:59:53 +03:00
Azat Khuzhin
1519985c98
Fix possible "Can't attach query to the thread, it is already attached"
...
After detachQueryIfNotDetached() had been removed it is not enough to
use attachTo() for ThreadPool (scheduleOrThrowOnError()) since the query
may be already attached, if the thread doing multiple jobs, so
CurrentThread::attachToIfDetached() should be used instead.
This should fix all the places from the failures on CI [1]:
$ fgrep DB::CurrentThread::attachTo -A1 ~/Downloads/47.txt | fgrep -v attachTo | cut -d' ' -f5,6 | sort | uniq -c
92 --
2 /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/deque:1393: DB::ParallelParsingInputFormat::parserThreadFunction(std::__1::shared_ptr<DB::ThreadGroupStatus>,
4 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeData.cpp:1595: void
87 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:993: void
[1]: https://github.com/ClickHouse/ClickHouse/runs/4954466034?check_suite_focus=true
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 16:25:33 +03:00
Azat Khuzhin
b0c862c297
Fix memory accounting for queries that uses < max_untracker_memory
...
MemoryTracker starts accounting memory directly only after per-thread
allocation exceeded max_untracker_memory (or memory_profiler_step).
But even memory under this limit should be accounted too, and there is
code to do this in ThreadStatus dtor, however due to
PullingAsyncPipelineExecutor detached the query from thread group that
memory was not accounted.
So remove CurrentThread::detachQueryIfNotDetached() from threads that
uses ThreadFromGlobalPool since it has ThreadStatus, and the query will
be detached using CurrentThread::defaultThreadDeleter.
Note, that before this patch memory accounting works for HTTP queries
due to it had been accounted from ParallelFormattingOutputFormat, but
not for TCP.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 16:25:33 +03:00
Nikolai Kochetov
1c9f026178
Merge branch 'master' into add-pool-to-s3-write-buffer
2022-01-28 16:01:42 +03:00
Alexander Tokmakov
b3ddc601a5
fix race between mergeSelectingTask and queue reinitialization
2022-01-28 15:50:58 +03:00
alexey-milovidov
f6684dbc62
Merge pull request #32304 from devcrafter/deduplication_token_7461
...
insert_deduplication_token setting for INSERT statement
2022-01-28 13:03:55 +03:00
Kruglov Pavel
efa8775ea6
Merge pull request #33960 from Avogar/autodetect-format
...
Detect format and schema from stdin in clickhouse-local
2022-01-28 12:02:15 +03:00
Azat Khuzhin
42ddf4d2b8
Fix negative StorageBufferBytes by proper accounting of empty blocks
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 11:10:00 +03:00
Azat Khuzhin
d77790182d
Tiny cleanup in StorageBuffer::flushBuffer()
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 11:10:00 +03:00
Azat Khuzhin
9948525816
Simplify different block sturcture (i.e. after ALTER) support for Buffer
...
v2: fix empty block in case of flush
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 11:09:56 +03:00
Nikolai Kochetov
94999e86c2
Merge pull request #33392 from azat/_shard_num
...
Replace old _shard_num implementation with shardNum() function
2022-01-28 11:02:19 +03:00
alexey-milovidov
63d8d75001
Merge pull request #33957 from azat/fix-getauxval
...
Fix getauxval() in glibc-compatibility and fix some leaks (after LSan started to work)
2022-01-27 23:05:23 +03:00
mergify[bot]
ad146a67ff
Merge branch 'master' into _shard_num
2022-01-27 13:11:42 +00:00
Nikolai Kochetov
9b2998c639
Review fixes.
2022-01-26 18:08:01 +00:00
Nikolai Kochetov
a8171269a1
Review fixes.
2022-01-26 17:55:24 +00:00
Raúl Marín
5a59d976dd
CurrentlyExecuting: Require mutex usage explicitly
2022-01-26 18:44:35 +01:00
Azat Khuzhin
08f4f45fd9
Fix Context leak (recursive context reference) in StorageSQLite
...
CI founds after LSan had been fixed [1]:
01889_sqlite_read_write: [ FAIL ] 8.32 sec. - return code: 1
=================================================================
==20649==ERROR: LeakSanitizer: detected memory leaks
Indirect leak of 1968 byte(s) in 1 object(s) allocated from:
0 0xc5c1ffd in operator new(unsigned long) (/usr/bin/clickhouse+0xc5c1ffd)
1 0x25e32d0d in std::__1::__unique_if<DB::StorageInMemoryMetadata>::__unique_single std::__1::make_unique<DB::StorageInMemoryMetadata, DB::StorageInMemoryMetadata const&>(DB::StorageInMemoryMetadata c>
2 0x25e32d0d in DB::IStorage::setInMemoryMetadata(DB::StorageInMemoryMetadata const&) obj-x86_64-linux-gnu/../src/Storages/IStorage.h:194:22
3 0x29bdee98 in DB::StorageSQLite::StorageSQLite(DB::StorageID const&, std::__1::shared_ptr<sqlite3>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std:>
4 0x25ee61d6 in std::__1::shared_ptr<DB::StorageSQLite> shared_ptr_helper<DB::StorageSQLite>::create<DB::StorageID, std::__1::shared_ptr<sqlite3> const&, std::__1::basic_string<char, std::__1::char_tr>
5 0x25ee61d6 in DB::TableFunctionSQLite::executeImpl(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1:>
SUMMARY: AddressSanitizer: 171256 byte(s) leaked in 130 allocation(s).
[1]: https://github.com/ClickHouse/ClickHouse/runs/4929706698?check_suite_focus=true
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-26 12:56:27 +03:00
Nikolai Kochetov
fcc29dbd15
Try to fix integration tests.
2022-01-25 15:26:36 +00:00
Kruglov Pavel
7873b4475f
Merge branch 'master' into autodetect-format
2022-01-25 10:56:52 +03:00
Kruglov Pavel
9f12f4af13
Merge pull request #33302 from Avogar/formats-with-suffixes
...
Allow to create new files on insert for File/S3/HDFS engines
2022-01-25 10:56:15 +03:00
avogar
a6740d2f9a
Detect format and schema for stdin in clickhouse-local
2022-01-25 10:25:37 +03:00