thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 20:02:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
97,002 Commits 9,330 Branches 1,783 Tags 2.1 GiB
81c98dadd2
Commit Graph

4097 Commits

Author SHA1 Message Date
FArthur-cmd
ad459bc6e4 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy-2 2022-07-14 14:10:44 +03:00
Azat Khuzhin
3e77b80b88 Fix fetch of in-memory part with allow_remote_fs_zero_copy_replication 
CI founds the following error during trying to fetch in-memory part [1]:

    2022.07.13 08:06:54.231033 [ 141093 ] {} <Error> InterserverIOHTTPHandler: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/disks/s3/store/886/88635b40-e4e3-4fe6-a0a0-1b6755463358/all_0_0_0/data.bin, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

<details>

<summary>stacktrace</summary>

    2022.07.13 08:06:54.101825 [ 189342 ] {8dbd11b3-f38a-4d5d-9ded-148987adb71d} <Debug> executeQuery: (from [::1]:54570) (comment: 01643_replicated_merge_tree_fsync_smoke.sql) select 'memory in_memory_parts_insert_sync'; (stage: Complete)
    2022.07.13 08:06:54.131309 [ 691 ] {} <Debug> test_26u6kx.rep_fsync_r2 (39c3823c-22e5-4c05-9dec-cdffd8872c40): Fetching part all_0_0_0 from /clickhouse/tables/test_26u6kx/rep_fsync/replicas/r1
    2022.07.13 08:06:54.231033 [ 141093 ] {} <Error> InterserverIOHTTPHandler: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/disks/s3/store/886/88635b40-e4e3-4fe6-a0a0-1b6755463358/all_0_0_0/data.bin, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):
    0. DB::Exception::Exception() @ 0xba0191a in /usr/bin/clickhouse
    1. DB::throwFromErrnoWithPath() @ 0xba029ca in /usr/bin/clickhouse
    2. DB::OpenedFile::open() const @ 0x156e7fb0 in /usr/bin/clickhouse
    3. DB::OpenedFile::getFD() const @ 0x156e8003 in /usr/bin/clickhouse
    4. DB::ReadBufferFromFilePReadWithDescriptorsCache::ReadBufferFromFilePReadWithDescriptorsCache() @ 0x156e5f23 in /usr/bin/clickhouse
    5. ? @ 0x156e53f0 in /usr/bin/clickhouse
    6. DB::createReadBufferFromFileBase() @ 0x156e52b5 in /usr/bin/clickhouse
    7. DB::DiskLocal::readFile() const @ 0x15e45ea8 in /usr/bin/clickhouse
    8. DB::MetadataStorageFromDisk::readFileToString() const @ 0x15e6ab8b in /usr/bin/clickhouse
    9. DB::MetadataStorageFromDisk::readMetadataUnlocked() const @ 0x15e6cdeb in /usr/bin/clickhouse
    10. DB::MetadataStorageFromDisk::getSerializedMetadata() const @ 0x15e6cfc4 in /usr/bin/clickhouse
    11. DB::DiskObjectStorage::getSerializedMetadata() const @ 0x15e19e2e in /usr/bin/clickhouse
    12. DB::DiskDecorator::getSerializedMetadata() const @ 0x15e54ed1 in /usr/bin/clickhouse
    13. DB::DiskDecorator::getSerializedMetadata() const @ 0x15e54ed1 in /usr/bin/clickhouse
    14. DB::DataPartsExchange::Service::sendPartFromDiskRemoteMeta() @ 0x1700bb9e in /usr/bin/clickhouse
    15. DB::DataPartsExchange::Service::processQuery(DB::HTMLForm const&, DB::ReadBuffer&, DB::WriteBuffer&, DB::HTTPServerResponse&) @ 0x1700a649 in /usr/bin/clickhouse
    16. DB::InterserverIOHTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTTPServerResponse&, DB::InterserverIOHTTPHandler::Output&) @ 0x17433c53 in /usr/bin/clickhouse
    17. DB::InterserverIOHTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x174344f1 in /usr/bin/clickhouse
    18. DB::HTTPServerConnection::run() @ 0x1768714d in /usr/bin/clickhouse
    19. Poco::Net::TCPServerConnection::start() @ 0x1a398093 in /usr/bin/clickhouse
    20. Poco::Net::TCPServerDispatcher::run() @ 0x1a399411 in /usr/bin/clickhouse
    21. Poco::PooledThread::run() @ 0x1a54b7bb in /usr/bin/clickhouse
    22. Poco::ThreadImpl::runnableEntry(void*) @ 0x1a548ec0 in /usr/bin/clickhouse
    23. ? @ 0x7fdf1c204609 in ?
    24. clone @ 0x7fdf1c129133 in ?
     (version 22.7.1.1781 (official build))

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/8b6e31cc615ca52c80724b6e5097777cb9514f07/stateless_tests__release__s3_storage__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 11:43:35 +03:00
Anton Popov
089009c996 fix reading from StorageLog with mmap 2022-07-13 20:36:18 +00:00
kssenii
615221f73b Minor changes 2022-07-13 19:22:47 +02:00
kssenii
0944183eec Fix merge 2022-07-13 16:50:31 +02:00
Azat Khuzhin
3720a3843c Add merge_algorithm to system.part_log 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-13 17:16:25 +03:00
Anton Popov
54a0985c42 randomize one more setting 2022-07-13 12:50:12 +00:00
FArthur-cmd
8ddc2578a2 refacroring 2022-07-13 14:48:02 +03:00
kssenii
0d2d3ac2b4 Merge master 2022-07-13 13:26:15 +02:00
Dmitry Novik
5f65b45269
Merge branch 'master' into group-by-use-nulls 2022-07-12 22:36:04 +02:00
Nikolai Kochetov
8efbe6d44d
Merge pull request #39119 from ClickHouse/removed-projection-and-old-part 
Properly remove projection from part in case it was removed from table metadata.
 2022-07-12 16:24:30 +02:00
Robert Schulze
69de9ee0e8
Merge pull request #39108 from ClickHouse/prefix_cmake_commands 
Prefix overridden add_executable() command with "clickhouse_"
 2022-07-12 15:05:48 +02:00
FArthur-cmd
6e2812490b refactoring 2022-07-12 14:29:11 +03:00
Alexander Tokmakov
c99fe6e6c8
Merge branch 'master' into removed-projection-and-old-part 2022-07-12 14:00:55 +03:00
FArthur-cmd
0e89fbff53 refactoring 2022-07-12 13:57:16 +03:00
Alexander Tokmakov
b35ad0dbdc
Merge pull request #39091 from ClickHouse/tavplubix-patch-1 
Follow-up to #39036
 2022-07-12 13:47:37 +03:00
Nikolai Kochetov
945b5d4b3f
Merge pull request #38993 from azat/fix-fsync_part_directory-on-fetch 
Fix fsync_part_directory for fetches
 2022-07-12 11:48:42 +02:00
Nikolai Kochetov
cd4211298e Fix typo. 2022-07-12 09:37:00 +00:00
Nikolai Kochetov
03d9301425 Fix log. 2022-07-12 08:31:09 +00:00
Nikolai Kochetov
631be9219b Properly remove projection from part in case it was removed from table metadata. 2022-07-12 08:28:22 +00:00
FArthur-cmd
783e6e398c merge with other working branch 2022-07-11 23:02:58 +03:00
FArthur-cmd
6ae227a6bb add test 2022-07-11 22:50:10 +03:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_" 
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
 2022-07-11 19:36:18 +02:00
FArthur-cmd
868628d3fd make stable prototype with arrays support 2022-07-11 18:02:50 +03:00
Alexander Tokmakov
e9ddbc247f
Merge pull request #39016 from Algunenano/clearOldLogs_zkerror 
clearOldLogs: Don't report KEEPER_EXCEPTION on concurrent deletes
 2022-07-11 15:16:18 +03:00
Alexander Tokmakov
048d27189e
Update MergeTreeData.cpp 2022-07-11 15:07:31 +03:00
FArthur-cmd
7b7a9521c7 change submodule path 2022-07-10 18:10:47 +03:00
FArthur-cmd
2f9f5f172a Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into annoy_3 2022-07-10 17:25:21 +03:00
Alexander Tokmakov
51130d723c
Update MergeTreeData.cpp 2022-07-08 20:17:29 +03:00
Raúl Marín
a965438eea clearOldLogs: Don't report errors on concurrent deletes 2022-07-08 13:57:53 +02:00
Azat Khuzhin
b2f7f9c0f0 Fix fsync_part_directory for fetches 
Before it produces the following error:

    2022.07.07 23:40:03.131044 [ 44869 ] {} <Error> default.rep_data: auto DB::StorageReplicatedMergeTree::processQueueEntry(ReplicatedMergeTreeQueue::SelectedEntryPtr)::(anonymous class)::operator()(DB::StorageReplicatedMergeTree::LogEntryPtr &) const: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/tmp-fetch_36_25458_25458_0, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

    0. Poco::Exception::Exception() @ 0x1764634c in /usr/bin/clickhouse
    1. DB::Exception::Exception() @ 0xb42caba in /usr/bin/clickhouse
    2. DB::throwFromErrnoWithPath() @ 0xb42d79b in /usr/bin/clickhouse
    3. DB::LocalDirectorySyncGuard::LocalDirectorySyncGuard() @ 0x1404bbd4 in /usr/bin/clickhouse
    4. DB::DiskLocal::getDirectorySyncGuard() const @ 0x140466a4 in /usr/bin/clickhouse
    5. DB::DataPartsExchange::Fetcher::downloadPartToDisk() @ 0x15122af1 in /usr/bin/clickhouse
    6. DB::DataPartsExchange::Fetcher::fetchPart() @ 0x1511de39 in /usr/bin/clickhouse

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-08 13:48:51 +03:00
Dmitry Novik
d1df66687b
Merge branch 'master' into group-by-use-nulls 2022-07-07 20:54:38 +02:00
Raúl Marín
33ffb71453 Avoid 'Node exists' errors when creating Ephemeral Locks 2022-07-07 16:10:04 +02:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
FArthur-cmd
8968ff8995 small inmprovements 2022-07-07 11:33:06 +03:00
Amos Bird
19bd3d8d9f
Nullable partition pruning fix 2022-07-07 16:02:38 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9 
Backup Improvements 9
 2022-07-07 02:24:47 +02:00
FArthur-cmd
a3407ba6db Merge branch 'master' of github.com:FArthur-cmd/ClickHouse into annoy-2 2022-07-06 19:42:55 +03:00
Vitaly Baranov
7f84cf3968 Fix style. 2022-07-06 16:36:59 +02:00
Maksim Kita
b94489d52c
Merge pull request #38859 from kitaisreal/merge-tree-merge-disable-batch-optimization 
MergeTree merge disable batch optimization
 2022-07-06 15:59:40 +02:00
FArthur-cmd
c94d6a57ec add arrays 2022-07-06 14:39:49 +03:00
Raúl Marín
86d1c319f4 Remove unnecessary log 2022-07-06 12:23:45 +02:00
Maksim Kita
bdc21737d5 MergeTree merge disable batch optimization 2022-07-05 16:15:00 +02:00
Vitaly Baranov
43d35eec1b Write unfinished mutations to backup. 2022-07-05 14:51:09 +02:00
kssenii
b52084265c Merge master 2022-07-04 21:37:43 +02:00
Dmitry Novik
c66e2c8095
Merge pull request #37155 from amosbird/projection-fix-two 
Fix intermediate header in aggregate projection
 2022-07-04 16:32:34 +02:00
FArthur-cmd
21df8ed12f refactoring 2022-07-04 17:21:50 +03:00
FArthur-cmd
21b99e7304 Merge branch 'master' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-07-04 14:54:21 +03:00
Vitaly Baranov
1dba3d9f9c Make temporary files in better places. 2022-07-04 11:02:58 +02:00
Vitaly Baranov
92e0ee0b6f More detailed error messages. 2022-07-03 14:20:19 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder 2022-07-02 15:39:44 +02:00
Dmitry Novik
81dd90893e Merge remote-tracking branch 'origin/master' into group-by-use-nulls 2022-07-01 16:24:05 +00:00
alesapin
0d7298f3a9 Fix typos 2022-07-01 15:26:27 +02:00
alesapin
57284e6a9d Fix possible deadlocks with MergeTreeData::Transaction 2022-07-01 15:16:32 +02:00
Vitaly Baranov
cadb496894
Merge pull request #38537 from vitlibar/backup-improvements-8 
Backup Improvements 8
 2022-07-01 11:20:46 +02:00
Amos Bird
d73f975849
Fix intermediate header in aggregate projection 2022-07-01 16:47:32 +08:00
alesapin
09faa1641e Some comments 2022-06-30 23:09:32 +02:00
alesapin
979565edf0 Merge branch 'master' into better_data_part_storage_builder 2022-06-30 22:54:42 +02:00
alesapin
eb5046ab26 Simplify everything 2022-06-30 22:51:27 +02:00
Alexander Tokmakov
d2cbdc7c53
Update ReplicatedMergeTreeQueue.cpp 2022-06-30 17:23:53 +03:00
Vitaly Baranov
e367d96964 Fix style. 2022-06-30 15:10:33 +02:00
alesapin
1363b478d4 Merge branch 'fix_disk_tx_write_op' into better_data_part_storage_builder 2022-06-30 14:38:32 +02:00
Anton Popov
7c721578c7
Merge pull request #38320 from CurtizJ/dynamic-columns-16 
Improve performace of insertion to columns of type JSON
 2022-06-30 14:18:20 +02:00
alesapin
cb90aca2ef Some changes in merge tree 2022-06-30 14:12:45 +02:00
Vitaly Baranov
031ca28fdc Add test for partition clause. More checks for data compatibility on restore. 2022-06-30 08:37:18 +02:00
Vitaly Baranov
01921ce9a3 Move most code from setTableStructure() to a separate function. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
7689e0c36f Improve gathering metadata for backup - part 6. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
47ac47350b Store projections in backups. 2022-06-30 08:37:17 +02:00
alesapin
67f2fa0ef0 Fix style 2022-06-29 15:18:48 +02:00
alesapin
2f03e821bd Fix stupid bug 2022-06-29 15:08:16 +02:00
alesapin
ac963a7663 Remove redundant check 2022-06-29 13:55:20 +02:00
Alexander Tokmakov
8246e55002
Merge pull request #38486 from azat/fix-parts-removal 
Fix parts removal after incorrect server shutdown
 2022-06-29 14:17:59 +03:00
Alexander Tokmakov
ceb66ade4b
Merge pull request #38335 from ClickHouse/deprecate_ordinary_database 
Deprecate Ordinary database and old *MergeTree syntax
 2022-06-29 13:42:59 +03:00
alesapin
c80a4c27be Merge branch 'master' into better_data_part_storage_builder 2022-06-29 12:33:42 +02:00
Alexey Milovidov
4cda5491f6
Merge pull request #38542 from ClickHouse/unique_lock-to-lock_guard 
Don't use std::unique_lock unless we need to
 2022-06-29 09:40:56 +03:00
Nikita Taranov
f5d26572df
Quick fix for aggregation pipeline (#38295) 2022-06-29 01:16:30 +02:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to 
Replace where possible by std::lock_guard which is more light-weight.
 2022-06-28 19:19:06 +00:00
alesapin
6429b72371 Fixes 2022-06-28 14:41:22 +02:00
alesapin
c7620b8217 Use builder instead of disk 2022-06-28 13:19:30 +02:00
alesapin
a68bf8fff2 Missed changes 2022-06-28 12:59:54 +02:00
alesapin
0a3fab1cb6 Some sad changes 2022-06-28 12:51:49 +02:00
alesapin
49add19842 Merge branch 'better_data_part_storage_builder' of github.com:ClickHouse/ClickHouse into better_data_part_storage_builder 2022-06-28 12:02:27 +02:00
alesapin
1a4aa8e52e Merge branch 'master' into better_data_part_storage_builder 2022-06-28 11:51:06 +02:00
Azat Khuzhin
e70cff2cea Fix parts removal after incorrect server shutdown 
Before this patch, and after transaction support had been added #24258,
if the server had not removed old parts on shutdown, then they will left
forever.

Fixes: #24258 (cc @tavplubix)
v2: use Tx::PrehistoricTID over removePartsFromWorkingSet() from ctor
v3: apply black to new test
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-27 23:56:25 +03:00
Azat Khuzhin
f0271dc486 Refactor deactive_part() a little 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-27 23:42:39 +03:00
alesapin
d88cdcd5c1 AAA 2022-06-27 21:41:29 +02:00
alesapin
180b4cd64d Merge branch 'master' into better_data_part_storage_builder 2022-06-27 19:46:17 +02:00
Alexander Tokmakov
88dbbcb15d fix tests 2022-06-27 16:10:00 +02:00
Alexander Tokmakov
7c37079a96 Merge branch 'master' into deprecate_ordinary_database 2022-06-27 14:46:59 +02:00
kssenii
5c26f99bc5 Merge master 2022-06-27 13:36:41 +02:00
Nikita Taranov
2487ba7f00
Move updateInputStream to ITransformingStep (#37393) 2022-06-27 13:16:52 +02:00
alesapin
83b584b2dd Something 2022-06-26 18:43:28 +02:00
alesapin
e2a779b315 Merge with new changes 2022-06-25 00:29:45 +02:00
alesapin
80d27f6381
Merge pull request #38380 from ClickHouse/simplify_parts_commit_merge_tree 
Simplify parts commit methods
 2022-06-25 00:12:52 +02:00
kssenii
708b93a03b Merge master 2022-06-25 00:08:24 +02:00
alesapin
8ccc47c202 Revert some changes 2022-06-24 23:54:10 +02:00
Alexander Tokmakov
31dcc7634e Merge branch 'master' into deprecate_ordinary_database 2022-06-24 18:16:07 +02:00
alesapin
d963f262f8 Fix style 2022-06-24 17:43:18 +02:00
alesapin
f685cf2268 Fix comment 2022-06-24 17:33:43 +02:00
alesapin
011d58d7a0 Simplify more 2022-06-24 17:19:59 +02:00
alesapin
612c4571d5 Split method into smaller 2022-06-24 15:41:09 +02:00
alesapin
7517e1f4d5 Remove some complexity 2022-06-24 15:24:02 +02:00
alesapin
c57e07cb49 Style 2022-06-24 14:51:47 +02:00
alesapin
37310dc9df Simpler 2022-06-24 14:10:15 +02:00
alesapin
36b810b076 Remove unused parameter 2022-06-24 13:42:36 +02:00
alesapin
af1a9d18ab Remove transaction argument 2022-06-24 13:34:00 +02:00
alesapin
9910395823 Simplify method signature 2022-06-24 13:19:29 +02:00
Alexander Tokmakov
dbf2763788 automatically convert system database to Atomic 2022-06-23 21:38:43 +02:00
alesapin
2bdedf5e0c Some changes 2022-06-23 18:21:46 +02:00
mergify[bot]
174d8577fa
Merge branch 'master' into merge_tree_reader 2022-06-23 15:16:51 +00:00
kssenii
f7b329ee57 Merge master 2022-06-23 14:56:48 +02:00
alesapin
5661280ef6 Remove some read methods 2022-06-23 14:01:26 +02:00
Alexander Tokmakov
f00e6b5a7a deprecate old MergeTree syntax 2022-06-23 11:24:54 +02:00
alesapin
1b495ec8ad Fix style 2022-06-23 00:17:37 +02:00
kssenii
e40d9bcf55 Merge master 2022-06-22 23:28:52 +02:00
Anton Popov
52db1b35a1 improve performace of insertion to columns of type JSON 2022-06-22 17:45:51 +00:00
Alexander Gololobov
0ee47363d4 Fixed includes 2022-06-22 19:08:18 +02:00
Alexander Gololobov
8138a3a36e Cleanups based on review comments 2022-06-22 19:08:01 +02:00
alesapin
ca0299e927 Ban projections for zero-copy replication in a right way 2022-06-22 19:01:46 +02:00
Alexander Gololobov
e5b55b965b Removed incorrect check 2022-06-22 17:23:09 +02:00
Alexander Gololobov
dbc6d1a159 Cleanups 2022-06-22 17:23:09 +02:00
Alexander Gololobov
aea5bc7527 Removed unused should_reorder flag 2022-06-22 17:23:09 +02:00
Alexander Gololobov
5427d5c51b Apply filter at row level security step 2022-06-22 17:23:09 +02:00
Alexander Gololobov
b3922461b3 Properly handle empty actions 2022-06-22 17:23:09 +02:00
Alexander Gololobov
ba89c3954c Do not add the same vitrual if it has been added by prev_reader 2022-06-22 17:23:09 +02:00
Alexander Gololobov
e5c0ebc7b5 Handle read from wide part when no columns are requested 2022-06-22 17:23:09 +02:00
Alexander Gololobov
a9e3b8d29e Don't read the same columns again 2022-06-22 17:23:09 +02:00
Alexander Gololobov
4e426c63cc Debuging test failures 2022-06-22 17:23:09 +02:00
Alexander Gololobov
6a26325fab Test dirty hacks for multiple PREWHERE steps 2022-06-22 17:23:05 +02:00
Alexander Gololobov
4360fd9798 Vector of structs instead of multiple vectors 2022-06-22 17:17:42 +02:00
Alexander Gololobov
87b669f439 Intermediate changes 2022-06-22 17:17:42 +02:00
Alexander Gololobov
159ab765fb Moved estimateNumRows from lambda to function 2022-06-22 17:17:42 +02:00
Alexander Gololobov
64a2f3734b Protect ReadResult internals from MergeTreeRangeReader clients 2022-06-22 17:17:42 +02:00
Nikolai Kochetov
cc6fdfe0eb
Merge pull request #36555 from ClickHouse/refactor-something-in-part-volumes 
Separate data storage abstraction for MergeTree
 2022-06-22 11:13:36 +02:00
kssenii
4178abc0a5 Merge master 2022-06-21 22:38:51 +02:00
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa 
Support for Clang Thread Safety Analysis (TSA)
 2022-06-21 20:19:33 +02:00
Nikolai Kochetov
dcf6e0d060 Roll back some changes from IPartMetadataManager 2022-06-21 13:05:49 +00:00
Nikolai Kochetov
854d148d73 Add some comments. Remove some commented code. 2022-06-21 12:31:02 +00:00
Nikolai Kochetov
46c4fd6cf7 Fixing build. 2022-06-21 11:15:06 +00:00
Kruglov Pavel
3c5a627bd7
Merge pull request #38239 from CurtizJ/fix-reading-from-s3 
Fix reading from s3 in some corner cases
 2022-06-21 12:57:55 +02:00
Nikolai Kochetov
1e8c9ecd4c Merge branch 'master' into refactor-something-in-part-volumes 2022-06-21 12:37:21 +02:00
alesapin
fafcb876be
Merge pull request #38182 from ClickHouse/disk_transaction 
Disk transaction
 2022-06-21 12:28:49 +02:00
Nikolai Kochetov
b8d27aa8dd
Merge pull request #37469 from azat/projections-optimize_aggregation_in_order 
Implement in order aggregation (optimize_aggregation_in_order) for projections for tables with fully materialized projections
 2022-06-21 12:17:35 +02:00
Nikolai Kochetov
e45d552f4d Do not create empty folder for in-memory parts. 2022-06-21 09:59:46 +00:00
Nikolai Kochetov
5232c17b89 Fix style 2022-06-21 07:30:45 +00:00
Nikolai Kochetov
3429a5e07a Fix 2022-06-21 07:26:43 +00:00
Nikolai Kochetov
dccf90b1ea Cleanup. 2022-06-20 18:18:17 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Anton Popov
59a9ce6def fix reading from s3 in some corner cases 2022-06-20 15:24:38 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
Nikolai Kochetov
39b02d4ab8 Continure resolving conflicts. 2022-06-20 15:39:05 +02:00
Nikolai Kochetov
7452d04e3a Merge branch 'master' into refactor-something-in-part-volumes 2022-06-20 15:31:02 +02:00
Nikolai Kochetov
ef8b967c3f Fix more tests. 2022-06-20 13:07:14 +00:00
kssenii
90be49faa4 Merge master 2022-06-20 14:26:56 +02:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6 
Backup Improvements 6
 2022-06-20 14:13:30 +02:00
alesapin
08b5dd091b Fix double negative 2022-06-19 17:12:04 +02:00
Vitaly Baranov
8a7c970ce0 Fix style. 2022-06-19 15:58:26 +02:00
alesapin
50801e41c5 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-19 14:05:46 +02:00
Alexander Tokmakov
83adf56383 fix race 2022-06-17 18:13:57 +02:00
Alexander Tokmakov
e9c2157f01 Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-17 15:57:30 +02:00
Alexander Tokmakov
5ffc38d9b9 fix style check, add comments 2022-06-17 13:47:14 +02:00
alesapin
e5b405701a Merge branch 'master' into disk_transaction 2022-06-17 12:16:12 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache 
Extract some diff from pr #36171
 2022-06-17 12:07:31 +02:00
Alexander Tokmakov
39c0219c11 fixes 2022-06-16 19:41:32 +02:00
Anton Popov
13ec7e3092
Merge pull request #37978 from CurtizJ/fix-sparse-s3 
Fix reading of sparse columns from s3
 2022-06-16 15:32:54 +02:00
alesapin
379fff7318 Merge branch 'master' into disk_transaction 2022-06-16 14:36:19 +02:00
Azat Khuzhin
4ff82eb9d5 Implement optimize_aggregation_in_order for projections 
v2: use real column name instead of aliases from GROUP BY
    Fixes the following error in 01710_projection_aggregation_in_order:

        Not found column a in block. There are only columns: toStartOfHour(ts), sum(value). (NOT_FOUND_COLUMN_IN_BLOCK)
v2.1: Get back support for projected and non-projected parts
v2.2: merge tests and rename
v3: Reduce copy-paste for optimize_aggregation_in_order for projections
v4: rebase on top of QueryPlanResourceHolder
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
14f9491619 MergeTreeData: preserve order for group_by_elements_order_descr 
This is required for proper optimize_aggregation_in_order for
projections.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
4694929623 Implement merging only for AggregatingStep 
v2: fill AggregateColumnsConstData only for only_merge
    (fixes 01291_aggregation_in_order and some other tests)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
682c93e2e1 Use proper setting compile_aggregate_expressions in MergeTreeDataSelectExecutor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:35 +03:00
Alexander Tokmakov
6ddfec47d2 maybe fix livelock on queue processing 2022-06-15 22:22:27 +02:00
Alexander Tokmakov
5e2b42c85a Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-15 20:47:02 +02:00
Vitaly Baranov
1198e86295 Fix storing temporary tables and skipping system tables while making a backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
21f3bed435 Simplify path calculations in backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
592f568f83 Move backup/restore code to storages and databases - part 2. 2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57 Move backup/restore code to storages and databases - part 1. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21 Rework collecting replicated parts. 2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Daniel Kutenin
a769dea8ef
Merge branch 'ClickHouse:master' into master 2022-06-15 19:12:10 +01:00
Nikolai Kochetov
25b9990737 Fixing build 2022-06-15 16:32:22 +00:00
Alexander Tokmakov
2ac72319bd
Merge pull request #37185 from amosbird/projection-fix-three 
Fix possible heap-use-after-free error when reading system.projection_parts and system.projection_parts_columns
 2022-06-15 19:00:10 +03:00
kssenii
416c6008ee Merge master 2022-06-15 16:50:25 +02:00
kssenii
500f49972b Extract diff from PR 36171 2022-06-15 16:40:18 +02:00
Alexander Tokmakov
e4d4d0a854 fix race on queue processing 2022-06-15 16:21:26 +02:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Danila Kutenin
08e3f77a9c Optimize most important parts with NEON SIMD 
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.

That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
 2022-06-15 13:19:29 +00:00
alesapin
1b66166501 Merge branch 'master' into disk_transaction 2022-06-15 13:54:10 +02:00
alesapin
b557c1dd80 Interm stage 2022-06-15 13:51:21 +02:00
Nikolai Kochetov
2ee349f29c Fix test_encrypted_disk 2022-06-15 11:42:59 +00:00
Nikolai Kochetov
a274bec0fa Fix test_encrypted_disk 2022-06-15 11:41:08 +00:00
Alexander Tokmakov
56f2121c1a
Revert "Add backoff to merges in replicated queue if always_fetch_merged_part is enabled" 2022-06-15 13:04:01 +03:00
alesapin
af1cd745e1
Merge pull request #37975 from kssenii/clean-up-broken-detached 
Clean up broken detached parts after timeout
 2022-06-14 20:53:31 +02:00
Amos Bird
d5a7a5be8e
Fix use-after-free in system.projection_parts 2022-06-14 23:41:42 +08:00
alesapin
3b832a25f6
Merge pull request #37995 from ClickHouse/fix_queue 
Add backoff to merges in replicated queue if `always_fetch_merged_part` is enabled
 2022-06-14 11:38:44 +02:00
kssenii
0d2dae0046 Add const, rename settings 2022-06-14 00:47:03 +02:00
kssenii
74cffaf186 Merge master 2022-06-14 00:45:19 +02:00
kssenii
4813b90281 Merge master 2022-06-13 21:32:00 +02:00
kssenii
34e63b20fb Review fixes 2022-06-13 21:27:54 +02:00
Anton Popov
e6517164f2 add test and comment 2022-06-13 17:30:58 +00:00
Antonio Andelic
b9cf6fe367 Merge branch 'master' into pp-system-unfreeze 2022-06-13 14:46:30 +00:00
alesapin
d984763b95
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-06-13 15:10:20 +02:00
kssenii
debe467ee7 Review fixes 2022-06-13 14:25:42 +02:00
kssenii
4859e11c27 Review fixes 2022-06-12 01:15:10 +02:00
alesapin
2a81904fbc Revert "Revert "Fix bug with removal of moving parts"" 
This reverts commit c0581bf741.
 2022-06-12 01:08:24 +02:00
kssenii
aaf9a46f07 Merge master 2022-06-11 15:56:05 +02:00
alesapin
c0581bf741 Revert "Fix bug with removal of moving parts" 
This reverts commit e32133ff8f.
 2022-06-10 19:35:57 +02:00
alesapin
301eac7d59 Comment 2022-06-10 19:34:03 +02:00
alesapin
ffff1c22a6 Add backoff for merges when always_fetch_merged_part turned on 2022-06-10 19:33:28 +02:00
alesapin
e32133ff8f Fix bug with removal of moving parts 2022-06-10 18:47:06 +02:00
kssenii
fd6c84b831 Review fixes 2022-06-10 14:23:13 +02:00
Anton Popov
6a0b305a14 fix reading of sparse columns from s3 2022-06-10 12:09:37 +00:00
kssenii
498f389c21 Merge master 2022-06-10 12:49:10 +02:00
kssenii
0fd2c4b2c0 Continuation 2022-06-10 12:33:29 +02:00
kssenii
7a2676c7ab Clean up broken detached parts with timeout 2022-06-10 12:27:57 +02:00
Kseniia Sumarokova
508676ab4e
Merge pull request #37804 from ClickHouse/metadata_storage 
Metadata storage abstraction for Disks
 2022-06-10 11:43:28 +02:00
Alexander Tokmakov
780f7c87c7
Merge pull request #36113 from ClickHouse/remove-useless-code-2 
Remove useless code in ReplicatedMergeTreeRestartingThread
 2022-06-09 17:56:28 +03:00
Vadim Volodin
637d293fbd Add SYSTEM UNFREEZE query 2022-06-08 15:21:14 +03:00
alesapin
81dca513de Review fixes 2022-06-08 13:28:33 +02:00
Nikolai Kochetov
5bc9b32025 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-08 11:10:06 +00:00
alesapin
b88e1a2bef Merge branch 'master' into metadata_storage 2022-06-08 13:02:21 +02:00
alesapin
9fdc783eac
Merge pull request #37906 from ClickHouse/remove_tmp_directories_on_replicated 
Remove temporary `delete_tmp_` on clickhouse server start
 2022-06-08 12:35:51 +02:00
alesapin
e32d36d790 Proper fix 2022-06-07 17:58:32 +02:00
Nikolai Kochetov
678d978acf Merge branch 'master' into refactor-something-in-part-volumes 2022-06-07 15:23:00 +00:00
mergify[bot]
2d01abf871
Merge branch 'master' into revert-37647-Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-07 13:32:30 +00:00
Nikita Taranov
d09ad5fcd5
Fix build 
https://s3.amazonaws.com/clickhouse-test-reports/36593/a4b9614385e0ecd1654d15f507159678758ec557/fast_test__actions_/build_log.txt
 2022-06-07 13:56:03 +02:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type" 2022-06-07 13:53:10 +02:00
Anton Popov
ef6f5a6500
Merge pull request #37570 from azat/column-ttl-expired-fix 
Do not write expired columns by TTL after subsequent merges
 2022-06-07 13:05:03 +02:00
mergify[bot]
dd0603a595
Merge branch 'master' into metadata_storage 2022-06-06 18:59:45 +00:00
Anton Popov
d40b23272e
Merge pull request #37755 from CurtizJ/fix-mutations-again 
Return back #37266
 2022-06-06 20:22:03 +02:00
Robert Schulze
2d87af2a15
Merge pull request #37647 from DevTeamBK/Fix-all-CheckTriviallyCopyableMove-Errors 
Fix errors of CheckTriviallyCopyableMove type
 2022-06-05 19:58:47 +02:00
alesapin
a25a01af16 Fix WAL 2022-06-05 16:15:11 +02:00
alesapin
38df94666b Fix merge tree wal 2022-06-05 15:09:39 +02:00
alesapin
a2704bc541 Trying to fix WAL log 2022-06-05 01:05:15 +02:00
Alexey Milovidov
f860407af5
Merge pull request #37777 from ClickHouse/avx512_tail_zero 
Use multitarget framework for numZerosInTail implementation
 2022-06-04 01:46:56 +03:00
Nikolai Kochetov
3f67abc3b0 Fix some tests. 2022-06-03 18:49:12 +00:00
mergify[bot]
ddf7210ecc
Merge branch 'master' into remove-useless-code-2 2022-06-03 13:58:45 +00:00
Maksim Kita
8469e0d10d
ExecutableLambdaAdapter forward job and callback (#37830) 2022-06-03 15:54:47 +02:00
alesapin
ac03e3af3e
Merge branch 'master' into metadata_storage 2022-06-03 14:42:21 +02:00
Anton Popov
5cf2c67410
Merge branch 'master' into fix-mutations-again 2022-06-03 14:17:05 +02:00
Nikolai Kochetov
8991f39412 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 17:00:08 +00:00
alesapin
46bccae078 Trying to create metadata layer 2022-06-02 18:09:40 +02:00
Nikolai Kochetov
25c8a437ea Fixing build. 2022-06-02 15:00:07 +00:00
HeenaBansal2009
e3080f2a97 Merge remote-tracking branch 'origin' into Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-02 07:30:08 -07:00
mergify[bot]
fbfaaa9c74
Merge branch 'master' into fix-mutations-again 2022-06-02 11:57:19 +00:00
Alexander Tokmakov
45657d203a
Merge pull request #37763 from ClickHouse/fix_replace_range_again 
Fix flaky 01154_move_partition_long
 2022-06-02 14:15:16 +03:00
Alexander Gololobov
0b004b907a Added AVX2 implementation of numZerosInTail 2022-06-02 13:01:13 +02:00
Nikolai Kochetov
fc726c236f
Merge pull request #37661 from ClickHouse/no_alias_actions 
Trying to remove prewhere_info.alias_actions as they seem to always be included in prewhere_actions
 2022-06-02 12:31:04 +02:00
Nikolai Kochetov
89c5855d20 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-02 12:19:07 +02:00
Nikita Mikhaylov
d34e051c69
Support for simultaneous read from local and remote parallel replica (#37204) 2022-06-02 11:46:33 +02:00
Nikolai Kochetov
edac3d6714 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 09:36:20 +00:00
Alexander Gololobov
d8e4dcf9e2 Use multitarget framework for numZerosInTail implementation 2022-06-02 11:23:44 +02:00
mergify[bot]
88bed9de20
Merge branch 'master' into fix_replace_range_again 2022-06-02 07:05:44 +00:00
Alexander Gololobov
a0cf902d49
Merge pull request #37588 from yaqi-zhao/avx512_tail_zero 
add avx512 support for mergetree reader
 2022-06-02 08:51:52 +03:00
lthaooo
6632616733
Fix TTL merge scheduling bug (#36387) 2022-06-01 21:09:53 +02:00
Alexander Tokmakov
06f80770b8 fix stuck REPALCE_RANGE 2022-06-01 20:11:53 +02:00
Sema Checherinda
2626a49616 FR: Expose what triggered the merge in system.part_log #26255 2022-06-01 16:58:06 +02:00
kssenii
aa60de8f63 Merge master 2022-06-01 15:32:01 +02:00
Anton Popov
cae2478b3f Revert "Merge pull request #37355 from ClickHouse/revert-37266-fix-mutations-with-object" 
This reverts commit a53cfa9fca, reversing
changes made to 9acb42fcdb.
 2022-06-01 10:57:20 +00:00
Alexander Gololobov
26609a1875 Style fixes 2022-05-31 21:41:10 +02:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00
Nikolai Kochetov
147a819221 Refactor a little bit more. 2022-05-31 14:43:38 +00:00
yaqi-zhao
a2857491c4 add avx512 support for mergetreereader 2022-05-30 20:53:00 -04:00
mergify[bot]
55913cf8e1
Merge branch 'master' into turn_on_s3_tests 2022-05-30 20:52:40 +00:00
kssenii
bdfd166eb1 Merge master 2022-05-30 21:31:47 +02:00
HeenaBansal2009
b7eb6bbd38 Fixed clang-tidy-CheckTriviallyCopyableMove-errors 2022-05-30 11:09:03 -07:00
Alexander Gololobov
e2dd6f6249 Removed prewhere_info.alias_actions 2022-05-30 19:58:23 +02:00
Nikolai Kochetov
5b4658aa5e Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-30 09:47:35 +00:00
alesapin
c32b6076fb Remove stranges from code 2022-05-30 00:12:33 +02:00
alesapin
0e8ab36913 Merge branch 'master' into turn_on_s3_tests 2022-05-29 14:37:10 +02:00
alesapin
d2cdbf3956 Fix refactoring issue 2022-05-29 14:09:49 +02:00
alesapin
9dc81e5cc8
Merge pull request #37598 from ClickHouse/revert-37545-revert-37424-fix_fetching_part_deadlock 
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part""
 2022-05-29 13:54:34 +02:00
Alexey Milovidov
97606c324c
Merge pull request #37574 from azat/mt-tiny-refactor 
Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm()
 2022-05-29 07:59:57 +03:00
Alexey Milovidov
73e2e63414
Merge pull request #37612 from ClickHouse/clang-tidy-14 
Fix clang-tidy-14, part 1
 2022-05-29 03:16:32 +03:00
Anton Popov
1d9b3be7da
Merge pull request #37536 from CurtizJ/profile-events-for-part-types 
Add profile events for introspection of part types
 2022-05-28 14:25:21 +02:00
Alexey Milovidov
c50791dd3b Fix clang-tidy-14, part 1 2022-05-27 22:52:14 +02:00
mergify[bot]
f5ee337bab
Merge branch 'master' into revert-37545-revert-37424-fix_fetching_part_deadlock 2022-05-27 16:52:00 +00:00
kssenii
25b1ed526d Merge master 2022-05-27 16:50:45 +02:00
alesapin
5a296aec01 Fix build 2022-05-27 16:34:16 +02:00
alesapin
be1c3c132b Fix some trash 2022-05-27 16:08:49 +02:00
alesapin
6d6779f17a
Merge pull request #37139 from ClickHouse/i_object_storage 
Separate object storage operations from disks
 2022-05-27 13:59:50 +02:00
alesapin
c79600c4c8 Fix build 2022-05-27 13:44:29 +02:00
alesapin
841858ec30
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part"" 2022-05-27 13:13:36 +02:00
Azat Khuzhin
c6c60364ae Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm() 
In favor of
MergeTask::ExecuteAndFinalizeHorizontalPart::chooseMergeAlgorithm()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 21:20:50 +03:00
Azat Khuzhin
73a99d4eee Improve error message for skipped/expired columns 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
f0cac5417a More optional skipping of fully expired columns (by TTL) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
46a94f395d Move the check into IMergedBlockOutputStream::removeEmptyColumnsFromPart() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
4288d09a85 Do not write expired columns by TTL after merge w/o TTL 
Usually second merge do not perform TTL, since everything is up to date,
however in this case TTLTransform is not used, and hence expired_columns
will not be filled for new part, and so those columns will be written
with default values.

Avoid this, by manually filling expired_columns.

Here is a simpler reproducer:

Simple reproducer:

    ```sql
    create table ttl_02262 (date Date, key Int, value String TTL date + interval 1 month) engine=MergeTree order by key settings min_bytes_for_wide_part=0, min_rows_for_wide_part=0;
    insert into ttl_02262 values ('2010-01-01', 2010, 'foo');
    ```

    ```sh
    # ls -l .server/data/default/ttl_02262/all_*
    .server/data/default/ttl_02262/all_1_1_0:
    total 48
    -rw-r----- 1 root root 335 May 26 14:19 checksums.txt
    -rw-r----- 1 root root  76 May 26 14:19 columns.txt
    -rw-r----- 1 root root   1 May 26 14:19 count.txt
    -rw-r----- 1 root root  28 May 26 14:19 date.bin
    -rw-r----- 1 root root  48 May 26 14:19 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:19 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:19 key.bin
    -rw-r----- 1 root root  48 May 26 14:19 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:19 primary.idx
    -rw-r----- 1 root root  99 May 26 14:19 ttl.txt
    -rw-r----- 1 root root  30 May 26 14:19 value.bin
    -rw-r----- 1 root root  48 May 26 14:19 value.mrk2
    ```

    ```sql
    optimize table ttl_02262 final;
    ```

    ```sh
    .server/data/default/ttl_02262/all_1_1_1:
    total 40
    -rw-r----- 1 root root 279 May 26 14:19 checksums.txt
    -rw-r----- 1 root root  61 May 26 14:19 columns.txt
    -rw-r----- 1 root root   1 May 26 14:19 count.txt
    -rw-r----- 1 root root  28 May 26 14:19 date.bin
    -rw-r----- 1 root root  48 May 26 14:19 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:19 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:19 key.bin
    -rw-r----- 1 root root  48 May 26 14:19 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:19 primary.idx
    -rw-r----- 1 root root  81 May 26 14:19 ttl.txt
    ```

    ```sql
    optimize table ttl_02262 final;
    ```

    ```sh
    .server/data/default/ttl_02262/all_1_1_2:
    total 48
    -rw-r----- 1 root root 349 May 26 14:20 checksums.txt
    -rw-r----- 1 root root  76 May 26 14:20 columns.txt
    -rw-r----- 1 root root   1 May 26 14:20 count.txt
    -rw-r----- 1 root root  28 May 26 14:20 date.bin
    -rw-r----- 1 root root  48 May 26 14:20 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:20 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:20 key.bin
    -rw-r----- 1 root root  48 May 26 14:20 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:20 primary.idx
    -rw-r----- 1 root root  81 May 26 14:20 ttl.txt
    -rw-r----- 1 root root  27 May 26 14:20 value.bin
    -rw-r----- 1 root root  48 May 26 14:20 value.mrk2
    ```

And now we have `value.*` for all_1_1_2, this should not happen.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
8328d7068b Fix updating of MergeTreeDataPartTTLInfo::finished 
Previously you cannot distinguish non-initialized finished with
initialized to false, so update() cannot do the correct thing.

Rename the field to avoid hidden usage.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
0de1a64436 Log empty parts in IMergedBlockOutputStream::removeEmptyColumnsFromPart() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:09 +03:00
Azat Khuzhin
dc9ca3d70c
Fix LOGICAL_ERROR in getMaxSourcePartsSizeForMerge during merges (#37413) 2022-05-26 14:14:58 +02:00
Nikolai Kochetov
84f97b53de Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-26 11:07:45 +00:00
Anton Popov
f488efd27e fix tests 2022-05-26 00:03:31 +00:00
alesapin
8f1aac0ce4 Fix merge with master 2022-05-26 00:44:45 +02:00
alesapin
1db9cf480b Merge remote-tracking branch 'origin/master' into i_object_storage 2022-05-25 22:50:22 +02:00
kssenii
50cd358c0e Merge master 2022-05-25 22:21:28 +02:00
Alexander Tokmakov
47820c216d
Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part" 2022-05-25 23:10:33 +03:00
alesapin
c7b16065e1 Merge with master 2022-05-25 21:47:05 +02:00
Nikolai Kochetov
6d4a26afac Update ReadProgressCallback. 2022-05-25 19:45:48 +00:00
kssenii
b89a4edc04 Merge master 2022-05-25 20:55:04 +02:00