thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-22 15:42:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into revive-sqlancer

Browse Source
This commit is contained in:
Ilya Yatsishin 2022-10-31 17:19:27 +01:00 committed by GitHub
commit 7c345f6094
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
123 changed files with 1722 additions and 1675 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

52
.github/workflows/nightly.yml vendored
View File

@ -122,3 +122,55 @@ jobs:
docker ps --quiet | xargs --no-run-if-empty docker kill ||:
docker ps --all --quiet | xargs --no-run-if-empty docker rm -f ||:
sudo rm -fr "$TEMP_PATH" "$CACHES_PATH"
SonarCloud:
runs-on: [self-hosted, builder]
env:
SONAR_SCANNER_VERSION: 4.7.0.2747
SONAR_SERVER_URL: "https://sonarcloud.io"
BUILD_WRAPPER_OUT_DIR: build_wrapper_output_directory # Directory where build-wrapper output will be placed
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
submodules: true
- name: Set up JDK 11
uses: actions/setup-java@v1
with:
java-version: 11
- name: Download and set up sonar-scanner
env:
SONAR_SCANNER_DOWNLOAD_URL: https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-${{ env.SONAR_SCANNER_VERSION }}-linux.zip
run: |
mkdir -p "$HOME/.sonar"
curl -sSLo "$HOME/.sonar/sonar-scanner.zip" "${{ env.SONAR_SCANNER_DOWNLOAD_URL }}"
unzip -o "$HOME/.sonar/sonar-scanner.zip" -d "$HOME/.sonar/"
echo "$HOME/.sonar/sonar-scanner-${{ env.SONAR_SCANNER_VERSION }}-linux/bin" >> "$GITHUB_PATH"
- name: Download and set up build-wrapper
env:
BUILD_WRAPPER_DOWNLOAD_URL: ${{ env.SONAR_SERVER_URL }}/static/cpp/build-wrapper-linux-x86.zip
run: |
curl -sSLo "$HOME/.sonar/build-wrapper-linux-x86.zip ${{ env.BUILD_WRAPPER_DOWNLOAD_URL }}"
unzip -o "$HOME/.sonar/build-wrapper-linux-x86.zip" -d "$HOME/.sonar/"
echo "$HOME/.sonar/build-wrapper-linux-x86" >> "$GITHUB_PATH"
- name: Set Up Build Tools
run: |
sudo apt-get update
sudo apt-get install -yq git cmake ccache python3 ninja-build
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
- name: Run build-wrapper
run: |
mkdir build
cd build
cmake ..
cd ..
build-wrapper-linux-x86-64 --out-dir ${{ env.BUILD_WRAPPER_OUT_DIR }} cmake --build build/
- name: Run sonar-scanner
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
run: |
sonar-scanner \
--define sonar.host.url="${{ env.SONAR_SERVER_URL }}" \
--define sonar.cfamily.build-wrapper-output="${{ env.BUILD_WRAPPER_OUT_DIR }}" \
--define sonar.projectKey="ClickHouse_ClickHouse" \
--define sonar.organization="clickhouse-java"

7
README.md
View File

@ -5,7 +5,7 @@ ClickHouse® is an open-source column-oriented database management system that a
## Useful Links
* [Official website](https://clickhouse.com/) has a quick high-level overview of ClickHouse on the main page.
* [ClickHouse Cloud](https://clickhouse.com/cloud) ClickHouse as a service, built by the creators and maintainers.
* [ClickHouse Cloud](https://clickhouse.cloud) ClickHouse as a service, built by the creators and maintainers.
* [Tutorial](https://clickhouse.com/docs/en/getting_started/tutorial/) shows how to set up and query a small ClickHouse cluster.
* [Documentation](https://clickhouse.com/docs/en/) provides more in-depth information.
* [YouTube channel](https://www.youtube.com/c/ClickHouseDB) has a lot of content about ClickHouse in video format.
@ -16,5 +16,6 @@ ClickHouse® is an open-source column-oriented database management system that a
* [Contacts](https://clickhouse.com/company/contact) can help to get your questions answered if there are any.
## Upcoming events
* [**v22.10 Release Webinar**](https://clickhouse.com/company/events/v22-10-release-webinar) Original creator, co-founder, and CTO of ClickHouse Alexey Milovidov will walk us through the highlights of the release, provide live demos, and share vision into what is coming in the roadmap.
* [**Introducing ClickHouse Cloud**](https://clickhouse.com/company/events/cloud-beta) Introducing ClickHouse as a service, built by creators and maintainers of the fastest OLAP database on earth. Join Tanya Bragin for a detailed walkthrough of ClickHouse Cloud capabilities, as well as a peek behind the curtain to understand the unique architecture that makes our service tick.
* [**v22.11 Release Webinar**](https://clickhouse.com/company/events/v22-11-release-webinar) Original creator, co-founder, and CTO of ClickHouse Alexey Milovidov will walk us through the highlights of the release, provide live demos, and share vision into what is coming in the roadmap.
* [**ClickHouse Meetup at the Deutsche Bank office in Berlin**](https://www.meetup.com/clickhouse-berlin-user-group/events/289311596/) Hear from Deutsche Bank on why they chose ClickHouse for big sensitive data in a regulated environment. The ClickHouse team will then present how ClickHouse is used for real time financial data analytics, including tick data, trade analytics and risk management.
* [**AWS re:Invent**](https://clickhouse.com/company/events/aws-reinvent) Core members of the ClickHouse team -- including 2 of our founders -- will be at re:Invent from November 29 to December 3. We are available on the show floor, but are also determining interest in holding an event during the time there.

26
docs/changelogs/v22.3.14.18-lts.md Normal file
View File

@ -0,0 +1,26 @@
---
sidebar_position: 1
sidebar_label: 2022
---
# 2022 Changelog
### ClickHouse release v22.3.14.18-lts (642946f61b2) FIXME as compared to v22.3.13.80-lts (e2708b01fba)
#### Bug Fix
* Backported in [#42432](https://github.com/ClickHouse/ClickHouse/issues/42432): - Choose correct aggregation method for LowCardinality with BigInt. [#42342](https://github.com/ClickHouse/ClickHouse/pull/42342) ([Duc Canh Le](https://github.com/canhld94)).
#### Build/Testing/Packaging Improvement
* Backported in [#42328](https://github.com/ClickHouse/ClickHouse/issues/42328): Update cctz to the latest master, update tzdb to 2020e. [#42273](https://github.com/ClickHouse/ClickHouse/pull/42273) ([Dom Del Nano](https://github.com/ddelnano)).
* Backported in [#42358](https://github.com/ClickHouse/ClickHouse/issues/42358): Update tzdata to 2022e to support the new timezone changes. Palestine transitions are now Saturdays at 02:00. Simplify three Ukraine zones into one. Jordan and Syria switch from +02/+03 with DST to year-round +03. (https://data.iana.org/time-zones/tzdb/NEWS). This closes [#42252](https://github.com/ClickHouse/ClickHouse/issues/42252). [#42327](https://github.com/ClickHouse/ClickHouse/pull/42327) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
#### Bug Fix (user-visible misbehavior in official stable or prestable release)
* Backported in [#42298](https://github.com/ClickHouse/ClickHouse/issues/42298): Fix a bug with projections and the `aggregate_functions_null_for_empty` setting. This bug is very rare and appears only if you enable the `aggregate_functions_null_for_empty` setting in the server's config. This closes [#41647](https://github.com/ClickHouse/ClickHouse/issues/41647). [#42198](https://github.com/ClickHouse/ClickHouse/pull/42198) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
* Backported in [#42592](https://github.com/ClickHouse/ClickHouse/issues/42592): This closes [#42453](https://github.com/ClickHouse/ClickHouse/issues/42453). [#42573](https://github.com/ClickHouse/ClickHouse/pull/42573) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
#### NOT FOR CHANGELOG / INSIGNIFICANT
* Add a warning message to release.py script, require release type [#41975](https://github.com/ClickHouse/ClickHouse/pull/41975) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
* Revert [#27787](https://github.com/ClickHouse/ClickHouse/issues/27787) [#42136](https://github.com/ClickHouse/ClickHouse/pull/42136) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).

29
docs/changelogs/v22.3.14.23-lts.md Normal file
View File

@ -0,0 +1,29 @@
---
sidebar_position: 1
sidebar_label: 2022
---
# 2022 Changelog
### ClickHouse release v22.3.14.23-lts (74956bfee4d) FIXME as compared to v22.3.13.80-lts (e2708b01fba)
#### Improvement
* Backported in [#42527](https://github.com/ClickHouse/ClickHouse/issues/42527): Fix issue with passing MySQL timeouts for MySQL database engine and MySQL table function. Closes [#34168](https://github.com/ClickHouse/ClickHouse/issues/34168)?notification_referrer_id=NT_kwDOAzsV57MzMDMxNjAzNTY5OjU0MjAzODc5. [#40751](https://github.com/ClickHouse/ClickHouse/pull/40751) ([Kseniia Sumarokova](https://github.com/kssenii)).
#### Bug Fix
* Backported in [#42432](https://github.com/ClickHouse/ClickHouse/issues/42432): - Choose correct aggregation method for LowCardinality with BigInt. [#42342](https://github.com/ClickHouse/ClickHouse/pull/42342) ([Duc Canh Le](https://github.com/canhld94)).
#### Build/Testing/Packaging Improvement
* Backported in [#42328](https://github.com/ClickHouse/ClickHouse/issues/42328): Update cctz to the latest master, update tzdb to 2020e. [#42273](https://github.com/ClickHouse/ClickHouse/pull/42273) ([Dom Del Nano](https://github.com/ddelnano)).
* Backported in [#42358](https://github.com/ClickHouse/ClickHouse/issues/42358): Update tzdata to 2022e to support the new timezone changes. Palestine transitions are now Saturdays at 02:00. Simplify three Ukraine zones into one. Jordan and Syria switch from +02/+03 with DST to year-round +03. (https://data.iana.org/time-zones/tzdb/NEWS). This closes [#42252](https://github.com/ClickHouse/ClickHouse/issues/42252). [#42327](https://github.com/ClickHouse/ClickHouse/pull/42327) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
#### Bug Fix (user-visible misbehavior in official stable or prestable release)
* Backported in [#42298](https://github.com/ClickHouse/ClickHouse/issues/42298): Fix a bug with projections and the `aggregate_functions_null_for_empty` setting. This bug is very rare and appears only if you enable the `aggregate_functions_null_for_empty` setting in the server's config. This closes [#41647](https://github.com/ClickHouse/ClickHouse/issues/41647). [#42198](https://github.com/ClickHouse/ClickHouse/pull/42198) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
* Backported in [#42592](https://github.com/ClickHouse/ClickHouse/issues/42592): This closes [#42453](https://github.com/ClickHouse/ClickHouse/issues/42453). [#42573](https://github.com/ClickHouse/ClickHouse/pull/42573) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
#### NOT FOR CHANGELOG / INSIGNIFICANT
* Add a warning message to release.py script, require release type [#41975](https://github.com/ClickHouse/ClickHouse/pull/41975) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
* Revert [#27787](https://github.com/ClickHouse/ClickHouse/issues/27787) [#42136](https://github.com/ClickHouse/ClickHouse/pull/42136) ([Nikolai Kochetov](https://github.com/KochetovNicolai)).

6
docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-polygon.md
View File

@ -14,8 +14,10 @@ Example of a polygon dictionary configuration:
<dictionary>
<structure>
<key>
<name>key</name>
<type>Array(Array(Array(Array(Float64))))</type>
<attribute>
<name>key</name>
<type>Array(Array(Array(Array(Float64))))</type>
</attribute>
</key>
<attribute>

4
docs/en/sql-reference/functions/date-time-functions.md
View File

@ -1068,7 +1068,7 @@ Example:
SELECT timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600));
SELECT timeSlots(toDateTime('1980-12-12 21:01:02', 'UTC'), toUInt32(600), 299);
SELECT timeSlots(toDateTime64('1980-12-12 21:01:02.1234', 4, 'UTC'), toDecimal64(600.1, 1), toDecimal64(299, 0));
```
```
``` text
┌─timeSlots(toDateTime('2012-01-01 12:20:00'), toUInt32(600))─┐
│ ['2012-01-01 12:00:00','2012-01-01 12:30:00'] │
@ -1244,7 +1244,7 @@ Result:
└──────────────────────────┘
```
When there are two arguments: first is an [Integer](../../sql-reference/data-types/int-uint.md) or [DateTime](../../sql-reference/data-types/datetime.md), second is a constant format string — it acts in the same way as [formatDateTime](#formatdatetime) and return [String](../../sql-reference/data-types/string.md#string) type.
When there are two or three arguments, the first an [Integer](../../sql-reference/data-types/int-uint.md), [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) or [DateTime64](../../sql-reference/data-types/datetime64.md), the second a constant format string and the third an optional constant time zone string — it acts in the same way as [formatDateTime](#formatdatetime) and return [String](../../sql-reference/data-types/string.md#string) type.
For example:

5
docs/en/sql-reference/statements/alter/column.md
View File

@ -107,7 +107,7 @@ ALTER TABLE visits RENAME COLUMN webBrowser TO browser
CLEAR COLUMN [IF EXISTS] name IN PARTITION partition_name
```
Resets all data in a column for a specified partition. Read more about setting the partition name in the section [How to specify the partition expression](#alter-how-to-specify-part-expr).
Resets all data in a column for a specified partition. Read more about setting the partition name in the section [How to set the partition expression](partition.md#how-to-set-partition-expression).
If the `IF EXISTS` clause is specified, the query won’t return an error if the column does not exist.
@ -204,8 +204,9 @@ It is used if it is necessary to add or update a column with a complicated expre
Syntax:
```sql
ALTER TABLE table MATERIALIZE COLUMN col;
ALTER TABLE [db.]table [ON CLUSTER cluster] MATERIALIZE COLUMN col [IN PARTITION partition | IN PARTITION ID 'partition_id'];
```
- If you specify a PARTITION, a column will be materialized with only the specified partition.
**Example**

10
docs/en/sql-reference/statements/alter/partition.md
View File

@ -39,7 +39,7 @@ ALTER TABLE mt DETACH PARTITION '2020-11-21';
ALTER TABLE mt DETACH PART 'all_2_2_0';
```
Read about setting the partition expression in a section [How to specify the partition expression](#alter-how-to-specify-part-expr).
Read about setting the partition expression in a section [How to set the partition expression](#how-to-set-partition-expression).
After the query is executed, you can do whatever you want with the data in the `detached` directory — delete it from the file system, or just leave it.
@ -53,7 +53,7 @@ ALTER TABLE table_name [ON CLUSTER cluster] DROP PARTITION|PART partition_expr
Deletes the specified partition from the table. This query tags the partition as inactive and deletes data completely, approximately in 10 minutes.
Read about setting the partition expression in a section [How to specify the partition expression](#alter-how-to-specify-part-expr).
Read about setting the partition expression in a section [How to set the partition expression](#how-to-set-partition-expression).
The query is replicated – it deletes data on all replicas.
@ -71,7 +71,7 @@ ALTER TABLE table_name [ON CLUSTER cluster] DROP DETACHED PARTITION|PART partiti
```
Removes the specified part or all parts of the specified partition from `detached`.
Read more about setting the partition expression in a section [How to specify the partition expression](#alter-how-to-specify-part-expr).
Read more about setting the partition expression in a section [How to set the partition expression](#how-to-set-partition-expression).
## ATTACH PARTITION\|PART
@ -86,7 +86,7 @@ ALTER TABLE visits ATTACH PARTITION 201901;
ALTER TABLE visits ATTACH PART 201901_2_2_0;
```
Read more about setting the partition expression in a section [How to specify the partition expression](#alter-how-to-specify-part-expr).
Read more about setting the partition expression in a section [How to set the partition expression](#how-to-set-partition-expression).
This query is replicated. The replica-initiator checks whether there is data in the `detached` directory.
If data exists, the query checks its integrity. If everything is correct, the query adds the data to the table.
@ -166,7 +166,7 @@ This query creates a local backup of a specified partition. If the `PARTITION` c
The entire backup process is performed without stopping the server.
:::
Note that for old-styled tables you can specify the prefix of the partition name (for example, `2019`) - then the query creates the backup for all the corresponding partitions. Read about setting the partition expression in a section [How to specify the partition expression](#alter-how-to-specify-part-expr).
Note that for old-styled tables you can specify the prefix of the partition name (for example, `2019`) - then the query creates the backup for all the corresponding partitions. Read about setting the partition expression in a section [How to set the partition expression](#how-to-set-partition-expression).
At the time of execution, for a data snapshot, the query creates hardlinks to a table data. Hardlinks are placed in the directory `/var/lib/clickhouse/shadow/N/...`, where:

2
docs/en/sql-reference/statements/create/database.md
View File

@ -31,7 +31,7 @@ By default, ClickHouse uses its own [Atomic](../../../engines/database-engines/a
### COMMENT
You can add a comment to the database when you creating it.
You can add a comment to the database when you are creating it.
The comment is supported for all database engines.

2
docs/en/sql-reference/statements/optimize.md
View File

@ -22,7 +22,7 @@ The `OPTIMIZE` query is supported for [MergeTree](../../engines/table-engines/me
When `OPTIMIZE` is used with the [ReplicatedMergeTree](../../engines/table-engines/mergetree-family/replication.md) family of table engines, ClickHouse creates a task for merging and waits for execution on all replicas (if the [replication_alter_partitions_sync](../../operations/settings/settings.md#replication-alter-partitions-sync) setting is set to `2`) or on current replica (if the [replication_alter_partitions_sync](../../operations/settings/settings.md#replication-alter-partitions-sync) setting is set to `1`).
- If `OPTIMIZE` does not perform a merge for any reason, it does not notify the client. To enable notifications, use the [optimize_throw_if_noop](../../operations/settings/settings.md#setting-optimize_throw_if_noop) setting.
- If you specify a `PARTITION`, only the specified partition is optimized. [How to set partition expression](../../sql-reference/statements/alter/index.md#alter-how-to-specify-part-expr).
- If you specify a `PARTITION`, only the specified partition is optimized. [How to set partition expression](alter/partition.md#how-to-set-partition-expression).
- If you specify `FINAL`, optimization is performed even when all the data is already in one part. Also merge is forced even if concurrent merges are performed.
- If you specify `DEDUPLICATE`, then completely identical rows (unless by-clause is specified) will be deduplicated (all columns are compared), it makes sense only for the MergeTree engine.

3
docs/ru/sql-reference/functions/date-time-functions.md
View File

@ -1126,8 +1126,7 @@ SELECT FROM_UNIXTIME(423543535);
└──────────────────────────┘
```
В случае, когда есть два аргумента: первый типа [Integer](../../sql-reference/data-types/int-uint.md) или [DateTime](../../sql-reference/data-types/datetime.md), а второй является строкой постоянного формата — функция работает также, как [formatDateTime](#formatdatetime), и возвращает значение типа [String](../../sql-reference/data-types/string.md#string).
В случае, когда есть два или три аргумента: первый типа [Integer](../../sql-reference/data-types/int-uint.md), [Date](../../sql-reference/data-types/date.md), [Date32](../../sql-reference/data-types/date32.md), [DateTime](../../sql-reference/data-types/datetime.md) или [DateTime64](../../sql-reference/data-types/datetime64.md), а второй является строкой постоянного формата и третий является строкой постоянной временной зоны — функция работает также, как [formatDateTime](#formatdatetime), и возвращает значение типа [String](../../sql-reference/data-types/string.md#string).
Запрос:

27
src/Analyzer/Passes/OrderByLimitByDuplicateEliminationPass.cpp
View File

@ -10,23 +10,34 @@ namespace DB
namespace
{
struct QueryTreeNodeHash
struct QueryTreeNodeWithHash
{
size_t operator()(const IQueryTreeNode * node) const
explicit QueryTreeNodeWithHash(const IQueryTreeNode * node_)
: node(node_)
, hash(node->getTreeHash().first)
{}
const IQueryTreeNode * node = nullptr;
size_t hash = 0;
};
struct QueryTreeNodeWithHashHash
{
size_t operator()(const QueryTreeNodeWithHash & node_with_hash) const
{
return node->getTreeHash().first;
return node_with_hash.hash;
}
};
struct QueryTreeNodeEqualTo
struct QueryTreeNodeWithHashEqualTo
{
size_t operator()(const IQueryTreeNode * lhs_node, const IQueryTreeNode * rhs_node) const
bool operator()(const QueryTreeNodeWithHash & lhs_node, const QueryTreeNodeWithHash & rhs_node) const
{
return lhs_node->isEqual(*rhs_node);
return lhs_node.hash == rhs_node.hash && lhs_node.node->isEqual(*rhs_node.node);
}
};
using QueryTreeNodeSet = std::unordered_set<const IQueryTreeNode *, QueryTreeNodeHash, QueryTreeNodeEqualTo>;
using QueryTreeNodeWithHashSet = std::unordered_set<QueryTreeNodeWithHash, QueryTreeNodeWithHashHash, QueryTreeNodeWithHashEqualTo>;
class OrderByLimitByDuplicateEliminationVisitor : public InDepthQueryTreeVisitor<OrderByLimitByDuplicateEliminationVisitor>
{
@ -82,7 +93,7 @@ public:
}
private:
QueryTreeNodeSet unique_expressions_nodes_set;
QueryTreeNodeWithHashSet unique_expressions_nodes_set;
};
}

2
src/Common/CaresPTRResolver.cpp
View File

@ -128,7 +128,7 @@ namespace DB
int number_of_fds_ready = 0;
if (!readable_sockets.empty())
{
number_of_fds_ready = poll(readable_sockets.data(), readable_sockets.size(), timeout);
number_of_fds_ready = poll(readable_sockets.data(), static_cast<nfds_t>(readable_sockets.size()), static_cast<int>(timeout));
}
if (number_of_fds_ready > 0)

8
src/Common/MemoryTracker.cpp
View File

@ -10,6 +10,7 @@
#include <Common/ProfileEvents.h>
#include <Common/thread_local_rng.h>
#include <Common/OvercommitTracker.h>
#include <Common/Stopwatch.h>
#include <Common/logger_useful.h>
#include "config.h"
@ -86,6 +87,8 @@ inline std::string_view toDescription(OvercommitResult result)
namespace ProfileEvents
{
extern const Event QueryMemoryLimitExceeded;
extern const Event MemoryAllocatorPurge;
extern const Event MemoryAllocatorPurgeTimeMicroseconds;
}
using namespace std::chrono_literals;
@ -229,7 +232,10 @@ void MemoryTracker::allocImpl(Int64 size, bool throw_if_memory_exceeded, MemoryT
{
if (free_memory_in_allocator_arenas.exchange(-current_free_memory_in_allocator_arenas) > 0)
{
Stopwatch watch;
mallctl("arena." STRINGIFY(MALLCTL_ARENAS_ALL) ".purge", nullptr, nullptr, nullptr, 0);
ProfileEvents::increment(ProfileEvents::MemoryAllocatorPurge);
ProfileEvents::increment(ProfileEvents::MemoryAllocatorPurgeTimeMicroseconds, watch.elapsedMicroseconds());
}
}
@ -432,7 +438,7 @@ void MemoryTracker::reset()
void MemoryTracker::setRSS(Int64 rss_, Int64 free_memory_in_allocator_arenas_)
{
Int64 new_amount = rss_; // - free_memory_in_allocator_arenas_;
Int64 new_amount = rss_;
total_memory_tracker.amount.store(new_amount, std::memory_order_relaxed);
free_memory_in_allocator_arenas.store(free_memory_in_allocator_arenas_, std::memory_order_relaxed);

2
src/Common/ProfileEvents.cpp
View File

@ -229,6 +229,8 @@ The server successfully detected this situation and will download merged part fr
M(UserTimeMicroseconds, "Total time spent in processing (queries and other tasks) threads executing CPU instructions in user space. This include time CPU pipeline was stalled due to cache misses, branch mispredictions, hyper-threading, etc.") \
M(SystemTimeMicroseconds, "Total time spent in processing (queries and other tasks) threads executing CPU instructions in OS kernel space. This include time CPU pipeline was stalled due to cache misses, branch mispredictions, hyper-threading, etc.") \
M(MemoryOvercommitWaitTimeMicroseconds, "Total time spent in waiting for memory to be freed in OvercommitTracker.") \
M(MemoryAllocatorPurge, "Total number of times memory allocator purge was requested") \
M(MemoryAllocatorPurgeTimeMicroseconds, "Total number of times memory allocator purge was requested") \
M(SoftPageFaults, "The number of soft page faults in query execution threads. Soft page fault usually means a miss in the memory allocator cache which required a new memory mapping from the OS and subsequent allocation of a page of physical memory.") \
M(HardPageFaults, "The number of hard page faults in query execution threads. High values indicate either that you forgot to turn off swap on your server, or eviction of memory pages of the ClickHouse binary during very high memory pressure, or successful usage of the 'mmap' read method for the tables data.") \
\

82
src/Common/Throttler.cpp
View File

@ -3,7 +3,6 @@
#include <Common/Exception.h>
#include <Common/Stopwatch.h>
#include <IO/WriteHelpers.h>
#include <cmath>
namespace ProfileEvents
{
@ -21,63 +20,56 @@ namespace ErrorCodes
/// Just 10^9.
static constexpr auto NS = 1000000000UL;
/// Tracking window. Actually the size is not really important. We just want to avoid
/// throttles when there are no actions for a long period time.
static const double window_ns = 1ULL * NS;
static const size_t default_burst_seconds = 1;
Throttler::Throttler(size_t max_speed_, const std::shared_ptr<Throttler> & parent_)
: max_speed(max_speed_)
, max_burst(max_speed_ * default_burst_seconds)
, limit_exceeded_exception_message("")
, tokens(max_burst)
, parent(parent_)
{}
Throttler::Throttler(size_t max_speed_, size_t limit_, const char * limit_exceeded_exception_message_,
const std::shared_ptr<Throttler> & parent_)
: max_speed(max_speed_)
, max_burst(max_speed_ * default_burst_seconds)
, limit(limit_)
, limit_exceeded_exception_message(limit_exceeded_exception_message_)
, tokens(max_burst)
, parent(parent_)
{}
void Throttler::add(size_t amount)
{
size_t new_count;
/// This outer variable is always equal to smoothed_speed.
/// We use to avoid race condition.
double current_speed = 0;
// Values obtained under lock to be checked after release
size_t count_value;
double tokens_value;
{
std::lock_guard lock(mutex);
auto now = clock_gettime_ns_adjusted(prev_ns);
/// If prev_ns is equal to zero (first `add` call) we known nothing about speed
/// and don't track anything.
if (max_speed && prev_ns != 0)
if (max_speed)
{
/// Time spent to process the amount of bytes
double time_spent = now - prev_ns;
/// The speed in bytes per second is equal to amount / time_spent in seconds
auto new_speed = amount / (time_spent / NS);
/// We want to make old values of speed less important for our smoothed value
/// so we decay it's value with coef.
auto decay_coeff = std::pow(0.5, time_spent / window_ns);
/// Weighted average between previous and new speed
smoothed_speed = smoothed_speed * decay_coeff + (1 - decay_coeff) * new_speed;
current_speed = smoothed_speed;
double delta_seconds = prev_ns ? static_cast<double>(now - prev_ns) / NS : 0;
tokens = std::min<double>(tokens + max_speed * delta_seconds - amount, max_burst);
}
count += amount;
new_count = count;
count_value = count;
tokens_value = tokens;
prev_ns = now;
}
if (limit && new_count > limit)
if (limit && count_value > limit)
throw Exception(limit_exceeded_exception_message + std::string(" Maximum: ") + toString(limit), ErrorCodes::LIMIT_EXCEEDED);
if (max_speed && current_speed > max_speed)
/// Wait unless there is positive amount of tokens - throttling
if (max_speed && tokens_value < 0)
{
/// If we was too fast then we have to sleep until our smoothed speed became <= max_speed
int64_t sleep_time = static_cast<int64_t>(-window_ns * std::log2(max_speed / current_speed));
if (sleep_time > 0)
{
accumulated_sleep += sleep_time;
sleepForNanoseconds(sleep_time);
accumulated_sleep -= sleep_time;
ProfileEvents::increment(ProfileEvents::ThrottlerSleepMicroseconds, sleep_time / 1000UL);
}
int64_t sleep_time = static_cast<int64_t>(-tokens_value / max_speed * NS);
accumulated_sleep += sleep_time;
sleepForNanoseconds(sleep_time);
accumulated_sleep -= sleep_time;
ProfileEvents::increment(ProfileEvents::ThrottlerSleepMicroseconds, sleep_time / 1000UL);
}
if (parent)
@ -89,9 +81,9 @@ void Throttler::reset()
std::lock_guard lock(mutex);
count = 0;
accumulated_sleep = 0;
smoothed_speed = 0;
tokens = max_burst;
prev_ns = 0;
// NOTE: do not zero `accumulated_sleep` to avoid races
}
bool Throttler::isThrottling() const

38
src/Common/Throttler.h
View File

@ -10,25 +10,26 @@
namespace DB
{
/** Allows you to limit the speed of something (in entities per second) using sleep.
* Specifics of work:
* Tracks exponentially (pow of 1/2) smoothed speed with hardcoded window.
* See more comments in .cpp file.
*
* Also allows you to set a limit on the maximum number of entities. If exceeded, an exception will be thrown.
/** Allows you to limit the speed of something (in tokens per second) using sleep.
* Implemented using Token Bucket Throttling algorithm.
* Also allows you to set a limit on the maximum number of tokens. If exceeded, an exception will be thrown.
*/
class Throttler
{
public:
explicit Throttler(size_t max_speed_, const std::shared_ptr<Throttler> & parent_ = nullptr)
: max_speed(max_speed_), limit_exceeded_exception_message(""), parent(parent_) {}
Throttler(size_t max_speed_, size_t max_burst_, const std::shared_ptr<Throttler> & parent_ = nullptr)
: max_speed(max_speed_), max_burst(max_burst_), limit_exceeded_exception_message(""), tokens(max_burst), parent(parent_) {}
explicit Throttler(size_t max_speed_, const std::shared_ptr<Throttler> & parent_ = nullptr);
Throttler(size_t max_speed_, size_t max_burst_, size_t limit_, const char * limit_exceeded_exception_message_,
const std::shared_ptr<Throttler> & parent_ = nullptr)
: max_speed(max_speed_), max_burst(max_burst_), limit(limit_), limit_exceeded_exception_message(limit_exceeded_exception_message_), tokens(max_burst), parent(parent_) {}
Throttler(size_t max_speed_, size_t limit_, const char * limit_exceeded_exception_message_,
const std::shared_ptr<Throttler> & parent_ = nullptr)
: max_speed(max_speed_), limit(limit_), limit_exceeded_exception_message(limit_exceeded_exception_message_), parent(parent_) {}
const std::shared_ptr<Throttler> & parent_ = nullptr);
/// Calculates the smoothed speed, sleeps if required and throws exception on
/// limit overflow.
/// Use `amount` tokens, sleeps if required or throws exception on limit overflow.
void add(size_t amount);
/// Not thread safe
@ -45,15 +46,14 @@ public:
private:
size_t count{0};
const size_t max_speed{0};
const uint64_t limit{0}; /// 0 - not limited.
const size_t max_speed{0}; /// in tokens per second.
const size_t max_burst{0}; /// in tokens.
const uint64_t limit{0}; /// 0 - not limited.
const char * limit_exceeded_exception_message = nullptr;
std::mutex mutex;
std::atomic<uint64_t> accumulated_sleep{0};
/// Smoothed value of current speed. Updated in `add` method.
double smoothed_speed{0};
/// previous `add` call time (in nanoseconds)
uint64_t prev_ns{0};
std::atomic<uint64_t> accumulated_sleep{0}; // Accumulated sleep time over all waiting threads
double tokens{0}; /// Amount of tokens available in token bucket. Updated in `add` method.
uint64_t prev_ns{0}; /// Previous `add` call time (in nanoseconds).
/// Used to implement a hierarchy of throttlers
std::shared_ptr<Throttler> parent;

4
src/Compression/getCompressionCodecForFile.cpp
View File

@ -13,9 +13,9 @@ namespace DB
using Checksum = CityHash_v1_0_2::uint128;
CompressionCodecPtr getCompressionCodecForFile(const DataPartStoragePtr & data_part_storage, const String & relative_path)
CompressionCodecPtr getCompressionCodecForFile(const IDataPartStorage & data_part_storage, const String & relative_path)
{
auto read_buffer = data_part_storage->readFile(relative_path, {}, std::nullopt, std::nullopt);
auto read_buffer = data_part_storage.readFile(relative_path, {}, std::nullopt, std::nullopt);
read_buffer->ignore(sizeof(Checksum));
UInt8 header_size = ICompressionCodec::getHeaderSize();

2
src/Compression/getCompressionCodecForFile.h
View File

@ -11,6 +11,6 @@ namespace DB
/// clickhouse fashion (with checksums, headers for each block, etc). This
/// method should be used as fallback when we cannot deduce compression codec
/// from metadata.
CompressionCodecPtr getCompressionCodecForFile(const DataPartStoragePtr & data_part_storage, const String & relative_path);
CompressionCodecPtr getCompressionCodecForFile(const IDataPartStorage & data_part_storage, const String & relative_path);
}

29
src/Disks/IStoragePolicy.cpp
View File

@ -31,4 +31,33 @@ VolumePtr IStoragePolicy::getVolumeByName(const String & volume_name) const
return volume;
}
size_t IStoragePolicy::getVolumeIndexByDiskName(const String & disk_name) const
{
auto index = tryGetVolumeIndexByDiskName(disk_name);
if (!index)
throw Exception(ErrorCodes::UNKNOWN_DISK,
"No disk {} in policy {}", backQuote(disk_name), backQuote(getName()));
return *index;
}
VolumePtr IStoragePolicy::tryGetVolumeByDiskName(const String & disk_name) const
{
auto index = tryGetVolumeIndexByDiskName(disk_name);
if (!index)
return nullptr;
return getVolume(*index);
}
VolumePtr IStoragePolicy::getVolumeByDiskName(const String & disk_name) const
{
auto volume = tryGetVolumeByDiskName(disk_name);
if (!volume)
throw Exception(ErrorCodes::UNKNOWN_DISK,
"No disk {} in policy {}", backQuote(disk_name), backQuote(getName()));
return volume;
}
}

10
src/Disks/IStoragePolicy.h
View File

@ -4,6 +4,7 @@
#include <memory>
#include <vector>
#include <optional>
#include <base/types.h>
namespace DB
@ -55,12 +56,15 @@ public:
/// Get volume by index.
virtual VolumePtr getVolume(size_t index) const = 0;
virtual VolumePtr tryGetVolumeByName(const String & volume_name) const = 0;
virtual VolumePtr tryGetVolumeByDisk(const DiskPtr & disk_ptr) const = 0;
VolumePtr getVolumeByName(const String & volume_name) const;
/// Checks if storage policy can be replaced by another one.
virtual void checkCompatibleWith(const StoragePolicyPtr & new_storage_policy) const = 0;
/// Find volume index, which contains disk
virtual size_t getVolumeIndexByDisk(const DiskPtr & disk_ptr) const = 0;
/// Finds a volume index, which contains disk
virtual std::optional<size_t> tryGetVolumeIndexByDiskName(const String & disk_name) const = 0;
size_t getVolumeIndexByDiskName(const String & disk_name) const;
/// Finds a volume which contains a specified disk.
VolumePtr tryGetVolumeByDiskName(const String & disk_name) const;
VolumePtr getVolumeByDiskName(const String & disk_name) const;
/// Check if we have any volume with stopped merges
virtual bool hasAnyVolumeWithDisabledMerges() const = 0;
virtual bool containsVolume(const String & volume_name) const = 0;

2
src/Disks/ObjectStorages/DiskObjectStorageTransaction.cpp
View File

@ -599,7 +599,7 @@ std::unique_ptr<WriteBufferFromFileBase> DiskObjectStorageTransaction::writeFile
auto write_operation = std::make_unique<WriteFileObjectStorageOperation>(object_storage, metadata_storage, object);
std::function<void(size_t count)> create_metadata_callback;
if (autocommit)
if (autocommit)
{
create_metadata_callback = [tx = shared_from_this(), mode, path, blob_name] (size_t count)
{

3
src/Disks/ObjectStorages/Web/MetadataStorageFromStaticFilesWebServer.cpp
View File

@ -55,6 +55,9 @@ bool MetadataStorageFromStaticFilesWebServer::exists(const std::string & path) c
path,
[](const auto & file, const std::string & path_) { return file.first < path_; }
);
if (it == object_storage.files.end())
return false;
if (startsWith(it->first, path)
|| (it != object_storage.files.begin() && startsWith(std::prev(it)->first, path)))
return true;

17
src/Disks/StoragePolicy.cpp
View File

@ -26,7 +26,6 @@ namespace ErrorCodes
extern const int BAD_ARGUMENTS;
extern const int EXCESSIVE_ELEMENT_IN_CONFIG;
extern const int NO_ELEMENTS_IN_CONFIG;
extern const int UNKNOWN_DISK;
extern const int UNKNOWN_POLICY;
extern const int UNKNOWN_VOLUME;
extern const int LOGICAL_ERROR;
@ -311,22 +310,12 @@ void StoragePolicy::checkCompatibleWith(const StoragePolicyPtr & new_storage_pol
}
size_t StoragePolicy::getVolumeIndexByDisk(const DiskPtr & disk_ptr) const
std::optional<size_t> StoragePolicy::tryGetVolumeIndexByDiskName(const String & disk_name) const
{
auto it = volume_index_by_disk_name.find(disk_ptr->getName());
auto it = volume_index_by_disk_name.find(disk_name);
if (it != volume_index_by_disk_name.end())
return it->second;
else
throw Exception("No disk " + backQuote(disk_ptr->getName()) + " in policy " + backQuote(name), ErrorCodes::UNKNOWN_DISK);
}
VolumePtr StoragePolicy::tryGetVolumeByDisk(const DiskPtr & disk_ptr) const
{
auto it = volume_index_by_disk_name.find(disk_ptr->getName());
if (it == volume_index_by_disk_name.end())
return nullptr;
return getVolume(it->second);
return {};
}

5
src/Disks/StoragePolicy.h
View File

@ -68,7 +68,7 @@ public:
ReservationPtr reserve(UInt64 bytes, size_t min_volume_index) const override;
/// Find volume index, which contains disk
size_t getVolumeIndexByDisk(const DiskPtr & disk_ptr) const override;
std::optional<size_t> tryGetVolumeIndexByDiskName(const String & disk_name) const override;
/// Reserves 0 bytes on disk with max available space
/// Do not use this function when it is possible to predict size.
@ -85,9 +85,6 @@ public:
VolumePtr tryGetVolumeByName(const String & volume_name) const override;
/// Finds a volume which contains a specified disk.
VolumePtr tryGetVolumeByDisk(const DiskPtr & disk_ptr) const override;
/// Checks if storage policy can be replaced by another one.
void checkCompatibleWith(const StoragePolicyPtr & new_storage_policy) const override;

100
src/Functions/formatDateTime.cpp
View File

@ -1,5 +1,6 @@
#include <DataTypes/DataTypeString.h>
#include <DataTypes/DataTypeDate.h>
#include <DataTypes/DataTypeDate32.h>
#include <DataTypes/DataTypeDateTime.h>
#include <DataTypes/DataTypeDateTime64.h>
#include <Columns/ColumnString.h>
@ -45,6 +46,7 @@ template <> struct ActionValueTypeMap<DataTypeUInt32> { using ActionValueTyp
template <> struct ActionValueTypeMap<DataTypeInt64> { using ActionValueType = UInt32; };
template <> struct ActionValueTypeMap<DataTypeUInt64> { using ActionValueType = UInt32; };
template <> struct ActionValueTypeMap<DataTypeDate> { using ActionValueType = UInt16; };
template <> struct ActionValueTypeMap<DataTypeDate32> { using ActionValueType = Int32; };
template <> struct ActionValueTypeMap<DataTypeDateTime> { using ActionValueType = UInt32; };
// TODO(vnemkov): to add sub-second format instruction, make that DateTime64 and do some math in Action<T>.
template <> struct ActionValueTypeMap<DataTypeDateTime64> { using ActionValueType = Int64; };
@ -315,44 +317,39 @@ public:
if constexpr (support_integer)
{
if (arguments.size() != 1 && arguments.size() != 2 && arguments.size() != 3)
throw Exception(
"Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size())
+ ", should be 1, 2 or 3",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
"Number of arguments for function {} doesn't match: passed {}, should be 1, 2 or 3",
getName(), arguments.size());
if (arguments.size() == 1 && !isInteger(arguments[0].type))
throw Exception(
"Illegal type " + arguments[0].type->getName() + " of 1 argument of function " + getName()
+ " when arguments size is 1. Should be integer",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
if (arguments.size() > 1 && !(isInteger(arguments[0].type) || isDate(arguments[0].type) || isDateTime(arguments[0].type) || isDateTime64(arguments[0].type)))
throw Exception(
"Illegal type " + arguments[0].type->getName() + " of 1 argument of function " + getName()
+ " when arguments size is 2 or 3. Should be a integer or a date with time",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
"Illegal type {} of first argument of function {} when arguments size is 1. Should be integer",
arguments[0].type->getName(), getName());
if (arguments.size() > 1 && !(isInteger(arguments[0].type) || isDate(arguments[0].type) || isDateTime(arguments[0].type) || isDate32(arguments[0].type) || isDateTime64(arguments[0].type)))
throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
"Illegal type {} of first argument of function {} when arguments size is 2 or 3. Should be a integer or a date with time",
arguments[0].type->getName(), getName());
}
else
{
if (arguments.size() != 2 && arguments.size() != 3)
throw Exception(
"Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size())
+ ", should be 2 or 3",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
if (!isDate(arguments[0].type) && !isDateTime(arguments[0].type) && !isDateTime64(arguments[0].type))
throw Exception(
"Illegal type " + arguments[0].type->getName() + " of 1 argument of function " + getName()
+ ". Should be a date or a date with time",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
"Number of arguments for function {} doesn't match: passed {}, should be 2 or 3",
getName(), arguments.size());
if (!isDate(arguments[0].type) && !isDateTime(arguments[0].type) && !isDate32(arguments[0].type) && !isDateTime64(arguments[0].type))
throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
"Illegal type {} of first argument of function {}. Should be a date or a date with time",
arguments[0].type->getName(), getName());
}
if (arguments.size() == 2 && !WhichDataType(arguments[1].type).isString())
throw Exception(
"Illegal type " + arguments[1].type->getName() + " of 2 argument of function " + getName() + ". Must be String.",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
"Illegal type {} of second argument of function {}. Must be String.",
arguments[1].type->getName(), getName());
if (arguments.size() == 3 && !WhichDataType(arguments[2].type).isString())
throw Exception(
"Illegal type " + arguments[2].type->getName() + " of 3 argument of function " + getName() + ". Must be String.",
ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT);
throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
"Illegal type {} of third argument of function {}. Must be String.",
arguments[2].type->getName(), getName());
if (arguments.size() == 1)
return std::make_shared<DataTypeDateTime>();
@ -373,10 +370,9 @@ public:
return true;
}))
{
throw Exception(
"Illegal column " + arguments[0].column->getName() + " of function " + getName()
+ ", must be Integer or DateTime when arguments size is 1.",
ErrorCodes::ILLEGAL_COLUMN);
throw Exception(ErrorCodes::ILLEGAL_COLUMN,
"Illegal column {} of function {}, must be Integer, Date, Date32, DateTime or DateTime64 when arguments size is 1.",
arguments[0].column->getName(), getName());
}
}
else
@ -385,32 +381,31 @@ public:
{
using FromDataType = std::decay_t<decltype(type)>;
if (!(res = executeType<FromDataType>(arguments, result_type)))
throw Exception(
"Illegal column " + arguments[0].column->getName() + " of function " + getName()
+ ", must be Integer or DateTime.",
ErrorCodes::ILLEGAL_COLUMN);
throw Exception(ErrorCodes::ILLEGAL_COLUMN,
"Illegal column {} of function {}, must be Integer, Date, Date32, DateTime or DateTime64.",
arguments[0].column->getName(), getName());
return true;
}))
{
if (!((res = executeType<DataTypeDate>(arguments, result_type))
|| (res = executeType<DataTypeDate32>(arguments, result_type))
|| (res = executeType<DataTypeDateTime>(arguments, result_type))
|| (res = executeType<DataTypeDateTime64>(arguments, result_type))))
throw Exception(
"Illegal column " + arguments[0].column->getName() + " of function " + getName()
+ ", must be Integer or DateTime.",
ErrorCodes::ILLEGAL_COLUMN);
throw Exception(ErrorCodes::ILLEGAL_COLUMN,
"Illegal column {} of function {}, must be Integer or DateTime.",
arguments[0].column->getName(), getName());
}
}
}
else
{
if (!((res = executeType<DataTypeDate>(arguments, result_type))
|| (res = executeType<DataTypeDate32>(arguments, result_type))
|| (res = executeType<DataTypeDateTime>(arguments, result_type))
|| (res = executeType<DataTypeDateTime64>(arguments, result_type))))
throw Exception(
"Illegal column " + arguments[0].column->getName() + " of function " + getName()
+ ", must be Date or DateTime.",
ErrorCodes::ILLEGAL_COLUMN);
throw Exception(ErrorCodes::ILLEGAL_COLUMN,
"Illegal column {} of function {}, must be Date or DateTime.",
arguments[0].column->getName(), getName());
}
return res;
@ -425,10 +420,9 @@ public:
const ColumnConst * pattern_column = checkAndGetColumnConst<ColumnString>(arguments[1].column.get());
if (!pattern_column)
throw Exception("Illegal column " + arguments[1].column->getName()
+ " of second ('format') argument of function " + getName()
+ ". Must be constant string.",
ErrorCodes::ILLEGAL_COLUMN);
throw Exception(ErrorCodes::ILLEGAL_COLUMN,
"Illegal column {} of second ('format') argument of function {}. Must be constant string.",
arguments[1].column->getName(), getName());
String pattern = pattern_column->getValue<String>();
@ -712,12 +706,14 @@ public:
// Unimplemented
case 'U': [[fallthrough]];
case 'W':
throw Exception("Wrong pattern '" + pattern + "', symbol '" + *pos + " is not implemented ' for function " + getName(),
ErrorCodes::NOT_IMPLEMENTED);
throw Exception(ErrorCodes::NOT_IMPLEMENTED,
"Wrong pattern '{}', symbol '{}' is not implemented for function {}",
pattern, *pos, getName());
default:
throw Exception(
"Wrong pattern '" + pattern + "', unexpected symbol '" + *pos + "' for function " + getName(), ErrorCodes::ILLEGAL_COLUMN);
throw Exception(ErrorCodes::ILLEGAL_COLUMN,
"Wrong pattern '{}', unexpected symbol '{}' for function {}",
pattern, *pos, getName());
}
++pos;

15
src/Interpreters/AsynchronousMetrics.cpp
View File

@ -703,19 +703,26 @@ void AsynchronousMetrics::update(TimePoint update_time)
Int64 free_memory_in_allocator_arenas = 0;
#if USE_JEMALLOC
/// This is a memory which is kept by allocator.
/// Will subsract it from RSS to decrease memory drift.
/// According to jemalloc man, pdirty is:
///
/// Number of pages within unused extents that are potentially
/// dirty, and for which madvise() or similar has not been called.
///
/// So they will be subtracted from RSS to make accounting more
/// accurate, since those pages are not really RSS but a memory
/// that can be used at anytime via jemalloc.
free_memory_in_allocator_arenas = je_malloc_pdirty * getPageSize();
#endif
Int64 difference = rss - free_memory_in_allocator_arenas - amount;
Int64 difference = rss - amount;
/// Log only if difference is high. This is for convenience. The threshold is arbitrary.
if (difference >= 1048576 || difference <= -1048576)
LOG_TRACE(log,
"MemoryTracking: was {}, peak {}, will set to {} (RSS), difference: {}",
"MemoryTracking: was {}, peak {}, free memory in arenas {}, will set to {} (RSS), difference: {}",
ReadableSize(amount),
ReadableSize(peak),
ReadableSize(free_memory_in_allocator_arenas),
ReadableSize(rss),
ReadableSize(difference));

2
src/Interpreters/InterpreterDropQuery.cpp
View File

@ -213,7 +213,7 @@ BlockIO InterpreterDropQuery::executeToTableImpl(ContextPtr context_, ASTDropQue
{
/// And for simple MergeTree we can stop merges before acquiring the lock
auto merges_blocker = table->getActionLock(ActionLocks::PartsMerge);
auto table_lock = table->lockExclusively(context_->getCurrentQueryId(), context_->getSettingsRef().lock_acquire_timeout);
table_lock = table->lockExclusively(context_->getCurrentQueryId(), context_->getSettingsRef().lock_acquire_timeout);
}
auto metadata_snapshot = table->getInMemoryMetadataPtr();

4
src/Interpreters/PartLog.cpp
View File

@ -207,8 +207,8 @@ bool PartLog::addNewParts(
elem.table_name = table_id.table_name;
elem.partition_id = part->info.partition_id;
elem.part_name = part->name;
elem.disk_name = part->data_part_storage->getDiskName();
elem.path_on_disk = part->data_part_storage->getFullPath();
elem.disk_name = part->getDataPartStorage().getDiskName();
elem.path_on_disk = part->getDataPartStorage().getFullPath();
elem.part_type = part->getType();
elem.bytes_compressed_on_disk = part->getBytesOnDisk();

5
src/Interpreters/convertFieldToType.cpp
View File

@ -236,10 +236,11 @@ Field convertFieldToTypeImpl(const Field & src, const IDataType & type, const ID
}
if (which_type.isDateTime64()
&& (which_from_type.isNativeInt() || which_from_type.isNativeUInt() || which_from_type.isDate() || which_from_type.isDate32() || which_from_type.isDateTime() || which_from_type.isDateTime64()))
&& (src.getType() == Field::Types::UInt64 || src.getType() == Field::Types::Int64 || src.getType() == Field::Types::Decimal64))
{
const auto scale = static_cast<const DataTypeDateTime64 &>(type).getScale();
const auto decimal_value = DecimalUtils::decimalFromComponents<DateTime64>(applyVisitor(FieldVisitorConvertToNumber<Int64>(), src), 0, scale);
const auto decimal_value
= DecimalUtils::decimalFromComponents<DateTime64>(applyVisitor(FieldVisitorConvertToNumber<Int64>(), src), 0, scale);
return Field(DecimalField<DateTime64>(decimal_value, scale));
}
}

5
src/Parsers/Access/ParserCreateUserQuery.cpp
View File

@ -295,11 +295,11 @@ namespace
}
bool parseHosts(IParserBase::Pos & pos, Expected & expected, const String & prefix, AllowedClientHosts & hosts)
bool parseHosts(IParserBase::Pos & pos, Expected & expected, std::string_view prefix, AllowedClientHosts & hosts)
{
return IParserBase::wrapParseImpl(pos, [&]
{
if (!prefix.empty() && !ParserKeyword{prefix.c_str()}.ignore(pos, expected))
if (!prefix.empty() && !ParserKeyword{prefix}.ignore(pos, expected))
return false;
if (!ParserKeyword{"HOST"}.ignore(pos, expected))
@ -492,7 +492,6 @@ bool ParserCreateUserQuery::parseImpl(Pos & pos, ASTPtr & node, Expected & expec
if (alter)
{
String maybe_new_name;
if (!new_name && (names->size() == 1) && parseRenameTo(pos, expected, new_name))
continue;

242
src/Parsers/ExpressionListParsers.cpp
View File

@ -561,13 +561,10 @@ public:
virtual bool getResult(ASTPtr & node)
{
if (elements.size() == 1)
{
node = std::move(elements[0]);
return true;
}
if (!finished)
return false;
return false;
return getResultImpl(node);
}
virtual bool parse(IParser::Pos & /*pos*/, Expected & /*expected*/, Action & /*action*/) = 0;
@ -746,6 +743,17 @@ public:
Checkpoint current_checkpoint = Checkpoint::None;
protected:
virtual bool getResultImpl(ASTPtr & node)
{
if (elements.size() == 1)
{
node = std::move(elements[0]);
return true;
}
return false;
}
std::vector<Operator> operators;
ASTs operands;
ASTs elements;
@ -766,17 +774,12 @@ public:
bool getResult(ASTPtr & node) override
{
/// We can exit the main cycle outside the parse() function,
/// so we need to merge the element here
/// so we need to merge the element here.
/// Because of this 'finished' flag can also not be set.
if (!mergeElement())
return false;
if (elements.size() == 1)
{
node = std::move(elements[0]);
return true;
}
return false;
return Layer::getResultImpl(node);
}
bool parse(IParser::Pos & pos, Expected & /*expected*/, Action & /*action*/) override
@ -1029,17 +1032,6 @@ private:
class RoundBracketsLayer : public Layer
{
public:
bool getResult(ASTPtr & node) override
{
// Round brackets can mean priority operator as well as function tuple()
if (!is_tuple && elements.size() == 1)
node = std::move(elements[0]);
else
node = makeASTFunction("tuple", std::move(elements));
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & action) override
{
if (ParserToken(TokenType::Comma).ignore(pos, expected))
@ -1069,6 +1061,19 @@ public:
return true;
}
protected:
bool getResultImpl(ASTPtr & node) override
{
// Round brackets can mean priority operator as well as function tuple()
if (!is_tuple && elements.size() == 1)
node = std::move(elements[0]);
else
node = makeASTFunction("tuple", std::move(elements));
return true;
}
private:
bool is_tuple = false;
};
@ -1077,16 +1082,17 @@ private:
class ArrayLayer : public LayerWithSeparator<TokenType::Comma, TokenType::ClosingSquareBracket>
{
public:
bool getResult(ASTPtr & node) override
{
node = makeASTFunction("array", std::move(elements));
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & action) override
{
return LayerWithSeparator::parse(pos, expected, action);
}
protected:
bool getResultImpl(ASTPtr & node) override
{
node = makeASTFunction("array", std::move(elements));
return true;
}
};
/// Layer for arrayElement square brackets operator
@ -1206,23 +1212,6 @@ class ExtractLayer : public LayerWithSeparator<TokenType::Comma, TokenType::Clos
public:
ExtractLayer() : LayerWithSeparator(/*allow_alias*/ true, /*allow_alias_without_as_keyword*/ true) {}
bool getResult(ASTPtr & node) override
{
if (state == 2)
{
if (elements.empty())
return false;
node = makeASTFunction(interval_kind.toNameOfFunctionExtractTimePart(), elements[0]);
}
else
{
node = makeASTFunction("extract", std::move(elements));
}
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & action) override
{
/// extract(haystack, pattern) or EXTRACT(DAY FROM Date)
@ -1268,6 +1257,25 @@ public:
return true;
}
protected:
bool getResultImpl(ASTPtr & node) override
{
if (state == 2)
{
if (elements.empty())
return false;
node = makeASTFunction(interval_kind.toNameOfFunctionExtractTimePart(), elements[0]);
}
else
{
node = makeASTFunction("extract", std::move(elements));
}
return true;
}
private:
IntervalKind interval_kind;
};
@ -1277,12 +1285,6 @@ class SubstringLayer : public Layer
public:
SubstringLayer() : Layer(/*allow_alias*/ true, /*allow_alias_without_as_keyword*/ true) {}
bool getResult(ASTPtr & node) override
{
node = makeASTFunction("substring", std::move(elements));
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & action) override
{
/// Either SUBSTRING(expr FROM start [FOR length]) or SUBSTRING(expr, start, length)
@ -1332,6 +1334,13 @@ public:
return true;
}
protected:
bool getResultImpl(ASTPtr & node) override
{
node = makeASTFunction("substring", std::move(elements));
return true;
}
};
class PositionLayer : public Layer
@ -1339,15 +1348,6 @@ class PositionLayer : public Layer
public:
PositionLayer() : Layer(/*allow_alias*/ true, /*allow_alias_without_as_keyword*/ true) {}
bool getResult(ASTPtr & node) override
{
if (state == 2)
std::swap(elements[1], elements[0]);
node = makeASTFunction("position", std::move(elements));
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & action) override
{
/// position(haystack, needle[, start_pos]) or position(needle IN haystack)
@ -1402,6 +1402,16 @@ public:
return true;
}
protected:
bool getResultImpl(ASTPtr & node) override
{
if (state == 2)
std::swap(elements[1], elements[0]);
node = makeASTFunction("position", std::move(elements));
return true;
}
};
class ExistsLayer : public Layer
@ -1436,12 +1446,6 @@ public:
TrimLayer(bool trim_left_, bool trim_right_)
: Layer(/*allow_alias*/ true, /*allow_alias_without_as_keyword*/ true), trim_left(trim_left_), trim_right(trim_right_) {}
bool getResult(ASTPtr & node) override
{
node = makeASTFunction(function_name, std::move(elements));
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & action) override
{
/// Handles all possible TRIM/LTRIM/RTRIM call variants
@ -1583,6 +1587,14 @@ public:
return true;
}
protected:
bool getResultImpl(ASTPtr & node) override
{
node = makeASTFunction(function_name, std::move(elements));
return true;
}
private:
bool trim_left;
bool trim_right;
@ -1598,23 +1610,6 @@ public:
explicit DateAddLayer(const char * function_name_)
: LayerWithSeparator(/*allow_alias*/ true, /*allow_alias_without_as_keyword*/ true), function_name(function_name_) {}
bool getResult(ASTPtr & node) override
{
if (parsed_interval_kind)
{
if (elements.size() < 2)
return false;
elements[0] = makeASTFunction(interval_kind.toNameOfFunctionToIntervalDataType(), elements[0]);
node = makeASTFunction(function_name, elements[1], elements[0]);
}
else
node = makeASTFunction(function_name, std::move(elements));
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & action) override
{
/// DATEADD(YEAR, 1, date) or DATEADD(INTERVAL 1 YEAR, date);
@ -1644,6 +1639,23 @@ public:
return true;
}
protected:
bool getResultImpl(ASTPtr & node) override
{
if (parsed_interval_kind)
{
if (elements.size() < 2)
return false;
elements[0] = makeASTFunction(interval_kind.toNameOfFunctionToIntervalDataType(), elements[0]);
node = makeASTFunction(function_name, elements[1], elements[0]);
}
else
node = makeASTFunction(function_name, std::move(elements));
return true;
}
private:
IntervalKind interval_kind;
const char * function_name;
@ -1655,24 +1667,6 @@ class DateDiffLayer : public LayerWithSeparator<TokenType::Comma, TokenType::Clo
public:
DateDiffLayer() : LayerWithSeparator(/*allow_alias*/ true, /*allow_alias_without_as_keyword*/ true) {}
bool getResult(ASTPtr & node) override
{
if (parsed_interval_kind)
{
if (elements.size() == 2)
node = makeASTFunction("dateDiff", std::make_shared<ASTLiteral>(interval_kind.toDateDiffUnit()), elements[0], elements[1]);
else if (elements.size() == 3)
node = makeASTFunction("dateDiff", std::make_shared<ASTLiteral>(interval_kind.toDateDiffUnit()), elements[0], elements[1], elements[2]);
else
return false;
}
else
{
node = makeASTFunction("dateDiff", std::move(elements));
}
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & action) override
{
/// 0. Try to parse interval_kind (-> 1)
@ -1699,6 +1693,25 @@ public:
return true;
}
protected:
bool getResultImpl(ASTPtr & node) override
{
if (parsed_interval_kind)
{
if (elements.size() == 2)
node = makeASTFunction("dateDiff", std::make_shared<ASTLiteral>(interval_kind.toDateDiffUnit()), elements[0], elements[1]);
else if (elements.size() == 3)
node = makeASTFunction("dateDiff", std::make_shared<ASTLiteral>(interval_kind.toDateDiffUnit()), elements[0], elements[1], elements[2]);
else
return false;
}
else
{
node = makeASTFunction("dateDiff", std::move(elements));
}
return true;
}
private:
IntervalKind interval_kind;
bool parsed_interval_kind = false;
@ -1882,16 +1895,6 @@ class ViewLayer : public Layer
public:
explicit ViewLayer(bool if_permitted_) : if_permitted(if_permitted_) {}
bool getResult(ASTPtr & node) override
{
if (if_permitted)
node = makeASTFunction("viewIfPermitted", std::move(elements));
else
node = makeASTFunction("view", std::move(elements));
return true;
}
bool parse(IParser::Pos & pos, Expected & expected, Action & /*action*/) override
{
/// view(SELECT ...)
@ -1948,6 +1951,17 @@ public:
return true;
}
protected:
bool getResultImpl(ASTPtr & node) override
{
if (if_permitted)
node = makeASTFunction("viewIfPermitted", std::move(elements));
else
node = makeASTFunction("view", std::move(elements));
return true;
}
private:
bool if_permitted;
};

2
src/Processors/Executors/ExecutingGraph.cpp
View File

@ -71,7 +71,7 @@ bool ExecutingGraph::addEdges(uint64_t node)
}
}
/// Add direct edges form output ports.
/// Add direct edges from output ports.
auto & outputs = from->getOutputs();
auto from_output = nodes[node]->direct_edges.size();

454
src/Storages/MergeTree/DataPartStorageOnDisk.cpp
View File

@ -6,12 +6,12 @@
#include <IO/ReadBufferFromFileBase.h>
#include <IO/ReadHelpers.h>
#include <Common/logger_useful.h>
#include <Disks/IStoragePolicy.h>
#include <Backups/BackupEntryFromSmallFile.h>
#include <Backups/BackupEntryFromImmutableFile.h>
#include <Storages/MergeTree/localBackup.h>
#include <Disks/SingleDiskVolume.h>
#include <Interpreters/TransactionVersionMetadata.h>
#include <memory>
namespace DB
{
@ -29,6 +29,16 @@ DataPartStorageOnDisk::DataPartStorageOnDisk(VolumePtr volume_, std::string root
{
}
DataPartStorageOnDisk::DataPartStorageOnDisk(
VolumePtr volume_, std::string root_path_, std::string part_dir_, DiskTransactionPtr transaction_)
: volume(std::move(volume_))
, root_path(std::move(root_path_))
, part_dir(std::move(part_dir_))
, transaction(std::move(transaction_))
, has_shared_transaction(transaction != nullptr)
{
}
std::string DataPartStorageOnDisk::getFullPath() const
{
return fs::path(volume->getDisk()->getPath()) / root_path / part_dir / "";
@ -49,6 +59,11 @@ std::string DataPartStorageOnDisk::getFullRootPath() const
return fs::path(volume->getDisk()->getPath()) / root_path / "";
}
MutableDataPartStoragePtr DataPartStorageOnDisk::getProjection(const std::string & name)
{
return std::shared_ptr<DataPartStorageOnDisk>(new DataPartStorageOnDisk(volume, std::string(fs::path(root_path) / part_dir), name, transaction));
}
DataPartStoragePtr DataPartStorageOnDisk::getProjection(const std::string & name) const
{
return std::make_shared<DataPartStorageOnDisk>(volume, std::string(fs::path(root_path) / part_dir), name);
@ -113,6 +128,7 @@ static UInt64 calculateTotalSizeOnDiskImpl(const DiskPtr & disk, const String &
{
if (disk->isFile(from))
return disk->getFileSize(from);
std::vector<std::string> files;
disk->listFiles(from, files);
UInt64 res = 0;
@ -135,75 +151,11 @@ std::unique_ptr<ReadBufferFromFileBase> DataPartStorageOnDisk::readFile(
return volume->getDisk()->readFile(fs::path(root_path) / part_dir / name, settings, read_hint, file_size);
}
static std::unique_ptr<ReadBufferFromFileBase> openForReading(const DiskPtr & disk, const String & path)
{
size_t file_size = disk->getFileSize(path);
return disk->readFile(path, ReadSettings().adjustBufferSize(file_size), file_size);
}
void DataPartStorageOnDisk::loadVersionMetadata(VersionMetadata & version, Poco::Logger * log) const
{
std::string version_file_name = fs::path(root_path) / part_dir / "txn_version.txt";
String tmp_version_file_name = version_file_name + ".tmp";
DiskPtr disk = volume->getDisk();
auto remove_tmp_file = [&]()
{
auto last_modified = disk->getLastModified(tmp_version_file_name);
auto buf = openForReading(disk, tmp_version_file_name);
String content;
readStringUntilEOF(content, *buf);
LOG_WARNING(log, "Found file {} that was last modified on {}, has size {} and the following content: {}",
tmp_version_file_name, last_modified.epochTime(), content.size(), content);
disk->removeFile(tmp_version_file_name);
};
if (disk->exists(version_file_name))
{
auto buf = openForReading(disk, version_file_name);
version.read(*buf);
if (disk->exists(tmp_version_file_name))
remove_tmp_file();
return;
}
/// Four (?) cases are possible:
/// 1. Part was created without transactions.
/// 2. Version metadata file was not renamed from *.tmp on part creation.
/// 3. Version metadata were written to *.tmp file, but hard restart happened before fsync.
/// 4. Fsyncs in storeVersionMetadata() work incorrectly.
if (!disk->exists(tmp_version_file_name))
{
/// Case 1.
/// We do not have version metadata and transactions history for old parts,
/// so let's consider that such parts were created by some ancient transaction
/// and were committed with some prehistoric CSN.
/// NOTE It might be Case 3, but version metadata file is written on part creation before other files,
/// so it's not Case 3 if part is not broken.
version.setCreationTID(Tx::PrehistoricTID, nullptr);
version.creation_csn = Tx::PrehistoricCSN;
return;
}
/// Case 2.
/// Content of *.tmp file may be broken, just use fake TID.
/// Transaction was not committed if *.tmp file was not renamed, so we should complete rollback by removing part.
version.setCreationTID(Tx::DummyTID, nullptr);
version.creation_csn = Tx::RolledBackCSN;
remove_tmp_file();
}
void DataPartStorageOnDisk::checkConsistency(const MergeTreeDataPartChecksums & checksums) const
{
checksums.checkSizes(volume->getDisk(), getRelativePath());
}
DataPartStorageBuilderPtr DataPartStorageOnDisk::getBuilder() const
{
return std::make_shared<DataPartStorageBuilderOnDisk>(volume, root_path, part_dir);
}
void DataPartStorageOnDisk::remove(
CanRemoveCallback && can_remove_callback,
const MergeTreeDataPartChecksums & checksums,
@ -273,7 +225,7 @@ void DataPartStorageOnDisk::remove(
try
{
disk->moveDirectory(from, to);
onRename(root_path, part_dir_without_slash);
part_dir = part_dir_without_slash;
}
catch (const Exception & e)
{
@ -488,11 +440,6 @@ bool DataPartStorageOnDisk::looksLikeBrokenDetachedPartHasTheSameContent(const S
return original_files_list == detached_files_list;
}
void DataPartStorageBuilderOnDisk::setRelativePath(const std::string & path)
{
part_dir = path;
}
std::string DataPartStorageOnDisk::getDiskName() const
{
return volume->getDisk()->getName();
@ -523,7 +470,7 @@ bool DataPartStorageOnDisk::isBroken() const
return volume->getDisk()->isBroken();
}
void DataPartStorageOnDisk::syncRevision(UInt64 revision)
void DataPartStorageOnDisk::syncRevision(UInt64 revision) const
{
volume->getDisk()->syncRevision(revision);
}
@ -543,11 +490,6 @@ std::string DataPartStorageOnDisk::getDiskPath() const
return volume->getDisk()->getPath();
}
DataPartStorageOnDisk::DisksSet::const_iterator DataPartStorageOnDisk::isStoredOnDisk(const DisksSet & disks) const
{
return disks.find(volume->getDisk());
}
ReservationPtr DataPartStorageOnDisk::reserve(UInt64 bytes) const
{
auto res = volume->reserve(bytes);
@ -562,159 +504,6 @@ ReservationPtr DataPartStorageOnDisk::tryReserve(UInt64 bytes) const
return volume->reserve(bytes);
}
size_t DataPartStorageOnDisk::getVolumeIndex(const IStoragePolicy & storage_policy) const
{
return storage_policy.getVolumeIndexByDisk(volume->getDisk());
}
void DataPartStorageOnDisk::writeChecksums(const MergeTreeDataPartChecksums & checksums, const WriteSettings & settings) const
{
std::string path = fs::path(root_path) / part_dir / "checksums.txt";
try
{
{
auto out = volume->getDisk()->writeFile(path + ".tmp", 4096, WriteMode::Rewrite, settings);
checksums.write(*out);
}
volume->getDisk()->moveFile(path + ".tmp", path);
}
catch (...)
{
try
{
if (volume->getDisk()->exists(path + ".tmp"))
volume->getDisk()->removeFile(path + ".tmp");
}
catch (...)
{
tryLogCurrentException("DataPartStorageOnDisk");
}
throw;
}
}
void DataPartStorageOnDisk::writeColumns(const NamesAndTypesList & columns, const WriteSettings & settings) const
{
std::string path = fs::path(root_path) / part_dir / "columns.txt";
try
{
auto buf = volume->getDisk()->writeFile(path + ".tmp", 4096, WriteMode::Rewrite, settings);
columns.writeText(*buf);
buf->finalize();
volume->getDisk()->moveFile(path + ".tmp", path);
}
catch (...)
{
try
{
if (volume->getDisk()->exists(path + ".tmp"))
volume->getDisk()->removeFile(path + ".tmp");
}
catch (...)
{
tryLogCurrentException("DataPartStorageOnDisk");
}
throw;
}
}
void DataPartStorageOnDisk::writeVersionMetadata(const VersionMetadata & version, bool fsync_part_dir) const
{
std::string path = fs::path(root_path) / part_dir / "txn_version.txt";
try
{
{
/// TODO IDisk interface does not allow to open file with O_EXCL flag (for DiskLocal),
/// so we create empty file at first (expecting that createFile throws if file already exists)
/// and then overwrite it.
volume->getDisk()->createFile(path + ".tmp");
auto buf = volume->getDisk()->writeFile(path + ".tmp", 256);
version.write(*buf);
buf->finalize();
buf->sync();
}
SyncGuardPtr sync_guard;
if (fsync_part_dir)
sync_guard = volume->getDisk()->getDirectorySyncGuard(getRelativePath());
volume->getDisk()->replaceFile(path + ".tmp", path);
}
catch (...)
{
try
{
if (volume->getDisk()->exists(path + ".tmp"))
volume->getDisk()->removeFile(path + ".tmp");
}
catch (...)
{
tryLogCurrentException("DataPartStorageOnDisk");
}
throw;
}
}
void DataPartStorageOnDisk::appendCSNToVersionMetadata(const VersionMetadata & version, VersionMetadata::WhichCSN which_csn) const
{
/// Small enough appends to file are usually atomic,
/// so we append new metadata instead of rewriting file to reduce number of fsyncs.
/// We don't need to do fsync when writing CSN, because in case of hard restart
/// we will be able to restore CSN from transaction log in Keeper.
std::string version_file_name = fs::path(root_path) / part_dir / "txn_version.txt";
DiskPtr disk = volume->getDisk();
auto out = disk->writeFile(version_file_name, 256, WriteMode::Append);
version.writeCSN(*out, which_csn);
out->finalize();
}
void DataPartStorageOnDisk::appendRemovalTIDToVersionMetadata(const VersionMetadata & version, bool clear) const
{
String version_file_name = fs::path(root_path) / part_dir / "txn_version.txt";
DiskPtr disk = volume->getDisk();
auto out = disk->writeFile(version_file_name, 256, WriteMode::Append);
version.writeRemovalTID(*out, clear);
out->finalize();
/// fsync is not required when we clearing removal TID, because after hard restart we will fix metadata
if (!clear)
out->sync();
}
void DataPartStorageOnDisk::writeDeleteOnDestroyMarker(Poco::Logger * log) const
{
String marker_path = fs::path(root_path) / part_dir / "delete-on-destroy.txt";
auto disk = volume->getDisk();
try
{
volume->getDisk()->createFile(marker_path);
}
catch (Poco::Exception & e)
{
LOG_ERROR(log, "{} (while creating DeleteOnDestroy marker: {})", e.what(), backQuote(fullPath(disk, marker_path)));
}
}
void DataPartStorageOnDisk::removeDeleteOnDestroyMarker() const
{
std::string delete_on_destroy_file_name = fs::path(root_path) / part_dir / "delete-on-destroy.txt";
volume->getDisk()->removeFileIfExists(delete_on_destroy_file_name);
}
void DataPartStorageOnDisk::removeVersionMetadata() const
{
std::string version_file_name = fs::path(root_path) / part_dir / "txn_version.txt";
volume->getDisk()->removeFileIfExists(version_file_name);
}
String DataPartStorageOnDisk::getUniqueId() const
{
auto disk = volume->getDisk();
@ -724,16 +513,6 @@ String DataPartStorageOnDisk::getUniqueId() const
return disk->getUniqueId(fs::path(getRelativePath()) / "checksums.txt");
}
bool DataPartStorageOnDisk::shallParticipateInMerges(const IStoragePolicy & storage_policy) const
{
/// `IMergeTreeDataPart::volume` describes space where current part belongs, and holds
/// `SingleDiskVolume` object which does not contain up-to-date settings of corresponding volume.
/// Therefore we shall obtain volume from storage policy.
auto volume_ptr = storage_policy.getVolume(storage_policy.getVolumeIndexByDisk(volume->getDisk()));
return !volume_ptr->areMergesAvoided();
}
void DataPartStorageOnDisk::backup(
const MergeTreeDataPartChecksums & checksums,
const NameSet & files_without_checksums,
@ -798,7 +577,7 @@ void DataPartStorageOnDisk::backup(
}
}
DataPartStoragePtr DataPartStorageOnDisk::freeze(
MutableDataPartStoragePtr DataPartStorageOnDisk::freeze(
const std::string & to,
const std::string & dir_path,
bool make_source_readonly,
@ -822,7 +601,7 @@ DataPartStoragePtr DataPartStorageOnDisk::freeze(
return std::make_shared<DataPartStorageOnDisk>(single_disk_volume, to, dir_path);
}
DataPartStoragePtr DataPartStorageOnDisk::clone(
MutableDataPartStoragePtr DataPartStorageOnDisk::clonePart(
const std::string & to,
const std::string & dir_path,
const DiskPtr & disk,
@ -835,6 +614,7 @@ DataPartStoragePtr DataPartStorageOnDisk::clone(
LOG_WARNING(log, "Path {} already exists. Will remove it and clone again.", fullPath(disk, path_to_clone));
disk->removeRecursive(path_to_clone);
}
disk->createDirectories(to);
volume->getDisk()->copy(getRelativePath(), disk, to);
volume->getDisk()->removeFileIfExists(fs::path(path_to_clone) / "delete-on-destroy.txt");
@ -843,13 +623,7 @@ DataPartStoragePtr DataPartStorageOnDisk::clone(
return std::make_shared<DataPartStorageOnDisk>(single_disk_volume, to, dir_path);
}
void DataPartStorageOnDisk::onRename(const std::string & new_root_path, const std::string & new_part_dir)
{
part_dir = new_part_dir;
root_path = new_root_path;
}
void DataPartStorageBuilderOnDisk::rename(
void DataPartStorageOnDisk::rename(
const std::string & new_root_path,
const std::string & new_part_dir,
Poco::Logger * log,
@ -870,7 +644,7 @@ void DataPartStorageBuilderOnDisk::rename(
"Part directory {} already exists and contains {} files. Removing it.",
fullPath(volume->getDisk(), to), files.size());
transaction->removeRecursive(to);
executeOperation([&](auto & disk) { disk.removeRecursive(to); });
}
else
{
@ -884,8 +658,12 @@ void DataPartStorageBuilderOnDisk::rename(
String from = getRelativePath();
/// Why?
transaction->setLastModified(from, Poco::Timestamp::fromEpochTime(time(nullptr)));
transaction->moveDirectory(from, to);
executeOperation([&](auto & disk)
{
disk.setLastModified(from, Poco::Timestamp::fromEpochTime(time(nullptr)));
disk.moveDirectory(from, to);
});
part_dir = new_part_dir;
root_path = new_root_path;
@ -907,7 +685,7 @@ void DataPartStorageOnDisk::changeRootPath(const std::string & from_root, const
--prefix_size;
if (prefix_size > root_path.size()
|| std::string_view(from_root).substr(0, prefix_size) != std::string_view(root_path).substr(0, prefix_size))
|| std::string_view(from_root).substr(0, prefix_size) != std::string_view(root_path).substr(0, prefix_size))
throw Exception(
ErrorCodes::LOGICAL_ERROR,
"Cannot change part root to {} because it is not a prefix of current root {}",
@ -920,51 +698,80 @@ void DataPartStorageOnDisk::changeRootPath(const std::string & from_root, const
root_path = to_root.substr(0, dst_size) + root_path.substr(prefix_size);
}
DataPartStorageBuilderOnDisk::DataPartStorageBuilderOnDisk(
VolumePtr volume_,
std::string root_path_,
std::string part_dir_)
: volume(std::move(volume_))
, root_path(std::move(root_path_))
, part_dir(std::move(part_dir_))
, transaction(volume->getDisk()->createTransaction())
{
}
std::unique_ptr<WriteBufferFromFileBase> DataPartStorageBuilderOnDisk::writeFile(
const String & name,
size_t buf_size,
const WriteSettings & settings)
{
return transaction->writeFile(fs::path(root_path) / part_dir / name, buf_size, WriteMode::Rewrite, settings, /* autocommit = */ false);
}
void DataPartStorageBuilderOnDisk::removeFile(const String & name)
{
transaction->removeFile(fs::path(root_path) / part_dir / name);
}
void DataPartStorageBuilderOnDisk::removeFileIfExists(const String & name)
{
transaction->removeFileIfExists(fs::path(root_path) / part_dir / name);
}
void DataPartStorageBuilderOnDisk::removeRecursive()
{
transaction->removeRecursive(fs::path(root_path) / part_dir);
}
void DataPartStorageBuilderOnDisk::removeSharedRecursive(bool keep_in_remote_fs)
{
transaction->removeSharedRecursive(fs::path(root_path) / part_dir, keep_in_remote_fs, {});
}
SyncGuardPtr DataPartStorageBuilderOnDisk::getDirectorySyncGuard() const
SyncGuardPtr DataPartStorageOnDisk::getDirectorySyncGuard() const
{
return volume->getDisk()->getDirectorySyncGuard(fs::path(root_path) / part_dir);
}
void DataPartStorageBuilderOnDisk::createHardLinkFrom(const IDataPartStorage & source, const std::string & from, const std::string & to) const
template <typename Op>
void DataPartStorageOnDisk::executeOperation(Op && op)
{
if (transaction)
op(*transaction);
else
op(*volume->getDisk());
}
std::unique_ptr<WriteBufferFromFileBase> DataPartStorageOnDisk::writeFile(
const String & name,
size_t buf_size,
const WriteSettings & settings)
{
if (transaction)
return transaction->writeFile(fs::path(root_path) / part_dir / name, buf_size, WriteMode::Rewrite, settings, /* autocommit = */ false);
return volume->getDisk()->writeFile(fs::path(root_path) / part_dir / name, buf_size, WriteMode::Rewrite, settings);
}
std::unique_ptr<WriteBufferFromFileBase> DataPartStorageOnDisk::writeTransactionFile(WriteMode mode) const
{
return volume->getDisk()->writeFile(fs::path(root_path) / part_dir / "txn_version.txt", 256, mode);
}
void DataPartStorageOnDisk::createFile(const String & name)
{
executeOperation([&](auto & disk) { disk.createFile(fs::path(root_path) / part_dir / name); });
}
void DataPartStorageOnDisk::moveFile(const String & from_name, const String & to_name)
{
executeOperation([&](auto & disk)
{
auto relative_path = fs::path(root_path) / part_dir;
disk.moveFile(relative_path / from_name, relative_path / to_name);
});
}
void DataPartStorageOnDisk::replaceFile(const String & from_name, const String & to_name)
{
executeOperation([&](auto & disk)
{
auto relative_path = fs::path(root_path) / part_dir;
disk.replaceFile(relative_path / from_name, relative_path / to_name);
});
}
void DataPartStorageOnDisk::removeFile(const String & name)
{
executeOperation([&](auto & disk) { disk.removeFile(fs::path(root_path) / part_dir / name); });
}
void DataPartStorageOnDisk::removeFileIfExists(const String & name)
{
executeOperation([&](auto & disk) { disk.removeFileIfExists(fs::path(root_path) / part_dir / name); });
}
void DataPartStorageOnDisk::removeRecursive()
{
executeOperation([&](auto & disk) { disk.removeRecursive(fs::path(root_path) / part_dir); });
}
void DataPartStorageOnDisk::removeSharedRecursive(bool keep_in_remote_fs)
{
executeOperation([&](auto & disk) { disk.removeSharedRecursive(fs::path(root_path) / part_dir, keep_in_remote_fs, {}); });
}
void DataPartStorageOnDisk::createHardLinkFrom(const IDataPartStorage & source, const std::string & from, const std::string & to)
{
const auto * source_on_disk = typeid_cast<const DataPartStorageOnDisk *>(&source);
if (!source_on_disk)
@ -973,58 +780,43 @@ void DataPartStorageBuilderOnDisk::createHardLinkFrom(const IDataPartStorage & s
"Cannot create hardlink from different storage. Expected DataPartStorageOnDisk, got {}",
typeid(source).name());
transaction->createHardLink(
fs::path(source_on_disk->getRelativePath()) / from,
fs::path(root_path) / part_dir / to);
executeOperation([&](auto & disk)
{
disk.createHardLink(
fs::path(source_on_disk->getRelativePath()) / from,
fs::path(root_path) / part_dir / to);
});
}
bool DataPartStorageBuilderOnDisk::exists() const
void DataPartStorageOnDisk::createDirectories()
{
return volume->getDisk()->exists(fs::path(root_path) / part_dir);
executeOperation([&](auto & disk) { disk.createDirectories(fs::path(root_path) / part_dir); });
}
std::string DataPartStorageBuilderOnDisk::getFullPath() const
void DataPartStorageOnDisk::createProjection(const std::string & name)
{
return fs::path(volume->getDisk()->getPath()) / root_path / part_dir;
executeOperation([&](auto & disk) { disk.createDirectory(fs::path(root_path) / part_dir / name); });
}
std::string DataPartStorageBuilderOnDisk::getRelativePath() const
void DataPartStorageOnDisk::beginTransaction()
{
return fs::path(root_path) / part_dir;
if (transaction)
throw Exception(ErrorCodes::LOGICAL_ERROR,
"Uncommitted {}transaction already exists", has_shared_transaction ? "shared " : "");
transaction = volume->getDisk()->createTransaction();
}
void DataPartStorageBuilderOnDisk::createDirectories()
void DataPartStorageOnDisk::commitTransaction()
{
transaction->createDirectories(fs::path(root_path) / part_dir);
}
if (!transaction)
throw Exception(ErrorCodes::LOGICAL_ERROR, "There is no uncommitted transaction");
void DataPartStorageBuilderOnDisk::createProjection(const std::string & name)
{
transaction->createDirectory(fs::path(root_path) / part_dir / name);
}
if (has_shared_transaction)
throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot commit shared transaction");
ReservationPtr DataPartStorageBuilderOnDisk::reserve(UInt64 bytes)
{
auto res = volume->reserve(bytes);
if (!res)
throw Exception(ErrorCodes::NOT_ENOUGH_SPACE, "Cannot reserve {}, not enough space", ReadableSize(bytes));
return res;
}
DataPartStorageBuilderPtr DataPartStorageBuilderOnDisk::getProjection(const std::string & name) const
{
return std::make_shared<DataPartStorageBuilderOnDisk>(volume, std::string(fs::path(root_path) / part_dir), name);
}
DataPartStoragePtr DataPartStorageBuilderOnDisk::getStorage() const
{
return std::make_shared<DataPartStorageOnDisk>(volume, root_path, part_dir);
}
void DataPartStorageBuilderOnDisk::commit()
{
transaction->commit();
transaction.reset();
}
}

119
src/Storages/MergeTree/DataPartStorageOnDisk.h
View File

@ -21,6 +21,7 @@ public:
std::string getPartDirectory() const override { return part_dir; }
std::string getFullRootPath() const override;
MutableDataPartStoragePtr getProjection(const std::string & name) override;
DataPartStoragePtr getProjection(const std::string & name) const override;
bool exists() const override;
@ -41,7 +42,6 @@ public:
std::optional<size_t> read_hint,
std::optional<size_t> file_size) const override;
void loadVersionMetadata(VersionMetadata & version, Poco::Logger * log) const override;
void checkConsistency(const MergeTreeDataPartChecksums & checksums) const override;
void remove(
@ -60,7 +60,6 @@ public:
std::optional<Strings> & original_files_list) const;
void setRelativePath(const std::string & path) override;
void onRename(const std::string & new_root_path, const std::string & new_part_dir) override;
std::string getDiskName() const override;
std::string getDiskType() const override;
@ -68,30 +67,14 @@ public:
bool supportZeroCopyReplication() const override;
bool supportParallelWrite() const override;
bool isBroken() const override;
void syncRevision(UInt64 revision) override;
void syncRevision(UInt64 revision) const override;
UInt64 getRevision() const override;
std::unordered_map<String, String> getSerializedMetadata(const std::vector<String> & paths) const override;
std::string getDiskPath() const override;
DisksSet::const_iterator isStoredOnDisk(const DisksSet & disks) const override;
ReservationPtr reserve(UInt64 bytes) const override;
ReservationPtr tryReserve(UInt64 bytes) const override;
size_t getVolumeIndex(const IStoragePolicy &) const override;
void writeChecksums(const MergeTreeDataPartChecksums & checksums, const WriteSettings & settings) const override;
void writeColumns(const NamesAndTypesList & columns, const WriteSettings & settings) const override;
void writeVersionMetadata(const VersionMetadata & version, bool fsync_part_dir) const override;
void appendCSNToVersionMetadata(const VersionMetadata & version, VersionMetadata::WhichCSN which_csn) const override;
void appendRemovalTIDToVersionMetadata(const VersionMetadata & version, bool clear) const override;
void writeDeleteOnDestroyMarker(Poco::Logger * log) const override;
void removeDeleteOnDestroyMarker() const override;
void removeVersionMetadata() const override;
String getUniqueId() const override;
bool shallParticipateInMerges(const IStoragePolicy &) const override;
void backup(
const MergeTreeDataPartChecksums & checksums,
const NameSet & files_without_checksums,
@ -100,7 +83,7 @@ public:
bool make_temporary_hard_links,
TemporaryFilesOnDisks * temp_dirs) const override;
DataPartStoragePtr freeze(
MutableDataPartStoragePtr freeze(
const std::string & to,
const std::string & dir_path,
bool make_source_readonly,
@ -108,7 +91,7 @@ public:
bool copy_instead_of_hardlink,
const NameSet & files_to_copy_instead_of_hardlinks) const override;
DataPartStoragePtr clone(
MutableDataPartStoragePtr clonePart(
const std::string & to,
const std::string & dir_path,
const DiskPtr & disk,
@ -116,11 +99,51 @@ public:
void changeRootPath(const std::string & from_root, const std::string & to_root) override;
DataPartStorageBuilderPtr getBuilder() const override;
void createDirectories() override;
void createProjection(const std::string & name) override;
std::unique_ptr<WriteBufferFromFileBase> writeFile(
const String & name,
size_t buf_size,
const WriteSettings & settings) override;
std::unique_ptr<WriteBufferFromFileBase> writeTransactionFile(WriteMode mode) const override;
void createFile(const String & name) override;
void moveFile(const String & from_name, const String & to_name) override;
void replaceFile(const String & from_name, const String & to_name) override;
void removeFile(const String & name) override;
void removeFileIfExists(const String & name) override;
void removeRecursive() override;
void removeSharedRecursive(bool keep_in_remote_fs) override;
SyncGuardPtr getDirectorySyncGuard() const override;
void createHardLinkFrom(const IDataPartStorage & source, const std::string & from, const std::string & to) override;
void rename(
const std::string & new_root_path,
const std::string & new_part_dir,
Poco::Logger * log,
bool remove_new_dir_if_exists,
bool fsync_part_dir) override;
void beginTransaction() override;
void commitTransaction() override;
bool hasActiveTransaction() const override { return transaction != nullptr; }
private:
VolumePtr volume;
std::string root_path;
std::string part_dir;
DiskTransactionPtr transaction;
bool has_shared_transaction = false;
DataPartStorageOnDisk(VolumePtr volume_, std::string root_path_, std::string part_dir_, DiskTransactionPtr transaction_);
template <typename Op>
void executeOperation(Op && op);
void clearDirectory(
const std::string & dir,
@ -134,56 +157,4 @@ private:
bool is_projection) const;
};
class DataPartStorageBuilderOnDisk final : public IDataPartStorageBuilder
{
public:
DataPartStorageBuilderOnDisk(VolumePtr volume_, std::string root_path_, std::string part_dir_);
void setRelativePath(const std::string & path) override;
bool exists() const override;
void createDirectories() override;
void createProjection(const std::string & name) override;
std::string getPartDirectory() const override { return part_dir; }
std::string getFullPath() const override;
std::string getRelativePath() const override;
std::unique_ptr<WriteBufferFromFileBase> writeFile(
const String & name,
size_t buf_size,
const WriteSettings & settings) override;
void removeFile(const String & name) override;
void removeFileIfExists(const String & name) override;
void removeRecursive() override;
void removeSharedRecursive(bool keep_in_remote_fs) override;
SyncGuardPtr getDirectorySyncGuard() const override;
void createHardLinkFrom(const IDataPartStorage & source, const std::string & from, const std::string & to) const override;
ReservationPtr reserve(UInt64 bytes) override;
DataPartStorageBuilderPtr getProjection(const std::string & name) const override;
DataPartStoragePtr getStorage() const override;
void rename(
const std::string & new_root_path,
const std::string & new_part_dir,
Poco::Logger * log,
bool remove_new_dir_if_exists,
bool fsync_part_dir) override;
void commit() override;
private:
VolumePtr volume;
std::string root_path;
std::string part_dir;
DiskTransactionPtr transaction;
};
}

98
src/Storages/MergeTree/DataPartsExchange.cpp
View File

@ -13,9 +13,9 @@
#include <Storages/MergeTree/MergedBlockOutputStream.h>
#include <Storages/MergeTree/ReplicatedFetchList.h>
#include <Storages/StorageReplicatedMergeTree.h>
#include <Storages/MergeTree/DataPartStorageOnDisk.h>
#include <Common/CurrentMetrics.h>
#include <Common/NetException.h>
#include <Storages/MergeTree/DataPartStorageOnDisk.h>
#include <Disks/IO/createReadBufferFromFileBase.h>
#include <base/scope_guard.h>
#include <Poco/Net/HTTPRequest.h>
@ -147,12 +147,13 @@ void Service::processQuery(const HTMLForm & params, ReadBuffer & /*body*/, Write
CurrentMetrics::Increment metric_increment{CurrentMetrics::ReplicatedSend};
if (part->data_part_storage->isStoredOnRemoteDisk())
if (part->getDataPartStorage().isStoredOnRemoteDisk())
{
UInt64 revision = parse<UInt64>(params.get("disk_revision", "0"));
if (revision)
part->data_part_storage->syncRevision(revision);
revision = part->data_part_storage->getRevision();
part->getDataPartStorage().syncRevision(revision);
revision = part->getDataPartStorage().getRevision();
if (revision)
response.addCookie({"disk_revision", toString(revision)});
}
@ -184,8 +185,8 @@ void Service::processQuery(const HTMLForm & params, ReadBuffer & /*body*/, Write
!isInMemoryPart(part) &&
client_protocol_version >= REPLICATION_PROTOCOL_VERSION_WITH_PARTS_ZERO_COPY)
{
auto disk_type = part->data_part_storage->getDiskType();
if (part->data_part_storage->supportZeroCopyReplication() && std::find(capability.begin(), capability.end(), disk_type) != capability.end())
auto disk_type = part->getDataPartStorage().getDiskType();
if (part->getDataPartStorage().supportZeroCopyReplication() && std::find(capability.begin(), capability.end(), disk_type) != capability.end())
{
/// Send metadata if the receiver's capability covers the source disk type.
response.addCookie({"remote_fs_metadata", disk_type});
@ -307,12 +308,12 @@ MergeTreeData::DataPart::Checksums Service::sendPartFromDisk(
{
String file_name = it.first;
UInt64 size = part->data_part_storage->getFileSize(file_name);
UInt64 size = part->getDataPartStorage().getFileSize(file_name);
writeStringBinary(it.first, out);
writeBinary(size, out);
auto file_in = part->data_part_storage->readFile(file_name, {}, std::nullopt, std::nullopt);
auto file_in = part->getDataPartStorage().readFile(file_name, {}, std::nullopt, std::nullopt);
HashingWriteBuffer hashing_out(out);
copyDataWithThrottler(*file_in, hashing_out, blocker.getCounter(), data.getSendsThrottler());
@ -323,7 +324,7 @@ MergeTreeData::DataPart::Checksums Service::sendPartFromDisk(
throw Exception(
ErrorCodes::BAD_SIZE_OF_FILE_IN_DATA_PART,
"Unexpected size of file {}, expected {} got {}",
std::string(fs::path(part->data_part_storage->getRelativePath()) / file_name),
std::string(fs::path(part->getDataPartStorage().getRelativePath()) / file_name),
hashing_out.count(), size);
writePODBinary(hashing_out.getHash(), out);
@ -342,9 +343,9 @@ MergeTreeData::DataPart::Checksums Service::sendPartFromDiskRemoteMeta(
bool send_part_id,
const std::map<String, std::shared_ptr<IMergeTreeDataPart>> & projections)
{
const auto * data_part_storage_on_disk = dynamic_cast<const DataPartStorageOnDisk *>(part->data_part_storage.get());
const auto * data_part_storage_on_disk = dynamic_cast<const DataPartStorageOnDisk *>(&part->getDataPartStorage());
if (!data_part_storage_on_disk)
throw Exception(ErrorCodes::LOGICAL_ERROR, "Storage '{}' doesn't support zero-copy replication", part->data_part_storage->getDiskName());
throw Exception(ErrorCodes::LOGICAL_ERROR, "Storage '{}' doesn't support zero-copy replication", part->getDataPartStorage().getDiskName());
if (!data_part_storage_on_disk->supportZeroCopyReplication())
throw Exception(ErrorCodes::LOGICAL_ERROR, "Disk '{}' doesn't support zero-copy replication", data_part_storage_on_disk->getDiskName());
@ -365,7 +366,7 @@ MergeTreeData::DataPart::Checksums Service::sendPartFromDiskRemoteMeta(
std::vector<std::string> paths;
paths.reserve(checksums.files.size());
for (const auto & it : checksums.files)
paths.push_back(fs::path(part->data_part_storage->getRelativePath()) / it.first);
paths.push_back(fs::path(part->getDataPartStorage().getRelativePath()) / it.first);
/// Serialized metadatadatas with zero ref counts.
auto metadatas = data_part_storage_on_disk->getSerializedMetadata(paths);
@ -399,7 +400,7 @@ MergeTreeData::DataPart::Checksums Service::sendPartFromDiskRemoteMeta(
for (const auto & it : checksums.files)
{
const String & file_name = it.first;
String file_path_prefix = fs::path(part->data_part_storage->getRelativePath()) / file_name;
String file_path_prefix = fs::path(part->getDataPartStorage().getRelativePath()) / file_name;
/// Just some additional checks
String metadata_file_path = fs::path(data_part_storage_on_disk->getDiskPath()) / file_path_prefix;
@ -728,13 +729,9 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToMemory(
data.getRelativeDataPath(),
part_name);
auto data_part_storage_builder = std::make_shared<DataPartStorageBuilderOnDisk>(
volume,
data.getRelativeDataPath(),
part_name);
MergeTreeData::MutableDataPartPtr new_data_part =
std::make_shared<MergeTreeDataPartInMemory>(data, part_name, data_part_storage);
new_data_part->version.setCreationTID(Tx::PrehistoricTID, nullptr);
for (auto i = 0ul; i < projections; ++i)
@ -750,7 +747,6 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToMemory(
throttler->add(block.bytes());
auto projection_part_storage = data_part_storage->getProjection(projection_name + ".proj");
auto projection_part_storage_builder = data_part_storage_builder->getProjection(projection_name + ".proj");
MergeTreePartInfo new_part_info("all", 0, 0, 0);
MergeTreeData::MutableDataPartPtr new_projection_part =
@ -764,7 +760,6 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToMemory(
MergedBlockOutputStream part_out(
new_projection_part,
projection_part_storage_builder,
metadata_snapshot->projections.get(projection_name).metadata,
block.getNamesAndTypesList(),
{},
@ -792,7 +787,7 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToMemory(
new_data_part->partition.create(metadata_snapshot, block, 0, context);
MergedBlockOutputStream part_out(
new_data_part, data_part_storage_builder, metadata_snapshot, block.getNamesAndTypesList(), {},
new_data_part, metadata_snapshot, block.getNamesAndTypesList(), {},
CompressionCodecFactory::instance().get("NONE", {}), NO_TRANSACTION_PTR);
part_out.write(block);
@ -804,7 +799,7 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToMemory(
void Fetcher::downloadBasePartOrProjectionPartToDiskRemoteMeta(
const String & replica_path,
DataPartStorageBuilderPtr & data_part_storage_builder,
const MutableDataPartStoragePtr & data_part_storage,
PooledReadWriteBufferFromHTTP & in,
MergeTreeData::DataPart::Checksums & checksums,
ThrottlerPtr throttler) const
@ -820,7 +815,7 @@ void Fetcher::downloadBasePartOrProjectionPartToDiskRemoteMeta(
readStringBinary(file_name, in);
readBinary(file_size, in);
String metadata_file = fs::path(data_part_storage_builder->getFullPath()) / file_name;
String metadata_file = fs::path(data_part_storage->getFullPath()) / file_name;
{
auto file_out = std::make_unique<WriteBufferFromFile>(metadata_file, DBMS_DEFAULT_BUFFER_SIZE, -1, 0666, nullptr, 0);
@ -834,8 +829,8 @@ void Fetcher::downloadBasePartOrProjectionPartToDiskRemoteMeta(
/// NOTE The is_cancelled flag also makes sense to check every time you read over the network,
/// performing a poll with a not very large timeout.
/// And now we check it only between read chunks (in the `copyData` function).
data_part_storage_builder->removeSharedRecursive(true);
data_part_storage_builder->commit();
data_part_storage->removeSharedRecursive(true);
data_part_storage->commitTransaction();
throw Exception("Fetching of part was cancelled", ErrorCodes::ABORTED);
}
@ -861,7 +856,7 @@ void Fetcher::downloadBasePartOrProjectionPartToDiskRemoteMeta(
void Fetcher::downloadBaseOrProjectionPartToDisk(
const String & replica_path,
DataPartStorageBuilderPtr & data_part_storage_builder,
const MutableDataPartStoragePtr & data_part_storage,
bool sync,
PooledReadWriteBufferFromHTTP & in,
MergeTreeData::DataPart::Checksums & checksums,
@ -880,14 +875,14 @@ void Fetcher::downloadBaseOrProjectionPartToDisk(
/// File must be inside "absolute_part_path" directory.
/// Otherwise malicious ClickHouse replica may force us to write to arbitrary path.
String absolute_file_path = fs::weakly_canonical(fs::path(data_part_storage_builder->getRelativePath()) / file_name);
if (!startsWith(absolute_file_path, fs::weakly_canonical(data_part_storage_builder->getRelativePath()).string()))
String absolute_file_path = fs::weakly_canonical(fs::path(data_part_storage->getRelativePath()) / file_name);
if (!startsWith(absolute_file_path, fs::weakly_canonical(data_part_storage->getRelativePath()).string()))
throw Exception(ErrorCodes::INSECURE_PATH,
"File path ({}) doesn't appear to be inside part path ({}). "
"This may happen if we are trying to download part from malicious replica or logical error.",
absolute_file_path, data_part_storage_builder->getRelativePath());
absolute_file_path, data_part_storage->getRelativePath());
auto file_out = data_part_storage_builder->writeFile(file_name, std::min<UInt64>(file_size, DBMS_DEFAULT_BUFFER_SIZE), {});
auto file_out = data_part_storage->writeFile(file_name, std::min<UInt64>(file_size, DBMS_DEFAULT_BUFFER_SIZE), {});
HashingWriteBuffer hashing_out(*file_out);
copyDataWithThrottler(in, hashing_out, file_size, blocker.getCounter(), throttler);
@ -896,7 +891,7 @@ void Fetcher::downloadBaseOrProjectionPartToDisk(
/// NOTE The is_cancelled flag also makes sense to check every time you read over the network,
/// performing a poll with a not very large timeout.
/// And now we check it only between read chunks (in the `copyData` function).
data_part_storage_builder->removeRecursive();
data_part_storage->removeRecursive();
throw Exception("Fetching of part was cancelled", ErrorCodes::ABORTED);
}
@ -906,7 +901,7 @@ void Fetcher::downloadBaseOrProjectionPartToDisk(
if (expected_hash != hashing_out.getHash())
throw Exception(ErrorCodes::CHECKSUM_DOESNT_MATCH,
"Checksum mismatch for file {} transferred from {}",
(fs::path(data_part_storage_builder->getFullPath()) / file_name).string(),
(fs::path(data_part_storage->getFullPath()) / file_name).string(),
replica_path);
if (file_name != "checksums.txt" &&
@ -951,15 +946,12 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk(
part_relative_path,
part_dir);
DataPartStorageBuilderPtr data_part_storage_builder = std::make_shared<DataPartStorageBuilderOnDisk>(
volume,
part_relative_path,
part_dir);
data_part_storage->beginTransaction();
if (data_part_storage_builder->exists())
if (data_part_storage->exists())
{
LOG_WARNING(log, "Directory {} already exists, probably result of a failed fetch. Will remove it before fetching part.",
data_part_storage_builder->getFullPath());
data_part_storage->getFullPath());
/// Even if it's a temporary part it could be downloaded with zero copy replication and this function
/// is executed as a callback.
@ -967,10 +959,10 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk(
/// We don't control the amount of refs for temporary parts so we cannot decide can we remove blobs
/// or not. So we are not doing it
bool keep_shared = disk->supportZeroCopyReplication() && data_settings->allow_remote_fs_zero_copy_replication;
data_part_storage_builder->removeSharedRecursive(keep_shared);
data_part_storage->removeSharedRecursive(keep_shared);
}
data_part_storage_builder->createDirectories();
data_part_storage->createDirectories();
SyncGuardPtr sync_guard;
if (data.getSettings()->fsync_part_directory)
@ -985,19 +977,18 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDisk(
MergeTreeData::DataPart::Checksums projection_checksum;
auto projection_part_storage = data_part_storage->getProjection(projection_name + ".proj");
auto projection_part_storage_builder = data_part_storage_builder->getProjection(projection_name + ".proj");
projection_part_storage_builder->createDirectories();
projection_part_storage->createDirectories();
downloadBaseOrProjectionPartToDisk(
replica_path, projection_part_storage_builder, sync, in, projection_checksum, throttler);
replica_path, projection_part_storage, sync, in, projection_checksum, throttler);
checksums.addFile(
projection_name + ".proj", projection_checksum.getTotalSizeOnDisk(), projection_checksum.getTotalChecksumUInt128());
}
// Download the base part
downloadBaseOrProjectionPartToDisk(replica_path, data_part_storage_builder, sync, in, checksums, throttler);
downloadBaseOrProjectionPartToDisk(replica_path, data_part_storage, sync, in, checksums, throttler);
assertEOF(in);
data_part_storage->commitTransaction();
MergeTreeData::MutableDataPartPtr new_data_part = data.createPart(part_name, data_part_storage);
new_data_part->version.setCreationTID(Tx::PrehistoricTID, nullptr);
new_data_part->is_temp = true;
@ -1043,17 +1034,14 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDiskRemoteMeta(
part_relative_path,
part_dir);
DataPartStorageBuilderPtr data_part_storage_builder = std::make_shared<DataPartStorageBuilderOnDisk>(
volume,
part_relative_path,
part_dir);
data_part_storage->beginTransaction();
if (data_part_storage->exists())
throw Exception(ErrorCodes::DIRECTORY_ALREADY_EXISTS, "Directory {} already exists.", data_part_storage->getFullPath());
CurrentMetrics::Increment metric_increment{CurrentMetrics::ReplicatedFetch};
volume->getDisk()->createDirectories(data_part_storage->getFullPath());
data_part_storage->createDirectories();
for (auto i = 0ul; i < projections; ++i)
{
@ -1062,24 +1050,22 @@ MergeTreeData::MutableDataPartPtr Fetcher::downloadPartToDiskRemoteMeta(
MergeTreeData::DataPart::Checksums projection_checksum;
auto projection_part_storage = data_part_storage->getProjection(projection_name + ".proj");
auto projection_part_storage_builder = data_part_storage_builder->getProjection(projection_name + ".proj");
projection_part_storage_builder->createDirectories();
projection_part_storage->createDirectories();
downloadBasePartOrProjectionPartToDiskRemoteMeta(
replica_path, projection_part_storage_builder, in, projection_checksum, throttler);
replica_path, projection_part_storage, in, projection_checksum, throttler);
checksums.addFile(
projection_name + ".proj", projection_checksum.getTotalSizeOnDisk(), projection_checksum.getTotalChecksumUInt128());
}
downloadBasePartOrProjectionPartToDiskRemoteMeta(
replica_path, data_part_storage_builder, in, checksums, throttler);
replica_path, data_part_storage, in, checksums, throttler);
assertEOF(in);
MergeTreeData::MutableDataPartPtr new_data_part;
try
{
data_part_storage_builder->commit();
data_part_storage->commitTransaction();
new_data_part = data.createPart(part_name, data_part_storage);
new_data_part->version.setCreationTID(Tx::PrehistoricTID, nullptr);

5
src/Storages/MergeTree/DataPartsExchange.h
View File

@ -94,7 +94,7 @@ public:
private:
void downloadBaseOrProjectionPartToDisk(
const String & replica_path,
DataPartStorageBuilderPtr & data_part_storage_builder,
const MutableDataPartStoragePtr & data_part_storage,
bool sync,
PooledReadWriteBufferFromHTTP & in,
MergeTreeData::DataPart::Checksums & checksums,
@ -102,12 +102,11 @@ private:
void downloadBasePartOrProjectionPartToDiskRemoteMeta(
const String & replica_path,
DataPartStorageBuilderPtr & data_part_storage_builder,
const MutableDataPartStoragePtr & data_part_storage,
PooledReadWriteBufferFromHTTP & in,
MergeTreeData::DataPart::Checksums & checksums,
ThrottlerPtr throttler) const;
MergeTreeData::MutableDataPartPtr downloadPartToDisk(
const String & part_name,
const String & replica_path,

129
src/Storages/MergeTree/IDataPartStorage.h
View File

@ -4,6 +4,9 @@
#include <Core/NamesAndTypes.h>
#include <Interpreters/TransactionVersionMetadata.h>
#include <Storages/MergeTree/MergeTreeDataPartState.h>
#include <Disks/WriteMode.h>
#include <boost/core/noncopyable.hpp>
#include <memory>
#include <optional>
namespace DB
@ -18,6 +21,7 @@ struct CanRemoveDescription
NameSet files_not_to_remove;
};
using CanRemoveCallback = std::function<CanRemoveDescription()>;
class IDataPartStorageIterator
@ -61,13 +65,10 @@ struct WriteSettings;
class TemporaryFileOnDisk;
class IDataPartStorageBuilder;
using DataPartStorageBuilderPtr = std::shared_ptr<IDataPartStorageBuilder>;
/// This is an abstraction of storage for data part files.
/// Ideally, it is assumed to contains read-only methods from IDisk.
/// It is not fulfilled now, but let's try our best.
class IDataPartStorage
class IDataPartStorage : public boost::noncopyable
{
public:
virtual ~IDataPartStorage() = default;
@ -81,16 +82,19 @@ public:
/// virtual std::string getRelativeRootPath() const = 0;
/// Get a storage for projection.
virtual std::shared_ptr<IDataPartStorage> getProjection(const std::string & name) const = 0;
virtual std::shared_ptr<IDataPartStorage> getProjection(const std::string & name) = 0;
virtual std::shared_ptr<const IDataPartStorage> getProjection(const std::string & name) const = 0;
/// Part directory exists.
virtual bool exists() const = 0;
/// File inside part directory exists. Specified path is relative to the part path.
virtual bool exists(const std::string & name) const = 0;
virtual bool isDirectory(const std::string & name) const = 0;
/// Modification time for part directory.
virtual Poco::Timestamp getLastModified() const = 0;
/// Iterate part directory. Iteration in subdirectory is not needed yet.
virtual DataPartStorageIteratorPtr iterate() const = 0;
@ -107,7 +111,6 @@ public:
std::optional<size_t> read_hint,
std::optional<size_t> file_size) const = 0;
virtual void loadVersionMetadata(VersionMetadata & version, Poco::Logger * log) const = 0;
virtual void checkConsistency(const MergeTreeDataPartChecksums & checksums) const = 0;
struct ProjectionChecksums
@ -129,12 +132,12 @@ public:
/// Get a name like 'prefix_partdir_tryN' which does not exist in a root dir.
/// TODO: remove it.
virtual std::optional<String> getRelativePathForPrefix(Poco::Logger * log, const String & prefix, bool detached, bool broken) const = 0;
virtual std::optional<String> getRelativePathForPrefix(
Poco::Logger * log, const String & prefix, bool detached, bool broken) const = 0;
/// Reset part directory, used for im-memory parts.
/// Reset part directory, used for in-memory parts.
/// TODO: remove it.
virtual void setRelativePath(const std::string & path) = 0;
virtual void onRename(const std::string & new_root_path, const std::string & new_part_dir) = 0;
/// Some methods from IDisk. Needed to avoid getting internal IDisk interface.
virtual std::string getDiskName() const = 0;
@ -143,41 +146,26 @@ public:
virtual bool supportZeroCopyReplication() const { return false; }
virtual bool supportParallelWrite() const = 0;
virtual bool isBroken() const = 0;
virtual void syncRevision(UInt64 revision) = 0;
/// TODO: remove or at least remove const.
virtual void syncRevision(UInt64 revision) const = 0;
virtual UInt64 getRevision() const = 0;
virtual std::unordered_map<String, String> getSerializedMetadata(const std::vector<String> & paths) const = 0;
/// Get a path for internal disk if relevant. It is used mainly for logging.
virtual std::string getDiskPath() const = 0;
/// Check if data part is stored on one of the specified disk in set.
using DisksSet = std::unordered_set<DiskPtr>;
virtual DisksSet::const_iterator isStoredOnDisk(const DisksSet & disks) const { return disks.end(); }
/// Reserve space on the same disk.
/// Probably we should try to remove it later.
virtual ReservationPtr reserve(UInt64 /*bytes*/) const { return nullptr; }
virtual ReservationPtr tryReserve(UInt64 /*bytes*/) const { return nullptr; }
virtual size_t getVolumeIndex(const IStoragePolicy &) const { return 0; }
/// Some methods which change data part internals possibly after creation.
/// Probably we should try to remove it later.
virtual void writeChecksums(const MergeTreeDataPartChecksums & checksums, const WriteSettings & settings) const = 0;
virtual void writeColumns(const NamesAndTypesList & columns, const WriteSettings & settings) const = 0;
virtual void writeVersionMetadata(const VersionMetadata & version, bool fsync_part_dir) const = 0;
virtual void appendCSNToVersionMetadata(const VersionMetadata & version, VersionMetadata::WhichCSN which_csn) const = 0;
virtual void appendRemovalTIDToVersionMetadata(const VersionMetadata & version, bool clear) const = 0;
virtual void writeDeleteOnDestroyMarker(Poco::Logger * log) const = 0;
virtual void removeDeleteOnDestroyMarker() const = 0;
virtual void removeVersionMetadata() const = 0;
/// TODO: remove constness
virtual ReservationPtr reserve(UInt64 /*bytes*/) const { return nullptr; }
virtual ReservationPtr tryReserve(UInt64 /*bytes*/) const { return nullptr; }
/// A leak of abstraction.
/// Return some uniq string for file.
/// Required for distinguish different copies of the same part on remote FS.
virtual String getUniqueId() const = 0;
/// A leak of abstraction
virtual bool shallParticipateInMerges(const IStoragePolicy &) const { return true; }
/// Create a backup of a data part.
/// This method adds a new entry to backup_entries.
/// Also creates a new tmp_dir for internal disk (if disk is mentioned the first time).
@ -205,7 +193,7 @@ public:
const NameSet & files_to_copy_instead_of_hardlinks) const = 0;
/// Make a full copy of a data part into 'to/dir_path' (possibly to a different disk).
virtual std::shared_ptr<IDataPartStorage> clone(
virtual std::shared_ptr<IDataPartStorage> clonePart(
const std::string & to,
const std::string & dir_path,
const DiskPtr & disk,
@ -215,33 +203,22 @@ public:
/// Right now, this is needed for rename table query.
virtual void changeRootPath(const std::string & from_root, const std::string & to_root) = 0;
/// Leak of abstraction as well. We should use builder as one-time object which allow
/// us to build parts, while storage should be read-only method to access part properties
/// related to disk. However our code is really tricky and sometimes we need ad-hoc builders.
virtual DataPartStorageBuilderPtr getBuilder() const = 0;
};
using DataPartStoragePtr = std::shared_ptr<IDataPartStorage>;
/// This interface is needed to write data part.
class IDataPartStorageBuilder
{
public:
virtual ~IDataPartStorageBuilder() = default;
/// Reset part directory, used for im-memory parts
virtual void setRelativePath(const std::string & path) = 0;
virtual std::string getPartDirectory() const = 0;
virtual std::string getFullPath() const = 0;
virtual std::string getRelativePath() const = 0;
virtual bool exists() const = 0;
virtual void createDirectories() = 0;
virtual void createProjection(const std::string & name) = 0;
virtual std::unique_ptr<WriteBufferFromFileBase> writeFile(const String & name, size_t buf_size, const WriteSettings & settings) = 0;
virtual std::unique_ptr<WriteBufferFromFileBase> writeFile(
const String & name,
size_t buf_size,
const WriteSettings & settings) = 0;
/// A special const method to write transaction file.
/// It's const, because file with transaction metadata
/// can be modified after part creation.
virtual std::unique_ptr<WriteBufferFromFileBase> writeTransactionFile(WriteMode mode) const = 0;
virtual void createFile(const String & name) = 0;
virtual void moveFile(const String & from_name, const String & to_name) = 0;
virtual void replaceFile(const String & from_name, const String & to_name) = 0;
virtual void removeFile(const String & name) = 0;
virtual void removeFileIfExists(const String & name) = 0;
@ -250,20 +227,12 @@ public:
virtual SyncGuardPtr getDirectorySyncGuard() const { return nullptr; }
virtual void createHardLinkFrom(const IDataPartStorage & source, const std::string & from, const std::string & to) const = 0;
virtual ReservationPtr reserve(UInt64 /*bytes*/) { return nullptr; }
virtual std::shared_ptr<IDataPartStorageBuilder> getProjection(const std::string & name) const = 0;
virtual DataPartStoragePtr getStorage() const = 0;
virtual void createHardLinkFrom(const IDataPartStorage & source, const std::string & from, const std::string & to) = 0;
/// Rename part.
/// Ideally, new_root_path should be the same as current root (but it is not true).
/// Examples are: 'all_1_2_1' -> 'detached/all_1_2_1'
/// 'moving/tmp_all_1_2_1' -> 'all_1_2_1'
///
/// To notify storage also call onRename for it with first two args
virtual void rename(
const std::string & new_root_path,
const std::string & new_part_dir,
@ -271,7 +240,35 @@ public:
bool remove_new_dir_if_exists,
bool fsync_part_dir) = 0;
virtual void commit() = 0;
/// Starts a transaction of mutable operations.
virtual void beginTransaction() = 0;
/// Commits a transaction of mutable operations.
virtual void commitTransaction() = 0;
virtual bool hasActiveTransaction() const = 0;
};
using DataPartStoragePtr = std::shared_ptr<const IDataPartStorage>;
using MutableDataPartStoragePtr = std::shared_ptr<IDataPartStorage>;
/// A holder that encapsulates data part storage and
/// gives access to const storage from const methods
/// and to mutable storage from non-const methods.
class DataPartStorageHolder : public boost::noncopyable
{
public:
explicit DataPartStorageHolder(MutableDataPartStoragePtr storage_)
: storage(std::move(storage_))
{
}
IDataPartStorage & getDataPartStorage() { return *storage; }
const IDataPartStorage & getDataPartStorage() const { return *storage; }
MutableDataPartStoragePtr getDataPartStoragePtr() { return storage; }
DataPartStoragePtr getDataPartStoragePtr() const { return storage; }
private:
MutableDataPartStoragePtr storage;
};
}

336
src/Storages/MergeTree/IMergeTreeDataPart.cpp
View File

@ -1,4 +1,5 @@
#include "IMergeTreeDataPart.h"
#include "Storages/MergeTree/IDataPartStorage.h"
#include <optional>
#include <boost/algorithm/string/join.hpp>
@ -101,7 +102,7 @@ void IMergeTreeDataPart::MinMaxIndex::load(const MergeTreeData & data, const Par
}
IMergeTreeDataPart::MinMaxIndex::WrittenFiles IMergeTreeDataPart::MinMaxIndex::store(
const MergeTreeData & data, const DataPartStorageBuilderPtr & data_part_storage_builder, Checksums & out_checksums) const
const MergeTreeData & data, IDataPartStorage & part_storage, Checksums & out_checksums) const
{
auto metadata_snapshot = data.getInMemoryMetadataPtr();
const auto & partition_key = metadata_snapshot->getPartitionKey();
@ -109,20 +110,20 @@ IMergeTreeDataPart::MinMaxIndex::WrittenFiles IMergeTreeDataPart::MinMaxIndex::s
auto minmax_column_names = data.getMinMaxColumnsNames(partition_key);
auto minmax_column_types = data.getMinMaxColumnsTypes(partition_key);
return store(minmax_column_names, minmax_column_types, data_part_storage_builder, out_checksums);
return store(minmax_column_names, minmax_column_types, part_storage, out_checksums);
}
IMergeTreeDataPart::MinMaxIndex::WrittenFiles IMergeTreeDataPart::MinMaxIndex::store(
const Names & column_names,
const DataTypes & data_types,
const DataPartStorageBuilderPtr & data_part_storage_builder,
IDataPartStorage & part_storage,
Checksums & out_checksums) const
{
if (!initialized)
throw Exception(
ErrorCodes::LOGICAL_ERROR,
"Attempt to store uninitialized MinMax index for part {}. This is a bug",
data_part_storage_builder->getFullPath());
part_storage.getFullPath());
WrittenFiles written_files;
@ -131,7 +132,7 @@ IMergeTreeDataPart::MinMaxIndex::WrittenFiles IMergeTreeDataPart::MinMaxIndex::s
String file_name = "minmax_" + escapeForFileName(column_names[i]) + ".idx";
auto serialization = data_types.at(i)->getDefaultSerialization();
auto out = data_part_storage_builder->writeFile(file_name, DBMS_DEFAULT_BUFFER_SIZE, {});
auto out = part_storage.writeFile(file_name, DBMS_DEFAULT_BUFFER_SIZE, {});
HashingWriteBuffer out_hashing(*out);
serialization->serializeBinary(hyperrectangle[i].left, out_hashing);
serialization->serializeBinary(hyperrectangle[i].right, out_hashing);
@ -301,13 +302,13 @@ static void decrementTypeMetric(MergeTreeDataPartType type)
IMergeTreeDataPart::IMergeTreeDataPart(
const MergeTreeData & storage_,
const String & name_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
Type part_type_,
const IMergeTreeDataPart * parent_part_)
: storage(storage_)
: DataPartStorageHolder(data_part_storage_)
, storage(storage_)
, name(name_)
, info(MergeTreePartInfo::fromPartName(name_, storage.format_version))
, data_part_storage(parent_part_ ? parent_part_->data_part_storage : data_part_storage_)
, index_granularity_info(storage_, part_type_)
, part_type(part_type_)
, parent_part(parent_part_)
@ -315,6 +316,7 @@ IMergeTreeDataPart::IMergeTreeDataPart(
{
if (parent_part)
state = MergeTreeDataPartState::Active;
incrementStateMetric(state);
incrementTypeMetric(part_type);
@ -328,13 +330,13 @@ IMergeTreeDataPart::IMergeTreeDataPart(
const MergeTreeData & storage_,
const String & name_,
const MergeTreePartInfo & info_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
Type part_type_,
const IMergeTreeDataPart * parent_part_)
: storage(storage_)
: DataPartStorageHolder(data_part_storage_)
, storage(storage_)
, name(name_)
, info(info_)
, data_part_storage(data_part_storage_)
, index_granularity_info(storage_, part_type_)
, part_type(part_type_)
, parent_part(parent_part_)
@ -342,6 +344,7 @@ IMergeTreeDataPart::IMergeTreeDataPart(
{
if (parent_part)
state = MergeTreeDataPartState::Active;
incrementStateMetric(state);
incrementTypeMetric(part_type);
@ -505,17 +508,17 @@ void IMergeTreeDataPart::removeIfNeeded()
std::string path;
try
{
path = data_part_storage->getRelativePath();
path = getDataPartStorage().getRelativePath();
if (!data_part_storage->exists()) // path
if (!getDataPartStorage().exists()) // path
return;
if (is_temp)
{
String file_name = fileName(data_part_storage->getPartDirectory());
String file_name = fileName(getDataPartStorage().getPartDirectory());
if (file_name.empty())
throw Exception(ErrorCodes::LOGICAL_ERROR, "relative_path {} of part {} is invalid or not set", data_part_storage->getPartDirectory(), name);
throw Exception(ErrorCodes::LOGICAL_ERROR, "relative_path {} of part {} is invalid or not set", getDataPartStorage().getPartDirectory(), name);
if (!startsWith(file_name, "tmp") && !endsWith(file_name, ".tmp_proj"))
{
@ -620,7 +623,7 @@ String IMergeTreeDataPart::getColumnNameWithMinimumCompressedSize(bool with_subc
}
if (!minimum_size_column)
throw Exception("Could not find a column of minimum size in MergeTree, part " + data_part_storage->getFullPath(), ErrorCodes::LOGICAL_ERROR);
throw Exception("Could not find a column of minimum size in MergeTree, part " + getDataPartStorage().getFullPath(), ErrorCodes::LOGICAL_ERROR);
return *minimum_size_column;
}
@ -698,9 +701,9 @@ void IMergeTreeDataPart::loadProjections(bool require_columns_checksums, bool ch
for (const auto & projection : metadata_snapshot->projections)
{
String path = /*getRelativePath() + */ projection.name + ".proj";
if (data_part_storage->exists(path))
if (getDataPartStorage().exists(path))
{
auto projection_part_storage = data_part_storage->getProjection(projection.name + ".proj");
auto projection_part_storage = getDataPartStorage().getProjection(projection.name + ".proj");
auto part = storage.createPart(projection.name, {"all", 0, 0, 0}, projection_part_storage, this);
part->loadColumnsChecksumsIndexes(require_columns_checksums, check_consistency);
projection_parts.emplace(projection.name, std::move(part));
@ -741,8 +744,8 @@ void IMergeTreeDataPart::loadIndex()
loaded_index[i]->reserve(index_granularity.getMarksCount());
}
String index_name = "primary" + getIndexExtensionFromFilesystem(data_part_storage).value();
String index_path = fs::path(data_part_storage->getRelativePath()) / index_name;
String index_name = "primary" + getIndexExtensionFromFilesystem(getDataPartStorage()).value();
String index_path = fs::path(getDataPartStorage().getRelativePath()) / index_name;
auto index_file = metadata_manager->read(index_name);
size_t marks_count = index_granularity.getMarksCount();
@ -781,7 +784,7 @@ void IMergeTreeDataPart::appendFilesOfIndex(Strings & files) const
if (metadata_snapshot->hasPrimaryKey())
{
String index_name = "primary" + getIndexExtensionFromFilesystem(data_part_storage).value();
String index_name = "primary" + getIndexExtensionFromFilesystem(getDataPartStorage()).value();
files.push_back(index_name);
}
}
@ -793,10 +796,10 @@ NameSet IMergeTreeDataPart::getFileNamesWithoutChecksums() const
NameSet result = {"checksums.txt", "columns.txt"};
if (data_part_storage->exists(DEFAULT_COMPRESSION_CODEC_FILE_NAME))
if (getDataPartStorage().exists(DEFAULT_COMPRESSION_CODEC_FILE_NAME))
result.emplace(DEFAULT_COMPRESSION_CODEC_FILE_NAME);
if (data_part_storage->exists(TXN_VERSION_METADATA_FILE_NAME))
if (getDataPartStorage().exists(TXN_VERSION_METADATA_FILE_NAME))
result.emplace(TXN_VERSION_METADATA_FILE_NAME);
return result;
@ -811,7 +814,7 @@ void IMergeTreeDataPart::loadDefaultCompressionCodec()
return;
}
String path = fs::path(data_part_storage->getRelativePath()) / DEFAULT_COMPRESSION_CODEC_FILE_NAME;
String path = fs::path(getDataPartStorage().getRelativePath()) / DEFAULT_COMPRESSION_CODEC_FILE_NAME;
bool exists = metadata_manager->exists(DEFAULT_COMPRESSION_CODEC_FILE_NAME);
if (!exists)
{
@ -851,6 +854,120 @@ void IMergeTreeDataPart::loadDefaultCompressionCodec()
}
}
template <typename Writer>
void IMergeTreeDataPart::writeMetadata(const String & filename, const WriteSettings & settings, Writer && writer)
{
auto & data_part_storage = getDataPartStorage();
auto tmp_filename = filename + ".tmp";
try
{
{
auto out = data_part_storage.writeFile(tmp_filename, 4096, settings);
writer(*out);
out->finalize();
}
data_part_storage.moveFile(tmp_filename, filename);
}
catch (...)
{
try
{
if (data_part_storage.exists(tmp_filename))
data_part_storage.removeFile(tmp_filename);
}
catch (...)
{
tryLogCurrentException("DataPartStorageOnDisk");
}
throw;
}
}
void IMergeTreeDataPart::writeChecksums(const MergeTreeDataPartChecksums & checksums_, const WriteSettings & settings)
{
writeMetadata("checksums.txt", settings, [&checksums_](auto & buffer)
{
checksums_.write(buffer);
});
}
void IMergeTreeDataPart::writeColumns(const NamesAndTypesList & columns_, const WriteSettings & settings)
{
writeMetadata("columns.txt", settings, [&columns_](auto & buffer)
{
columns_.writeText(buffer);
});
}
void IMergeTreeDataPart::writeVersionMetadata(const VersionMetadata & version_, bool fsync_part_dir) const
{
static constexpr auto filename = "txn_version.txt";
static constexpr auto tmp_filename = "txn_version.txt.tmp";
auto & data_part_storage = const_cast<IDataPartStorage &>(getDataPartStorage());
try
{
{
/// TODO IDisk interface does not allow to open file with O_EXCL flag (for DiskLocal),
/// so we create empty file at first (expecting that createFile throws if file already exists)
/// and then overwrite it.
data_part_storage.createFile(tmp_filename);
auto write_settings = storage.getContext()->getWriteSettings();
auto buf = data_part_storage.writeFile(tmp_filename, 256, write_settings);
version_.write(*buf);
buf->finalize();
buf->sync();
}
SyncGuardPtr sync_guard;
if (fsync_part_dir)
sync_guard = data_part_storage.getDirectorySyncGuard();
data_part_storage.replaceFile(tmp_filename, filename);
}
catch (...)
{
try
{
if (data_part_storage.exists(tmp_filename))
data_part_storage.removeFile(tmp_filename);
}
catch (...)
{
tryLogCurrentException("DataPartStorageOnDisk");
}
throw;
}
}
void IMergeTreeDataPart::writeDeleteOnDestroyMarker()
{
static constexpr auto marker_path = "delete-on-destroy.txt";
try
{
getDataPartStorage().createFile(marker_path);
}
catch (Poco::Exception & e)
{
LOG_ERROR(storage.log, "{} (while creating DeleteOnDestroy marker: {})",
e.what(), (fs::path(getDataPartStorage().getFullPath()) / marker_path).string());
}
}
void IMergeTreeDataPart::removeDeleteOnDestroyMarker()
{
getDataPartStorage().removeFileIfExists("delete-on-destroy.txt");
}
void IMergeTreeDataPart::removeVersionMetadata()
{
getDataPartStorage().removeFileIfExists("txn_version.txt");
}
void IMergeTreeDataPart::appendFilesOfDefaultCompressionCodec(Strings & files)
{
files.push_back(DEFAULT_COMPRESSION_CODEC_FILE_NAME);
@ -880,7 +997,7 @@ CompressionCodecPtr IMergeTreeDataPart::detectDefaultCompressionCodec() const
String candidate_path = /*fs::path(getRelativePath()) */ (ISerialization::getFileNameForStream(part_column, substream_path) + ".bin");
/// We can have existing, but empty .bin files. Example: LowCardinality(Nullable(...)) columns and column_name.dict.null.bin file.
if (data_part_storage->exists(candidate_path) && data_part_storage->getFileSize(candidate_path) != 0)
if (getDataPartStorage().exists(candidate_path) && getDataPartStorage().getFileSize(candidate_path) != 0)
path_to_data_file = candidate_path;
}
});
@ -891,7 +1008,7 @@ CompressionCodecPtr IMergeTreeDataPart::detectDefaultCompressionCodec() const
continue;
}
result = getCompressionCodecForFile(data_part_storage, path_to_data_file);
result = getCompressionCodecForFile(getDataPartStorage(), path_to_data_file);
break;
}
}
@ -936,7 +1053,7 @@ void IMergeTreeDataPart::loadPartitionAndMinMaxIndex()
String calculated_partition_id = partition.getID(metadata_snapshot->getPartitionKey().sample_block);
if (calculated_partition_id != info.partition_id)
throw Exception(
"While loading part " + data_part_storage->getFullPath() + ": calculated partition ID: " + calculated_partition_id
"While loading part " + getDataPartStorage().getFullPath() + ": calculated partition ID: " + calculated_partition_id
+ " differs from partition ID in part name: " + info.partition_id,
ErrorCodes::CORRUPTED_DATA);
}
@ -965,7 +1082,7 @@ void IMergeTreeDataPart::loadChecksums(bool require)
bytes_on_disk = checksums.getTotalSizeOnDisk();
}
else
bytes_on_disk = data_part_storage->calculateTotalSizeOnDisk();
bytes_on_disk = getDataPartStorage().calculateTotalSizeOnDisk();
}
else
{
@ -977,7 +1094,7 @@ void IMergeTreeDataPart::loadChecksums(bool require)
LOG_WARNING(storage.log, "Checksums for part {} not found. Will calculate them from data on disk.", name);
checksums = checkDataPart(shared_from_this(), false);
data_part_storage->writeChecksums(checksums, {});
writeChecksums(checksums, {});
bytes_on_disk = checksums.getTotalSizeOnDisk();
}
@ -990,8 +1107,6 @@ void IMergeTreeDataPart::appendFilesOfChecksums(Strings & files)
void IMergeTreeDataPart::loadRowsCount()
{
//String path = fs::path(getRelativePath()) / "count.txt";
auto read_rows_count = [&]()
{
auto buf = metadata_manager->read("count.txt");
@ -1062,7 +1177,7 @@ void IMergeTreeDataPart::loadRowsCount()
}
else
{
if (data_part_storage->exists("count.txt"))
if (getDataPartStorage().exists("count.txt"))
{
read_rows_count();
return;
@ -1161,7 +1276,7 @@ void IMergeTreeDataPart::appendFilesOfUUID(Strings & files)
void IMergeTreeDataPart::loadColumns(bool require)
{
String path = fs::path(data_part_storage->getRelativePath()) / "columns.txt";
String path = fs::path(getDataPartStorage().getRelativePath()) / "columns.txt";
auto metadata_snapshot = storage.getInMemoryMetadataPtr();
if (parent_part)
metadata_snapshot = metadata_snapshot->projections.get(name).metadata;
@ -1172,18 +1287,18 @@ void IMergeTreeDataPart::loadColumns(bool require)
{
/// We can get list of columns only from columns.txt in compact parts.
if (require || part_type == Type::Compact)
throw Exception("No columns.txt in part " + name + ", expected path " + path + " on drive " + data_part_storage->getDiskName(),
throw Exception("No columns.txt in part " + name + ", expected path " + path + " on drive " + getDataPartStorage().getDiskName(),
ErrorCodes::NO_FILE_IN_DATA_PART);
/// If there is no file with a list of columns, write it down.
for (const NameAndTypePair & column : metadata_snapshot->getColumns().getAllPhysical())
if (data_part_storage->exists(getFileNameForColumn(column) + ".bin"))
if (getDataPartStorage().exists(getFileNameForColumn(column) + ".bin"))
loaded_columns.push_back(column);
if (columns.empty())
throw Exception("No columns in part " + name, ErrorCodes::NO_FILE_IN_DATA_PART);
data_part_storage->writeColumns(loaded_columns, {});
writeColumns(loaded_columns, {});
}
else
{
@ -1227,7 +1342,7 @@ void IMergeTreeDataPart::assertHasVersionMetadata(MergeTreeTransaction * txn) co
name, storage.getStorageID().getNameForLogs(), version.creation_tid, txn ? txn->dumpDescription() : "<none>");
assert(!txn || storage.supportsTransactions());
assert(!txn || data_part_storage->exists(TXN_VERSION_METADATA_FILE_NAME));
assert(!txn || getDataPartStorage().exists(TXN_VERSION_METADATA_FILE_NAME));
}
void IMergeTreeDataPart::storeVersionMetadata(bool force) const
@ -1242,7 +1357,7 @@ void IMergeTreeDataPart::storeVersionMetadata(bool force) const
throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Transactions are not supported for in-memory parts (table: {}, part: {})",
storage.getStorageID().getNameForLogs(), name);
data_part_storage->writeVersionMetadata(version, storage.getSettings()->fsync_part_directory);
writeVersionMetadata(version, storage.getSettings()->fsync_part_directory);
}
void IMergeTreeDataPart::appendCSNToVersionMetadata(VersionMetadata::WhichCSN which_csn) const
@ -1254,7 +1369,14 @@ void IMergeTreeDataPart::appendCSNToVersionMetadata(VersionMetadata::WhichCSN wh
chassert(!(which_csn == VersionMetadata::WhichCSN::REMOVAL && version.removal_csn == 0));
chassert(isStoredOnDisk());
data_part_storage->appendCSNToVersionMetadata(version, which_csn);
/// Small enough appends to file are usually atomic,
/// so we append new metadata instead of rewriting file to reduce number of fsyncs.
/// We don't need to do fsync when writing CSN, because in case of hard restart
/// we will be able to restore CSN from transaction log in Keeper.
auto out = getDataPartStorage().writeTransactionFile(WriteMode::Append);
version.writeCSN(*out, which_csn);
out->finalize();
}
void IMergeTreeDataPart::appendRemovalTIDToVersionMetadata(bool clear) const
@ -1277,13 +1399,74 @@ void IMergeTreeDataPart::appendRemovalTIDToVersionMetadata(bool clear) const
else
LOG_TEST(storage.log, "Appending removal TID for {} (creation: {}, removal {})", name, version.creation_tid, version.removal_tid);
data_part_storage->appendRemovalTIDToVersionMetadata(version, clear);
auto out = getDataPartStorage().writeTransactionFile(WriteMode::Append);
version.writeRemovalTID(*out, clear);
out->finalize();
/// fsync is not required when we clearing removal TID, because after hard restart we will fix metadata
if (!clear)
out->sync();
}
static std::unique_ptr<ReadBufferFromFileBase> openForReading(const IDataPartStorage & part_storage, const String & filename)
{
size_t file_size = part_storage.getFileSize(filename);
return part_storage.readFile(filename, ReadSettings().adjustBufferSize(file_size), file_size, file_size);
}
void IMergeTreeDataPart::loadVersionMetadata() const
try
{
data_part_storage->loadVersionMetadata(version, storage.log);
static constexpr auto version_file_name = "txn_version.txt";
static constexpr auto tmp_version_file_name = "txn_version.txt.tmp";
auto & data_part_storage = const_cast<IDataPartStorage &>(getDataPartStorage());
auto remove_tmp_file = [&]()
{
auto last_modified = data_part_storage.getLastModified();
auto buf = openForReading(data_part_storage, tmp_version_file_name);
String content;
readStringUntilEOF(content, *buf);
LOG_WARNING(storage.log, "Found file {} that was last modified on {}, has size {} and the following content: {}",
tmp_version_file_name, last_modified.epochTime(), content.size(), content);
data_part_storage.removeFile(tmp_version_file_name);
};
if (data_part_storage.exists(version_file_name))
{
auto buf = openForReading(data_part_storage, version_file_name);
version.read(*buf);
if (data_part_storage.exists(tmp_version_file_name))
remove_tmp_file();
return;
}
/// Four (?) cases are possible:
/// 1. Part was created without transactions.
/// 2. Version metadata file was not renamed from *.tmp on part creation.
/// 3. Version metadata were written to *.tmp file, but hard restart happened before fsync.
/// 4. Fsyncs in storeVersionMetadata() work incorrectly.
if (!data_part_storage.exists(tmp_version_file_name))
{
/// Case 1.
/// We do not have version metadata and transactions history for old parts,
/// so let's consider that such parts were created by some ancient transaction
/// and were committed with some prehistoric CSN.
/// NOTE It might be Case 3, but version metadata file is written on part creation before other files,
/// so it's not Case 3 if part is not broken.
version.setCreationTID(Tx::PrehistoricTID, nullptr);
version.creation_csn = Tx::PrehistoricCSN;
return;
}
/// Case 2.
/// Content of *.tmp file may be broken, just use fake TID.
/// Transaction was not committed if *.tmp file was not renamed, so we should complete rollback by removing part.
version.setCreationTID(Tx::DummyTID, nullptr);
version.creation_csn = Tx::RolledBackCSN;
remove_tmp_file();
}
catch (Exception & e)
{
@ -1320,15 +1503,15 @@ bool IMergeTreeDataPart::assertHasValidVersionMetadata() const
if (state == MergeTreeDataPartState::Temporary)
return true;
if (!data_part_storage->exists())
if (!getDataPartStorage().exists())
return true;
String content;
String version_file_name = TXN_VERSION_METADATA_FILE_NAME;
try
{
size_t file_size = data_part_storage->getFileSize(TXN_VERSION_METADATA_FILE_NAME);
auto buf = data_part_storage->readFile(TXN_VERSION_METADATA_FILE_NAME, ReadSettings().adjustBufferSize(file_size), file_size, std::nullopt);
size_t file_size = getDataPartStorage().getFileSize(TXN_VERSION_METADATA_FILE_NAME);
auto buf = getDataPartStorage().readFile(TXN_VERSION_METADATA_FILE_NAME, ReadSettings().adjustBufferSize(file_size), file_size, std::nullopt);
readStringUntilEOF(content, *buf);
ReadBufferFromString str_buf{content};
@ -1362,10 +1545,11 @@ void IMergeTreeDataPart::appendFilesOfColumns(Strings & files)
bool IMergeTreeDataPart::shallParticipateInMerges(const StoragePolicyPtr & storage_policy) const
{
return data_part_storage->shallParticipateInMerges(*storage_policy);
auto disk_name = getDataPartStorage().getDiskName();
return !storage_policy->getVolumeByDiskName(disk_name)->areMergesAvoided();
}
void IMergeTreeDataPart::renameTo(const String & new_relative_path, bool remove_new_dir_if_exists, DataPartStorageBuilderPtr builder) const
void IMergeTreeDataPart::renameTo(const String & new_relative_path, bool remove_new_dir_if_exists)
try
{
assertOnDisk();
@ -1376,22 +1560,21 @@ try
if (parent_part)
{
/// For projections, move is only possible inside parent part dir.
relative_path = parent_part->data_part_storage->getRelativePath();
relative_path = parent_part->getDataPartStorage().getRelativePath();
}
String from = data_part_storage->getRelativePath();
auto old_projection_root_path = getDataPartStorage().getRelativePath();
auto to = fs::path(relative_path) / new_relative_path;
metadata_manager->deleteAll(true);
metadata_manager->assertAllDeleted(true);
builder->rename(to.parent_path(), to.filename(), storage.log, remove_new_dir_if_exists, fsync_dir);
data_part_storage->onRename(to.parent_path(), to.filename());
getDataPartStorage().rename(to.parent_path(), to.filename(), storage.log, remove_new_dir_if_exists, fsync_dir);
metadata_manager->updateAll(true);
for (const auto & [p_name, part] : projection_parts)
{
part->data_part_storage = data_part_storage->getProjection(p_name + ".proj");
}
auto new_projection_root_path = to.string();
for (const auto & [_, part] : projection_parts)
part->getDataPartStorage().changeRootPath(old_projection_root_path, new_projection_root_path);
}
catch (...)
{
@ -1432,14 +1615,14 @@ void IMergeTreeDataPart::initializePartMetadataManager()
void IMergeTreeDataPart::initializeIndexGranularityInfo()
{
auto mrk_ext = MergeTreeIndexGranularityInfo::getMarksExtensionFromFilesystem(data_part_storage);
auto mrk_ext = MergeTreeIndexGranularityInfo::getMarksExtensionFromFilesystem(getDataPartStorage());
if (mrk_ext)
index_granularity_info = MergeTreeIndexGranularityInfo(storage, MarkType{*mrk_ext});
else
index_granularity_info = MergeTreeIndexGranularityInfo(storage, part_type);
}
void IMergeTreeDataPart::remove() const
void IMergeTreeDataPart::remove()
{
assert(assertHasValidVersionMetadata());
part_is_probably_removed_from_disk = true;
@ -1456,7 +1639,6 @@ void IMergeTreeDataPart::remove() const
return CanRemoveDescription{.can_remove_anything = can_remove, .files_not_to_remove = files_not_to_remove };
};
if (!isStoredOnDisk())
return;
@ -1475,7 +1657,7 @@ void IMergeTreeDataPart::remove() const
projection_checksums.emplace_back(IDataPartStorage::ProjectionChecksums{.name = p_name, .checksums = projection_part->checksums});
}
data_part_storage->remove(std::move(can_remove_callback), checksums, projection_checksums, is_temp, getState(), storage.log);
getDataPartStorage().remove(std::move(can_remove_callback), checksums, projection_checksums, is_temp, getState(), storage.log);
}
std::optional<String> IMergeTreeDataPart::getRelativePathForPrefix(const String & prefix, bool detached, bool broken) const
@ -1492,7 +1674,7 @@ std::optional<String> IMergeTreeDataPart::getRelativePathForPrefix(const String
if (detached && parent_part)
throw Exception(ErrorCodes::LOGICAL_ERROR, "Cannot detach projection");
return data_part_storage->getRelativePathForPrefix(storage.log, prefix, detached, broken);
return getDataPartStorage().getRelativePathForPrefix(storage.log, prefix, detached, broken);
}
std::optional<String> IMergeTreeDataPart::getRelativePathForDetachedPart(const String & prefix, bool broken) const
@ -1507,11 +1689,11 @@ std::optional<String> IMergeTreeDataPart::getRelativePathForDetachedPart(const S
return {};
}
void IMergeTreeDataPart::renameToDetached(const String & prefix, DataPartStorageBuilderPtr builder) const
void IMergeTreeDataPart::renameToDetached(const String & prefix)
{
auto path_to_detach = getRelativePathForDetachedPart(prefix, /* broken */ false);
assert(path_to_detach);
renameTo(path_to_detach.value(), true, builder);
renameTo(path_to_detach.value(), true);
part_is_probably_removed_from_disk = true;
}
@ -1530,7 +1712,7 @@ void IMergeTreeDataPart::makeCloneInDetached(const String & prefix, const Storag
if (!maybe_path_in_detached)
return;
data_part_storage->freeze(
getDataPartStorage().freeze(
storage.relative_data_path,
*maybe_path_in_detached,
/*make_source_readonly*/ true,
@ -1539,17 +1721,17 @@ void IMergeTreeDataPart::makeCloneInDetached(const String & prefix, const Storag
{});
}
DataPartStoragePtr IMergeTreeDataPart::makeCloneOnDisk(const DiskPtr & disk, const String & directory_name) const
MutableDataPartStoragePtr IMergeTreeDataPart::makeCloneOnDisk(const DiskPtr & disk, const String & directory_name) const
{
assertOnDisk();
if (disk->getName() == data_part_storage->getDiskName())
throw Exception("Can not clone data part " + name + " to same disk " + data_part_storage->getDiskName(), ErrorCodes::LOGICAL_ERROR);
if (disk->getName() == getDataPartStorage().getDiskName())
throw Exception("Can not clone data part " + name + " to same disk " + getDataPartStorage().getDiskName(), ErrorCodes::LOGICAL_ERROR);
if (directory_name.empty())
throw Exception("Can not clone data part " + name + " to empty directory.", ErrorCodes::LOGICAL_ERROR);
String path_to_clone = fs::path(storage.relative_data_path) / directory_name / "";
return data_part_storage->clone(path_to_clone, data_part_storage->getPartDirectory(), disk, storage.log);
return getDataPartStorage().clonePart(path_to_clone, getDataPartStorage().getPartDirectory(), disk, storage.log);
}
void IMergeTreeDataPart::checkConsistencyBase() const
@ -1590,26 +1772,26 @@ void IMergeTreeDataPart::checkConsistencyBase() const
}
}
data_part_storage->checkConsistency(checksums);
getDataPartStorage().checkConsistency(checksums);
}
else
{
auto check_file_not_empty = [this](const String & file_path)
{
UInt64 file_size;
if (!data_part_storage->exists(file_path) || (file_size = data_part_storage->getFileSize(file_path)) == 0)
if (!getDataPartStorage().exists(file_path) || (file_size = getDataPartStorage().getFileSize(file_path)) == 0)
throw Exception(
ErrorCodes::BAD_SIZE_OF_FILE_IN_DATA_PART,
"Part {} is broken: {} is empty",
data_part_storage->getFullPath(),
std::string(fs::path(data_part_storage->getFullPath()) / file_path));
getDataPartStorage().getFullPath(),
std::string(fs::path(getDataPartStorage().getFullPath()) / file_path));
return file_size;
};
/// Check that the primary key index is not empty.
if (!pk.column_names.empty())
{
String index_name = "primary" + getIndexExtensionFromFilesystem(data_part_storage).value();
String index_name = "primary" + getIndexExtensionFromFilesystem(getDataPartStorage()).value();
check_file_not_empty(index_name);
}
@ -1753,7 +1935,7 @@ bool IMergeTreeDataPart::checkAllTTLCalculated(const StorageMetadataPtr & metada
String IMergeTreeDataPart::getUniqueId() const
{
return data_part_storage->getUniqueId();
return getDataPartStorage().getUniqueId();
}
String IMergeTreeDataPart::getZeroLevelPartBlockID(std::string_view token) const
@ -1792,11 +1974,11 @@ IMergeTreeDataPart::uint128 IMergeTreeDataPart::getActualChecksumByFile(const St
return it->second.file_hash;
}
if (!data_part_storage->exists(file_name))
if (!getDataPartStorage().exists(file_name))
{
return {};
}
std::unique_ptr<ReadBufferFromFileBase> in_file = data_part_storage->readFile(file_name, {}, std::nullopt, std::nullopt);
std::unique_ptr<ReadBufferFromFileBase> in_file = getDataPartStorage().readFile(file_name, {}, std::nullopt, std::nullopt);
HashingReadBuffer in_hash(*in_file);
String value;
@ -1824,11 +2006,11 @@ bool isInMemoryPart(const MergeTreeDataPartPtr & data_part)
return (data_part && data_part->getType() == MergeTreeDataPartType::InMemory);
}
std::optional<std::string> getIndexExtensionFromFilesystem(const DataPartStoragePtr & data_part_storage)
std::optional<std::string> getIndexExtensionFromFilesystem(const IDataPartStorage & data_part_storage)
{
if (data_part_storage->exists())
if (data_part_storage.exists())
{
for (auto it = data_part_storage->iterate(); it->isValid(); it->next())
for (auto it = data_part_storage.iterate(); it->isValid(); it->next())
{
const auto & extension = fs::path(it->name()).extension();
if (extension == getIndexExtension(false)

41
src/Storages/MergeTree/IMergeTreeDataPart.h
View File

@ -1,5 +1,6 @@
#pragma once
#include "IO/WriteSettings.h"
#include <Core/Block.h>
#include <base/types.h>
#include <Core/NamesAndTypes.h>
@ -46,7 +47,7 @@ class UncompressedCache;
class MergeTreeTransaction;
/// Description of the data part.
class IMergeTreeDataPart : public std::enable_shared_from_this<IMergeTreeDataPart>
class IMergeTreeDataPart : public std::enable_shared_from_this<IMergeTreeDataPart>, public DataPartStorageHolder
{
public:
static constexpr auto DATA_FILE_EXTENSION = ".bin";
@ -67,19 +68,18 @@ public:
using uint128 = IPartMetadataManager::uint128;
IMergeTreeDataPart(
const MergeTreeData & storage_,
const String & name_,
const MergeTreePartInfo & info_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
Type part_type_,
const IMergeTreeDataPart * parent_part_);
IMergeTreeDataPart(
const MergeTreeData & storage_,
const String & name_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
Type part_type_,
const IMergeTreeDataPart * parent_part_);
@ -94,13 +94,12 @@ public:
const ReadBufferFromFileBase::ProfileCallback & profile_callback_) const = 0;
virtual MergeTreeWriterPtr getWriter(
DataPartStorageBuilderPtr data_part_storage_builder,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & writer_settings,
const MergeTreeIndexGranularity & computed_index_granularity) const = 0;
const MergeTreeIndexGranularity & computed_index_granularity) = 0;
virtual bool isStoredOnDisk() const = 0;
@ -152,7 +151,7 @@ public:
/// Throws an exception if part is not stored in on-disk format.
void assertOnDisk() const;
void remove() const;
void remove();
/// Initialize columns (from columns.txt if exists, or create from column files if not).
/// Load checksums from checksums.txt if exists. Load index if required.
@ -200,10 +199,6 @@ public:
/// processed by multiple shards.
UUID uuid = UUIDHelpers::Nil;
/// This is an object which encapsulates all the operations with disk.
/// Contains a path to stored data.
DataPartStoragePtr data_part_storage;
MergeTreeIndexGranularityInfo index_granularity_info;
size_t rows_count = 0;
@ -289,8 +284,8 @@ public:
using WrittenFiles = std::vector<std::unique_ptr<WriteBufferFromFileBase>>;
[[nodiscard]] WrittenFiles store(const MergeTreeData & data, const DataPartStorageBuilderPtr & data_part_storage_builder, Checksums & checksums) const;
[[nodiscard]] WrittenFiles store(const Names & column_names, const DataTypes & data_types, const DataPartStorageBuilderPtr & data_part_storage_builder, Checksums & checksums) const;
[[nodiscard]] WrittenFiles store(const MergeTreeData & data, IDataPartStorage & part_storage, Checksums & checksums) const;
[[nodiscard]] WrittenFiles store(const Names & column_names, const DataTypes & data_types, IDataPartStorage & part_storage, Checksums & checksums) const;
void update(const Block & block, const Names & column_names);
void merge(const MinMaxIndex & other);
@ -321,17 +316,17 @@ public:
size_t getFileSizeOrZero(const String & file_name) const;
/// Moves a part to detached/ directory and adds prefix to its name
void renameToDetached(const String & prefix, DataPartStorageBuilderPtr builder) const;
void renameToDetached(const String & prefix);
/// Makes checks and move part to new directory
/// Changes only relative_dir_name, you need to update other metadata (name, is_temp) explicitly
virtual void renameTo(const String & new_relative_path, bool remove_new_dir_if_exists, DataPartStorageBuilderPtr builder) const;
virtual void renameTo(const String & new_relative_path, bool remove_new_dir_if_exists);
/// Makes clone of a part in detached/ directory via hard links
virtual void makeCloneInDetached(const String & prefix, const StorageMetadataPtr & metadata_snapshot) const;
/// Makes full clone of part in specified subdirectory (relative to storage data directory, e.g. "detached") on another disk
DataPartStoragePtr makeCloneOnDisk(const DiskPtr & disk, const String & directory_name) const;
MutableDataPartStoragePtr makeCloneOnDisk(const DiskPtr & disk, const String & directory_name) const;
/// Checks that .bin and .mrk files exist.
///
@ -445,6 +440,12 @@ public:
/// True if here is lightweight deleted mask file in part.
bool hasLightweightDelete() const { return columns.contains(LightweightDeleteDescription::FILTER_COLUMN.name); }
void writeChecksums(const MergeTreeDataPartChecksums & checksums_, const WriteSettings & settings);
void writeDeleteOnDestroyMarker();
void removeDeleteOnDestroyMarker();
void removeVersionMetadata();
protected:
/// Total size of all columns, calculated once in calcuateColumnSizesOnDisk
@ -566,6 +567,12 @@ private:
/// any specifial compression.
void loadDefaultCompressionCodec();
void writeColumns(const NamesAndTypesList & columns_, const WriteSettings & settings);
void writeVersionMetadata(const VersionMetadata & version_, bool fsync_part_dir) const;
template <typename Writer>
void writeMetadata(const String & filename, const WriteSettings & settings, Writer && writer);
static void appendFilesOfDefaultCompressionCodec(Strings & files);
/// Found column without specific compression and return codec
@ -585,7 +592,7 @@ bool isCompactPart(const MergeTreeDataPartPtr & data_part);
bool isWidePart(const MergeTreeDataPartPtr & data_part);
bool isInMemoryPart(const MergeTreeDataPartPtr & data_part);
inline String getIndexExtension(bool is_compressed_primary_key) { return is_compressed_primary_key ? ".cidx" : ".idx"; }
std::optional<String> getIndexExtensionFromFilesystem(const DataPartStoragePtr & data_part_storage);
std::optional<String> getIndexExtensionFromFilesystem(const IDataPartStorage & data_part_storage);
bool isCompressedFromIndexExtension(const String & index_extension);
}

5
src/Storages/MergeTree/IMergeTreeDataPartInfoForReader.h
View File

@ -7,7 +7,8 @@ namespace DB
{
class IDataPartStorage;
using DataPartStoragePtr = std::shared_ptr<IDataPartStorage>;
using DataPartStoragePtr = std::shared_ptr<const IDataPartStorage>;
class MergeTreeIndexGranularity;
struct MergeTreeDataPartChecksums;
struct MergeTreeIndexGranularityInfo;
@ -36,7 +37,7 @@ public:
virtual bool isProjectionPart() const = 0;
virtual const DataPartStoragePtr & getDataPartStorage() const = 0;
virtual DataPartStoragePtr getDataPartStorage() const = 0;
virtual const NamesAndTypesList & getColumns() const = 0;

4
src/Storages/MergeTree/IMergeTreeDataPartWriter.cpp
View File

@ -38,14 +38,12 @@ Block permuteBlockIfNeeded(const Block & block, const IColumn::Permutation * per
}
IMergeTreeDataPartWriter::IMergeTreeDataPartWriter(
const MergeTreeData::DataPartPtr & data_part_,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part_,
const NamesAndTypesList & columns_list_,
const StorageMetadataPtr & metadata_snapshot_,
const MergeTreeWriterSettings & settings_,
const MergeTreeIndexGranularity & index_granularity_)
: data_part(data_part_)
, data_part_storage_builder(std::move(data_part_storage_builder_))
, storage(data_part_->storage)
, metadata_snapshot(metadata_snapshot_)
, columns_list(columns_list_)

6
src/Storages/MergeTree/IMergeTreeDataPartWriter.h
View File

@ -22,8 +22,7 @@ class IMergeTreeDataPartWriter : private boost::noncopyable
{
public:
IMergeTreeDataPartWriter(
const MergeTreeData::DataPartPtr & data_part_,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part_,
const NamesAndTypesList & columns_list_,
const StorageMetadataPtr & metadata_snapshot_,
const MergeTreeWriterSettings & settings_,
@ -42,8 +41,7 @@ public:
protected:
const MergeTreeData::DataPartPtr data_part;
DataPartStorageBuilderPtr data_part_storage_builder;
const MergeTreeMutableDataPartPtr data_part;
const MergeTreeData & storage;
const StorageMetadataPtr metadata_snapshot;
const NamesAndTypesList columns_list;

5
src/Storages/MergeTree/IMergedBlockOutputStream.cpp
View File

@ -6,14 +6,13 @@ namespace DB
{
IMergedBlockOutputStream::IMergedBlockOutputStream(
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeDataPartPtr & data_part,
const MergeTreeMutableDataPartPtr & data_part,
const StorageMetadataPtr & metadata_snapshot_,
const NamesAndTypesList & columns_list,
bool reset_columns_)
: storage(data_part->storage)
, metadata_snapshot(metadata_snapshot_)
, data_part_storage_builder(std::move(data_part_storage_builder_))
, data_part_storage(data_part->getDataPartStoragePtr())
, reset_columns(reset_columns_)
{
if (reset_columns)

6
src/Storages/MergeTree/IMergedBlockOutputStream.h
View File

@ -1,5 +1,6 @@
#pragma once
#include "Storages/MergeTree/IDataPartStorage.h"
#include <Storages/MergeTree/MergeTreeIndexGranularity.h>
#include <Storages/MergeTree/MergeTreeData.h>
#include <Storages/MergeTree/IMergeTreeDataPart.h>
@ -12,8 +13,7 @@ class IMergedBlockOutputStream
{
public:
IMergedBlockOutputStream(
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeDataPartPtr & data_part,
const MergeTreeMutableDataPartPtr & data_part,
const StorageMetadataPtr & metadata_snapshot_,
const NamesAndTypesList & columns_list,
bool reset_columns_);
@ -42,7 +42,7 @@ protected:
const MergeTreeData & storage;
StorageMetadataPtr metadata_snapshot;
DataPartStorageBuilderPtr data_part_storage_builder;
MutableDataPartStoragePtr data_part_storage;
IMergeTreeDataPart::MergeTreeWriterPtr writer;
bool reset_columns = false;

3
src/Storages/MergeTree/KeyCondition.cpp
View File

@ -1197,7 +1197,8 @@ bool KeyCondition::transformConstantWithValidFunctions(
if (is_valid_chain)
{
auto const_type = cur_node->result_type;
out_type = removeLowCardinality(out_type);
auto const_type = removeLowCardinality(cur_node->result_type);
auto const_column = out_type->createColumnConst(1, out_value);
auto const_value = (*castColumnAccurateOrNull({const_column, out_type, ""}, const_type))[0];

5
src/Storages/MergeTree/LoadedMergeTreeDataPartInfoForReader.h
View File

@ -12,7 +12,8 @@ public:
explicit LoadedMergeTreeDataPartInfoForReader(MergeTreeData::DataPartPtr data_part_)
: IMergeTreeDataPartInfoForReader(data_part_->storage.getContext())
, data_part(data_part_)
{}
{
}
bool isCompactPart() const override { return DB::isCompactPart(data_part); }
@ -22,7 +23,7 @@ public:
bool isProjectionPart() const override { return data_part->isProjectionPart(); }
const DataPartStoragePtr & getDataPartStorage() const override { return data_part->data_part_storage; }
DataPartStoragePtr getDataPartStorage() const override { return data_part->getDataPartStoragePtr(); }
const NamesAndTypesList & getColumns() const override { return data_part->getColumns(); }

8
src/Storages/MergeTree/MergeFromLogEntryTask.cpp
View File

@ -160,7 +160,9 @@ ReplicatedMergeMutateTaskBase::PrepareResult MergeFromLogEntryTask::prepare()
for (auto & part_ptr : parts)
{
ttl_infos.update(part_ptr->ttl_infos);
max_volume_index = std::max(max_volume_index, part_ptr->data_part_storage->getVolumeIndex(*storage.getStoragePolicy()));
auto disk_name = part_ptr->getDataPartStorage().getDiskName();
size_t volume_index = storage.getStoragePolicy()->getVolumeIndexByDiskName(disk_name);
max_volume_index = std::max(max_volume_index, volume_index);
}
/// It will live until the whole task is being destroyed
@ -294,12 +296,10 @@ ReplicatedMergeMutateTaskBase::PrepareResult MergeFromLogEntryTask::prepare()
bool MergeFromLogEntryTask::finalize(ReplicatedMergeMutateTaskBase::PartLogWriter write_part_log)
{
part = merge_task->getFuture().get();
auto builder = merge_task->getBuilder();
/// Task is not needed
merge_task.reset();
storage.merger_mutator.renameMergedTemporaryPart(part, parts, NO_TRANSACTION_PTR, *transaction_ptr, builder);
storage.merger_mutator.renameMergedTemporaryPart(part, parts, NO_TRANSACTION_PTR, *transaction_ptr);
try
{

2
src/Storages/MergeTree/MergeList.cpp
View File

@ -65,7 +65,7 @@ MergeListElement::MergeListElement(
for (const auto & source_part : future_part->parts)
{
source_part_names.emplace_back(source_part->name);
source_part_paths.emplace_back(source_part->data_part_storage->getFullPath());
source_part_paths.emplace_back(source_part->getDataPartStorage().getFullPath());
total_size_bytes_compressed += source_part->getBytesOnDisk();
total_size_marks += source_part->getMarksCount();

3
src/Storages/MergeTree/MergePlainMergeTreeTask.cpp
View File

@ -115,10 +115,9 @@ void MergePlainMergeTreeTask::prepare()
void MergePlainMergeTreeTask::finish()
{
new_part = merge_task->getFuture().get();
auto builder = merge_task->getBuilder();
MergeTreeData::Transaction transaction(storage, txn.get());
storage.merger_mutator.renameMergedTemporaryPart(new_part, future_part->parts, txn, transaction, builder);
storage.merger_mutator.renameMergedTemporaryPart(new_part, future_part->parts, txn, transaction);
transaction.commit();
write_part_log({});

19
src/Storages/MergeTree/MergeTask.cpp
View File

@ -1,3 +1,4 @@
#include "Storages/MergeTree/IDataPartStorage.h"
#include <Storages/MergeTree/MergeTask.h>
#include <memory>
@ -125,23 +126,26 @@ bool MergeTask::ExecuteAndFinalizeHorizontalPart::prepare()
ctx->disk = global_ctx->space_reservation->getDisk();
String local_tmp_part_basename = local_tmp_prefix + global_ctx->future_part->name + local_tmp_suffix;
MutableDataPartStoragePtr data_part_storage;
if (global_ctx->parent_path_storage_builder)
if (global_ctx->parent_part)
{
global_ctx->data_part_storage_builder = global_ctx->parent_path_storage_builder->getProjection(local_tmp_part_basename);
data_part_storage = global_ctx->parent_part->getDataPartStorage().getProjection(local_tmp_part_basename);
}
else
{
auto local_single_disk_volume = std::make_shared<SingleDiskVolume>("volume_" + global_ctx->future_part->name, ctx->disk, 0);
global_ctx->data_part_storage_builder = std::make_shared<DataPartStorageBuilderOnDisk>(
data_part_storage = std::make_shared<DataPartStorageOnDisk>(
local_single_disk_volume,
global_ctx->data->relative_data_path,
local_tmp_part_basename);
data_part_storage->beginTransaction();
}
if (global_ctx->data_part_storage_builder->exists())
throw Exception("Directory " + global_ctx->data_part_storage_builder->getFullPath() + " already exists", ErrorCodes::DIRECTORY_ALREADY_EXISTS);
if (data_part_storage->exists())
throw Exception("Directory " + data_part_storage->getFullPath() + " already exists", ErrorCodes::DIRECTORY_ALREADY_EXISTS);
if (!global_ctx->parent_part)
global_ctx->temporary_directory_lock = global_ctx->data->getTemporaryPartDirectoryHolder(local_tmp_part_basename);
@ -163,8 +167,6 @@ bool MergeTask::ExecuteAndFinalizeHorizontalPart::prepare()
global_ctx->merging_columns,
global_ctx->merging_column_names);
auto data_part_storage = global_ctx->data_part_storage_builder->getStorage();
global_ctx->new_data_part = global_ctx->data->createPart(
global_ctx->future_part->name,
global_ctx->future_part->type,
@ -302,7 +304,6 @@ bool MergeTask::ExecuteAndFinalizeHorizontalPart::prepare()
global_ctx->to = std::make_shared<MergedBlockOutputStream>(
global_ctx->new_data_part,
global_ctx->data_part_storage_builder,
global_ctx->metadata_snapshot,
global_ctx->merging_columns,
MergeTreeIndexFactory::instance().getMany(global_ctx->metadata_snapshot->getSecondaryIndices()),
@ -501,7 +502,6 @@ void MergeTask::VerticalMergeStage::prepareVerticalMergeForOneColumn() const
ctx->executor = std::make_unique<PullingPipelineExecutor>(ctx->column_parts_pipeline);
ctx->column_to = std::make_unique<MergedColumnOnlyOutputStream>(
global_ctx->data_part_storage_builder,
global_ctx->new_data_part,
global_ctx->metadata_snapshot,
ctx->executor->getHeader(),
@ -654,7 +654,6 @@ bool MergeTask::MergeProjectionsStage::mergeMinMaxIndexAndPrepareProjections() c
global_ctx->deduplicate_by_columns,
projection_merging_params,
global_ctx->new_data_part.get(),
global_ctx->data_part_storage_builder.get(),
".proj",
NO_TRANSACTION_PTR,
global_ctx->data,

13
src/Storages/MergeTree/MergeTask.h
View File

@ -59,8 +59,7 @@ public:
bool deduplicate_,
Names deduplicate_by_columns_,
MergeTreeData::MergingParams merging_params_,
const IMergeTreeDataPart * parent_part_,
const IDataPartStorageBuilder * parent_path_storage_builder_,
IMergeTreeDataPart * parent_part_,
String suffix_,
MergeTreeTransactionPtr txn,
MergeTreeData * data_,
@ -82,7 +81,6 @@ public:
global_ctx->deduplicate = std::move(deduplicate_);
global_ctx->deduplicate_by_columns = std::move(deduplicate_by_columns_);
global_ctx->parent_part = std::move(parent_part_);
global_ctx->parent_path_storage_builder = std::move(parent_path_storage_builder_);
global_ctx->data = std::move(data_);
global_ctx->mutator = std::move(mutator_);
global_ctx->merges_blocker = std::move(merges_blocker_);
@ -102,11 +100,6 @@ public:
return global_ctx->promise.get_future();
}
DataPartStorageBuilderPtr getBuilder()
{
return global_ctx->data_part_storage_builder;
}
bool execute();
private:
@ -141,8 +134,7 @@ private:
StorageMetadataPtr metadata_snapshot{nullptr};
FutureMergedMutatedPartPtr future_part{nullptr};
/// This will be either nullptr or new_data_part, so raw pointer is ok.
const IMergeTreeDataPart * parent_part{nullptr};
const IDataPartStorageBuilder * parent_path_storage_builder{nullptr};
IMergeTreeDataPart * parent_part{nullptr};
ContextPtr context{nullptr};
time_t time_of_merge{0};
ReservationSharedPtr space_reservation{nullptr};
@ -168,7 +160,6 @@ private:
std::unique_ptr<PullingPipelineExecutor> merging_executor;
MergeTreeData::MutableDataPartPtr new_data_part{nullptr};
DataPartStorageBuilderPtr data_part_storage_builder;
/// If lightweight delete mask is present then some input rows are filtered out right after reading.
std::shared_ptr<std::atomic<size_t>> input_rows_filtered{std::make_shared<std::atomic<size_t>>(0)};

227
src/Storages/MergeTree/MergeTreeData.cpp
View File

@ -943,8 +943,8 @@ Int64 MergeTreeData::getMaxBlockNumber() const
}
void MergeTreeData::loadDataPartsFromDisk(
DataPartsVector & broken_parts_to_detach,
DataPartsVector & duplicate_parts_to_remove,
MutableDataPartsVector & broken_parts_to_detach,
MutableDataPartsVector & duplicate_parts_to_remove,
ThreadPool & pool,
size_t num_parts,
std::queue<std::vector<std::pair<String, DiskPtr>>> & parts_queue,
@ -1082,7 +1082,6 @@ void MergeTreeData::loadDataPartsFromDisk(
if (size_of_part.has_value())
part_size_str = formatReadableSizeWithBinarySuffix(*size_of_part);
LOG_ERROR(log,
"Detaching broken part {}{} (size: {}). "
"If it happened after update, it is likely because of backward incompatibility. "
@ -1200,8 +1199,7 @@ void MergeTreeData::loadDataPartsFromDisk(
void MergeTreeData::loadDataPartsFromWAL(
DataPartsVector & /* broken_parts_to_detach */,
DataPartsVector & duplicate_parts_to_remove,
MutableDataPartsVector & duplicate_parts_to_remove,
MutableDataPartsVector & parts_from_wal)
{
for (auto & part : parts_from_wal)
@ -1215,7 +1213,7 @@ void MergeTreeData::loadDataPartsFromWAL(
{
if ((*it)->checksums.getTotalChecksumHex() == part->checksums.getTotalChecksumHex())
{
LOG_ERROR(log, "Remove duplicate part {}", part->data_part_storage->getFullPath());
LOG_ERROR(log, "Remove duplicate part {}", part->getDataPartStorage().getFullPath());
duplicate_parts_to_remove.push_back(part);
}
else
@ -1329,8 +1327,8 @@ void MergeTreeData::loadDataParts(bool skip_sanity_checks)
auto part_lock = lockParts();
data_parts_indexes.clear();
DataPartsVector broken_parts_to_detach;
DataPartsVector duplicate_parts_to_remove;
MutableDataPartsVector broken_parts_to_detach;
MutableDataPartsVector duplicate_parts_to_remove;
if (num_parts > 0)
loadDataPartsFromDisk(
@ -1384,7 +1382,7 @@ void MergeTreeData::loadDataParts(bool skip_sanity_checks)
parts_from_wal.insert(
parts_from_wal.end(), std::make_move_iterator(disk_wal_parts.begin()), std::make_move_iterator(disk_wal_parts.end()));
loadDataPartsFromWAL(broken_parts_to_detach, duplicate_parts_to_remove, parts_from_wal);
loadDataPartsFromWAL(duplicate_parts_to_remove, parts_from_wal);
num_parts += parts_from_wal.size();
}
@ -1397,11 +1395,7 @@ void MergeTreeData::loadDataParts(bool skip_sanity_checks)
}
for (auto & part : broken_parts_to_detach)
{
auto builder = part->data_part_storage->getBuilder();
part->renameToDetached("broken-on-start", builder); /// detached parts must not have '_' in prefixes
builder->commit();
}
part->renameToDetached("broken-on-start"); /// detached parts must not have '_' in prefixes
for (auto & part : duplicate_parts_to_remove)
part->remove();
@ -1689,6 +1683,15 @@ scope_guard MergeTreeData::getTemporaryPartDirectoryHolder(const String & part_d
return [this, part_dir_name]() { temporary_parts.remove(part_dir_name); };
}
MergeTreeData::MutableDataPartPtr MergeTreeData::preparePartForRemoval(const DataPartPtr & part)
{
auto state = part->getState();
if (state != DataPartState::Deleting && state != DataPartState::DeleteOnDestroy)
throw Exception(ErrorCodes::LOGICAL_ERROR,
"Cannot remove part {}, because it has state: {}", part->name, magic_enum::enum_name(part->getState()));
return std::const_pointer_cast<IMergeTreeDataPart>(part);
}
MergeTreeData::DataPartsVector MergeTreeData::grabOldParts(bool force)
{
@ -1864,7 +1867,7 @@ void MergeTreeData::flushAllInMemoryPartsIfNeeded()
{
if (auto part_in_memory = asInMemoryPart(part))
{
part_in_memory->flushToDisk(part_in_memory->data_part_storage->getPartDirectory(), metadata_snapshot);
part_in_memory->flushToDisk(part_in_memory->getDataPartStorage().getPartDirectory(), metadata_snapshot);
}
}
}
@ -1948,7 +1951,7 @@ void MergeTreeData::clearPartsFromFilesystemImpl(const DataPartsVector & parts_t
if (thread_group)
CurrentThread::attachToIfDetached(thread_group);
part->remove();
preparePartForRemoval(part)->remove();
if (part_names_succeed)
{
std::lock_guard lock(part_names_mutex);
@ -1964,7 +1967,7 @@ void MergeTreeData::clearPartsFromFilesystemImpl(const DataPartsVector & parts_t
LOG_DEBUG(log, "Removing {} parts from filesystem: {}", parts_to_remove.size(), fmt::join(parts_to_remove, ", "));
for (const DataPartPtr & part : parts_to_remove)
{
part->remove();
preparePartForRemoval(part)->remove();
if (part_names_succeed)
part_names_succeed->insert(part->name);
}
@ -2144,11 +2147,14 @@ void MergeTreeData::rename(const String & new_table_path, const StorageID & new_
if (!getStorageID().hasUUID())
getContext()->dropCaches();
/// TODO: remove const_cast
for (const auto & part : data_parts_by_info)
part->data_part_storage->changeRootPath(relative_data_path, new_table_path);
{
auto & part_mutable = const_cast<IMergeTreeDataPart &>(*part);
part_mutable.getDataPartStorage().changeRootPath(relative_data_path, new_table_path);
}
relative_data_path = new_table_path;
renameInMemory(new_table_id);
}
@ -2166,7 +2172,12 @@ void MergeTreeData::dropAllData()
auto lock = lockParts();
DataPartsVector all_parts(data_parts_by_info.begin(), data_parts_by_info.end());
DataPartsVector all_parts;
for (auto it = data_parts_by_info.begin(); it != data_parts_by_info.end(); ++it)
{
modifyPartState(it, DataPartState::Deleting);
all_parts.push_back(*it);
}
{
std::lock_guard wal_lock(write_ahead_log_mutex);
@ -2179,7 +2190,6 @@ void MergeTreeData::dropAllData()
if (!getStorageID().hasUUID())
getContext()->dropCaches();
/// Removing of each data part before recursive removal of directory is to speed-up removal, because there will be less number of syscalls.
NameSet part_names_failed;
try
@ -2189,6 +2199,7 @@ void MergeTreeData::dropAllData()
LOG_TRACE(log, "dropAllData: removing all data parts from memory.");
data_parts_indexes.clear();
all_data_dropped = true;
}
catch (...)
{
@ -2726,7 +2737,7 @@ MergeTreeDataPartType MergeTreeData::choosePartTypeOnDisk(size_t bytes_uncompres
MergeTreeData::MutableDataPartPtr MergeTreeData::createPart(const String & name,
MergeTreeDataPartType type, const MergeTreePartInfo & part_info,
const DataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part) const
const MutableDataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part) const
{
if (type == MergeTreeDataPartType::Compact)
return std::make_shared<MergeTreeDataPartCompact>(*this, name, part_info, data_part_storage, parent_part);
@ -2739,17 +2750,17 @@ MergeTreeData::MutableDataPartPtr MergeTreeData::createPart(const String & name,
}
MergeTreeData::MutableDataPartPtr MergeTreeData::createPart(
const String & name, const DataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part) const
const String & name, const MutableDataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part) const
{
return createPart(name, MergeTreePartInfo::fromPartName(name, format_version), data_part_storage, parent_part);
}
MergeTreeData::MutableDataPartPtr MergeTreeData::createPart(
const String & name, const MergeTreePartInfo & part_info,
const DataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part) const
const MutableDataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part) const
{
MergeTreeDataPartType type;
auto mrk_ext = MergeTreeIndexGranularityInfo::getMarksExtensionFromFilesystem(data_part_storage);
auto mrk_ext = MergeTreeIndexGranularityInfo::getMarksExtensionFromFilesystem(*data_part_storage);
if (mrk_ext)
{
@ -2943,12 +2954,11 @@ MergeTreeData::DataPartsVector MergeTreeData::getActivePartsToReplace(
bool MergeTreeData::renameTempPartAndAdd(
MutableDataPartPtr & part,
Transaction & out_transaction,
DataPartStorageBuilderPtr builder,
DataPartsLock & lock)
{
DataPartsVector covered_parts;
if (!renameTempPartAndReplaceImpl(part, out_transaction, lock, builder, &covered_parts))
if (!renameTempPartAndReplaceImpl(part, out_transaction, lock, &covered_parts))
return false;
if (!covered_parts.empty())
@ -2982,32 +2992,31 @@ void MergeTreeData::checkPartCanBeAddedToTable(MutableDataPartPtr & part, DataPa
}
}
void MergeTreeData::preparePartForCommit(MutableDataPartPtr & part, Transaction & out_transaction, DataPartStorageBuilderPtr builder)
void MergeTreeData::preparePartForCommit(MutableDataPartPtr & part, Transaction & out_transaction)
{
part->is_temp = false;
part->setState(DataPartState::PreActive);
assert([&]()
{
String dir_name = fs::path(part->data_part_storage->getRelativePath()).filename();
String dir_name = fs::path(part->getDataPartStorage().getRelativePath()).filename();
bool may_be_cleaned_up = dir_name.starts_with("tmp_") || dir_name.starts_with("tmp-fetch_");
return !may_be_cleaned_up || temporary_parts.contains(dir_name);
}());
part->renameTo(part->name, true, builder);
part->renameTo(part->name, true);
data_parts_indexes.insert(part);
out_transaction.addPart(part, builder);
out_transaction.addPart(part);
}
bool MergeTreeData::renameTempPartAndReplaceImpl(
MutableDataPartPtr & part,
Transaction & out_transaction,
DataPartsLock & lock,
DataPartStorageBuilderPtr builder,
DataPartsVector * out_covered_parts)
{
LOG_TRACE(log, "Renaming temporary part {} to {}.", part->data_part_storage->getPartDirectory(), part->name);
LOG_TRACE(log, "Renaming temporary part {} to {}.", part->getDataPartStorage().getPartDirectory(), part->name);
if (&out_transaction.data != this)
throw Exception("MergeTreeData::Transaction for one table cannot be used with another. It is a bug.",
@ -3029,7 +3038,7 @@ bool MergeTreeData::renameTempPartAndReplaceImpl(
/// All checks are passed. Now we can rename the part on disk.
/// So, we maintain invariant: if a non-temporary part in filesystem then it is in data_parts
preparePartForCommit(part, out_transaction, builder);
preparePartForCommit(part, out_transaction);
if (out_covered_parts)
{
@ -3045,21 +3054,19 @@ bool MergeTreeData::renameTempPartAndReplaceImpl(
MergeTreeData::DataPartsVector MergeTreeData::renameTempPartAndReplaceUnlocked(
MutableDataPartPtr & part,
Transaction & out_transaction,
DataPartStorageBuilderPtr builder,
DataPartsLock & lock)
{
DataPartsVector covered_parts;
renameTempPartAndReplaceImpl(part, out_transaction, lock, builder, &covered_parts);
renameTempPartAndReplaceImpl(part, out_transaction, lock, &covered_parts);
return covered_parts;
}
MergeTreeData::DataPartsVector MergeTreeData::renameTempPartAndReplace(
MutableDataPartPtr & part,
Transaction & out_transaction,
DataPartStorageBuilderPtr builder)
Transaction & out_transaction)
{
auto part_lock = lockParts();
return renameTempPartAndReplaceUnlocked(part, out_transaction, builder, part_lock);
return renameTempPartAndReplaceUnlocked(part, out_transaction, part_lock);
}
void MergeTreeData::removePartsFromWorkingSet(MergeTreeTransaction * txn, const MergeTreeData::DataPartsVector & remove, bool clear_without_timeout, DataPartsLock & acquired_lock)
@ -3136,7 +3143,7 @@ void MergeTreeData::removePartsInRangeFromWorkingSet(MergeTreeTransaction * txn,
removePartsInRangeFromWorkingSetAndGetPartsToRemoveFromZooKeeper(txn, drop_range, lock);
}
MergeTreeData::DataPartsVector MergeTreeData::removePartsInRangeFromWorkingSetAndGetPartsToRemoveFromZooKeeper(
MergeTreeData::PartsToRemoveFromZooKeeper MergeTreeData::removePartsInRangeFromWorkingSetAndGetPartsToRemoveFromZooKeeper(
MergeTreeTransaction * txn, const MergeTreePartInfo & drop_range, DataPartsLock & lock)
{
DataPartsVector parts_to_remove;
@ -3214,15 +3221,20 @@ MergeTreeData::DataPartsVector MergeTreeData::removePartsInRangeFromWorkingSetAn
/// FIXME refactor removePartsFromWorkingSet(...), do not remove parts twice
removePartsFromWorkingSet(txn, parts_to_remove, clear_without_timeout, lock);
/// Since we can return parts in Deleting state, we have to use a wrapper that restricts access to such parts.
PartsToRemoveFromZooKeeper parts_to_remove_from_zookeeper;
for (auto & part : parts_to_remove)
parts_to_remove_from_zookeeper.emplace_back(std::move(part));
for (auto & part : inactive_parts_to_remove_immediately)
{
if (!drop_range.contains(part->info))
continue;
part->remove_time.store(0, std::memory_order_relaxed);
parts_to_remove.push_back(std::move(part));
parts_to_remove_from_zookeeper.emplace_back(std::move(part), /* was_active */ false);
}
return parts_to_remove;
return parts_to_remove_from_zookeeper;
}
void MergeTreeData::restoreAndActivatePart(const DataPartPtr & part, DataPartsLock * acquired_lock)
@ -3240,9 +3252,9 @@ void MergeTreeData::outdateBrokenPartAndCloneToDetached(const DataPartPtr & part
{
auto metadata_snapshot = getInMemoryMetadataPtr();
if (prefix.empty())
LOG_INFO(log, "Cloning part {} to {} and making it obsolete.", part_to_detach->data_part_storage->getPartDirectory(), part_to_detach->name);
LOG_INFO(log, "Cloning part {} to {} and making it obsolete.", part_to_detach->getDataPartStorage().getPartDirectory(), part_to_detach->name);
else
LOG_INFO(log, "Cloning part {} to {}_{} and making it obsolete.", part_to_detach->data_part_storage->getPartDirectory(), prefix, part_to_detach->name);
LOG_INFO(log, "Cloning part {} to {}_{} and making it obsolete.", part_to_detach->getDataPartStorage().getPartDirectory(), prefix, part_to_detach->name);
part_to_detach->makeCloneInDetached(prefix, metadata_snapshot);
@ -3254,9 +3266,9 @@ void MergeTreeData::outdateBrokenPartAndCloneToDetached(const DataPartPtr & part
void MergeTreeData::forcefullyMovePartToDetachedAndRemoveFromMemory(const MergeTreeData::DataPartPtr & part_to_detach, const String & prefix, bool restore_covered)
{
if (prefix.empty())
LOG_INFO(log, "Renaming {} to {} and forgetting it.", part_to_detach->data_part_storage->getPartDirectory(), part_to_detach->name);
LOG_INFO(log, "Renaming {} to {} and forgetting it.", part_to_detach->getDataPartStorage().getPartDirectory(), part_to_detach->name);
else
LOG_INFO(log, "Renaming {} to {}_{} and forgetting it.", part_to_detach->data_part_storage->getPartDirectory(), prefix, part_to_detach->name);
LOG_INFO(log, "Renaming {} to {}_{} and forgetting it.", part_to_detach->getDataPartStorage().getPartDirectory(), prefix, part_to_detach->name);
auto lock = lockParts();
bool removed_active_part = false;
@ -3279,11 +3291,7 @@ void MergeTreeData::forcefullyMovePartToDetachedAndRemoveFromMemory(const MergeT
}
modifyPartState(it_part, DataPartState::Deleting);
auto builder = part->data_part_storage->getBuilder();
part->renameToDetached(prefix, builder);
builder->commit();
preparePartForRemoval(part)->renameToDetached(prefix);
data_parts_indexes.erase(it_part);
if (restore_covered && part->info.level == 0)
@ -3437,7 +3445,7 @@ void MergeTreeData::tryRemovePartImmediately(DataPartPtr && part)
try
{
part_to_delete->remove();
preparePartForRemoval(part_to_delete)->remove();
}
catch (...)
{
@ -3647,9 +3655,9 @@ void MergeTreeData::swapActivePart(MergeTreeData::DataPartPtr part_copy)
/// when allow_remote_fs_zero_copy_replication turned on and off again
original_active_part->force_keep_shared_data = false;
if (original_active_part->data_part_storage->supportZeroCopyReplication() &&
part_copy->data_part_storage->supportZeroCopyReplication() &&
original_active_part->data_part_storage->getUniqueId() == part_copy->data_part_storage->getUniqueId())
if (original_active_part->getDataPartStorage().supportZeroCopyReplication() &&
part_copy->getDataPartStorage().supportZeroCopyReplication() &&
original_active_part->getDataPartStorage().getUniqueId() == part_copy->getDataPartStorage().getUniqueId())
{
/// May be when several volumes use the same S3/HDFS storage
original_active_part->force_keep_shared_data = true;
@ -3669,7 +3677,7 @@ void MergeTreeData::swapActivePart(MergeTreeData::DataPartPtr part_copy)
/// All other locks are taken in StorageReplicatedMergeTree
lockSharedData(*part_copy);
original_active_part->data_part_storage->writeDeleteOnDestroyMarker(log);
preparePartForRemoval(original_active_part)->writeDeleteOnDestroyMarker();
return;
}
}
@ -3803,9 +3811,9 @@ MergeTreeData::DataPartPtr MergeTreeData::getPartIfExists(const String & part_na
static void loadPartAndFixMetadataImpl(MergeTreeData::MutableDataPartPtr part)
{
part->loadColumnsChecksumsIndexes(false, true);
part->modification_time = part->data_part_storage->getLastModified().epochTime();
part->data_part_storage->removeDeleteOnDestroyMarker();
part->data_part_storage->removeVersionMetadata();
part->modification_time = part->getDataPartStorage().getLastModified().epochTime();
part->removeDeleteOnDestroyMarker();
part->removeVersionMetadata();
}
void MergeTreeData::calculateColumnAndSecondaryIndexSizesImpl()
@ -3965,7 +3973,7 @@ void MergeTreeData::movePartitionToDisk(const ASTPtr & partition, const String &
auto disk = getStoragePolicy()->getDiskByName(name);
std::erase_if(parts, [&](auto part_ptr)
{
return part_ptr->data_part_storage->getDiskName() == disk->getName();
return part_ptr->getDataPartStorage().getDiskName() == disk->getName();
});
if (parts.empty())
@ -4015,7 +4023,7 @@ void MergeTreeData::movePartitionToVolume(const ASTPtr & partition, const String
{
for (const auto & disk : volume->getDisks())
{
if (part_ptr->data_part_storage->getDiskName() == disk->getName())
if (part_ptr->getDataPartStorage().getDiskName() == disk->getName())
{
return true;
}
@ -4212,7 +4220,7 @@ BackupEntries MergeTreeData::backupParts(const DataPartsVector & data_parts, con
make_temporary_hard_links = false;
hold_storage_and_part_ptrs = true;
}
else if (supportsReplication() && part->data_part_storage->supportZeroCopyReplication() && getSettings()->allow_remote_fs_zero_copy_replication)
else if (supportsReplication() && part->getDataPartStorage().supportZeroCopyReplication() && getSettings()->allow_remote_fs_zero_copy_replication)
{
/// Hard links don't work correctly with zero copy replication.
make_temporary_hard_links = false;
@ -4224,7 +4232,7 @@ BackupEntries MergeTreeData::backupParts(const DataPartsVector & data_parts, con
table_lock = lockForShare(local_context->getCurrentQueryId(), local_context->getSettingsRef().lock_acquire_timeout);
BackupEntries backup_entries_from_part;
part->data_part_storage->backup(
part->getDataPartStorage().backup(
part->checksums,
part->getFileNamesWithoutChecksums(),
data_path_in_backup,
@ -4235,7 +4243,7 @@ BackupEntries MergeTreeData::backupParts(const DataPartsVector & data_parts, con
auto projection_parts = part->getProjectionParts();
for (const auto & [projection_name, projection_part] : projection_parts)
{
projection_part->data_part_storage->backup(
projection_part->getDataPartStorage().backup(
projection_part->checksums,
projection_part->getFileNamesWithoutChecksums(),
fs::path{data_path_in_backup} / part->name,
@ -4911,22 +4919,16 @@ ReservationPtr MergeTreeData::reserveSpace(UInt64 expected_size, SpacePtr space)
return checkAndReturnReservation(expected_size, std::move(reservation));
}
ReservationPtr MergeTreeData::reserveSpace(UInt64 expected_size, const DataPartStoragePtr & data_part_storage)
ReservationPtr MergeTreeData::reserveSpace(UInt64 expected_size, const IDataPartStorage & data_part_storage)
{
expected_size = std::max(RESERVATION_MIN_ESTIMATION_SIZE, expected_size);
return data_part_storage->reserve(expected_size);
return data_part_storage.reserve(expected_size);
}
ReservationPtr MergeTreeData::reserveSpace(UInt64 expected_size, const DataPartStorageBuilderPtr & data_part_storage_builder)
ReservationPtr MergeTreeData::tryReserveSpace(UInt64 expected_size, const IDataPartStorage & data_part_storage)
{
expected_size = std::max(RESERVATION_MIN_ESTIMATION_SIZE, expected_size);
return data_part_storage_builder->reserve(expected_size);
}
ReservationPtr MergeTreeData::tryReserveSpace(UInt64 expected_size, const DataPartStoragePtr & data_part_storage)
{
expected_size = std::max(RESERVATION_MIN_ESTIMATION_SIZE, expected_size);
return data_part_storage->tryReserve(expected_size);
return data_part_storage.tryReserve(expected_size);
}
ReservationPtr MergeTreeData::tryReserveSpace(UInt64 expected_size, SpacePtr space)
@ -5063,7 +5065,7 @@ bool MergeTreeData::shouldPerformTTLMoveOnInsert(const SpacePtr & move_destinati
if (move_destination->isDisk())
{
auto disk = std::static_pointer_cast<IDisk>(move_destination);
if (auto volume = getStoragePolicy()->tryGetVolumeByDisk(disk))
if (auto volume = getStoragePolicy()->tryGetVolumeByDiskName(disk->getName()))
return volume->perform_ttl_move_on_insert;
}
return false;
@ -5075,11 +5077,11 @@ bool MergeTreeData::isPartInTTLDestination(const TTLDescription & ttl, const IMe
if (ttl.destination_type == DataDestinationType::VOLUME)
{
for (const auto & disk : policy->getVolumeByName(ttl.destination_name)->getDisks())
if (disk->getName() == part.data_part_storage->getDiskName())
if (disk->getName() == part.getDataPartStorage().getDiskName())
return true;
}
else if (ttl.destination_type == DataDestinationType::DISK)
return policy->getDiskByName(ttl.destination_name)->getName() == part.data_part_storage->getDiskName();
return policy->getDiskByName(ttl.destination_name)->getName() == part.getDataPartStorage().getDiskName();
return false;
}
@ -5151,7 +5153,7 @@ void MergeTreeData::Transaction::rollbackPartsToTemporaryState()
WriteBufferFromOwnString buf;
buf << " Rollbacking parts state to temporary and removing from working set:";
for (const auto & part : precommitted_parts)
buf << " " << part->data_part_storage->getPartDirectory();
buf << " " << part->getDataPartStorage().getPartDirectory();
buf << ".";
LOG_DEBUG(data.log, "Undoing transaction.{}", buf.str());
@ -5162,12 +5164,11 @@ void MergeTreeData::Transaction::rollbackPartsToTemporaryState()
clear();
}
void MergeTreeData::Transaction::addPart(MutableDataPartPtr & part, DataPartStorageBuilderPtr builder)
void MergeTreeData::Transaction::addPart(MutableDataPartPtr & part)
{
precommitted_parts.insert(part);
if (asInMemoryPart(part))
has_in_memory_parts = true;
part_builders.push_back(builder);
}
void MergeTreeData::Transaction::rollback()
@ -5177,13 +5178,31 @@ void MergeTreeData::Transaction::rollback()
WriteBufferFromOwnString buf;
buf << " Removing parts:";
for (const auto & part : precommitted_parts)
buf << " " << part->data_part_storage->getPartDirectory();
buf << " " << part->getDataPartStorage().getPartDirectory();
buf << ".";
LOG_DEBUG(data.log, "Undoing transaction.{}", buf.str());
data.removePartsFromWorkingSet(txn,
DataPartsVector(precommitted_parts.begin(), precommitted_parts.end()),
/* clear_without_timeout = */ true);
auto lock = data.lockParts();
if (data.data_parts_indexes.empty())
{
/// Table was dropped concurrently and all parts (including PreActive parts) were cleared, so there's nothing to rollback
if (!data.all_data_dropped)
{
Strings part_names;
for (const auto & part : precommitted_parts)
part_names.emplace_back(part->name);
throw Exception(ErrorCodes::LOGICAL_ERROR, "There are some PreActive parts ({}) to rollback, "
"but data parts set is empty and table {} was not dropped. It's a bug",
fmt::join(part_names, ", "), data.getStorageID().getNameForLogs());
}
}
else
{
data.removePartsFromWorkingSet(txn,
DataPartsVector(precommitted_parts.begin(), precommitted_parts.end()),
/* clear_without_timeout = */ true, &lock);
}
}
clear();
@ -5205,8 +5224,9 @@ MergeTreeData::DataPartsVector MergeTreeData::Transaction::commit(MergeTreeData:
auto parts_lock = acquired_parts_lock ? MergeTreeData::DataPartsLock() : data.lockParts();
auto * owing_parts_lock = acquired_parts_lock ? acquired_parts_lock : &parts_lock;
for (auto & builder : part_builders)
builder->commit();
for (const auto & part : precommitted_parts)
if (part->getDataPartStorage().hasActiveTransaction())
part->getDataPartStorage().commitTransaction();
bool commit_to_wal = has_in_memory_parts && settings->in_memory_parts_enable_wal;
if (txn || commit_to_wal)
@ -5215,7 +5235,7 @@ MergeTreeData::DataPartsVector MergeTreeData::Transaction::commit(MergeTreeData:
if (commit_to_wal)
wal = data.getWriteAheadLog();
for (const DataPartPtr & part : precommitted_parts)
for (const auto & part : precommitted_parts)
{
if (txn)
{
@ -5240,7 +5260,7 @@ MergeTreeData::DataPartsVector MergeTreeData::Transaction::commit(MergeTreeData:
size_t reduce_rows = 0;
size_t reduce_parts = 0;
for (const DataPartPtr & part : precommitted_parts)
for (const auto & part : precommitted_parts)
{
DataPartPtr covering_part;
DataPartsVector covered_parts = data.getActivePartsToReplace(part->info, part->name, covering_part, *owing_parts_lock);
@ -6232,7 +6252,7 @@ std::pair<MergeTreeData::MutableDataPartPtr, scope_guard> MergeTreeData::cloneAn
bool does_storage_policy_allow_same_disk = false;
for (const DiskPtr & disk : getStoragePolicy()->getDisks())
{
if (disk->getName() == src_part->data_part_storage->getDiskName())
if (disk->getName() == src_part->getDataPartStorage().getDiskName())
{
does_storage_policy_allow_same_disk = true;
break;
@ -6242,7 +6262,7 @@ std::pair<MergeTreeData::MutableDataPartPtr, scope_guard> MergeTreeData::cloneAn
throw Exception(
ErrorCodes::BAD_ARGUMENTS,
"Could not clone and load part {} because disk does not belong to storage policy",
quoteString(src_part->data_part_storage->getFullPath()));
quoteString(src_part->getDataPartStorage().getFullPath()));
String dst_part_name = src_part->getNewName(dst_part_info);
assert(!tmp_part_prefix.empty());
@ -6250,9 +6270,8 @@ std::pair<MergeTreeData::MutableDataPartPtr, scope_guard> MergeTreeData::cloneAn
auto temporary_directory_lock = getTemporaryPartDirectoryHolder(tmp_dst_part_name);
/// Why it is needed if we only hardlink files?
auto reservation = src_part->data_part_storage->reserve(src_part->getBytesOnDisk());
auto src_part_storage = src_part->data_part_storage;
auto reservation = src_part->getDataPartStorage().reserve(src_part->getBytesOnDisk());
auto src_part_storage = src_part->getDataPartStoragePtr();
/// If source part is in memory, flush it to disk and clone it already in on-disk format
if (auto src_part_in_memory = asInMemoryPart(src_part))
@ -6279,7 +6298,7 @@ std::pair<MergeTreeData::MutableDataPartPtr, scope_guard> MergeTreeData::cloneAn
hardlinked_files->source_part_name = src_part->name;
hardlinked_files->source_table_shared_id = src_part->storage.getTableSharedID();
for (auto it = src_part->data_part_storage->iterate(); it->isValid(); it->next())
for (auto it = src_part->getDataPartStorage().iterate(); it->isValid(); it->next())
{
if (!files_to_copy_instead_of_hardlinks.contains(it->name())
&& it->name() != IMergeTreeDataPart::DELETE_ON_DESTROY_MARKER_FILE_NAME
@ -6338,14 +6357,14 @@ Strings MergeTreeData::getDataPaths() const
void MergeTreeData::reportBrokenPart(MergeTreeData::DataPartPtr & data_part) const
{
if (data_part->data_part_storage && data_part->data_part_storage->isBroken())
if (data_part->getDataPartStorage().isBroken())
{
auto parts = getDataPartsForInternalUsage();
LOG_WARNING(log, "Scanning parts to recover on broken disk {}@{}.", data_part->data_part_storage->getDiskName(), data_part->data_part_storage->getDiskPath());
LOG_WARNING(log, "Scanning parts to recover on broken disk {}@{}.", data_part->getDataPartStorage().getDiskName(), data_part->getDataPartStorage().getDiskPath());
for (const auto & part : parts)
{
if (part->data_part_storage && part->data_part_storage->getDiskName() == data_part->data_part_storage->getDiskName())
if (part->getDataPartStorage().getDiskName() == data_part->getDataPartStorage().getDiskName())
broken_part_callback(part->name);
}
}
@ -6436,7 +6455,7 @@ PartitionCommandsResultInfo MergeTreeData::freezePartitionsByMatcher(
LOG_DEBUG(log, "Freezing part {} snapshot will be placed at {}", part->name, backup_path);
auto data_part_storage = part->data_part_storage;
auto data_part_storage = part->getDataPartStoragePtr();
String src_part_path = data_part_storage->getRelativePath();
String backup_part_path = fs::path(backup_path) / relative_data_path;
if (auto part_in_memory = asInMemoryPart(part))
@ -6450,12 +6469,12 @@ PartitionCommandsResultInfo MergeTreeData::freezePartitionsByMatcher(
// Store metadata for replicated table.
// Do nothing for non-replicated.
createAndStoreFreezeMetadata(disk, part, fs::path(backup_part_path) / part->data_part_storage->getPartDirectory());
createAndStoreFreezeMetadata(disk, part, fs::path(backup_part_path) / part->getDataPartStorage().getPartDirectory());
};
auto new_storage = data_part_storage->freeze(
backup_part_path,
part->data_part_storage->getPartDirectory(),
part->getDataPartStorage().getPartDirectory(),
/*make_source_readonly*/ true,
callback,
/*copy_instead_of_hardlink*/ false,
@ -6577,8 +6596,8 @@ try
if (result_part)
{
part_log_elem.disk_name = result_part->data_part_storage->getDiskName();
part_log_elem.path_on_disk = result_part->data_part_storage->getFullPath();
part_log_elem.disk_name = result_part->getDataPartStorage().getDiskName();
part_log_elem.path_on_disk = result_part->getDataPartStorage().getFullPath();
part_log_elem.bytes_compressed_on_disk = result_part->getBytesOnDisk();
part_log_elem.rows = result_part->rows_count;
part_log_elem.part_type = result_part->getType();
@ -6734,7 +6753,7 @@ bool MergeTreeData::moveParts(const CurrentlyMovingPartsTaggerPtr & moving_tagge
for (const auto & moving_part : moving_tagger->parts_to_move)
{
Stopwatch stopwatch;
DataPartPtr cloned_part;
MutableDataPartPtr cloned_part;
auto write_part_log = [&](const ExecutionStatus & execution_status)
{
@ -6997,7 +7016,7 @@ ReservationPtr MergeTreeData::balancedReservation(
if (part->isStoredOnDisk() && part->getBytesOnDisk() >= min_bytes_to_rebalance_partition_over_jbod
&& part_info.partition_id == part->info.partition_id)
{
auto name = part->data_part_storage->getDiskName();
auto name = part->getDataPartStorage().getDiskName();
auto it = disk_occupation.find(name);
if (it != disk_occupation.end())
{

67
src/Storages/MergeTree/MergeTreeData.h
View File

@ -214,6 +214,7 @@ public:
};
using DataParts = std::set<DataPartPtr, LessDataPart>;
using MutableDataParts = std::set<MutableDataPartPtr, LessDataPart>;
using DataPartsVector = std::vector<DataPartPtr>;
using DataPartsLock = std::unique_lock<std::mutex>;
@ -225,15 +226,15 @@ public:
/// After this method setColumns must be called
MutableDataPartPtr createPart(const String & name,
MergeTreeDataPartType type, const MergeTreePartInfo & part_info,
const DataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part = nullptr) const;
const MutableDataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part = nullptr) const;
/// Create part, that already exists on filesystem.
/// After this methods 'loadColumnsChecksumsIndexes' must be called.
MutableDataPartPtr createPart(const String & name,
const DataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part = nullptr) const;
const MutableDataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part = nullptr) const;
MutableDataPartPtr createPart(const String & name, const MergeTreePartInfo & part_info,
const DataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part = nullptr) const;
const MutableDataPartStoragePtr & data_part_storage, const IMergeTreeDataPart * parent_part = nullptr) const;
/// Auxiliary object to add a set of parts into the working set in two steps:
/// * First, as PreActive parts (the parts are ready, but not yet in the active set).
@ -247,7 +248,7 @@ public:
DataPartsVector commit(MergeTreeData::DataPartsLock * acquired_parts_lock = nullptr);
void addPart(MutableDataPartPtr & part, DataPartStorageBuilderPtr builder);
void addPart(MutableDataPartPtr & part);
void rollback();
@ -275,9 +276,8 @@ public:
MergeTreeData & data;
MergeTreeTransaction * txn;
DataParts precommitted_parts;
std::vector<DataPartStorageBuilderPtr> part_builders;
DataParts locked_parts;
MutableDataParts precommitted_parts;
MutableDataParts locked_parts;
bool has_in_memory_parts = false;
void clear();
@ -414,9 +414,8 @@ public:
SelectQueryInfo & info) const override;
ReservationPtr reserveSpace(UInt64 expected_size, VolumePtr & volume) const;
static ReservationPtr tryReserveSpace(UInt64 expected_size, const DataPartStoragePtr & data_part_storage);
static ReservationPtr reserveSpace(UInt64 expected_size, const DataPartStoragePtr & data_part_storage);
static ReservationPtr reserveSpace(UInt64 expected_size, const DataPartStorageBuilderPtr & data_part_storage_builder);
static ReservationPtr tryReserveSpace(UInt64 expected_size, const IDataPartStorage & data_part_storage);
static ReservationPtr reserveSpace(UInt64 expected_size, const IDataPartStorage & data_part_storage);
static bool partsContainSameProjections(const DataPartPtr & left, const DataPartPtr & right);
@ -555,21 +554,18 @@ public:
bool renameTempPartAndAdd(
MutableDataPartPtr & part,
Transaction & transaction,
DataPartStorageBuilderPtr builder,
DataPartsLock & lock);
/// The same as renameTempPartAndAdd but the block range of the part can contain existing parts.
/// Returns all parts covered by the added part (in ascending order).
DataPartsVector renameTempPartAndReplace(
MutableDataPartPtr & part,
Transaction & out_transaction,
DataPartStorageBuilderPtr builder);
Transaction & out_transaction);
/// Unlocked version of previous one. Useful when added multiple parts with a single lock.
DataPartsVector renameTempPartAndReplaceUnlocked(
MutableDataPartPtr & part,
Transaction & out_transaction,
DataPartStorageBuilderPtr builder,
DataPartsLock & lock);
/// Remove parts from working set immediately (without wait for background
@ -588,10 +584,33 @@ public:
/// Used in REPLACE PARTITION command.
void removePartsInRangeFromWorkingSet(MergeTreeTransaction * txn, const MergeTreePartInfo & drop_range, DataPartsLock & lock);
/// This wrapper is required to restrict access to parts in Deleting state
class PartToRemoveFromZooKeeper
{
DataPartPtr part;
bool was_active;
public:
explicit PartToRemoveFromZooKeeper(DataPartPtr && part_, bool was_active_ = true)
: part(std::move(part_)), was_active(was_active_)
{
}
/// It's safe to get name of any part
const String & getPartName() const { return part->name; }
DataPartPtr getPartIfItWasActive() const
{
return was_active ? part : nullptr;
}
};
using PartsToRemoveFromZooKeeper = std::vector<PartToRemoveFromZooKeeper>;
/// Same as above, but also returns list of parts to remove from ZooKeeper.
/// It includes parts that have been just removed by these method
/// and Outdated parts covered by drop_range that were removed earlier for any reason.
DataPartsVector removePartsInRangeFromWorkingSetAndGetPartsToRemoveFromZooKeeper(
PartsToRemoveFromZooKeeper removePartsInRangeFromWorkingSetAndGetPartsToRemoveFromZooKeeper(
MergeTreeTransaction * txn, const MergeTreePartInfo & drop_range, DataPartsLock & lock);
/// Restores Outdated part and adds it to working set
@ -644,6 +663,9 @@ public:
/// Deletes the data directory and flushes the uncompressed blocks cache and the marks cache.
void dropAllData();
/// This flag is for hardening and assertions.
bool all_data_dropped = false;
/// Drop data directories if they are empty. It is safe to call this method if table creation was unsuccessful.
void dropIfEmpty();
@ -979,7 +1001,7 @@ public:
/// Fetch part only if some replica has it on shared storage like S3
/// Overridden in StorageReplicatedMergeTree
virtual DataPartStoragePtr tryToFetchIfShared(const IMergeTreeDataPart &, const DiskPtr &, const String &) { return nullptr; }
virtual MutableDataPartStoragePtr tryToFetchIfShared(const IMergeTreeDataPart &, const DiskPtr &, const String &) { return nullptr; }
/// Check shared data usage on other replicas for detached/freezed part
/// Remove local files and remote files if needed
@ -1264,13 +1286,12 @@ protected:
static void incrementMergedPartsProfileEvent(MergeTreeDataPartType type);
private:
/// Checking that candidate part doesn't break invariants: correct partition and doesn't exist already
void checkPartCanBeAddedToTable(MutableDataPartPtr & part, DataPartsLock & lock) const;
/// Preparing itself to be committed in memory: fill some fields inside part, add it to data_parts_indexes
/// in precommitted state and to transaction
void preparePartForCommit(MutableDataPartPtr & part, Transaction & out_transaction, DataPartStorageBuilderPtr builder);
void preparePartForCommit(MutableDataPartPtr & part, Transaction & out_transaction);
/// Low-level method for preparing parts for commit (in-memory).
/// FIXME Merge MergeTreeTransaction and Transaction
@ -1278,7 +1299,6 @@ private:
MutableDataPartPtr & part,
Transaction & out_transaction,
DataPartsLock & lock,
DataPartStorageBuilderPtr builder,
DataPartsVector * out_covered_parts);
/// RAII Wrapper for atomic work with currently moving parts
@ -1334,8 +1354,8 @@ private:
virtual std::unique_ptr<MergeTreeSettings> getDefaultSettings() const = 0;
void loadDataPartsFromDisk(
DataPartsVector & broken_parts_to_detach,
DataPartsVector & duplicate_parts_to_remove,
MutableDataPartsVector & broken_parts_to_detach,
MutableDataPartsVector & duplicate_parts_to_remove,
ThreadPool & pool,
size_t num_parts,
std::queue<std::vector<std::pair<String, DiskPtr>>> & parts_queue,
@ -1343,8 +1363,7 @@ private:
const MergeTreeSettingsPtr & settings);
void loadDataPartsFromWAL(
DataPartsVector & broken_parts_to_detach,
DataPartsVector & duplicate_parts_to_remove,
MutableDataPartsVector & duplicate_parts_to_remove,
MutableDataPartsVector & parts_from_wal);
/// Create zero-copy exclusive lock for part and disk. Useful for coordination of
@ -1356,6 +1375,8 @@ private:
/// Otherwise, in non-parallel case will break and return.
void clearPartsFromFilesystemImpl(const DataPartsVector & parts, NameSet * part_names_succeed);
static MutableDataPartPtr preparePartForRemoval(const DataPartPtr & part);
TemporaryParts temporary_parts;
};

9
src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp
View File

@ -483,8 +483,7 @@ MergeTaskPtr MergeTreeDataMergerMutator::mergePartsToTemporaryPart(
const Names & deduplicate_by_columns,
const MergeTreeData::MergingParams & merging_params,
const MergeTreeTransactionPtr & txn,
const IMergeTreeDataPart * parent_part,
const IDataPartStorageBuilder * parent_path_storage_builder,
IMergeTreeDataPart * parent_part,
const String & suffix)
{
return std::make_shared<MergeTask>(
@ -499,7 +498,6 @@ MergeTaskPtr MergeTreeDataMergerMutator::mergePartsToTemporaryPart(
deduplicate_by_columns,
merging_params,
parent_part,
parent_path_storage_builder,
suffix,
txn,
&data,
@ -541,8 +539,7 @@ MergeTreeData::DataPartPtr MergeTreeDataMergerMutator::renameMergedTemporaryPart
MergeTreeData::MutableDataPartPtr & new_data_part,
const MergeTreeData::DataPartsVector & parts,
const MergeTreeTransactionPtr & txn,
MergeTreeData::Transaction & out_transaction,
DataPartStorageBuilderPtr builder)
MergeTreeData::Transaction & out_transaction)
{
/// Some of source parts was possibly created in transaction, so non-transactional merge may break isolation.
if (data.transactions_enabled.load(std::memory_order_relaxed) && !txn)
@ -550,7 +547,7 @@ MergeTreeData::DataPartPtr MergeTreeDataMergerMutator::renameMergedTemporaryPart
"but transactions were enabled for this table");
/// Rename new part, add to the set and remove original parts.
auto replaced_parts = data.renameTempPartAndReplace(new_data_part, out_transaction, builder);
auto replaced_parts = data.renameTempPartAndReplace(new_data_part, out_transaction);
/// Let's check that all original parts have been deleted and only them.
if (replaced_parts.size() != parts.size())

6
src/Storages/MergeTree/MergeTreeDataMergerMutator.h
View File

@ -113,8 +113,7 @@ public:
const Names & deduplicate_by_columns,
const MergeTreeData::MergingParams & merging_params,
const MergeTreeTransactionPtr & txn,
const IMergeTreeDataPart * parent_part = nullptr,
const IDataPartStorageBuilder * parent_path_storage_builder = nullptr,
IMergeTreeDataPart * parent_part = nullptr,
const String & suffix = "");
/// Mutate a single data part with the specified commands. Will create and return a temporary part.
@ -133,8 +132,7 @@ public:
MergeTreeData::MutableDataPartPtr & new_data_part,
const MergeTreeData::DataPartsVector & parts,
const MergeTreeTransactionPtr & txn,
MergeTreeData::Transaction & out_transaction,
DataPartStorageBuilderPtr builder);
MergeTreeData::Transaction & out_transaction);
/// The approximate amount of disk space needed for merge or mutation. With a surplus.

47
src/Storages/MergeTree/MergeTreeDataPartCompact.cpp
View File

@ -22,7 +22,7 @@ namespace ErrorCodes
MergeTreeDataPartCompact::MergeTreeDataPartCompact(
MergeTreeData & storage_,
const String & name_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_)
: IMergeTreeDataPart(storage_, name_, data_part_storage_, Type::Compact, parent_part_)
{
@ -32,7 +32,7 @@ MergeTreeDataPartCompact::MergeTreeDataPartCompact(
const MergeTreeData & storage_,
const String & name_,
const MergeTreePartInfo & info_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_)
: IMergeTreeDataPart(storage_, name_, info_, data_part_storage_, Type::Compact, parent_part_)
{
@ -58,13 +58,12 @@ IMergeTreeDataPart::MergeTreeReaderPtr MergeTreeDataPartCompact::getReader(
}
IMergeTreeDataPart::MergeTreeWriterPtr MergeTreeDataPartCompact::getWriter(
DataPartStorageBuilderPtr data_part_storage_builder,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & writer_settings,
const MergeTreeIndexGranularity & computed_index_granularity) const
const MergeTreeIndexGranularity & computed_index_granularity)
{
NamesAndTypesList ordered_columns_list;
std::copy_if(columns_list.begin(), columns_list.end(), std::back_inserter(ordered_columns_list),
@ -75,7 +74,7 @@ IMergeTreeDataPart::MergeTreeWriterPtr MergeTreeDataPartCompact::getWriter(
{ return *getColumnPosition(lhs.name) < *getColumnPosition(rhs.name); });
return std::make_unique<MergeTreeDataPartWriterCompact>(
shared_from_this(), std::move(data_part_storage_builder), ordered_columns_list, metadata_snapshot,
shared_from_this(), ordered_columns_list, metadata_snapshot,
indices_to_recalc, getMarksFileExtension(),
default_codec_, writer_settings, computed_index_granularity);
}
@ -97,21 +96,21 @@ void MergeTreeDataPartCompact::calculateEachColumnSizes(ColumnSizeByName & /*eac
void MergeTreeDataPartCompact::loadIndexGranularityImpl(
MergeTreeIndexGranularity & index_granularity_, const MergeTreeIndexGranularityInfo & index_granularity_info_,
size_t columns_count, const DataPartStoragePtr & data_part_storage_)
size_t columns_count, const IDataPartStorage & data_part_storage_)
{
if (!index_granularity_info_.mark_type.adaptive)
throw Exception("MergeTreeDataPartCompact cannot be created with non-adaptive granulary.", ErrorCodes::NOT_IMPLEMENTED);
auto marks_file_path = index_granularity_info_.getMarksFilePath("data");
if (!data_part_storage_->exists(marks_file_path))
if (!data_part_storage_.exists(marks_file_path))
throw Exception(
ErrorCodes::NO_FILE_IN_DATA_PART,
"Marks file '{}' doesn't exist",
std::string(fs::path(data_part_storage_->getFullPath()) / marks_file_path));
std::string(fs::path(data_part_storage_.getFullPath()) / marks_file_path));
size_t marks_file_size = data_part_storage_->getFileSize(marks_file_path);
size_t marks_file_size = data_part_storage_.getFileSize(marks_file_path);
std::unique_ptr<ReadBufferFromFileBase> buffer = data_part_storage_->readFile(
std::unique_ptr<ReadBufferFromFileBase> buffer = data_part_storage_.readFile(
marks_file_path, ReadSettings().adjustBufferSize(marks_file_size), marks_file_size, std::nullopt);
std::unique_ptr<ReadBuffer> marks_reader;
@ -140,7 +139,7 @@ void MergeTreeDataPartCompact::loadIndexGranularity()
if (columns.empty())
throw Exception("No columns in part " + name, ErrorCodes::NO_FILE_IN_DATA_PART);
loadIndexGranularityImpl(index_granularity, index_granularity_info, columns.size(), data_part_storage);
loadIndexGranularityImpl(index_granularity, index_granularity_info, columns.size(), getDataPartStorage());
}
bool MergeTreeDataPartCompact::hasColumnFiles(const NameAndTypePair & column) const
@ -171,12 +170,12 @@ void MergeTreeDataPartCompact::checkConsistency(bool require_part_metadata) cons
throw Exception(
ErrorCodes::NO_FILE_IN_DATA_PART,
"No marks file checksum for column in part {}",
data_part_storage->getFullPath());
getDataPartStorage().getFullPath());
if (!checksums.files.contains(DATA_FILE_NAME_WITH_EXTENSION))
throw Exception(
ErrorCodes::NO_FILE_IN_DATA_PART,
"No data file checksum for in part {}",
data_part_storage->getFullPath());
getDataPartStorage().getFullPath());
}
}
else
@ -184,33 +183,33 @@ void MergeTreeDataPartCompact::checkConsistency(bool require_part_metadata) cons
{
/// count.txt should be present even in non custom-partitioned parts
std::string file_path = "count.txt";
if (!data_part_storage->exists(file_path) || data_part_storage->getFileSize(file_path) == 0)
if (!getDataPartStorage().exists(file_path) || getDataPartStorage().getFileSize(file_path) == 0)
throw Exception(
ErrorCodes::BAD_SIZE_OF_FILE_IN_DATA_PART,
"Part {} is broken: {} is empty",
data_part_storage->getRelativePath(),
std::string(fs::path(data_part_storage->getFullPath()) / file_path));
getDataPartStorage().getRelativePath(),
std::string(fs::path(getDataPartStorage().getFullPath()) / file_path));
}
/// Check that marks are nonempty and have the consistent size with columns number.
if (data_part_storage->exists(mrk_file_name))
if (getDataPartStorage().exists(mrk_file_name))
{
UInt64 file_size = data_part_storage->getFileSize(mrk_file_name);
UInt64 file_size = getDataPartStorage().getFileSize(mrk_file_name);
if (!file_size)
throw Exception(
ErrorCodes::BAD_SIZE_OF_FILE_IN_DATA_PART,
"Part {} is broken: {} is empty.",
data_part_storage->getRelativePath(),
std::string(fs::path(data_part_storage->getFullPath()) / mrk_file_name));
getDataPartStorage().getRelativePath(),
std::string(fs::path(getDataPartStorage().getFullPath()) / mrk_file_name));
UInt64 expected_file_size = index_granularity_info.getMarkSizeInBytes(columns.size()) * index_granularity.getMarksCount();
if (expected_file_size != file_size)
throw Exception(
ErrorCodes::BAD_SIZE_OF_FILE_IN_DATA_PART,
"Part {} is broken: bad size of marks file '{}': {}, must be: {}",
data_part_storage->getRelativePath(),
std::string(fs::path(data_part_storage->getFullPath()) / mrk_file_name),
getDataPartStorage().getRelativePath(),
std::string(fs::path(getDataPartStorage().getFullPath()) / mrk_file_name),
std::to_string(file_size), std::to_string(expected_file_size));
}
}
@ -218,12 +217,12 @@ void MergeTreeDataPartCompact::checkConsistency(bool require_part_metadata) cons
bool MergeTreeDataPartCompact::isStoredOnRemoteDisk() const
{
return data_part_storage->isStoredOnRemoteDisk();
return getDataPartStorage().isStoredOnRemoteDisk();
}
bool MergeTreeDataPartCompact::isStoredOnRemoteDiskWithZeroCopySupport() const
{
return data_part_storage->supportZeroCopyReplication();
return getDataPartStorage().supportZeroCopyReplication();
}
MergeTreeDataPartCompact::~MergeTreeDataPartCompact()

9
src/Storages/MergeTree/MergeTreeDataPartCompact.h
View File

@ -25,13 +25,13 @@ public:
const MergeTreeData & storage_,
const String & name_,
const MergeTreePartInfo & info_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_ = nullptr);
MergeTreeDataPartCompact(
MergeTreeData & storage_,
const String & name_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_ = nullptr);
MergeTreeReaderPtr getReader(
@ -45,13 +45,12 @@ public:
const ReadBufferFromFileBase::ProfileCallback & profile_callback) const override;
MergeTreeWriterPtr getWriter(
DataPartStorageBuilderPtr data_part_storage_builder,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & writer_settings,
const MergeTreeIndexGranularity & computed_index_granularity) const override;
const MergeTreeIndexGranularity & computed_index_granularity) override;
bool isStoredOnDisk() const override { return true; }
@ -68,7 +67,7 @@ public:
protected:
static void loadIndexGranularityImpl(
MergeTreeIndexGranularity & index_granularity_, const MergeTreeIndexGranularityInfo & index_granularity_info_,
size_t columns_count, const DataPartStoragePtr & data_part_storage_);
size_t columns_count, const IDataPartStorage & data_part_storage_);
private:
void checkConsistency(bool require_part_metadata) const override;

58
src/Storages/MergeTree/MergeTreeDataPartInMemory.cpp
View File

@ -1,10 +1,12 @@
#include "MergeTreeDataPartInMemory.h"
#include <Storages/MergeTree/MergeTreeDataPartInMemory.h>
#include <Storages/MergeTree/MergeTreeReaderInMemory.h>
#include <Storages/MergeTree/MergedBlockOutputStream.h>
#include <Storages/MergeTree/MergeTreeDataPartWriterInMemory.h>
#include <Storages/MergeTree/IMergeTreeReader.h>
#include <Storages/MergeTree/LoadedMergeTreeDataPartInfoForReader.h>
#include <Storages/MergeTree/DataPartStorageOnDisk.h>
#include <DataTypes/NestedUtils.h>
#include <Disks/createVolume.h>
#include <Interpreters/Context.h>
#include <Poco/Logger.h>
#include <Common/logger_useful.h>
@ -21,7 +23,7 @@ namespace ErrorCodes
MergeTreeDataPartInMemory::MergeTreeDataPartInMemory(
MergeTreeData & storage_,
const String & name_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_)
: IMergeTreeDataPart(storage_, name_, data_part_storage_, Type::InMemory, parent_part_)
{
@ -32,7 +34,7 @@ MergeTreeDataPartInMemory::MergeTreeDataPartInMemory(
const MergeTreeData & storage_,
const String & name_,
const MergeTreePartInfo & info_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_)
: IMergeTreeDataPart(storage_, name_, info_, data_part_storage_, Type::InMemory, parent_part_)
{
@ -56,27 +58,33 @@ IMergeTreeDataPart::MergeTreeReaderPtr MergeTreeDataPartInMemory::getReader(
}
IMergeTreeDataPart::MergeTreeWriterPtr MergeTreeDataPartInMemory::getWriter(
DataPartStorageBuilderPtr data_part_storage_builder_,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot,
const std::vector<MergeTreeIndexPtr> & /* indices_to_recalc */,
const CompressionCodecPtr & /* default_codec */,
const MergeTreeWriterSettings & writer_settings,
const MergeTreeIndexGranularity & /* computed_index_granularity */) const
const MergeTreeIndexGranularity & /* computed_index_granularity */)
{
data_part_storage_builder = data_part_storage_builder_;
auto ptr = std::static_pointer_cast<const MergeTreeDataPartInMemory>(shared_from_this());
auto ptr = std::static_pointer_cast<MergeTreeDataPartInMemory>(shared_from_this());
return std::make_unique<MergeTreeDataPartWriterInMemory>(
ptr, columns_list, metadata_snapshot, writer_settings);
}
DataPartStoragePtr MergeTreeDataPartInMemory::flushToDisk(const String & new_relative_path, const StorageMetadataPtr & metadata_snapshot) const
MutableDataPartStoragePtr MergeTreeDataPartInMemory::flushToDisk(const String & new_relative_path, const StorageMetadataPtr & metadata_snapshot) const
{
auto current_full_path = data_part_storage_builder->getFullPath();
data_part_storage_builder->setRelativePath(new_relative_path);
auto reservation = storage.reserveSpace(block.bytes(), getDataPartStorage());
VolumePtr volume = storage.getStoragePolicy()->getVolume(0);
VolumePtr data_part_volume = createVolumeFromReservation(reservation, volume);
auto new_data_part_storage = std::make_shared<DataPartStorageOnDisk>(
data_part_volume,
storage.getRelativeDataPath(),
new_relative_path);
new_data_part_storage->beginTransaction();
auto current_full_path = getDataPartStorage().getFullPath();
auto new_type = storage.choosePartTypeOnDisk(block.bytes(), rows_count);
auto new_data_part_storage = data_part_storage_builder->getStorage();
auto new_data_part = storage.createPart(name, new_type, info, new_data_part_storage);
new_data_part->uuid = uuid;
@ -84,50 +92,50 @@ DataPartStoragePtr MergeTreeDataPartInMemory::flushToDisk(const String & new_rel
new_data_part->partition.value = partition.value;
new_data_part->minmax_idx = minmax_idx;
if (data_part_storage_builder->exists())
if (new_data_part_storage->exists())
{
throw Exception(
ErrorCodes::DIRECTORY_ALREADY_EXISTS,
"Could not flush part {}. Part in {} already exists",
quoteString(current_full_path),
data_part_storage_builder->getFullPath());
new_data_part_storage->getFullPath());
}
data_part_storage_builder->createDirectories();
new_data_part_storage->createDirectories();
auto compression_codec = storage.getContext()->chooseCompressionCodec(0, 0);
auto indices = MergeTreeIndexFactory::instance().getMany(metadata_snapshot->getSecondaryIndices());
MergedBlockOutputStream out(new_data_part, data_part_storage_builder, metadata_snapshot, columns, indices, compression_codec, NO_TRANSACTION_PTR);
MergedBlockOutputStream out(new_data_part, metadata_snapshot, columns, indices, compression_codec, NO_TRANSACTION_PTR);
out.write(block);
const auto & projections = metadata_snapshot->getProjections();
for (const auto & [projection_name, projection] : projection_parts)
{
if (projections.has(projection_name))
{
auto projection_part_storage_builder = data_part_storage_builder->getProjection(projection_name + ".proj");
if (projection_part_storage_builder->exists())
auto projection_part_storage = new_data_part_storage->getProjection(projection_name + ".proj");
if (projection_part_storage->exists())
{
throw Exception(
ErrorCodes::DIRECTORY_ALREADY_EXISTS,
"Could not flush projection part {}. Projection part in {} already exists",
projection_name,
projection_part_storage_builder->getFullPath());
projection_part_storage->getFullPath());
}
auto projection_part = asInMemoryPart(projection);
auto projection_type = storage.choosePartTypeOnDisk(projection_part->block.bytes(), rows_count);
MergeTreePartInfo projection_info("all", 0, 0, 0);
auto projection_data_part
= storage.createPart(projection_name, projection_type, projection_info, projection_part_storage_builder->getStorage(), parent_part);
= storage.createPart(projection_name, projection_type, projection_info, projection_part_storage, parent_part);
projection_data_part->is_temp = false; // clean up will be done on parent part
projection_data_part->setColumns(projection->getColumns(), {});
projection_part_storage_builder->createDirectories();
projection_part_storage->createDirectories();
const auto & desc = projections.get(name);
auto projection_compression_codec = storage.getContext()->chooseCompressionCodec(0, 0);
auto projection_indices = MergeTreeIndexFactory::instance().getMany(desc.metadata->getSecondaryIndices());
MergedBlockOutputStream projection_out(
projection_data_part, projection_part_storage_builder, desc.metadata, projection_part->columns, projection_indices,
projection_data_part, desc.metadata, projection_part->columns, projection_indices,
projection_compression_codec, NO_TRANSACTION_PTR);
projection_out.write(projection_part->block);
@ -137,6 +145,7 @@ DataPartStoragePtr MergeTreeDataPartInMemory::flushToDisk(const String & new_rel
}
out.finalizePart(new_data_part, false);
new_data_part_storage->commitTransaction();
return new_data_part_storage;
}
@ -146,12 +155,9 @@ void MergeTreeDataPartInMemory::makeCloneInDetached(const String & prefix, const
flushToDisk(detached_path, metadata_snapshot);
}
void MergeTreeDataPartInMemory::renameTo(const String & new_relative_path, bool /* remove_new_dir_if_exists */, DataPartStorageBuilderPtr) const
void MergeTreeDataPartInMemory::renameTo(const String & new_relative_path, bool /* remove_new_dir_if_exists */)
{
data_part_storage->setRelativePath(new_relative_path);
if (data_part_storage_builder)
data_part_storage_builder->setRelativePath(new_relative_path);
getDataPartStorage().setRelativePath(new_relative_path);
}
void MergeTreeDataPartInMemory::calculateEachColumnSizes(ColumnSizeByName & each_columns_size, ColumnSize & total_size) const

14
src/Storages/MergeTree/MergeTreeDataPartInMemory.h
View File

@ -14,13 +14,13 @@ public:
const MergeTreeData & storage_,
const String & name_,
const MergeTreePartInfo & info_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_ = nullptr);
MergeTreeDataPartInMemory(
MergeTreeData & storage_,
const String & name_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_ = nullptr);
MergeTreeReaderPtr getReader(
@ -34,29 +34,27 @@ public:
const ReadBufferFromFileBase::ProfileCallback & profile_callback) const override;
MergeTreeWriterPtr getWriter(
DataPartStorageBuilderPtr data_part_storage_builder_,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & writer_settings,
const MergeTreeIndexGranularity & computed_index_granularity) const override;
const MergeTreeIndexGranularity & computed_index_granularity) override;
bool isStoredOnDisk() const override { return false; }
bool isStoredOnRemoteDisk() const override { return false; }
bool isStoredOnRemoteDiskWithZeroCopySupport() const override { return false; }
bool hasColumnFiles(const NameAndTypePair & column) const override { return !!getColumnPosition(column.getNameInStorage()); }
String getFileNameForColumn(const NameAndTypePair & /* column */) const override { return ""; }
void renameTo(const String & new_relative_path, bool remove_new_dir_if_exists, DataPartStorageBuilderPtr) const override;
void renameTo(const String & new_relative_path, bool remove_new_dir_if_exists) override;
void makeCloneInDetached(const String & prefix, const StorageMetadataPtr & metadata_snapshot) const override;
DataPartStoragePtr flushToDisk(const String & new_relative_path, const StorageMetadataPtr & metadata_snapshot) const;
MutableDataPartStoragePtr flushToDisk(const String & new_relative_path, const StorageMetadataPtr & metadata_snapshot) const;
/// Returns hash of parts's block
Checksum calculateBlockChecksum() const;
mutable Block block;
mutable DataPartStorageBuilderPtr data_part_storage_builder;
private:
mutable std::condition_variable is_merged;
@ -66,6 +64,8 @@ private:
};
using DataPartInMemoryPtr = std::shared_ptr<const MergeTreeDataPartInMemory>;
using MutableDataPartInMemoryPtr = std::shared_ptr<MergeTreeDataPartInMemory>;
DataPartInMemoryPtr asInMemoryPart(const MergeTreeDataPartPtr & part);
}

41
src/Storages/MergeTree/MergeTreeDataPartWide.cpp
View File

@ -21,7 +21,7 @@ namespace ErrorCodes
MergeTreeDataPartWide::MergeTreeDataPartWide(
MergeTreeData & storage_,
const String & name_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_)
: IMergeTreeDataPart(storage_, name_, data_part_storage_, Type::Wide, parent_part_)
{
@ -31,7 +31,7 @@ MergeTreeDataPartWide::MergeTreeDataPartWide(
const MergeTreeData & storage_,
const String & name_,
const MergeTreePartInfo & info_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_)
: IMergeTreeDataPart(storage_, name_, info_, data_part_storage_, Type::Wide, parent_part_)
{
@ -56,17 +56,16 @@ IMergeTreeDataPart::MergeTreeReaderPtr MergeTreeDataPartWide::getReader(
}
IMergeTreeDataPart::MergeTreeWriterPtr MergeTreeDataPartWide::getWriter(
DataPartStorageBuilderPtr data_part_storage_builder,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & writer_settings,
const MergeTreeIndexGranularity & computed_index_granularity) const
const MergeTreeIndexGranularity & computed_index_granularity)
{
return std::make_unique<MergeTreeDataPartWriterWide>(
shared_from_this(), data_part_storage_builder,
columns_list, metadata_snapshot, indices_to_recalc,
shared_from_this(), columns_list,
metadata_snapshot, indices_to_recalc,
getMarksFileExtension(),
default_codec_, writer_settings, computed_index_granularity);
}
@ -105,18 +104,18 @@ ColumnSize MergeTreeDataPartWide::getColumnSizeImpl(
void MergeTreeDataPartWide::loadIndexGranularityImpl(
MergeTreeIndexGranularity & index_granularity_, MergeTreeIndexGranularityInfo & index_granularity_info_,
const DataPartStoragePtr & data_part_storage_, const std::string & any_column_file_name)
const IDataPartStorage & data_part_storage_, const std::string & any_column_file_name)
{
index_granularity_info_.changeGranularityIfRequired(data_part_storage_);
/// We can use any column, it doesn't matter
std::string marks_file_path = index_granularity_info_.getMarksFilePath(any_column_file_name);
if (!data_part_storage_->exists(marks_file_path))
if (!data_part_storage_.exists(marks_file_path))
throw Exception(
ErrorCodes::NO_FILE_IN_DATA_PART, "Marks file '{}' doesn't exist",
std::string(fs::path(data_part_storage_->getFullPath()) / marks_file_path));
std::string(fs::path(data_part_storage_.getFullPath()) / marks_file_path));
size_t marks_file_size = data_part_storage_->getFileSize(marks_file_path);
size_t marks_file_size = data_part_storage_.getFileSize(marks_file_path);
if (!index_granularity_info_.mark_type.adaptive && !index_granularity_info_.mark_type.compressed)
{
@ -126,7 +125,7 @@ void MergeTreeDataPartWide::loadIndexGranularityImpl(
}
else
{
auto marks_file = data_part_storage_->readFile(marks_file_path, ReadSettings().adjustBufferSize(marks_file_size), marks_file_size, std::nullopt);
auto marks_file = data_part_storage_.readFile(marks_file_path, ReadSettings().adjustBufferSize(marks_file_size), marks_file_size, std::nullopt);
std::unique_ptr<ReadBuffer> marks_reader;
if (!index_granularity_info_.mark_type.compressed)
@ -163,18 +162,18 @@ void MergeTreeDataPartWide::loadIndexGranularity()
if (columns.empty())
throw Exception("No columns in part " + name, ErrorCodes::NO_FILE_IN_DATA_PART);
loadIndexGranularityImpl(index_granularity, index_granularity_info, data_part_storage, getFileNameForColumn(columns.front()));
loadIndexGranularityImpl(index_granularity, index_granularity_info, getDataPartStorage(), getFileNameForColumn(columns.front()));
}
bool MergeTreeDataPartWide::isStoredOnRemoteDisk() const
{
return data_part_storage->isStoredOnRemoteDisk();
return getDataPartStorage().isStoredOnRemoteDisk();
}
bool MergeTreeDataPartWide::isStoredOnRemoteDiskWithZeroCopySupport() const
{
return data_part_storage->supportZeroCopyReplication();
return getDataPartStorage().supportZeroCopyReplication();
}
MergeTreeDataPartWide::~MergeTreeDataPartWide()
@ -203,13 +202,13 @@ void MergeTreeDataPartWide::checkConsistency(bool require_part_metadata) const
throw Exception(
ErrorCodes::NO_FILE_IN_DATA_PART,
"No {} file checksum for column {} in part {} ",
mrk_file_name, name_type.name, data_part_storage->getFullPath());
mrk_file_name, name_type.name, getDataPartStorage().getFullPath());
if (!checksums.files.contains(bin_file_name))
throw Exception(
ErrorCodes::NO_FILE_IN_DATA_PART,
"No {} file checksum for column {} in part ",
bin_file_name, name_type.name, data_part_storage->getFullPath());
bin_file_name, name_type.name, getDataPartStorage().getFullPath());
});
}
}
@ -225,23 +224,23 @@ void MergeTreeDataPartWide::checkConsistency(bool require_part_metadata) const
auto file_path = ISerialization::getFileNameForStream(name_type, substream_path) + marks_file_extension;
/// Missing file is Ok for case when new column was added.
if (data_part_storage->exists(file_path))
if (getDataPartStorage().exists(file_path))
{
UInt64 file_size = data_part_storage->getFileSize(file_path);
UInt64 file_size = getDataPartStorage().getFileSize(file_path);
if (!file_size)
throw Exception(
ErrorCodes::BAD_SIZE_OF_FILE_IN_DATA_PART,
"Part {} is broken: {} is empty.",
data_part_storage->getFullPath(),
std::string(fs::path(data_part_storage->getFullPath()) / file_path));
getDataPartStorage().getFullPath(),
std::string(fs::path(getDataPartStorage().getFullPath()) / file_path));
if (!marks_size)
marks_size = file_size;
else if (file_size != *marks_size)
throw Exception(
ErrorCodes::BAD_SIZE_OF_FILE_IN_DATA_PART,
"Part {} is broken: marks have different sizes.", data_part_storage->getFullPath());
"Part {} is broken: marks have different sizes.", getDataPartStorage().getFullPath());
}
});
}

10
src/Storages/MergeTree/MergeTreeDataPartWide.h
View File

@ -1,5 +1,6 @@
#pragma once
#include "Storages/MergeTree/IDataPartStorage.h"
#include <Storages/MergeTree/IMergeTreeDataPart.h>
namespace DB
@ -19,13 +20,13 @@ public:
const MergeTreeData & storage_,
const String & name_,
const MergeTreePartInfo & info_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_ = nullptr);
MergeTreeDataPartWide(
MergeTreeData & storage_,
const String & name_,
const DataPartStoragePtr & data_part_storage_,
const MutableDataPartStoragePtr & data_part_storage_,
const IMergeTreeDataPart * parent_part_ = nullptr);
MergeTreeReaderPtr getReader(
@ -39,13 +40,12 @@ public:
const ReadBufferFromFileBase::ProfileCallback & profile_callback) const override;
MergeTreeWriterPtr getWriter(
DataPartStorageBuilderPtr data_part_storage_builder,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & writer_settings,
const MergeTreeIndexGranularity & computed_index_granularity) const override;
const MergeTreeIndexGranularity & computed_index_granularity) override;
bool isStoredOnDisk() const override { return true; }
@ -64,7 +64,7 @@ public:
protected:
static void loadIndexGranularityImpl(
MergeTreeIndexGranularity & index_granularity_, MergeTreeIndexGranularityInfo & index_granularity_info_,
const DataPartStoragePtr & data_part_storage_, const std::string & any_column_file_name);
const IDataPartStorage & data_part_storage_, const std::string & any_column_file_name);
private:
void checkConsistency(bool require_part_metadata) const override;

9
src/Storages/MergeTree/MergeTreeDataPartWriterCompact.cpp
View File

@ -10,8 +10,7 @@ namespace ErrorCodes
}
MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact(
const MergeTreeData::DataPartPtr & data_part_,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part_,
const NamesAndTypesList & columns_list_,
const StorageMetadataPtr & metadata_snapshot_,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc_,
@ -19,16 +18,16 @@ MergeTreeDataPartWriterCompact::MergeTreeDataPartWriterCompact(
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & settings_,
const MergeTreeIndexGranularity & index_granularity_)
: MergeTreeDataPartWriterOnDisk(data_part_, std::move(data_part_storage_builder_), columns_list_, metadata_snapshot_,
: MergeTreeDataPartWriterOnDisk(data_part_, columns_list_, metadata_snapshot_,
indices_to_recalc_, marks_file_extension_,
default_codec_, settings_, index_granularity_)
, plain_file(data_part_storage_builder->writeFile(
, plain_file(data_part_->getDataPartStorage().writeFile(
MergeTreeDataPartCompact::DATA_FILE_NAME_WITH_EXTENSION,
settings.max_compress_block_size,
settings_.query_write_settings))
, plain_hashing(*plain_file)
{
marks_file = data_part_storage_builder->writeFile(
marks_file = data_part_->getDataPartStorage().writeFile(
MergeTreeDataPartCompact::DATA_FILE_NAME + marks_file_extension_,
4096,
settings_.query_write_settings);

3
src/Storages/MergeTree/MergeTreeDataPartWriterCompact.h
View File

@ -11,8 +11,7 @@ class MergeTreeDataPartWriterCompact : public MergeTreeDataPartWriterOnDisk
{
public:
MergeTreeDataPartWriterCompact(
const MergeTreeData::DataPartPtr & data_part,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot_,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,

4
src/Storages/MergeTree/MergeTreeDataPartWriterInMemory.cpp
View File

@ -11,11 +11,11 @@ namespace ErrorCodes
}
MergeTreeDataPartWriterInMemory::MergeTreeDataPartWriterInMemory(
const DataPartInMemoryPtr & part_,
const MutableDataPartInMemoryPtr & part_,
const NamesAndTypesList & columns_list_,
const StorageMetadataPtr & metadata_snapshot_,
const MergeTreeWriterSettings & settings_)
: IMergeTreeDataPartWriter(part_, nullptr, columns_list_, metadata_snapshot_, settings_)
: IMergeTreeDataPartWriter(part_, columns_list_, metadata_snapshot_, settings_)
, part_in_memory(part_) {}
void MergeTreeDataPartWriterInMemory::write(

4
src/Storages/MergeTree/MergeTreeDataPartWriterInMemory.h
View File

@ -10,7 +10,7 @@ class MergeTreeDataPartWriterInMemory : public IMergeTreeDataPartWriter
{
public:
MergeTreeDataPartWriterInMemory(
const DataPartInMemoryPtr & part_,
const MutableDataPartInMemoryPtr & part_,
const NamesAndTypesList & columns_list_,
const StorageMetadataPtr & metadata_snapshot,
const MergeTreeWriterSettings & settings_);
@ -24,7 +24,7 @@ public:
private:
void calculateAndSerializePrimaryIndex(const Block & primary_index_block);
DataPartInMemoryPtr part_in_memory;
MutableDataPartInMemoryPtr part_in_memory;
};
}

20
src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.cpp
View File

@ -48,7 +48,7 @@ void MergeTreeDataPartWriterOnDisk::Stream::sync() const
MergeTreeDataPartWriterOnDisk::Stream::Stream(
const String & escaped_column_name_,
const DataPartStorageBuilderPtr & data_part_storage_builder,
const MutableDataPartStoragePtr & data_part_storage,
const String & data_path_,
const std::string & data_file_extension_,
const std::string & marks_path_,
@ -61,11 +61,11 @@ MergeTreeDataPartWriterOnDisk::Stream::Stream(
escaped_column_name(escaped_column_name_),
data_file_extension{data_file_extension_},
marks_file_extension{marks_file_extension_},
plain_file(data_part_storage_builder->writeFile(data_path_ + data_file_extension, max_compress_block_size_, query_write_settings)),
plain_file(data_part_storage->writeFile(data_path_ + data_file_extension, max_compress_block_size_, query_write_settings)),
plain_hashing(*plain_file),
compressor(plain_hashing, compression_codec_, max_compress_block_size_),
compressed_hashing(compressor),
marks_file(data_part_storage_builder->writeFile(marks_path_ + marks_file_extension, 4096, query_write_settings)),
marks_file(data_part_storage->writeFile(marks_path_ + marks_file_extension, 4096, query_write_settings)),
marks_hashing(*marks_file),
marks_compressor(marks_hashing, marks_compression_codec_, marks_compress_block_size_),
marks_compressed_hashing(marks_compressor),
@ -96,8 +96,7 @@ void MergeTreeDataPartWriterOnDisk::Stream::addToChecksums(MergeTreeData::DataPa
MergeTreeDataPartWriterOnDisk::MergeTreeDataPartWriterOnDisk(
const MergeTreeData::DataPartPtr & data_part_,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part_,
const NamesAndTypesList & columns_list_,
const StorageMetadataPtr & metadata_snapshot_,
const MergeTreeIndices & indices_to_recalc_,
@ -105,8 +104,7 @@ MergeTreeDataPartWriterOnDisk::MergeTreeDataPartWriterOnDisk(
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & settings_,
const MergeTreeIndexGranularity & index_granularity_)
: IMergeTreeDataPartWriter(data_part_, std::move(data_part_storage_builder_),
columns_list_, metadata_snapshot_, settings_, index_granularity_)
: IMergeTreeDataPartWriter(data_part_, columns_list_, metadata_snapshot_, settings_, index_granularity_)
, skip_indices(indices_to_recalc_)
, marks_file_extension(marks_file_extension_)
, default_codec(default_codec_)
@ -116,8 +114,8 @@ MergeTreeDataPartWriterOnDisk::MergeTreeDataPartWriterOnDisk(
if (settings.blocks_are_granules_size && !index_granularity.empty())
throw Exception("Can't take information about index granularity from blocks, when non empty index_granularity array specified", ErrorCodes::LOGICAL_ERROR);
if (!data_part_storage_builder->exists())
data_part_storage_builder->createDirectories();
if (!data_part->getDataPartStorage().exists())
data_part->getDataPartStorage().createDirectories();
if (settings.rewrite_primary_key)
initPrimaryIndex();
@ -178,7 +176,7 @@ void MergeTreeDataPartWriterOnDisk::initPrimaryIndex()
if (metadata_snapshot->hasPrimaryKey())
{
String index_name = "primary" + getIndexExtension(compress_primary_key);
index_file_stream = data_part_storage_builder->writeFile(index_name, DBMS_DEFAULT_BUFFER_SIZE, settings.query_write_settings);
index_file_stream = data_part->getDataPartStorage().writeFile(index_name, DBMS_DEFAULT_BUFFER_SIZE, settings.query_write_settings);
index_file_hashing_stream = std::make_unique<HashingWriteBuffer>(*index_file_stream);
if (compress_primary_key)
@ -204,7 +202,7 @@ void MergeTreeDataPartWriterOnDisk::initSkipIndices()
skip_indices_streams.emplace_back(
std::make_unique<MergeTreeDataPartWriterOnDisk::Stream>(
stream_name,
data_part_storage_builder,
data_part->getDataPartStoragePtr(),
stream_name, index_helper->getSerializedFileExtension(),
stream_name, marks_file_extension,
default_codec, settings.max_compress_block_size,

5
src/Storages/MergeTree/MergeTreeDataPartWriterOnDisk.h
View File

@ -50,7 +50,7 @@ public:
{
Stream(
const String & escaped_column_name_,
const DataPartStorageBuilderPtr & data_part_storage_builder,
const MutableDataPartStoragePtr & data_part_storage,
const String & data_path_,
const std::string & data_file_extension_,
const std::string & marks_path_,
@ -92,8 +92,7 @@ public:
using StreamPtr = std::unique_ptr<Stream>;
MergeTreeDataPartWriterOnDisk(
const MergeTreeData::DataPartPtr & data_part_,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part_,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot_,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,

17
src/Storages/MergeTree/MergeTreeDataPartWriterWide.cpp
View File

@ -71,8 +71,7 @@ Granules getGranulesToWrite(const MergeTreeIndexGranularity & index_granularity,
}
MergeTreeDataPartWriterWide::MergeTreeDataPartWriterWide(
const MergeTreeData::DataPartPtr & data_part_,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part_,
const NamesAndTypesList & columns_list_,
const StorageMetadataPtr & metadata_snapshot_,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc_,
@ -80,7 +79,7 @@ MergeTreeDataPartWriterWide::MergeTreeDataPartWriterWide(
const CompressionCodecPtr & default_codec_,
const MergeTreeWriterSettings & settings_,
const MergeTreeIndexGranularity & index_granularity_)
: MergeTreeDataPartWriterOnDisk(data_part_, std::move(data_part_storage_builder_), columns_list_, metadata_snapshot_,
: MergeTreeDataPartWriterOnDisk(data_part_, columns_list_, metadata_snapshot_,
indices_to_recalc_, marks_file_extension_,
default_codec_, settings_, index_granularity_)
{
@ -117,7 +116,7 @@ void MergeTreeDataPartWriterWide::addStreams(
column_streams[stream_name] = std::make_unique<Stream>(
stream_name,
data_part_storage_builder,
data_part->getDataPartStoragePtr(),
stream_name, DATA_FILE_EXTENSION,
stream_name, marks_file_extension,
compression_codec,
@ -421,20 +420,18 @@ void MergeTreeDataPartWriterWide::validateColumnOfFixedSize(const NameAndTypePai
String mrk_path = escaped_name + marks_file_extension;
String bin_path = escaped_name + DATA_FILE_EXTENSION;
auto data_part_storage = data_part_storage_builder->getStorage();
/// Some columns may be removed because of ttl. Skip them.
if (!data_part_storage->exists(mrk_path))
if (!data_part->getDataPartStorage().exists(mrk_path))
return;
auto mrk_file_in = data_part_storage->readFile(mrk_path, {}, std::nullopt, std::nullopt);
auto mrk_file_in = data_part->getDataPartStorage().readFile(mrk_path, {}, std::nullopt, std::nullopt);
std::unique_ptr<ReadBuffer> mrk_in;
if (data_part->index_granularity_info.mark_type.compressed)
mrk_in = std::make_unique<CompressedReadBufferFromFile>(std::move(mrk_file_in));
else
mrk_in = std::move(mrk_file_in);
DB::CompressedReadBufferFromFile bin_in(data_part_storage->readFile(bin_path, {}, std::nullopt, std::nullopt));
DB::CompressedReadBufferFromFile bin_in(data_part->getDataPartStorage().readFile(bin_path, {}, std::nullopt, std::nullopt));
bool must_be_last = false;
UInt64 offset_in_compressed_file = 0;
UInt64 offset_in_decompressed_block = 0;
@ -485,7 +482,7 @@ void MergeTreeDataPartWriterWide::validateColumnOfFixedSize(const NameAndTypePai
if (index_granularity_rows != index_granularity.getMarkRows(mark_num))
throw Exception(
ErrorCodes::LOGICAL_ERROR, "Incorrect mark rows for part {} for mark #{} (compressed offset {}, decompressed offset {}), in-memory {}, on disk {}, total marks {}",
data_part_storage_builder->getFullPath(), mark_num, offset_in_compressed_file, offset_in_decompressed_block, index_granularity.getMarkRows(mark_num), index_granularity_rows, index_granularity.getMarksCount());
data_part->getDataPartStorage().getFullPath(), mark_num, offset_in_compressed_file, offset_in_decompressed_block, index_granularity.getMarkRows(mark_num), index_granularity_rows, index_granularity.getMarksCount());
auto column = type->createColumn();

3
src/Storages/MergeTree/MergeTreeDataPartWriterWide.h
View File

@ -18,8 +18,7 @@ class MergeTreeDataPartWriterWide : public MergeTreeDataPartWriterOnDisk
{
public:
MergeTreeDataPartWriterWide(
const MergeTreeData::DataPartPtr & data_part,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part,
const NamesAndTypesList & columns_list,
const StorageMetadataPtr & metadata_snapshot,
const std::vector<MergeTreeIndexPtr> & indices_to_recalc,

6
src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp
View File

@ -1639,10 +1639,10 @@ MarkRanges MergeTreeDataSelectExecutor::filterMarksUsingIndex(
UncompressedCache * uncompressed_cache,
Poco::Logger * log)
{
if (!index_helper->getDeserializedFormat(part->data_part_storage, index_helper->getFileName()))
if (!index_helper->getDeserializedFormat(part->getDataPartStorage(), index_helper->getFileName()))
{
LOG_DEBUG(log, "File for index {} does not exist ({}.*). Skipping it.", backQuote(index_helper->index.name),
(fs::path(part->data_part_storage->getFullPath()) / index_helper->getFileName()).string());
(fs::path(part->getDataPartStorage().getFullPath()) / index_helper->getFileName()).string());
return ranges;
}
@ -1757,7 +1757,7 @@ MarkRanges MergeTreeDataSelectExecutor::filterMarksUsingMergedIndex(
{
for (const auto & index_helper : indices)
{
if (!part->data_part_storage->exists(index_helper->getFileName() + ".idx"))
if (!part->getDataPartStorage().exists(index_helper->getFileName() + ".idx"))
{
LOG_DEBUG(log, "File for index {} does not exist. Skipping it.", backQuote(index_helper->index.name));
return ranges;

48
src/Storages/MergeTree/MergeTreeDataWriter.cpp
View File

@ -378,10 +378,7 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeTempPart(
data.relative_data_path,
TMP_PREFIX + part_name);
auto data_part_storage_builder = std::make_shared<DataPartStorageBuilderOnDisk>(
data_part_volume,
data.relative_data_path,
TMP_PREFIX + part_name);
data_part_storage->beginTransaction();
auto new_data_part = data.createPart(
part_name,
@ -408,15 +405,15 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeTempPart(
if (new_data_part->isStoredOnDisk())
{
/// The name could be non-unique in case of stale files from previous runs.
String full_path = new_data_part->data_part_storage->getFullPath();
String full_path = new_data_part->getDataPartStorage().getFullPath();
if (new_data_part->data_part_storage->exists())
if (new_data_part->getDataPartStorage().exists())
{
LOG_WARNING(log, "Removing old temporary directory {}", full_path);
data_part_storage_builder->removeRecursive();
data_part_storage->removeRecursive();
}
data_part_storage_builder->createDirectories();
data_part_storage->createDirectories();
if (data.getSettings()->fsync_part_directory)
{
@ -448,7 +445,7 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeTempPart(
auto compression_codec = data.getContext()->chooseCompressionCodec(0, 0);
const auto & index_factory = MergeTreeIndexFactory::instance();
auto out = std::make_unique<MergedBlockOutputStream>(new_data_part, data_part_storage_builder, metadata_snapshot, columns,
auto out = std::make_unique<MergedBlockOutputStream>(new_data_part, metadata_snapshot, columns,
index_factory.getMany(metadata_snapshot->getSecondaryIndices()), compression_codec,
context->getCurrentTransaction(), false, false, context->getWriteSettings());
@ -459,9 +456,8 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeTempPart(
auto projection_block = projection.calculate(block, context);
if (projection_block.rows())
{
auto proj_temp_part = writeProjectionPart(data, log, projection_block, projection, data_part_storage_builder, new_data_part.get());
auto proj_temp_part = writeProjectionPart(data, log, projection_block, projection, new_data_part.get());
new_data_part->addProjectionPart(projection.name, std::move(proj_temp_part.part));
proj_temp_part.builder->commit();
for (auto & stream : proj_temp_part.streams)
temp_part.streams.emplace_back(std::move(stream));
}
@ -473,7 +469,6 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeTempPart(
nullptr, nullptr);
temp_part.part = new_data_part;
temp_part.builder = data_part_storage_builder;
temp_part.streams.emplace_back(TemporaryPart::Stream{.stream = std::move(out), .finalizer = std::move(finalizer)});
ProfileEvents::increment(ProfileEvents::MergeTreeDataWriterRows, block.rows());
@ -487,9 +482,8 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPartImpl(
const String & part_name,
MergeTreeDataPartType part_type,
const String & relative_path,
const DataPartStorageBuilderPtr & data_part_storage_builder,
bool is_temp,
const IMergeTreeDataPart * parent_part,
IMergeTreeDataPart * parent_part,
const MergeTreeData & data,
Poco::Logger * log,
Block block,
@ -498,7 +492,8 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPartImpl(
TemporaryPart temp_part;
const StorageMetadataPtr & metadata_snapshot = projection.metadata;
MergeTreePartInfo new_part_info("all", 0, 0, 0);
auto projection_part_storage = parent_part->data_part_storage->getProjection(relative_path);
auto projection_part_storage = parent_part->getDataPartStorage().getProjection(relative_path);
auto new_data_part = data.createPart(
part_name,
part_type,
@ -506,7 +501,6 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPartImpl(
projection_part_storage,
parent_part);
auto projection_part_storage_builder = data_part_storage_builder->getProjection(relative_path);
new_data_part->is_temp = is_temp;
NamesAndTypesList columns = metadata_snapshot->getColumns().getAllPhysical().filter(block.getNames());
@ -522,10 +516,10 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPartImpl(
if (projection_part_storage->exists())
{
LOG_WARNING(log, "Removing old temporary directory {}", projection_part_storage->getFullPath());
projection_part_storage_builder->removeRecursive();
projection_part_storage->removeRecursive();
}
projection_part_storage_builder->createDirectories();
projection_part_storage->createDirectories();
}
/// If we need to calculate some columns to sort.
@ -569,7 +563,6 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPartImpl(
auto out = std::make_unique<MergedBlockOutputStream>(
new_data_part,
projection_part_storage_builder,
metadata_snapshot,
columns,
MergeTreeIndices{},
@ -580,7 +573,6 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPartImpl(
out->writeWithPermutation(block, perm_ptr);
auto finalizer = out->finalizePartAsync(new_data_part, false);
temp_part.part = new_data_part;
temp_part.builder = projection_part_storage_builder;
temp_part.streams.emplace_back(TemporaryPart::Stream{.stream = std::move(out), .finalizer = std::move(finalizer)});
ProfileEvents::increment(ProfileEvents::MergeTreeDataProjectionWriterRows, block.rows());
@ -595,8 +587,7 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPart(
Poco::Logger * log,
Block block,
const ProjectionDescription & projection,
const DataPartStorageBuilderPtr & data_part_storage_builder,
const IMergeTreeDataPart * parent_part)
IMergeTreeDataPart * parent_part)
{
String part_name = projection.name;
MergeTreeDataPartType part_type;
@ -609,7 +600,7 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPart(
/// Size of part would not be greater than block.bytes() + epsilon
size_t expected_size = block.bytes();
// just check if there is enough space on parent volume
data.reserveSpace(expected_size, data_part_storage_builder);
data.reserveSpace(expected_size, parent_part->getDataPartStorage());
part_type = data.choosePartTypeOnDisk(expected_size, block.rows());
}
@ -617,7 +608,6 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeProjectionPart(
part_name,
part_type,
part_name + ".proj" /* relative_path */,
data_part_storage_builder,
false /* is_temp */,
parent_part,
data,
@ -633,8 +623,7 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeTempProjectionPart(
Poco::Logger * log,
Block block,
const ProjectionDescription & projection,
const DataPartStorageBuilderPtr & data_part_storage_builder,
const IMergeTreeDataPart * parent_part,
IMergeTreeDataPart * parent_part,
size_t block_num)
{
String part_name = fmt::format("{}_{}", projection.name, block_num);
@ -648,7 +637,7 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeTempProjectionPart(
/// Size of part would not be greater than block.bytes() + epsilon
size_t expected_size = block.bytes();
// just check if there is enough space on parent volume
data.reserveSpace(expected_size, data_part_storage_builder);
data.reserveSpace(expected_size, parent_part->getDataPartStorage());
part_type = data.choosePartTypeOnDisk(expected_size, block.rows());
}
@ -656,7 +645,6 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeTempProjectionPart(
part_name,
part_type,
part_name + ".tmp_proj" /* relative_path */,
data_part_storage_builder,
true /* is_temp */,
parent_part,
data,
@ -670,14 +658,12 @@ MergeTreeDataWriter::TemporaryPart MergeTreeDataWriter::writeInMemoryProjectionP
Poco::Logger * log,
Block block,
const ProjectionDescription & projection,
const DataPartStorageBuilderPtr & data_part_storage_builder,
const IMergeTreeDataPart * parent_part)
IMergeTreeDataPart * parent_part)
{
return writeProjectionPartImpl(
projection.name,
MergeTreeDataPartType::InMemory,
projection.name + ".proj" /* relative_path */,
data_part_storage_builder,
false /* is_temp */,
parent_part,
data,

13
src/Storages/MergeTree/MergeTreeDataWriter.h
View File

@ -52,7 +52,6 @@ public:
struct TemporaryPart
{
MergeTreeData::MutableDataPartPtr part;
DataPartStorageBuilderPtr builder;
struct Stream
{
@ -78,8 +77,7 @@ public:
Poco::Logger * log,
Block block,
const ProjectionDescription & projection,
const DataPartStorageBuilderPtr & data_part_storage_builder,
const IMergeTreeDataPart * parent_part);
IMergeTreeDataPart * parent_part);
/// For mutation: MATERIALIZE PROJECTION.
static TemporaryPart writeTempProjectionPart(
@ -87,8 +85,7 @@ public:
Poco::Logger * log,
Block block,
const ProjectionDescription & projection,
const DataPartStorageBuilderPtr & data_part_storage_builder,
const IMergeTreeDataPart * parent_part,
IMergeTreeDataPart * parent_part,
size_t block_num);
/// For WriteAheadLog AddPart.
@ -97,8 +94,7 @@ public:
Poco::Logger * log,
Block block,
const ProjectionDescription & projection,
const DataPartStorageBuilderPtr & data_part_storage_builder,
const IMergeTreeDataPart * parent_part);
IMergeTreeDataPart * parent_part);
static Block mergeBlock(
const Block & block,
@ -112,9 +108,8 @@ private:
const String & part_name,
MergeTreeDataPartType part_type,
const String & relative_path,
const DataPartStorageBuilderPtr & data_part_storage_builder,
bool is_temp,
const IMergeTreeDataPart * parent_part,
IMergeTreeDataPart * parent_part,
const MergeTreeData & data,
Poco::Logger * log,
Block block,

8
src/Storages/MergeTree/MergeTreeIndexGranularityInfo.cpp
View File

@ -89,10 +89,10 @@ std::string MarkType::getFileExtension() const
}
std::optional<std::string> MergeTreeIndexGranularityInfo::getMarksExtensionFromFilesystem(const DataPartStoragePtr & data_part_storage)
std::optional<std::string> MergeTreeIndexGranularityInfo::getMarksExtensionFromFilesystem(const IDataPartStorage & data_part_storage)
{
if (data_part_storage->exists())
for (auto it = data_part_storage->iterate(); it->isValid(); it->next())
if (data_part_storage.exists())
for (auto it = data_part_storage.iterate(); it->isValid(); it->next())
if (it->isFile())
if (std::string ext = fs::path(it->name()).extension(); MarkType::isMarkFileExtension(ext))
return ext;
@ -110,7 +110,7 @@ MergeTreeIndexGranularityInfo::MergeTreeIndexGranularityInfo(const MergeTreeData
fixed_index_granularity = storage.getSettings()->index_granularity;
}
void MergeTreeIndexGranularityInfo::changeGranularityIfRequired(const DataPartStoragePtr & data_part_storage)
void MergeTreeIndexGranularityInfo::changeGranularityIfRequired(const IDataPartStorage & data_part_storage)
{
auto mrk_ext = getMarksExtensionFromFilesystem(data_part_storage);
if (mrk_ext && !MarkType(*mrk_ext).adaptive)

4
src/Storages/MergeTree/MergeTreeIndexGranularityInfo.h
View File

@ -48,7 +48,7 @@ public:
MergeTreeIndexGranularityInfo(MergeTreeDataPartType type_, bool is_adaptive_, size_t index_granularity_, size_t index_granularity_bytes_);
void changeGranularityIfRequired(const DataPartStoragePtr & data_part_storage);
void changeGranularityIfRequired(const IDataPartStorage & data_part_storage);
String getMarksFilePath(const String & path_prefix) const
{
@ -57,7 +57,7 @@ public:
size_t getMarkSizeInBytes(size_t columns_num = 1) const;
static std::optional<std::string> getMarksExtensionFromFilesystem(const DataPartStoragePtr & data_part_storage);
static std::optional<std::string> getMarksExtensionFromFilesystem(const IDataPartStorage & data_part_storage);
};
constexpr inline auto getNonAdaptiveMrkSizeWide() { return sizeof(UInt64) * 2; }

6
src/Storages/MergeTree/MergeTreeIndexMinMax.cpp
View File

@ -211,11 +211,11 @@ bool MergeTreeIndexMinMax::mayBenefitFromIndexForIn(const ASTPtr & node) const
return false;
}
MergeTreeIndexFormat MergeTreeIndexMinMax::getDeserializedFormat(const DataPartStoragePtr & data_part_storage, const std::string & relative_path_prefix) const
MergeTreeIndexFormat MergeTreeIndexMinMax::getDeserializedFormat(const IDataPartStorage & data_part_storage, const std::string & relative_path_prefix) const
{
if (data_part_storage->exists(relative_path_prefix + ".idx2"))
if (data_part_storage.exists(relative_path_prefix + ".idx2"))
return {2, ".idx2"};
else if (data_part_storage->exists(relative_path_prefix + ".idx"))
else if (data_part_storage.exists(relative_path_prefix + ".idx"))
return {1, ".idx"};
return {0 /* unknown */, ""};
}

2
src/Storages/MergeTree/MergeTreeIndexMinMax.h
View File

@ -83,7 +83,7 @@ public:
bool mayBenefitFromIndexForIn(const ASTPtr & node) const override;
const char* getSerializedFileExtension() const override { return ".idx2"; }
MergeTreeIndexFormat getDeserializedFormat(const DataPartStoragePtr & data_part_storage, const std::string & path_prefix) const override; /// NOLINT
MergeTreeIndexFormat getDeserializedFormat(const IDataPartStorage & data_part_storage, const std::string & path_prefix) const override; /// NOLINT
};
}

4
src/Storages/MergeTree/MergeTreeIndexReader.cpp
View File

@ -20,7 +20,7 @@ std::unique_ptr<MergeTreeReaderStream> makeIndexReader(
auto * load_marks_threadpool = settings.read_settings.load_marks_asynchronously ? &context->getLoadMarksThreadpool() : nullptr;
return std::make_unique<MergeTreeReaderStream>(
part->data_part_storage,
part->getDataPartStoragePtr(),
index->getFileName(), extension, marks_count,
all_mark_ranges,
std::move(settings), mark_cache, uncompressed_cache,
@ -44,7 +44,7 @@ MergeTreeIndexReader::MergeTreeIndexReader(
MergeTreeReaderSettings settings)
: index(index_)
{
auto index_format = index->getDeserializedFormat(part_->data_part_storage, index->getFileName());
auto index_format = index->getDeserializedFormat(part_->getDataPartStorage(), index->getFileName());
stream = makeIndexReader(
index_format.extension,

4
src/Storages/MergeTree/MergeTreeIndices.h
View File

@ -148,9 +148,9 @@ struct IMergeTreeIndex
/// Returns extension for deserialization.
///
/// Return pair<extension, version>.
virtual MergeTreeIndexFormat getDeserializedFormat(const DataPartStoragePtr & data_part_storage, const std::string & relative_path_prefix) const
virtual MergeTreeIndexFormat getDeserializedFormat(const IDataPartStorage & data_part_storage, const std::string & relative_path_prefix) const
{
if (data_part_storage->exists(relative_path_prefix + ".idx"))
if (data_part_storage.exists(relative_path_prefix + ".idx"))
return {1, ".idx"};
return {0 /*unknown*/, ""};
}

8
src/Storages/MergeTree/MergeTreePartition.cpp
View File

@ -382,20 +382,20 @@ void MergeTreePartition::load(const MergeTreeData & storage, const PartMetadataM
partition_key_sample.getByPosition(i).type->getDefaultSerialization()->deserializeBinary(value[i], *file);
}
std::unique_ptr<WriteBufferFromFileBase> MergeTreePartition::store(const MergeTreeData & storage, const DataPartStorageBuilderPtr & data_part_storage_builder, MergeTreeDataPartChecksums & checksums) const
std::unique_ptr<WriteBufferFromFileBase> MergeTreePartition::store(const MergeTreeData & storage, IDataPartStorage & data_part_storage, MergeTreeDataPartChecksums & checksums) const
{
auto metadata_snapshot = storage.getInMemoryMetadataPtr();
const auto & context = storage.getContext();
const auto & partition_key_sample = adjustPartitionKey(metadata_snapshot, storage.getContext()).sample_block;
return store(partition_key_sample, data_part_storage_builder, checksums, context->getWriteSettings());
return store(partition_key_sample, data_part_storage, checksums, context->getWriteSettings());
}
std::unique_ptr<WriteBufferFromFileBase> MergeTreePartition::store(const Block & partition_key_sample, const DataPartStorageBuilderPtr & data_part_storage_builder, MergeTreeDataPartChecksums & checksums, const WriteSettings & settings) const
std::unique_ptr<WriteBufferFromFileBase> MergeTreePartition::store(const Block & partition_key_sample, IDataPartStorage & data_part_storage, MergeTreeDataPartChecksums & checksums, const WriteSettings & settings) const
{
if (!partition_key_sample)
return nullptr;
auto out = data_part_storage_builder->writeFile("partition.dat", DBMS_DEFAULT_BUFFER_SIZE, settings);
auto out = data_part_storage.writeFile("partition.dat", DBMS_DEFAULT_BUFFER_SIZE, settings);
HashingWriteBuffer out_hashing(*out);
for (size_t i = 0; i < value.size(); ++i)
{

8
src/Storages/MergeTree/MergeTreePartition.h
View File

@ -15,10 +15,10 @@ class MergeTreeData;
struct FormatSettings;
struct MergeTreeDataPartChecksums;
struct StorageInMemoryMetadata;
class IDataPartStorageBuilder;
class IDataPartStorage;
using StorageMetadataPtr = std::shared_ptr<const StorageInMemoryMetadata>;
using DataPartStorageBuilderPtr = std::shared_ptr<IDataPartStorageBuilder>;
using MutableDataPartStoragePtr = std::shared_ptr<IDataPartStorage>;
/// This class represents a partition value of a single part and encapsulates its loading/storing logic.
struct MergeTreePartition
@ -44,8 +44,8 @@ public:
/// Store functions return write buffer with written but not finalized data.
/// User must call finish() for returned object.
[[nodiscard]] std::unique_ptr<WriteBufferFromFileBase> store(const MergeTreeData & storage, const DataPartStorageBuilderPtr & data_part_storage_builder, MergeTreeDataPartChecksums & checksums) const;
[[nodiscard]] std::unique_ptr<WriteBufferFromFileBase> store(const Block & partition_key_sample, const DataPartStorageBuilderPtr & data_part_storage_builder, MergeTreeDataPartChecksums & checksums, const WriteSettings & settings) const;
[[nodiscard]] std::unique_ptr<WriteBufferFromFileBase> store(const MergeTreeData & storage, IDataPartStorage & data_part_storage, MergeTreeDataPartChecksums & checksums) const;
[[nodiscard]] std::unique_ptr<WriteBufferFromFileBase> store(const Block & partition_key_sample, IDataPartStorage & data_part_storage, MergeTreeDataPartChecksums & checksums, const WriteSettings & settings) const;
void assign(const MergeTreePartition & other) { value = other.value; }

50
src/Storages/MergeTree/MergeTreePartsMover.cpp
View File

@ -100,7 +100,6 @@ bool MergeTreePartsMover::selectPartsForMove(
return false;
std::unordered_map<DiskPtr, LargestPartsWithRequiredSize> need_to_move;
std::unordered_set<DiskPtr> need_to_move_disks;
const auto policy = data->getStoragePolicy();
const auto & volumes = policy->getVolumes();
@ -115,10 +114,7 @@ bool MergeTreePartsMover::selectPartsForMove(
UInt64 unreserved_space = disk->getUnreservedSpace();
if (unreserved_space < required_maximum_available_space && !disk->isBroken())
{
need_to_move.emplace(disk, required_maximum_available_space - unreserved_space);
need_to_move_disks.emplace(disk);
}
}
}
}
@ -140,8 +136,16 @@ bool MergeTreePartsMover::selectPartsForMove(
auto ttl_entry = selectTTLDescriptionForTTLInfos(metadata_snapshot->getMoveTTLs(), part->ttl_infos.moves_ttl, time_of_move, true);
auto to_insert = need_to_move.end();
if (auto disk_it = part->data_part_storage->isStoredOnDisk(need_to_move_disks); disk_it != need_to_move_disks.end())
to_insert = need_to_move.find(*disk_it);
auto part_disk_name = part->getDataPartStorage().getDiskName();
for (auto it = need_to_move.begin(); it != need_to_move.end(); ++it)
{
if (it->first->getName() == part_disk_name)
{
to_insert = it;
break;
}
}
ReservationPtr reservation;
if (ttl_entry)
@ -158,9 +162,8 @@ bool MergeTreePartsMover::selectPartsForMove(
/// In order to not over-move, we need to "release" required space on this disk,
/// possibly to zero.
if (to_insert != need_to_move.end())
{
to_insert->second.decreaseRequiredSizeAndRemoveRedundantParts(part->getBytesOnDisk());
}
++parts_to_move_by_ttl_rules;
parts_to_move_total_size_bytes += part->getBytesOnDisk();
}
@ -173,7 +176,7 @@ bool MergeTreePartsMover::selectPartsForMove(
for (auto && move : need_to_move)
{
auto min_volume_index = policy->getVolumeIndexByDisk(move.first) + 1;
auto min_volume_index = policy->getVolumeIndexByDiskName(move.first->getName()) + 1;
for (auto && part : move.second.getAccumulatedParts())
{
auto reservation = policy->reserve(part->getBytesOnDisk(), min_volume_index);
@ -199,7 +202,7 @@ bool MergeTreePartsMover::selectPartsForMove(
return false;
}
MergeTreeData::DataPartPtr MergeTreePartsMover::clonePart(const MergeTreeMoveEntry & moving_part) const
MergeTreeMutableDataPartPtr MergeTreePartsMover::clonePart(const MergeTreeMoveEntry & moving_part) const
{
if (moves_blocker.isCancelled())
throw Exception("Cancelled moving parts.", ErrorCodes::ABORTED);
@ -207,16 +210,15 @@ MergeTreeData::DataPartPtr MergeTreePartsMover::clonePart(const MergeTreeMoveEnt
auto settings = data->getSettings();
auto part = moving_part.part;
auto disk = moving_part.reserved_space->getDisk();
LOG_DEBUG(log, "Cloning part {} from '{}' to '{}'", part->name, part->data_part_storage->getDiskName(), disk->getName());
DataPartStoragePtr cloned_part_storage;
LOG_DEBUG(log, "Cloning part {} from '{}' to '{}'", part->name, part->getDataPartStorage().getDiskName(), disk->getName());
MutableDataPartStoragePtr cloned_part_storage;
if (disk->supportZeroCopyReplication() && settings->allow_remote_fs_zero_copy_replication)
{
/// Try zero-copy replication and fallback to default copy if it's not possible
moving_part.part->assertOnDisk();
String path_to_clone = fs::path(data->getRelativeDataPath()) / MergeTreeData::MOVING_DIR_NAME / "";
String relative_path = part->data_part_storage->getPartDirectory();
String relative_path = part->getDataPartStorage().getPartDirectory();
if (disk->exists(path_to_clone + relative_path))
{
LOG_WARNING(log, "Path {} already exists. Will remove it and clone again.", fullPath(disk, path_to_clone + relative_path));
@ -230,7 +232,7 @@ MergeTreeData::DataPartPtr MergeTreePartsMover::clonePart(const MergeTreeMoveEnt
if (!cloned_part_storage)
{
LOG_INFO(log, "Part {} was not fetched, we are the first who move it to another disk, so we will copy it", part->name);
cloned_part_storage = part->data_part_storage->clone(path_to_clone, part->data_part_storage->getPartDirectory(), disk, log);
cloned_part_storage = part->getDataPartStorage().clonePart(path_to_clone, part->getDataPartStorage().getPartDirectory(), disk, log);
}
}
else
@ -238,18 +240,17 @@ MergeTreeData::DataPartPtr MergeTreePartsMover::clonePart(const MergeTreeMoveEnt
cloned_part_storage = part->makeCloneOnDisk(disk, MergeTreeData::MOVING_DIR_NAME);
}
MergeTreeData::MutableDataPartPtr cloned_part = data->createPart(part->name, cloned_part_storage);
LOG_TRACE(log, "Part {} was cloned to {}", part->name, cloned_part->data_part_storage->getFullPath());
auto cloned_part = data->createPart(part->name, cloned_part_storage);
LOG_TRACE(log, "Part {} was cloned to {}", part->name, cloned_part->getDataPartStorage().getFullPath());
cloned_part->loadColumnsChecksumsIndexes(true, true);
cloned_part->loadVersionMetadata();
cloned_part->modification_time = cloned_part->data_part_storage->getLastModified().epochTime();
cloned_part->modification_time = cloned_part->getDataPartStorage().getLastModified().epochTime();
return cloned_part;
}
void MergeTreePartsMover::swapClonedPart(const MergeTreeData::DataPartPtr & cloned_part) const
void MergeTreePartsMover::swapClonedPart(const MergeTreeMutableDataPartPtr & cloned_part) const
{
if (moves_blocker.isCancelled())
throw Exception("Cancelled moving parts.", ErrorCodes::ABORTED);
@ -259,20 +260,17 @@ void MergeTreePartsMover::swapClonedPart(const MergeTreeData::DataPartPtr & clon
/// It's ok, because we don't block moving parts for merges or mutations
if (!active_part || active_part->name != cloned_part->name)
{
LOG_INFO(log, "Failed to swap {}. Active part doesn't exist. Possible it was merged or mutated. Will remove copy on path '{}'.", cloned_part->name, cloned_part->data_part_storage->getFullPath());
LOG_INFO(log, "Failed to swap {}. Active part doesn't exist. Possible it was merged or mutated. Will remove copy on path '{}'.", cloned_part->name, cloned_part->getDataPartStorage().getFullPath());
return;
}
auto builder = cloned_part->data_part_storage->getBuilder();
/// Don't remove new directory but throw an error because it may contain part which is currently in use.
cloned_part->renameTo(active_part->name, false, builder);
builder->commit();
cloned_part->renameTo(active_part->name, false);
/// TODO what happen if server goes down here?
data->swapActivePart(cloned_part);
LOG_TRACE(log, "Part {} was moved to {}", cloned_part->name, cloned_part->data_part_storage->getFullPath());
LOG_TRACE(log, "Part {} was moved to {}", cloned_part->name, cloned_part->getDataPartStorage().getFullPath());
}
}

4
src/Storages/MergeTree/MergeTreePartsMover.h
View File

@ -50,14 +50,14 @@ public:
const std::lock_guard<std::mutex> & moving_parts_lock);
/// Copies part to selected reservation in detached folder. Throws exception if part already exists.
MergeTreeDataPartPtr clonePart(const MergeTreeMoveEntry & moving_part) const;
MergeTreeMutableDataPartPtr clonePart(const MergeTreeMoveEntry & moving_part) const;
/// Replaces cloned part from detached directory into active data parts set.
/// Replacing part changes state to DeleteOnDestroy and will be removed from disk after destructor of
///IMergeTreeDataPart called. If replacing part doesn't exists or not active (committed) than
/// cloned part will be removed and log message will be reported. It may happen in case of concurrent
/// merge or mutation.
void swapClonedPart(const MergeTreeDataPartPtr & cloned_parts) const;
void swapClonedPart(const MergeTreeMutableDataPartPtr & cloned_parts) const;
/// Can stop background moves and moves from queries
ActionBlocker moves_blocker;

2
src/Storages/MergeTree/MergeTreeReadPool.cpp
View File

@ -263,7 +263,7 @@ void MergeTreeReadPool::fillPerThreadInfo(
{
PartInfo part_info{parts[i], per_part_sum_marks[i], i};
if (parts[i].data_part->isStoredOnDisk())
parts_per_disk[parts[i].data_part->data_part_storage->getDiskName()].push_back(std::move(part_info));
parts_per_disk[parts[i].data_part->getDataPartStorage().getDiskName()].push_back(std::move(part_info));
else
parts_per_disk[""].push_back(std::move(part_info));
}

10
src/Storages/MergeTree/MergeTreeReaderCompact.cpp
View File

@ -59,13 +59,15 @@ MergeTreeReaderCompact::MergeTreeReaderCompact(
throw Exception(ErrorCodes::CANNOT_READ_ALL_DATA, "Cannot read to empty buffer.");
const String path = MergeTreeDataPartCompact::DATA_FILE_NAME_WITH_EXTENSION;
auto data_part_storage = data_part_info_for_read->getDataPartStorage();
if (uncompressed_cache)
{
auto buffer = std::make_unique<CachedCompressedReadBuffer>(
std::string(fs::path(data_part_info_for_read->getDataPartStorage()->getFullPath()) / path),
[this, path]()
std::string(fs::path(data_part_storage->getFullPath()) / path),
[this, path, data_part_storage]()
{
return data_part_info_for_read->getDataPartStorage()->readFile(
return data_part_storage->readFile(
path,
settings.read_settings,
std::nullopt, std::nullopt);
@ -87,7 +89,7 @@ MergeTreeReaderCompact::MergeTreeReaderCompact(
{
auto buffer =
std::make_unique<CompressedReadBufferFromFile>(
data_part_info_for_read->getDataPartStorage()->readFile(
data_part_storage->readFile(
path,
settings.read_settings,
std::nullopt, std::nullopt),

4
src/Storages/MergeTree/MergeTreeSink.cpp
View File

@ -81,7 +81,7 @@ void MergeTreeSink::consume(Chunk chunk)
if (!temp_part.part)
continue;
if (!support_parallel_write && temp_part.part->data_part_storage->supportParallelWrite())
if (!support_parallel_write && temp_part.part->getDataPartStorage().supportParallelWrite())
support_parallel_write = true;
if (storage.getDeduplicationLog())
@ -160,7 +160,7 @@ void MergeTreeSink::finishDelayedChunk()
}
}
added = storage.renameTempPartAndAdd(part, transaction, partition.temp_part.builder, lock);
added = storage.renameTempPartAndAdd(part, transaction, lock);
transaction.commit(&lock);
}

6
src/Storages/MergeTree/MergeTreeWriteAheadLog.cpp
View File

@ -150,7 +150,6 @@ MergeTreeData::MutableDataPartsVector MergeTreeWriteAheadLog::restore(
while (!in->eof())
{
MergeTreeData::MutableDataPartPtr part;
DataPartStorageBuilderPtr data_part_storage_builder;
UInt8 version;
String part_name;
Block block;
@ -177,7 +176,6 @@ MergeTreeData::MutableDataPartsVector MergeTreeWriteAheadLog::restore(
{
auto single_disk_volume = std::make_shared<SingleDiskVolume>("volume_" + part_name, disk, 0);
auto data_part_storage = std::make_shared<DataPartStorageOnDisk>(single_disk_volume, storage.getRelativeDataPath(), part_name);
data_part_storage_builder = std::make_shared<DataPartStorageBuilderOnDisk>(single_disk_volume, storage.getRelativeDataPath(), part_name);
part = storage.createPart(
part_name,
@ -222,7 +220,6 @@ MergeTreeData::MutableDataPartsVector MergeTreeWriteAheadLog::restore(
{
MergedBlockOutputStream part_out(
part,
data_part_storage_builder,
metadata_snapshot,
block.getNamesAndTypesList(),
{},
@ -240,11 +237,12 @@ MergeTreeData::MutableDataPartsVector MergeTreeWriteAheadLog::restore(
for (const auto & projection : metadata_snapshot->getProjections())
{
auto projection_block = projection.calculate(block, context);
auto temp_part = MergeTreeDataWriter::writeInMemoryProjectionPart(storage, log, projection_block, projection, data_part_storage_builder, part.get());
auto temp_part = MergeTreeDataWriter::writeInMemoryProjectionPart(storage, log, projection_block, projection, part.get());
temp_part.finalize();
if (projection_block.rows())
part->addProjectionPart(projection.name, std::move(temp_part.part));
}
part_out.finalizePart(part, false);
min_block_number = std::min(min_block_number, part->info.min_block);

59
src/Storages/MergeTree/MergedBlockOutputStream.cpp
View File

@ -14,8 +14,7 @@ namespace ErrorCodes
MergedBlockOutputStream::MergedBlockOutputStream(
const MergeTreeDataPartPtr & data_part,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part,
const StorageMetadataPtr & metadata_snapshot_,
const NamesAndTypesList & columns_list_,
const MergeTreeIndices & skip_indices,
@ -24,7 +23,7 @@ MergedBlockOutputStream::MergedBlockOutputStream(
bool reset_columns_,
bool blocks_are_granules_size,
const WriteSettings & write_settings_)
: IMergedBlockOutputStream(std::move(data_part_storage_builder_), data_part, metadata_snapshot_, columns_list_, reset_columns_)
: IMergedBlockOutputStream(data_part, metadata_snapshot_, columns_list_, reset_columns_)
, columns_list(columns_list_)
, default_codec(default_codec_)
, write_settings(write_settings_)
@ -38,7 +37,7 @@ MergedBlockOutputStream::MergedBlockOutputStream(
blocks_are_granules_size);
if (data_part->isStoredOnDisk())
data_part_storage_builder->createDirectories();
data_part_storage->createDirectories();
/// We should write version metadata on part creation to distinguish it from parts that were created without transaction.
TransactionID tid = txn ? txn->tid : Tx::PrehistoricTID;
@ -47,7 +46,7 @@ MergedBlockOutputStream::MergedBlockOutputStream(
data_part->version.setCreationTID(tid, nullptr);
data_part->storeVersionMetadata();
writer = data_part->getWriter(data_part_storage_builder, columns_list, metadata_snapshot, skip_indices, default_codec, writer_settings, {});
writer = data_part->getWriter(columns_list, metadata_snapshot, skip_indices, default_codec, writer_settings, {});
}
/// If data is pre-sorted.
@ -68,17 +67,17 @@ struct MergedBlockOutputStream::Finalizer::Impl
{
IMergeTreeDataPartWriter & writer;
MergeTreeData::MutableDataPartPtr part;
DataPartStorageBuilderPtr data_part_storage_builder;
NameSet files_to_remove_after_finish;
std::vector<std::unique_ptr<WriteBufferFromFileBase>> written_files;
bool sync;
Impl(IMergeTreeDataPartWriter & writer_, MergeTreeData::MutableDataPartPtr part_, DataPartStorageBuilderPtr data_part_storage_builder_, const NameSet & files_to_remove_after_finish_, bool sync_)
Impl(IMergeTreeDataPartWriter & writer_, MergeTreeData::MutableDataPartPtr part_, const NameSet & files_to_remove_after_finish_, bool sync_)
: writer(writer_)
, part(std::move(part_))
, data_part_storage_builder(std::move(data_part_storage_builder_))
, files_to_remove_after_finish(files_to_remove_after_finish_)
, sync(sync_) {}
, sync(sync_)
{
}
void finish();
};
@ -95,7 +94,7 @@ void MergedBlockOutputStream::Finalizer::Impl::finish()
writer.finish(sync);
for (const auto & file_name : files_to_remove_after_finish)
data_part_storage_builder->removeFile(file_name);
part->getDataPartStorage().removeFile(file_name);
for (auto & file : written_files)
{
@ -122,19 +121,19 @@ MergedBlockOutputStream::Finalizer & MergedBlockOutputStream::Finalizer::operato
MergedBlockOutputStream::Finalizer::Finalizer(std::unique_ptr<Impl> impl_) : impl(std::move(impl_)) {}
void MergedBlockOutputStream::finalizePart(
MergeTreeData::MutableDataPartPtr & new_part,
bool sync,
const NamesAndTypesList * total_columns_list,
MergeTreeData::DataPart::Checksums * additional_column_checksums)
const MergeTreeMutableDataPartPtr & new_part,
bool sync,
const NamesAndTypesList * total_columns_list,
MergeTreeData::DataPart::Checksums * additional_column_checksums)
{
finalizePartAsync(new_part, sync, total_columns_list, additional_column_checksums).finish();
}
MergedBlockOutputStream::Finalizer MergedBlockOutputStream::finalizePartAsync(
MergeTreeData::MutableDataPartPtr & new_part,
bool sync,
const NamesAndTypesList * total_columns_list,
MergeTreeData::DataPart::Checksums * additional_column_checksums)
const MergeTreeMutableDataPartPtr & new_part,
bool sync,
const NamesAndTypesList * total_columns_list,
MergeTreeData::DataPart::Checksums * additional_column_checksums)
{
/// Finish write and get checksums.
MergeTreeData::DataPart::Checksums checksums;
@ -165,7 +164,7 @@ MergedBlockOutputStream::Finalizer MergedBlockOutputStream::finalizePartAsync(
new_part->setColumns(part_columns, serialization_infos);
}
auto finalizer = std::make_unique<Finalizer::Impl>(*writer, new_part, data_part_storage_builder, files_to_remove_after_sync, sync);
auto finalizer = std::make_unique<Finalizer::Impl>(*writer, new_part, files_to_remove_after_sync, sync);
if (new_part->isStoredOnDisk())
finalizer->written_files = finalizePartOnDisk(new_part, checksums);
@ -184,7 +183,7 @@ MergedBlockOutputStream::Finalizer MergedBlockOutputStream::finalizePartAsync(
}
MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDisk(
const MergeTreeData::DataPartPtr & new_part,
const MergeTreeMutableDataPartPtr & new_part,
MergeTreeData::DataPart::Checksums & checksums)
{
WrittenFiles written_files;
@ -192,7 +191,7 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
{
if (storage.format_version >= MERGE_TREE_DATA_MIN_FORMAT_VERSION_WITH_CUSTOM_PARTITIONING || isCompactPart(new_part))
{
auto count_out = data_part_storage_builder->writeFile("count.txt", 4096, write_settings);
auto count_out = new_part->getDataPartStorage().writeFile("count.txt", 4096, write_settings);
HashingWriteBuffer count_out_hashing(*count_out);
writeIntText(rows_count, count_out_hashing);
count_out_hashing.next();
@ -206,7 +205,7 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
{
if (new_part->uuid != UUIDHelpers::Nil)
{
auto out = data_part_storage_builder->writeFile(IMergeTreeDataPart::UUID_FILE_NAME, 4096, write_settings);
auto out = new_part->getDataPartStorage().writeFile(IMergeTreeDataPart::UUID_FILE_NAME, 4096, write_settings);
HashingWriteBuffer out_hashing(*out);
writeUUIDText(new_part->uuid, out_hashing);
checksums.files[IMergeTreeDataPart::UUID_FILE_NAME].file_size = out_hashing.count();
@ -217,12 +216,12 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
if (storage.format_version >= MERGE_TREE_DATA_MIN_FORMAT_VERSION_WITH_CUSTOM_PARTITIONING)
{
if (auto file = new_part->partition.store(storage, data_part_storage_builder, checksums))
if (auto file = new_part->partition.store(storage, new_part->getDataPartStorage(), checksums))
written_files.emplace_back(std::move(file));
if (new_part->minmax_idx->initialized)
{
auto files = new_part->minmax_idx->store(storage, data_part_storage_builder, checksums);
auto files = new_part->minmax_idx->store(storage, new_part->getDataPartStorage(), checksums);
for (auto & file : files)
written_files.emplace_back(std::move(file));
}
@ -232,7 +231,7 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
}
{
auto count_out = data_part_storage_builder->writeFile("count.txt", 4096, write_settings);
auto count_out = new_part->getDataPartStorage().writeFile("count.txt", 4096, write_settings);
HashingWriteBuffer count_out_hashing(*count_out);
writeIntText(rows_count, count_out_hashing);
count_out_hashing.next();
@ -246,7 +245,7 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
if (!new_part->ttl_infos.empty())
{
/// Write a file with ttl infos in json format.
auto out = data_part_storage_builder->writeFile("ttl.txt", 4096, write_settings);
auto out = new_part->getDataPartStorage().writeFile("ttl.txt", 4096, write_settings);
HashingWriteBuffer out_hashing(*out);
new_part->ttl_infos.write(out_hashing);
checksums.files["ttl.txt"].file_size = out_hashing.count();
@ -257,7 +256,7 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
if (!new_part->getSerializationInfos().empty())
{
auto out = data_part_storage_builder->writeFile(IMergeTreeDataPart::SERIALIZATION_FILE_NAME, 4096, write_settings);
auto out = new_part->getDataPartStorage().writeFile(IMergeTreeDataPart::SERIALIZATION_FILE_NAME, 4096, write_settings);
HashingWriteBuffer out_hashing(*out);
new_part->getSerializationInfos().writeJSON(out_hashing);
checksums.files[IMergeTreeDataPart::SERIALIZATION_FILE_NAME].file_size = out_hashing.count();
@ -268,7 +267,7 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
{
/// Write a file with a description of columns.
auto out = data_part_storage_builder->writeFile("columns.txt", 4096, write_settings);
auto out = new_part->getDataPartStorage().writeFile("columns.txt", 4096, write_settings);
new_part->getColumns().writeText(*out);
out->preFinalize();
written_files.emplace_back(std::move(out));
@ -276,7 +275,7 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
if (default_codec != nullptr)
{
auto out = data_part_storage_builder->writeFile(IMergeTreeDataPart::DEFAULT_COMPRESSION_CODEC_FILE_NAME, 4096, write_settings);
auto out = new_part->getDataPartStorage().writeFile(IMergeTreeDataPart::DEFAULT_COMPRESSION_CODEC_FILE_NAME, 4096, write_settings);
DB::writeText(queryToString(default_codec->getFullCodecDesc()), *out);
out->preFinalize();
written_files.emplace_back(std::move(out));
@ -289,7 +288,7 @@ MergedBlockOutputStream::WrittenFiles MergedBlockOutputStream::finalizePartOnDis
{
/// Write file with checksums.
auto out = data_part_storage_builder->writeFile("checksums.txt", 4096, write_settings);
auto out = new_part->getDataPartStorage().writeFile("checksums.txt", 4096, write_settings);
checksums.write(*out);
out->preFinalize();
written_files.emplace_back(std::move(out));

23
src/Storages/MergeTree/MergedBlockOutputStream.h
View File

@ -15,8 +15,7 @@ class MergedBlockOutputStream final : public IMergedBlockOutputStream
{
public:
MergedBlockOutputStream(
const MergeTreeDataPartPtr & data_part,
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeMutableDataPartPtr & data_part,
const StorageMetadataPtr & metadata_snapshot_,
const NamesAndTypesList & columns_list_,
const MergeTreeIndices & skip_indices,
@ -55,16 +54,16 @@ public:
/// Finalize writing part and fill inner structures
/// If part is new and contains projections, they should be added before invoking this method.
Finalizer finalizePartAsync(
MergeTreeData::MutableDataPartPtr & new_part,
bool sync,
const NamesAndTypesList * total_columns_list = nullptr,
MergeTreeData::DataPart::Checksums * additional_column_checksums = nullptr);
const MergeTreeMutableDataPartPtr & new_part,
bool sync,
const NamesAndTypesList * total_columns_list = nullptr,
MergeTreeData::DataPart::Checksums * additional_column_checksums = nullptr);
void finalizePart(
MergeTreeData::MutableDataPartPtr & new_part,
bool sync,
const NamesAndTypesList * total_columns_list = nullptr,
MergeTreeData::DataPart::Checksums * additional_column_checksums = nullptr);
const MergeTreeMutableDataPartPtr & new_part,
bool sync,
const NamesAndTypesList * total_columns_list = nullptr,
MergeTreeData::DataPart::Checksums * additional_column_checksums = nullptr);
private:
/** If `permutation` is given, it rearranges the values in the columns when writing.
@ -74,8 +73,8 @@ private:
using WrittenFiles = std::vector<std::unique_ptr<WriteBufferFromFileBase>>;
WrittenFiles finalizePartOnDisk(
const MergeTreeData::DataPartPtr & new_part,
MergeTreeData::DataPart::Checksums & checksums);
const MergeTreeMutableDataPartPtr & new_part,
MergeTreeData::DataPart::Checksums & checksums);
NamesAndTypesList columns_list;
IMergeTreeDataPart::MinMaxIndex minmax_idx;

8
src/Storages/MergeTree/MergedColumnOnlyOutputStream.cpp
View File

@ -11,8 +11,7 @@ namespace ErrorCodes
}
MergedColumnOnlyOutputStream::MergedColumnOnlyOutputStream(
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeDataPartPtr & data_part,
const MergeTreeMutableDataPartPtr & data_part,
const StorageMetadataPtr & metadata_snapshot_,
const Block & header_,
CompressionCodecPtr default_codec,
@ -20,7 +19,7 @@ MergedColumnOnlyOutputStream::MergedColumnOnlyOutputStream(
WrittenOffsetColumns * offset_columns_,
const MergeTreeIndexGranularity & index_granularity,
const MergeTreeIndexGranularityInfo * index_granularity_info)
: IMergedBlockOutputStream(std::move(data_part_storage_builder_), data_part, metadata_snapshot_, header_.getNamesAndTypesList(), /*reset_columns=*/ true)
: IMergedBlockOutputStream(data_part, metadata_snapshot_, header_.getNamesAndTypesList(), /*reset_columns=*/ true)
, header(header_)
{
const auto & global_settings = data_part->storage.getContext()->getSettings();
@ -34,7 +33,6 @@ MergedColumnOnlyOutputStream::MergedColumnOnlyOutputStream(
/* rewrite_primary_key = */ false);
writer = data_part->getWriter(
data_part_storage_builder,
header.getNamesAndTypesList(),
metadata_snapshot_,
indices_to_recalc,
@ -81,7 +79,7 @@ MergedColumnOnlyOutputStream::fillChecksums(
for (const String & removed_file : removed_files)
{
data_part_storage_builder->removeFileIfExists(removed_file);
new_part->getDataPartStorage().removeFileIfExists(removed_file);
if (all_checksums.files.contains(removed_file))
all_checksums.files.erase(removed_file);

3
src/Storages/MergeTree/MergedColumnOnlyOutputStream.h
View File

@ -14,8 +14,7 @@ public:
/// Pass empty 'already_written_offset_columns' first time then and pass the same object to subsequent instances of MergedColumnOnlyOutputStream
/// if you want to serialize elements of Nested data structure in different instances of MergedColumnOnlyOutputStream.
MergedColumnOnlyOutputStream(
DataPartStorageBuilderPtr data_part_storage_builder_,
const MergeTreeDataPartPtr & data_part,
const MergeTreeMutableDataPartPtr & data_part,
const StorageMetadataPtr & metadata_snapshot_,
const Block & header_,
CompressionCodecPtr default_codec_,

9
src/Storages/MergeTree/MutateFromLogEntryTask.cpp
View File

@ -92,7 +92,7 @@ ReplicatedMergeMutateTaskBase::PrepareResult MutateFromLogEntryTask::prepare()
/// Once we mutate part, we must reserve space on the same disk, because mutations can possibly create hardlinks.
/// Can throw an exception.
reserved_space = storage.reserveSpace(estimated_space_for_result, source_part->data_part_storage);
reserved_space = storage.reserveSpace(estimated_space_for_result, source_part->getDataPartStorage());
table_lock_holder = storage.lockForShare(
RWLockImpl::NO_QUERY, storage_settings_ptr->lock_acquire_timeout_for_background_operations);
@ -193,12 +193,7 @@ ReplicatedMergeMutateTaskBase::PrepareResult MutateFromLogEntryTask::prepare()
bool MutateFromLogEntryTask::finalize(ReplicatedMergeMutateTaskBase::PartLogWriter write_part_log)
{
new_part = mutate_task->getFuture().get();
auto builder = mutate_task->getBuilder();
if (!builder)
builder = new_part->data_part_storage->getBuilder();
storage.renameTempPartAndReplace(new_part, *transaction_ptr, builder);
storage.renameTempPartAndReplace(new_part, *transaction_ptr);
try
{

7
src/Storages/MergeTree/MutatePlainMergeTreeTask.cpp
View File

@ -83,14 +83,9 @@ bool MutatePlainMergeTreeTask::executeStep()
new_part = mutate_task->getFuture().get();
auto builder = mutate_task->getBuilder();
if (!builder)
builder = new_part->data_part_storage->getBuilder();
MergeTreeData::Transaction transaction(storage, merge_mutate_entry->txn.get());
/// FIXME Transactions: it's too optimistic, better to lock parts before starting transaction
storage.renameTempPartAndReplace(new_part, transaction, builder);
storage.renameTempPartAndReplace(new_part, transaction);
transaction.commit();
storage.updateMutationEntriesErrors(future_part, true, "");

62
src/Storages/MergeTree/MutateTask.cpp
View File

@ -619,7 +619,6 @@ static NameToNameVector collectFilesForRenames(
/// Initialize and write to disk new part fields like checksums, columns, etc.
void finalizeMutatedPart(
const MergeTreeDataPartPtr & source_part,
const DataPartStorageBuilderPtr & data_part_storage_builder,
MergeTreeData::MutableDataPartPtr new_data_part,
ExecuteTTLType execute_ttl_type,
const CompressionCodecPtr & codec,
@ -627,7 +626,7 @@ void finalizeMutatedPart(
{
if (new_data_part->uuid != UUIDHelpers::Nil)
{
auto out = data_part_storage_builder->writeFile(IMergeTreeDataPart::UUID_FILE_NAME, 4096, context->getWriteSettings());
auto out = new_data_part->getDataPartStorage().writeFile(IMergeTreeDataPart::UUID_FILE_NAME, 4096, context->getWriteSettings());
HashingWriteBuffer out_hashing(*out);
writeUUIDText(new_data_part->uuid, out_hashing);
new_data_part->checksums.files[IMergeTreeDataPart::UUID_FILE_NAME].file_size = out_hashing.count();
@ -637,7 +636,7 @@ void finalizeMutatedPart(
if (execute_ttl_type != ExecuteTTLType::NONE)
{
/// Write a file with ttl infos in json format.
auto out_ttl = data_part_storage_builder->writeFile("ttl.txt", 4096, context->getWriteSettings());
auto out_ttl = new_data_part->getDataPartStorage().writeFile("ttl.txt", 4096, context->getWriteSettings());
HashingWriteBuffer out_hashing(*out_ttl);
new_data_part->ttl_infos.write(out_hashing);
new_data_part->checksums.files["ttl.txt"].file_size = out_hashing.count();
@ -646,7 +645,7 @@ void finalizeMutatedPart(
if (!new_data_part->getSerializationInfos().empty())
{
auto out = data_part_storage_builder->writeFile(IMergeTreeDataPart::SERIALIZATION_FILE_NAME, 4096, context->getWriteSettings());
auto out = new_data_part->getDataPartStorage().writeFile(IMergeTreeDataPart::SERIALIZATION_FILE_NAME, 4096, context->getWriteSettings());
HashingWriteBuffer out_hashing(*out);
new_data_part->getSerializationInfos().writeJSON(out_hashing);
new_data_part->checksums.files[IMergeTreeDataPart::SERIALIZATION_FILE_NAME].file_size = out_hashing.count();
@ -655,18 +654,18 @@ void finalizeMutatedPart(
{
/// Write file with checksums.
auto out_checksums = data_part_storage_builder->writeFile("checksums.txt", 4096, context->getWriteSettings());
auto out_checksums = new_data_part->getDataPartStorage().writeFile("checksums.txt", 4096, context->getWriteSettings());
new_data_part->checksums.write(*out_checksums);
} /// close fd
{
auto out = data_part_storage_builder->writeFile(IMergeTreeDataPart::DEFAULT_COMPRESSION_CODEC_FILE_NAME, 4096, context->getWriteSettings());
auto out = new_data_part->getDataPartStorage().writeFile(IMergeTreeDataPart::DEFAULT_COMPRESSION_CODEC_FILE_NAME, 4096, context->getWriteSettings());
DB::writeText(queryToString(codec->getFullCodecDesc()), *out);
} /// close fd
{
/// Write a file with a description of columns.
auto out_columns = data_part_storage_builder->writeFile("columns.txt", 4096, context->getWriteSettings());
auto out_columns = new_data_part->getDataPartStorage().writeFile("columns.txt", 4096, context->getWriteSettings());
new_data_part->getColumns().writeText(*out_columns);
} /// close fd
@ -734,8 +733,6 @@ struct MutationContext
= MutationsInterpreter::MutationKind::MutationKindEnum::MUTATE_UNKNOWN;
MergeTreeData::MutableDataPartPtr new_data_part;
DataPartStorageBuilderPtr data_part_storage_builder;
IMergedBlockOutputStreamPtr out{nullptr};
String mrk_extension;
@ -816,11 +813,9 @@ public:
if (next_level_parts.empty())
{
LOG_DEBUG(log, "Merged a projection part in level {}", current_level);
auto builder = selected_parts[0]->data_part_storage->getBuilder();
selected_parts[0]->renameTo(projection.name + ".proj", true, builder);
selected_parts[0]->renameTo(projection.name + ".proj", true);
selected_parts[0]->name = projection.name;
selected_parts[0]->is_temp = false;
builder->commit();
ctx->new_data_part->addProjectionPart(name, std::move(selected_parts[0]));
/// Task is finished
@ -865,7 +860,6 @@ public:
projection_merging_params,
NO_TRANSACTION_PTR,
ctx->new_data_part.get(),
ctx->data_part_storage_builder.get(),
".tmp_proj");
next_level_parts.push_back(executeHere(tmp_part_merge_task));
@ -1025,8 +1019,7 @@ bool PartMergerWriter::mutateOriginalPartAndPrepareProjections()
if (projection_block)
{
auto tmp_part = MergeTreeDataWriter::writeTempProjectionPart(
*ctx->data, ctx->log, projection_block, projection, ctx->data_part_storage_builder, ctx->new_data_part.get(), ++block_num);
tmp_part.builder->commit();
*ctx->data, ctx->log, projection_block, projection, ctx->new_data_part.get(), ++block_num);
tmp_part.finalize();
projection_parts[projection.name].emplace_back(std::move(tmp_part.part));
}
@ -1048,8 +1041,7 @@ bool PartMergerWriter::mutateOriginalPartAndPrepareProjections()
if (projection_block)
{
auto temp_part = MergeTreeDataWriter::writeTempProjectionPart(
*ctx->data, ctx->log, projection_block, projection, ctx->data_part_storage_builder, ctx->new_data_part.get(), ++block_num);
temp_part.builder->commit();
*ctx->data, ctx->log, projection_block, projection, ctx->new_data_part.get(), ++block_num);
temp_part.finalize();
projection_parts[projection.name].emplace_back(std::move(temp_part.part));
}
@ -1149,7 +1141,7 @@ private:
void prepare()
{
ctx->data_part_storage_builder->createDirectories();
ctx->new_data_part->getDataPartStorage().createDirectories();
/// Note: this is done before creating input streams, because otherwise data.data_parts_mutex
/// (which is locked in data.getTotalActiveSizeInBytes())
@ -1184,7 +1176,6 @@ private:
ctx->out = std::make_shared<MergedBlockOutputStream>(
ctx->new_data_part,
ctx->data_part_storage_builder,
ctx->metadata_snapshot,
ctx->new_data_part->getColumns(),
skip_part_indices,
@ -1280,7 +1271,7 @@ private:
if (ctx->execute_ttl_type != ExecuteTTLType::NONE)
ctx->files_to_skip.insert("ttl.txt");
ctx->data_part_storage_builder->createDirectories();
ctx->new_data_part->getDataPartStorage().createDirectories();
/// We should write version metadata on part creation to distinguish it from parts that were created without transaction.
TransactionID tid = ctx->txn ? ctx->txn->tid : Tx::PrehistoricTID;
@ -1291,7 +1282,7 @@ private:
NameSet hardlinked_files;
/// Create hardlinks for unchanged files
for (auto it = ctx->source_part->data_part_storage->iterate(); it->isValid(); it->next())
for (auto it = ctx->source_part->getDataPartStorage().iterate(); it->isValid(); it->next())
{
if (ctx->files_to_skip.contains(it->name()))
continue;
@ -1317,22 +1308,22 @@ private:
if (it->isFile())
{
ctx->data_part_storage_builder->createHardLinkFrom(
*ctx->source_part->data_part_storage, it->name(), destination);
ctx->new_data_part->getDataPartStorage().createHardLinkFrom(
ctx->source_part->getDataPartStorage(), it->name(), destination);
hardlinked_files.insert(it->name());
}
else if (!endsWith(it->name(), ".tmp_proj")) // ignore projection tmp merge dir
{
// it's a projection part directory
ctx->data_part_storage_builder->createProjection(destination);
ctx->new_data_part->getDataPartStorage().createProjection(destination);
auto projection_data_part_storage = ctx->source_part->data_part_storage->getProjection(destination);
auto projection_data_part_storage_builder = ctx->data_part_storage_builder->getProjection(destination);
auto projection_data_part_storage_src = ctx->source_part->getDataPartStorage().getProjection(destination);
auto projection_data_part_storage_dst = ctx->new_data_part->getDataPartStorage().getProjection(destination);
for (auto p_it = projection_data_part_storage->iterate(); p_it->isValid(); p_it->next())
for (auto p_it = projection_data_part_storage_src->iterate(); p_it->isValid(); p_it->next())
{
projection_data_part_storage_builder->createHardLinkFrom(
*projection_data_part_storage, p_it->name(), p_it->name());
projection_data_part_storage_dst->createHardLinkFrom(
*projection_data_part_storage_src, p_it->name(), p_it->name());
hardlinked_files.insert(p_it->name());
}
}
@ -1362,7 +1353,6 @@ private:
builder.addTransform(std::make_shared<TTLCalcTransform>(builder.getHeader(), *ctx->data, ctx->metadata_snapshot, ctx->new_data_part, ctx->time_of_mutation, true));
ctx->out = std::make_shared<MergedColumnOnlyOutputStream>(
ctx->data_part_storage_builder,
ctx->new_data_part,
ctx->metadata_snapshot,
ctx->updated_header,
@ -1414,7 +1404,7 @@ private:
}
}
MutationHelpers::finalizeMutatedPart(ctx->source_part, ctx->data_part_storage_builder, ctx->new_data_part, ctx->execute_ttl_type, ctx->compression_codec, ctx->context);
MutationHelpers::finalizeMutatedPart(ctx->source_part, ctx->new_data_part, ctx->execute_ttl_type, ctx->compression_codec, ctx->context);
}
@ -1584,10 +1574,7 @@ bool MutateTask::prepare()
ctx->data->getRelativeDataPath(),
tmp_part_dir_name);
ctx->data_part_storage_builder = std::make_shared<DataPartStorageBuilderOnDisk>(
single_disk_volume,
ctx->data->getRelativeDataPath(),
tmp_part_dir_name);
data_part_storage->beginTransaction();
ctx->new_data_part = ctx->data->createPart(
ctx->future_part->name, ctx->future_part->type, ctx->future_part->part_info, data_part_storage);
@ -1690,9 +1677,4 @@ const MergeTreeData::HardlinkedFiles & MutateTask::getHardlinkedFiles() const
return ctx->hardlinked_files;
}
DataPartStorageBuilderPtr MutateTask::getBuilder() const
{
return ctx->data_part_storage_builder;
}
}

2
src/Storages/MergeTree/MutateTask.h
View File

@ -46,7 +46,7 @@ public:
const MergeTreeData::HardlinkedFiles & getHardlinkedFiles() const;
DataPartStorageBuilderPtr getBuilder() const;
MutableDataPartStoragePtr getBuilder() const;
private:

16
src/Storages/MergeTree/PartMetadataManagerOrdinary.cpp
View File

@ -8,20 +8,10 @@
namespace DB
{
static std::unique_ptr<ReadBufferFromFileBase> openForReading(const DataPartStoragePtr & data_part_storage, const String & path)
{
size_t file_size = data_part_storage->getFileSize(path);
return data_part_storage->readFile(path, ReadSettings().adjustBufferSize(file_size), file_size, std::nullopt);
}
PartMetadataManagerOrdinary::PartMetadataManagerOrdinary(const IMergeTreeDataPart * part_) : IPartMetadataManager(part_)
{
}
std::unique_ptr<ReadBuffer> PartMetadataManagerOrdinary::read(const String & file_name) const
{
auto res = openForReading(part->data_part_storage, file_name);
size_t file_size = part->getDataPartStorage().getFileSize(file_name);
auto res = part->getDataPartStorage().readFile(file_name, ReadSettings().adjustBufferSize(file_size), file_size, std::nullopt);
if (isCompressedFromFileName(file_name))
return std::make_unique<CompressedReadBufferFromFile>(std::move(res));
@ -31,7 +21,7 @@ std::unique_ptr<ReadBuffer> PartMetadataManagerOrdinary::read(const String & fil
bool PartMetadataManagerOrdinary::exists(const String & file_name) const
{
return part->data_part_storage->exists(file_name);
return part->getDataPartStorage().exists(file_name);
}

2
src/Storages/MergeTree/PartMetadataManagerOrdinary.h
View File

@ -8,7 +8,7 @@ namespace DB
class PartMetadataManagerOrdinary : public IPartMetadataManager
{
public:
explicit PartMetadataManagerOrdinary(const IMergeTreeDataPart * part_);
explicit PartMetadataManagerOrdinary(const IMergeTreeDataPart * part_) : IPartMetadataManager(part_) {}
~PartMetadataManagerOrdinary() override = default;

28
src/Storages/MergeTree/PartMetadataManagerWithCache.cpp
View File

@ -31,24 +31,24 @@ PartMetadataManagerWithCache::PartMetadataManagerWithCache(const IMergeTreeDataP
String PartMetadataManagerWithCache::getKeyFromFilePath(const String & file_path) const
{
return part->data_part_storage->getDiskName() + ":" + file_path;
return part->getDataPartStorage().getDiskName() + ":" + file_path;
}
String PartMetadataManagerWithCache::getFilePathFromKey(const String & key) const
{
return key.substr(part->data_part_storage->getDiskName().size() + 1);
return key.substr(part->getDataPartStorage().getDiskName().size() + 1);
}
std::unique_ptr<ReadBuffer> PartMetadataManagerWithCache::read(const String & file_name) const
{
String file_path = fs::path(part->data_part_storage->getRelativePath()) / file_name;
String file_path = fs::path(part->getDataPartStorage().getRelativePath()) / file_name;
String key = getKeyFromFilePath(file_path);
String value;
auto status = cache->get(key, value);
if (!status.ok())
{
ProfileEvents::increment(ProfileEvents::MergeTreeMetadataCacheMiss);
auto in = part->data_part_storage->readFile(file_name, {}, std::nullopt, std::nullopt);
auto in = part->getDataPartStorage().readFile(file_name, {}, std::nullopt, std::nullopt);
std::unique_ptr<ReadBuffer> reader;
if (!isCompressedFromFileName(file_name))
reader = std::move(in);
@ -67,7 +67,7 @@ std::unique_ptr<ReadBuffer> PartMetadataManagerWithCache::read(const String & fi
bool PartMetadataManagerWithCache::exists(const String & file_name) const
{
String file_path = fs::path(part->data_part_storage->getRelativePath()) / file_name;
String file_path = fs::path(part->getDataPartStorage().getRelativePath()) / file_name;
String key = getKeyFromFilePath(file_path);
String value;
auto status = cache->get(key, value);
@ -79,7 +79,7 @@ bool PartMetadataManagerWithCache::exists(const String & file_name) const
else
{
ProfileEvents::increment(ProfileEvents::MergeTreeMetadataCacheMiss);
return part->data_part_storage->exists(file_name);
return part->getDataPartStorage().exists(file_name);
}
}
@ -91,7 +91,7 @@ void PartMetadataManagerWithCache::deleteAll(bool include_projection)
String value;
for (const auto & file_name : file_names)
{
String file_path = fs::path(part->data_part_storage->getRelativePath()) / file_name;
String file_path = fs::path(part->getDataPartStorage().getRelativePath()) / file_name;
String key = getKeyFromFilePath(file_path);
auto status = cache->del(key);
if (!status.ok())
@ -119,10 +119,10 @@ void PartMetadataManagerWithCache::updateAll(bool include_projection)
String read_value;
for (const auto & file_name : file_names)
{
String file_path = fs::path(part->data_part_storage->getRelativePath()) / file_name;
if (!part->data_part_storage->exists(file_name))
String file_path = fs::path(part->getDataPartStorage().getRelativePath()) / file_name;
if (!part->getDataPartStorage().exists(file_name))
continue;
auto in = part->data_part_storage->readFile(file_name, {}, std::nullopt, std::nullopt);
auto in = part->getDataPartStorage().readFile(file_name, {}, std::nullopt, std::nullopt);
readStringUntilEOF(value, *in);
String key = getKeyFromFilePath(file_path);
@ -159,7 +159,7 @@ void PartMetadataManagerWithCache::assertAllDeleted(bool include_projection) con
file_name = fs::path(file_path).filename();
/// Metadata file belongs to current part
if (fs::path(part->data_part_storage->getRelativePath()) / file_name == file_path)
if (fs::path(part->getDataPartStorage().getRelativePath()) / file_name == file_path)
throw Exception(
ErrorCodes::LOGICAL_ERROR,
"Data part {} with type {} with meta file {} still in cache",
@ -173,7 +173,7 @@ void PartMetadataManagerWithCache::assertAllDeleted(bool include_projection) con
const auto & projection_parts = part->getProjectionParts();
for (const auto & [projection_name, projection_part] : projection_parts)
{
if (fs::path(part->data_part_storage->getRelativePath()) / (projection_name + ".proj") / file_name == file_path)
if (fs::path(part->getDataPartStorage().getRelativePath()) / (projection_name + ".proj") / file_name == file_path)
{
throw Exception(
ErrorCodes::LOGICAL_ERROR,
@ -190,7 +190,7 @@ void PartMetadataManagerWithCache::assertAllDeleted(bool include_projection) con
void PartMetadataManagerWithCache::getKeysAndCheckSums(Strings & keys, std::vector<uint128> & checksums) const
{
String prefix = getKeyFromFilePath(fs::path(part->data_part_storage->getRelativePath()) / "");
String prefix = getKeyFromFilePath(fs::path(part->getDataPartStorage().getRelativePath()) / "");
Strings values;
cache->getByPrefix(prefix, keys, values);
size_t size = keys.size();
@ -225,7 +225,7 @@ std::unordered_map<String, IPartMetadataManager::uint128> PartMetadataManagerWit
results.emplace(file_name, cache_checksums[i]);
/// File belongs to normal part
if (fs::path(part->data_part_storage->getRelativePath()) / file_name == file_path)
if (fs::path(part->getDataPartStorage().getRelativePath()) / file_name == file_path)
{
auto disk_checksum = part->getActualChecksumByFile(file_name);
if (disk_checksum != cache_checksums[i])

12
src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp
View File

@ -268,7 +268,7 @@ void ReplicatedMergeTreeSink::finishDelayedChunk(zkutil::ZooKeeperPtr & zookeepe
try
{
commitPart(zookeeper, part, partition.block_id, partition.temp_part.builder, delayed_chunk->replicas_num);
commitPart(zookeeper, part, partition.block_id, delayed_chunk->replicas_num);
last_block_is_duplicate = last_block_is_duplicate || part->is_duplicate;
@ -301,7 +301,7 @@ void ReplicatedMergeTreeSink::writeExistingPart(MergeTreeData::MutableDataPartPt
try
{
part->version.setCreationTID(Tx::PrehistoricTID, nullptr);
commitPart(zookeeper, part, "", part->data_part_storage->getBuilder(), replicas_num);
commitPart(zookeeper, part, "", replicas_num);
PartLog::addNewPart(storage.getContext(), part, watch.elapsed());
}
catch (...)
@ -315,7 +315,6 @@ void ReplicatedMergeTreeSink::commitPart(
zkutil::ZooKeeperPtr & zookeeper,
MergeTreeData::MutableDataPartPtr & part,
const String & block_id,
DataPartStorageBuilderPtr builder,
size_t replicas_num)
{
/// It is possible that we alter a part with different types of source columns.
@ -326,7 +325,7 @@ void ReplicatedMergeTreeSink::commitPart(
assertSessionIsNotExpired(zookeeper);
String temporary_part_relative_path = part->data_part_storage->getPartDirectory();
String temporary_part_relative_path = part->getDataPartStorage().getPartDirectory();
/// There is one case when we need to retry transaction in a loop.
/// But don't do it too many times - just as defensive measure.
@ -499,7 +498,7 @@ void ReplicatedMergeTreeSink::commitPart(
try
{
auto lock = storage.lockParts();
renamed = storage.renameTempPartAndAdd(part, transaction, builder, lock);
renamed = storage.renameTempPartAndAdd(part, transaction, lock);
}
catch (const Exception & e)
{
@ -563,8 +562,7 @@ void ReplicatedMergeTreeSink::commitPart(
transaction.rollbackPartsToTemporaryState();
part->is_temp = true;
part->renameTo(temporary_part_relative_path, false, builder);
builder->commit();
part->renameTo(temporary_part_relative_path, false);
/// If this part appeared on other replica than it's better to try to write it locally one more time. If it's our part
/// than it will be ignored on the next itration.

1
src/Storages/MergeTree/ReplicatedMergeTreeSink.h
View File

@ -79,7 +79,6 @@ private:
zkutil::ZooKeeperPtr & zookeeper,
MergeTreeData::MutableDataPartPtr & part,
const String & block_id,
DataPartStorageBuilderPtr part_builder,
size_t replicas_num);
/// Wait for quorum to be satisfied on path (quorum_path) form part (part_name)

Some files were not shown because too many files have changed in this diff Show More