thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-23 08:02:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge master

Browse Source
This commit is contained in:
kssenii 2022-03-11 12:24:03 +01:00
commit d70d7aac41
17 changed files with 148 additions and 194 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
.github/codecov.yml vendored
View File

@ -1,17 +0,0 @@
codecov:
max_report_age: "off"
strict_yaml_branch: "master"
ignore:
- "contrib"
- "docs"
- "benchmark"
- "tests"
- "docker"
- "debian"
- "cmake"
comment: false
github_checks:
annotations: false

43
.github/workflows/anchore-analysis.yml vendored
View File

@ -1,43 +0,0 @@
# This workflow checks out code, performs an Anchore container image
# vulnerability and compliance scan, and integrates the results with
# GitHub Advanced Security code scanning feature. For more information on
# the Anchore scan action usage and parameters, see
# https://github.com/anchore/scan-action. For more information on
# Anchore container image scanning in general, see
# https://docs.anchore.com.
name: Docker Container Scan (clickhouse-server)
env:
# Force the stdout and stderr streams to be unbuffered
PYTHONUNBUFFERED: 1
"on":
pull_request:
paths:
- docker/server/Dockerfile
- .github/workflows/anchore-analysis.yml
schedule:
- cron: '0 21 * * *'
jobs:
Anchore-Build-Scan:
runs-on: ubuntu-latest
steps:
- name: Checkout the code
uses: actions/checkout@v2
- name: Build the Docker image
run: |
cd docker/server
perl -pi -e 's|=\$version||g' Dockerfile
docker build . --file Dockerfile --tag localbuild/testimage:latest
- name: Run the local Anchore scan action itself with GitHub Advanced Security code scanning integration enabled
uses: anchore/scan-action@v2
id: scan
with:
image: "localbuild/testimage:latest"
acs-report-enable: true
- name: Upload Anchore Scan Report
uses: github/codeql-action/upload-sarif@v1
with:
sarif_file: ${{ steps.scan.outputs.sarif }}

49
docs/en/engines/table-engines/mergetree-family/custom-partitioning-key.md
View File

@ -55,27 +55,28 @@ WHERE table = 'visits'
```
``` text
┌─partition─┬─name───────────┬─active─┐
│ 201901 │ 201901_1_3_1 │ 0 │
│ 201901 │ 201901_1_9_2 │ 1 │
│ 201901 │ 201901_8_8_0 │ 0 │
│ 201901 │ 201901_9_9_0 │ 0 │
│ 201902 │ 201902_4_6_1 │ 1 │
│ 201902 │ 201902_10_10_0 │ 1 │
│ 201902 │ 201902_11_11_0 │ 1 │
└───────────┴────────────────┴────────┘
┌─partition─┬─name──────────────┬─active─┐
│ 201901 │ 201901_1_3_1 │ 0 │
│ 201901 │ 201901_1_9_2_11 │ 1 │
│ 201901 │ 201901_8_8_0 │ 0 │
│ 201901 │ 201901_9_9_0 │ 0 │
│ 201902 │ 201902_4_6_1_11 │ 1 │
│ 201902 │ 201902_10_10_0_11 │ 1 │
│ 201902 │ 201902_11_11_0_11 │ 1 │
└───────────┴───────────────────┴────────┘
```
The `partition` column contains the names of the partitions. There are two partitions in this example: `201901` and `201902`. You can use this column value to specify the partition name in [ALTER … PARTITION](../../../sql-reference/statements/alter/partition.md) queries.
The `name` column contains the names of the partition data parts. You can use this column to specify the name of the part in the [ALTER ATTACH PART](../../../sql-reference/statements/alter/partition.md#alter_attach-partition) query.
Let’s break down the name of the first part: `201901_1_3_1`:
Let’s break down the name of the part: `201901_1_9_2_11`:
- `201901` is the partition name.
- `1` is the minimum number of the data block.
- `3` is the maximum number of the data block.
- `1` is the chunk level (the depth of the merge tree it is formed from).
- `9` is the maximum number of the data block.
- `2` is the chunk level (the depth of the merge tree it is formed from).
- `11` is the mutation version (if a part mutated)
!!! info "Info"
The parts of old-type tables have the name: `20190117_20190123_2_2_0` (minimum date - maximum date - minimum block number - maximum block number - level).
@ -89,16 +90,16 @@ OPTIMIZE TABLE visits PARTITION 201902;
```
``` text
┌─partition─┬─name───────────┬─active─┐
│ 201901 │ 201901_1_3_1 │ 0 │
│ 201901 │ 201901_1_9_2 │ 1 │
│ 201901 │ 201901_8_8_0 │ 0 │
│ 201901 │ 201901_9_9_0 │ 0 │
│ 201902 │ 201902_4_6_1 │ 0 │
│ 201902 │ 201902_4_11_2 │ 1 │
│ 201902 │ 201902_10_10_0 │ 0 │
│ 201902 │ 201902_11_11_0 │ 0 │
└───────────┴────────────────┴────────┘
┌─partition─┬─name─────────────┬─active─┐
│ 201901 │ 201901_1_3_1 │ 0 │
│ 201901 │ 201901_1_9_2_11 │ 1 │
│ 201901 │ 201901_8_8_0 │ 0 │
│ 201901 │ 201901_9_9_0 │ 0 │
│ 201902 │ 201902_4_6_1 │ 0 │
│ 201902 │ 201902_4_11_2_11 │ 1 │
│ 201902 │ 201902_10_10_0 │ 0 │
│ 201902 │ 201902_11_11_0 │ 0 │
└───────────┴──────────────────┴────────┘
```
Inactive parts will be deleted approximately 10 minutes after merging.
@ -109,12 +110,12 @@ Another way to view a set of parts and partitions is to go into the directory of
/var/lib/clickhouse/data/default/visits$ ls -l
total 40
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 1 16:48 201901_1_3_1
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:17 201901_1_9_2
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:17 201901_1_9_2_11
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 15:52 201901_8_8_0
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 15:52 201901_9_9_0
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:17 201902_10_10_0
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:17 201902_11_11_0
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:19 201902_4_11_2
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:19 201902_4_11_2_11
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 12:09 201902_4_6_1
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 1 16:48 detached
```

49
docs/ru/engines/table-engines/mergetree-family/custom-partitioning-key.md
View File

@ -53,15 +53,15 @@ WHERE table = 'visits'
```
``` text
┌─partition─┬─name───────────┬─active─┐
│ 201901 │ 201901_1_3_1 │ 0 │
│ 201901 │ 201901_1_9_2 │ 1 │
│ 201901 │ 201901_8_8_0 │ 0 │
│ 201901 │ 201901_9_9_0 │ 0 │
│ 201902 │ 201902_4_6_1 │ 1 │
│ 201902 │ 201902_10_10_0 │ 1 │
│ 201902 │ 201902_11_11_0 │ 1 │
└───────────┴────────────────┴────────┘
┌─partition─┬─name──────────────┬─active─┐
│ 201901 │ 201901_1_3_1 │ 0 │
│ 201901 │ 201901_1_9_2_11 │ 1 │
│ 201901 │ 201901_8_8_0 │ 0 │
│ 201901 │ 201901_9_9_0 │ 0 │
│ 201902 │ 201902_4_6_1_11 │ 1 │
│ 201902 │ 201902_10_10_0_11 │ 1 │
│ 201902 │ 201902_11_11_0_11 │ 1 │
└───────────┴───────────────────┴────────┘
```
Столбец `partition` содержит имена всех партиций таблицы. Таблица `visits` из нашего примера содержит две партиции: `201901` и `201902`. Используйте значения из этого столбца в запросах [ALTER … PARTITION](../../../sql-reference/statements/alter/partition.md).
@ -70,12 +70,13 @@ WHERE table = 'visits'
Столбец `active` отображает состояние куска. `1` означает, что кусок активен; `0` – неактивен. К неактивным можно отнести куски, оставшиеся после слияния данных. Поврежденные куски также отображаются как неактивные. Неактивные куски удаляются приблизительно через 10 минут после того, как было выполнено слияние.
Рассмотрим детальнее имя первого куска `201901_1_3_1`:
Рассмотрим детальнее имя куска `201901_1_9_2_11`:
- `201901` имя партиции;
- `1` – минимальный номер блока данных;
- `3` – максимальный номер блока данных;
- `1` – уровень куска (глубина дерева слияний, которыми этот кусок образован).
- `9` – максимальный номер блока данных;
- `2` – уровень куска (глубина дерева слияний, которыми этот кусок образован).
- `11` - версия мутации (если парт мутировал)
!!! info "Info"
Названия кусков для таблиц старого типа образуются следующим образом: `20190117_20190123_2_2_0` (минимальная дата _ максимальная дата _ номер минимального блока _ номер максимального блока _ уровень).
@ -89,16 +90,16 @@ OPTIMIZE TABLE visits PARTITION 201902;
```
``` text
┌─partition─┬─name───────────┬─active─┐
│ 201901 │ 201901_1_3_1 │ 0 │
│ 201901 │ 201901_1_9_2 │ 1 │
│ 201901 │ 201901_8_8_0 │ 0 │
│ 201901 │ 201901_9_9_0 │ 0 │
│ 201902 │ 201902_4_6_1 │ 0 │
│ 201902 │ 201902_4_11_2 │ 1 │
│ 201902 │ 201902_10_10_0 │ 0 │
│ 201902 │ 201902_11_11_0 │ 0 │
└───────────┴────────────────┴────────┘
┌─partition─┬─name─────────────┬─active─┐
│ 201901 │ 201901_1_3_1 │ 0 │
│ 201901 │ 201901_1_9_2_11 │ 1 │
│ 201901 │ 201901_8_8_0 │ 0 │
│ 201901 │ 201901_9_9_0 │ 0 │
│ 201902 │ 201902_4_6_1 │ 0 │
│ 201902 │ 201902_4_11_2_11 │ 1 │
│ 201902 │ 201902_10_10_0 │ 0 │
│ 201902 │ 201902_11_11_0 │ 0 │
└───────────┴──────────────────┴────────┘
```
Неактивные куски будут удалены примерно через 10 минут после слияния.
@ -109,12 +110,12 @@ OPTIMIZE TABLE visits PARTITION 201902;
/var/lib/clickhouse/data/default/visits$ ls -l
total 40
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 1 16:48 201901_1_3_1
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:17 201901_1_9_2
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:17 201901_1_9_2_11
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 15:52 201901_8_8_0
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 15:52 201901_9_9_0
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:17 201902_10_10_0
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:17 201902_11_11_0
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:19 201902_4_11_2
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 16:19 201902_4_11_2_11
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 5 12:09 201902_4_6_1
drwxr-xr-x 2 clickhouse clickhouse 4096 Feb 1 16:48 detached
```

2
src/Core/examples/mysql_protocol.cpp
View File

@ -330,7 +330,7 @@ int main(int argc, char ** argv)
/// Connect to the master.
slave.connect();
slave.startBinlogDumpGTID(slave_id, replicate_db, gtid_sets, binlog_checksum);
slave.startBinlogDumpGTID(slave_id, replicate_db, {}, gtid_sets, binlog_checksum);
WriteBufferFromOStream cerr(std::cerr);

8
src/Interpreters/RewriteFunctionToSubcolumnVisitor.cpp
View File

@ -83,6 +83,7 @@ void RewriteFunctionToSubcolumnData::visit(ASTFunction & function, ASTPtr & ast)
const auto & column_type = columns.get(name_in_storage).type;
TypeIndex column_type_id = column_type->getTypeId();
const auto & alias = function.tryGetAlias();
if (arguments.size() == 1)
{
@ -91,7 +92,10 @@ void RewriteFunctionToSubcolumnData::visit(ASTFunction & function, ASTPtr & ast)
{
const auto & [type_id, subcolumn_name, transformer] = it->second;
if (column_type_id == type_id)
{
ast = transformer(name_in_storage, subcolumn_name);
ast->setAlias(alias);
}
}
}
else
@ -116,6 +120,7 @@ void RewriteFunctionToSubcolumnData::visit(ASTFunction & function, ASTPtr & ast)
return;
ast = transformToSubcolumn(name_in_storage, subcolumn_name);
ast->setAlias(alias);
}
else
{
@ -124,7 +129,10 @@ void RewriteFunctionToSubcolumnData::visit(ASTFunction & function, ASTPtr & ast)
{
const auto & [type_id, subcolumn_name, transformer] = it->second;
if (column_type_id == type_id)
{
ast = transformer(name_in_storage, subcolumn_name, arguments[1]);
ast->setAlias(alias);
}
}
}
}

4
src/Processors/Transforms/TotalsHavingTransform.cpp
View File

@ -138,7 +138,7 @@ IProcessor::Status TotalsHavingTransform::prepare()
if (!totals_output.canPush())
return Status::PortFull;
if (!totals)
if (!total_prepared)
return Status::Ready;
totals_output.push(std::move(totals));
@ -312,6 +312,8 @@ void TotalsHavingTransform::prepareTotals()
/// Note: after expression totals may have several rows if `arrayJoin` was used in expression.
totals = Chunk(block.getColumns(), num_rows);
}
total_prepared = true;
}
}

1
src/Processors/Transforms/TotalsHavingTransform.h
View File

@ -46,6 +46,7 @@ protected:
void transform(Chunk & chunk) override;
bool finished_transform = false;
bool total_prepared = false;
Chunk totals;
private:

20
src/Processors/Transforms/WindowTransform.cpp
View File

@ -7,6 +7,7 @@
#include <Columns/ColumnLowCardinality.h>
#include <base/arithmeticOverflow.h>
#include <Columns/ColumnConst.h>
#include <Columns/ColumnAggregateFunction.h>
#include <DataTypes/DataTypesNumber.h>
#include <DataTypes/getLeastSupertype.h>
#include <DataTypes/DataTypeLowCardinality.h>
@ -22,6 +23,7 @@ namespace ErrorCodes
{
extern const int BAD_ARGUMENTS;
extern const int NOT_IMPLEMENTED;
extern const int ILLEGAL_COLUMN;
}
// Interface for true window functions. It's not much of an interface, they just
@ -986,7 +988,23 @@ void WindowTransform::writeOutCurrentRow()
auto * buf = ws.aggregate_function_state.data();
// FIXME does it also allocate the result on the arena?
// We'll have to pass it out with blocks then...
a->insertResultInto(buf, *result_column, arena.get());
if (a->isState())
{
/// AggregateFunction's states should be inserted into column using specific way
auto * res_col_aggregate_function = typeid_cast<ColumnAggregateFunction *>(result_column);
if (!res_col_aggregate_function)
{
throw Exception("State function " + a->getName() + " inserts results into non-state column ",
ErrorCodes::ILLEGAL_COLUMN);
}
res_col_aggregate_function->insertFrom(buf);
}
else
{
a->insertResultInto(buf, *result_column, arena.get());
}
}
}

83
tests/ci/ci_config.json
View File

@ -1,83 +0,0 @@
{
"build_config": [
{
"compiler": "clang-13",
"build-type": "",
"sanitizer": "",
"package-type": "deb",
"bundled": "bundled",
"splitted": "unsplitted",
"alien_pkgs": true,
"tidy": "disable",
"with_coverage": false
},
{
"compiler": "clang-13",
"build-type": "",
"sanitizer": "",
"package-type": "performance",
"bundled": "bundled",
"splitted": "unsplitted",
"tidy": "disable",
"with_coverage": false
},
{
"compiler": "gcc-11",
"build-type": "",
"sanitizer": "",
"package-type": "binary",
"bundled": "bundled",
"splitted": "unsplitted",
"tidy": "disable",
"with_coverage": false
},
{
"compiler": "clang-13",
"build-type": "",
"sanitizer": "",
"package-type": "binary",
"bundled": "bundled",
"splitted": "unsplitted",
"tidy": "disable",
"with_coverage": false
}
],
"tests_config": {
"Testflows check": {
"required_build_properties": {
"compiler": "clang-13",
"package_type": "deb",
"build_type": "relwithdebuginfo",
"sanitizer": "none",
"bundled": "bundled",
"splitted": "unsplitted",
"clang-tidy": "disable",
"with_coverage": false
}
},
"Release": {
"required_build_properties": {
"compiler": "clang-13",
"package_type": "deb",
"build_type": "relwithdebuginfo",
"sanitizer": "none",
"bundled": "bundled",
"splitted": "unsplitted",
"clang-tidy": "disable",
"with_coverage": false
}
},
"ClickHouse Keeper Jepsen": {
"required_build_properties": {
"compiler": "clang-13",
"package_type": "binary",
"build_type": "relwithdebuginfo",
"sanitizer": "none",
"bundled": "bundled",
"splitted": "unsplitted",
"clang-tidy": "disable",
"with_coverage": false
}
}
}
}

22
tests/queries/0_stateless/01591_window_functions.reference
View File

@ -1141,6 +1141,28 @@ from (
from numbers_mt(10000)
) settings max_block_size = 7;
49995000
-- a test with aggregate function which is -state type
select bitmapCardinality(bs)
from
(
select groupBitmapMergeState(bm) over (order by k asc rows between unbounded preceding and current row) as bs
from
(
select
groupBitmapState(number) as bm, k
from
(
select
number,
number % 3 as k
from numbers(3)
)
group by k
)
);
1
2
3
-- -INT_MIN row offset that can lead to problems with negation, found when fuzzing
-- under UBSan. Should be limited to at most INT_MAX.
select count() over (rows between 2147483648 preceding and 2147493648 following) from numbers(2); -- { serverError 36 }

22
tests/queries/0_stateless/01591_window_functions.sql
View File

@ -1,3 +1,5 @@
-- Tags: long
-- { echo }
-- just something basic
@ -442,6 +444,26 @@ from (
from numbers_mt(10000)
) settings max_block_size = 7;
-- a test with aggregate function which is -state type
select bitmapCardinality(bs)
from
(
select groupBitmapMergeState(bm) over (order by k asc rows between unbounded preceding and current row) as bs
from
(
select
groupBitmapState(number) as bm, k
from
(
select
number,
number % 3 as k
from numbers(3)
)
group by k
)
);
-- -INT_MIN row offset that can lead to problems with negation, found when fuzzing
-- under UBSan. Should be limited to at most INT_MAX.
select count() over (rows between 2147483648 preceding and 2147493648 following) from numbers(2); -- { serverError 36 }

8
tests/queries/0_stateless/02232_functions_to_subcolumns_alias.reference Normal file
View File

@ -0,0 +1,8 @@
cnt
2
t0 t0
100 100
0 0
hit
1
0

10
tests/queries/0_stateless/02232_functions_to_subcolumns_alias.sql Normal file
View File

@ -0,0 +1,10 @@
DROP TABLE IF EXISTS t_functions_to_subcolumns_alias;
CREATE TABLE t_functions_to_subcolumns_alias (id UInt64, t Tuple(UInt64, String), m Map(String, UInt64)) ENGINE = Memory;
INSERT INTO t_functions_to_subcolumns_alias VALUES (1, (100, 'abc'), map('foo', 1, 'bar', 2)) (2, NULL, map());
SELECT count(id) AS cnt FROM t_functions_to_subcolumns_alias FORMAT TSVWithNames;
SELECT tupleElement(t, 1) as t0, t0 FROM t_functions_to_subcolumns_alias FORMAT TSVWithNames;
SELECT mapContains(m, 'foo') AS hit FROM t_functions_to_subcolumns_alias FORMAT TSVWithNames;
DROP TABLE t_functions_to_subcolumns_alias;

0
tests/queries/0_stateless/02233_with_total_empty_chunk.reference Normal file
View File

1
tests/queries/0_stateless/02233_with_total_empty_chunk.sql Normal file
View File

@ -0,0 +1 @@
SELECT (NULL, NULL, NULL, NULL, NULL, NULL, NULL) FROM numbers(0) GROUP BY number WITH TOTALS HAVING sum(number) <= arrayJoin([]);

3
website/js/base.js
View File

@ -85,6 +85,9 @@
$(element).append(
'<script async src="https://www.googletagmanager.com/gtag/js?id=G-KF1LLRTQ5Q"></script><script>window.dataLayer = window.dataLayer || [];function gtag(){dataLayer.push(arguments);}gtag(\'js\', new Date());gtag(\'config\', \'G-KF1LLRTQ5Q\');</script>'
);
$(element).append(
'<script>!function(){var analytics=window.analytics=window.analytics||[];if(!analytics.initialize)if(analytics.invoked)window.console&&console.error&&console.error("Segment snippet included twice.");else{analytics.invoked=!0;analytics.methods=["trackSubmit","trackClick","trackLink","trackForm","pageview","identify","reset","group","track","ready","alias","debug","page","once","off","on","addSourceMiddleware","addIntegrationMiddleware","setAnonymousId","addDestinationMiddleware"];analytics.factory=function(e){return function(){var t=Array.prototype.slice.call(arguments);t.unshift(e);analytics.push(t);return analytics}};for(var e=0;e<analytics.methods.length;e++){var key=analytics.methods[e];analytics[key]=analytics.factory(key)}analytics.load=function(key,e){var t=document.createElement("script");t.type="text/javascript";t.async=!0;t.src="https://cdn.segment.com/analytics.js/v1/" + key + "/analytics.min.js";var n=document.getElementsByTagName("script")[0];n.parentNode.insertBefore(t,n);analytics._loadOptions=e};analytics._writeKey="dZuEnmCPmWqDuSEzCvLUSBBRt8Xrh2el";;analytics.SNIPPET_VERSION="4.15.3";analytics.load("dZuEnmCPmWqDuSEzCvLUSBBRt8Xrh2el");analytics.page();}}();</script>'
);
});
}
});