thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-21 23:21:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-dynamic-sizes

Browse Source
This commit is contained in:
avogar 2024-11-07 20:00:54 +00:00
commit cab460d7c3
59 changed files with 804 additions and 102 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
docs/en/operations/query-cache.md
View File

@ -25,9 +25,10 @@ Query caches can generally be viewed as transactionally consistent or inconsiste
slowly enough that the database only needs to compute the report once (represented by the first `SELECT` query). Further queries can be
served directly from the query cache. In this example, a reasonable validity period could be 30 min.
Transactionally inconsistent caching is traditionally provided by client tools or proxy packages interacting with the database. As a result,
the same caching logic and configuration is often duplicated. With ClickHouse's query cache, the caching logic moves to the server side.
This reduces maintenance effort and avoids redundancy.
Transactionally inconsistent caching is traditionally provided by client tools or proxy packages (e.g.
[chproxy](https://www.chproxy.org/configuration/caching/)) interacting with the database. As a result, the same caching logic and
configuration is often duplicated. With ClickHouse's query cache, the caching logic moves to the server side. This reduces maintenance
effort and avoids redundancy.
## Configuration Settings and Usage
@ -138,7 +139,10 @@ is only cached if the query runs longer than 5 seconds. It is also possible to s
cached - for that use setting [query_cache_min_query_runs](settings/settings.md#query-cache-min-query-runs).
Entries in the query cache become stale after a certain time period (time-to-live). By default, this period is 60 seconds but a different
value can be specified at session, profile or query level using setting [query_cache_ttl](settings/settings.md#query-cache-ttl).
value can be specified at session, profile or query level using setting [query_cache_ttl](settings/settings.md#query-cache-ttl). The query
cache evicts entries "lazily", i.e. when an entry becomes stale, it is not immediately removed from the cache. Instead, when a new entry
is to be inserted into the query cache, the database checks whether the cache has enough free space for the new entry. If this is not the
case, the database tries to remove all stale entries. If the cache still has not enough free space, the new entry is not inserted.
Entries in the query cache are compressed by default. This reduces the overall memory consumption at the cost of slower writes into / reads
from the query cache. To disable compression, use setting [query_cache_compress_entries](settings/settings.md#query-cache-compress-entries).
@ -188,14 +192,9 @@ Also, results of queries with non-deterministic functions are not cached by defa
To force caching of results of queries with non-deterministic functions regardless, use setting
[query_cache_nondeterministic_function_handling](settings/settings.md#query-cache-nondeterministic-function-handling).
Results of queries that involve system tables, e.g. `system.processes` or `information_schema.tables`, are not cached by default. To force
caching of results of queries with system tables regardless, use setting
[query_cache_system_table_handling](settings/settings.md#query-cache-system-table-handling).
:::note
Prior to ClickHouse v23.11, setting 'query_cache_store_results_of_queries_with_nondeterministic_functions = 0 / 1' controlled whether
results of queries with non-deterministic results were cached. In newer ClickHouse versions, this setting is obsolete and has no effect.
:::
Results of queries that involve system tables (e.g. [system.processes](system-tables/processes.md)` or
[information_schema.tables](system-tables/information_schema.md)) are not cached by default. To force caching of results of queries with
system tables regardless, use setting [query_cache_system_table_handling](settings/settings.md#query-cache-system-table-handling).
Finally, entries in the query cache are not shared between users due to security reasons. For example, user A must not be able to bypass a
row policy on a table by running the same query as another user B for whom no such policy exists. However, if necessary, cache entries can

10
docs/en/sql-reference/data-types/dynamic.md
View File

@ -512,6 +512,8 @@ The result of operator `<` for values `d1` with underlying type `T1` and `d2` wi
- If `T1 = T2 = T`, the result will be `d1.T < d2.T` (underlying values will be compared).
- If `T1 != T2`, the result will be `T1 < T2` (type names will be compared).
By default `Dynamic` type is not allowed in `GROUP BY`/`ORDER BY` keys, if you want to use it consider its special comparison rule and enable `allow_suspicious_types_in_group_by`/`allow_suspicious_types_in_order_by` settings.
Examples:
```sql
CREATE TABLE test (d Dynamic) ENGINE=Memory;
@ -535,7 +537,7 @@ SELECT d, dynamicType(d) FROM test;
```
```sql
SELECT d, dynamicType(d) FROM test ORDER BY d;
SELECT d, dynamicType(d) FROM test ORDER BY d SETTINGS allow_suspicious_types_in_order_by=1;
```
```sql
@ -557,7 +559,7 @@ Example:
```sql
CREATE TABLE test (d Dynamic) ENGINE=Memory;
INSERT INTO test VALUES (1::UInt32), (1::Int64), (100::UInt32), (100::Int64);
SELECT d, dynamicType(d) FROM test ORDER by d;
SELECT d, dynamicType(d) FROM test ORDER BY d SETTINGS allow_suspicious_types_in_order_by=1;
```
```text
@ -570,7 +572,7 @@ SELECT d, dynamicType(d) FROM test ORDER by d;
```
```sql
SELECT d, dynamicType(d) FROM test GROUP by d;
SELECT d, dynamicType(d) FROM test GROUP by d SETTINGS allow_suspicious_types_in_group_by=1;
```
```text
@ -582,7 +584,7 @@ SELECT d, dynamicType(d) FROM test GROUP by d;
└─────┴────────────────┘
```
**Note**: the described comparison rule is not applied during execution of comparison functions like `<`/`>`/`=` and others because of [special work](#using-dynamic-type-in-functions) of functions with `Dynamic` type
**Note:** the described comparison rule is not applied during execution of comparison functions like `<`/`>`/`=` and others because of [special work](#using-dynamic-type-in-functions) of functions with `Dynamic` type
## Reaching the limit in number of different data types stored inside Dynamic

2
docs/en/sql-reference/data-types/variant.md
View File

@ -441,6 +441,8 @@ SELECT v, variantType(v) FROM test ORDER by v;
└─────┴────────────────┘
```
**Note** by default `Variant` type is not allowed in `GROUP BY`/`ORDER BY` keys, if you want to use it consider its special comparison rule and enable `allow_suspicious_types_in_group_by`/`allow_suspicious_types_in_order_by` settings.
## JSONExtract functions with Variant
All `JSONExtract*` functions support `Variant` type:

12
src/Analyzer/QueryNode.h
View File

@ -602,9 +602,21 @@ public:
return projection_columns;
}
/// Returns true if query node is resolved, false otherwise
bool isResolved() const
{
return !projection_columns.empty();
}
/// Resolve query node projection columns
void resolveProjectionColumns(NamesAndTypes projection_columns_value);
/// Clear query node projection columns
void clearProjectionColumns()
{
projection_columns.clear();
}
/// Remove unused projection columns
void removeUnusedProjectionColumns(const std::unordered_set<std::string> & used_projection_columns);

120
src/Analyzer/Resolve/QueryAnalyzer.cpp
View File

@ -103,6 +103,8 @@ namespace Setting
extern const SettingsBool single_join_prefer_left_table;
extern const SettingsBool transform_null_in;
extern const SettingsUInt64 use_structure_from_insertion_table_in_table_functions;
extern const SettingsBool allow_suspicious_types_in_group_by;
extern const SettingsBool allow_suspicious_types_in_order_by;
extern const SettingsBool use_concurrency_control;
}
@ -2966,27 +2968,29 @@ ProjectionNames QueryAnalyzer::resolveFunction(QueryTreeNodePtr & node, Identifi
/// Replace storage with values storage of insertion block
if (StoragePtr storage = scope.context->getViewSource())
{
QueryTreeNodePtr table_expression;
/// Process possibly nested sub-selects
for (auto * query_node = in_second_argument->as<QueryNode>(); query_node; query_node = table_expression->as<QueryNode>())
table_expression = extractLeftTableExpression(query_node->getJoinTree());
QueryTreeNodePtr table_expression = in_second_argument;
if (table_expression)
/// Process possibly nested sub-selects
while (table_expression)
{
if (auto * query_table_node = table_expression->as<TableNode>())
{
if (query_table_node->getStorageID().getFullNameNotQuoted() == storage->getStorageID().getFullNameNotQuoted())
{
auto replacement_table_expression = std::make_shared<TableNode>(storage, scope.context);
if (std::optional<TableExpressionModifiers> table_expression_modifiers = query_table_node->getTableExpressionModifiers())
replacement_table_expression->setTableExpressionModifiers(*table_expression_modifiers);
in_second_argument = in_second_argument->cloneAndReplace(table_expression, std::move(replacement_table_expression));
}
}
if (auto * query_node = table_expression->as<QueryNode>())
table_expression = extractLeftTableExpression(query_node->getJoinTree());
else if (auto * union_node = table_expression->as<UnionNode>())
table_expression = union_node->getQueries().getNodes().at(0);
else
break;
}
TableNode * table_expression_table_node = table_expression ? table_expression->as<TableNode>() : nullptr;
if (table_expression_table_node &&
table_expression_table_node->getStorageID().getFullNameNotQuoted() == storage->getStorageID().getFullNameNotQuoted())
{
auto replacement_table_expression_table_node = table_expression_table_node->clone();
replacement_table_expression_table_node->as<TableNode &>().updateStorage(storage, scope.context);
in_second_argument = in_second_argument->cloneAndReplace(table_expression, std::move(replacement_table_expression_table_node));
}
}
resolveExpressionNode(in_second_argument, scope, false /*allow_lambda_expression*/, true /*allow_table_expression*/);
}
/// Edge case when the first argument of IN is scalar subquery.
@ -4028,6 +4032,8 @@ ProjectionNames QueryAnalyzer::resolveSortNodeList(QueryTreeNodePtr & sort_node_
sort_node.getExpression() = sort_column_list_node->getNodes().front();
}
validateSortingKeyType(sort_node.getExpression()->getResultType(), scope);
size_t sort_expression_projection_names_size = sort_expression_projection_names.size();
if (sort_expression_projection_names_size != 1)
throw Exception(ErrorCodes::LOGICAL_ERROR,
@ -4142,6 +4148,26 @@ ProjectionNames QueryAnalyzer::resolveSortNodeList(QueryTreeNodePtr & sort_node_
return result_projection_names;
}
void QueryAnalyzer::validateSortingKeyType(const DataTypePtr & sorting_key_type, const IdentifierResolveScope & scope) const
{
if (scope.context->getSettingsRef()[Setting::allow_suspicious_types_in_order_by])
return;
auto check = [](const IDataType & type)
{
if (isDynamic(type) || isVariant(type))
throw Exception(
ErrorCodes::ILLEGAL_COLUMN,
"Data types Variant/Dynamic are not allowed in ORDER BY keys, because it can lead to unexpected results. "
"Consider using a subcolumn with a specific data type instead (for example 'column.Int64' or 'json.some.path.:Int64' if "
"its a JSON path subcolumn) or casting this column to a specific data type. "
"Set setting allow_suspicious_types_in_order_by = 1 in order to allow it");
};
check(*sorting_key_type);
sorting_key_type->forEachChild(check);
}
namespace
{
@ -4181,11 +4207,12 @@ void QueryAnalyzer::resolveGroupByNode(QueryNode & query_node_typed, IdentifierR
expandTuplesInList(group_by_list);
}
if (scope.group_by_use_nulls)
for (const auto & grouping_set : query_node_typed.getGroupBy().getNodes())
{
for (const auto & grouping_set : query_node_typed.getGroupBy().getNodes())
for (const auto & group_by_elem : grouping_set->as<ListNode>()->getNodes())
{
for (const auto & group_by_elem : grouping_set->as<ListNode>()->getNodes())
validateGroupByKeyType(group_by_elem->getResultType(), scope);
if (scope.group_by_use_nulls)
scope.nullable_group_by_keys.insert(group_by_elem);
}
}
@ -4201,14 +4228,37 @@ void QueryAnalyzer::resolveGroupByNode(QueryNode & query_node_typed, IdentifierR
auto & group_by_list = query_node_typed.getGroupBy().getNodes();
expandTuplesInList(group_by_list);
if (scope.group_by_use_nulls)
for (const auto & group_by_elem : query_node_typed.getGroupBy().getNodes())
{
for (const auto & group_by_elem : query_node_typed.getGroupBy().getNodes())
validateGroupByKeyType(group_by_elem->getResultType(), scope);
if (scope.group_by_use_nulls)
scope.nullable_group_by_keys.insert(group_by_elem);
}
}
}
/** Validate data types of GROUP BY key.
*/
void QueryAnalyzer::validateGroupByKeyType(const DataTypePtr & group_by_key_type, const IdentifierResolveScope & scope) const
{
if (scope.context->getSettingsRef()[Setting::allow_suspicious_types_in_group_by])
return;
auto check = [](const IDataType & type)
{
if (isDynamic(type) || isVariant(type))
throw Exception(
ErrorCodes::ILLEGAL_COLUMN,
"Data types Variant/Dynamic are not allowed in GROUP BY keys, because it can lead to unexpected results. "
"Consider using a subcolumn with a specific data type instead (for example 'column.Int64' or 'json.some.path.:Int64' if "
"its a JSON path subcolumn) or casting this column to a specific data type. "
"Set setting allow_suspicious_types_in_group_by = 1 in order to allow it");
};
check(*group_by_key_type);
group_by_key_type->forEachChild(check);
}
/** Resolve interpolate columns nodes list.
*/
void QueryAnalyzer::resolveInterpolateColumnsNodeList(QueryTreeNodePtr & interpolate_node_list, IdentifierResolveScope & scope)
@ -5347,6 +5397,16 @@ void QueryAnalyzer::resolveQuery(const QueryTreeNodePtr & query_node, Identifier
auto & query_node_typed = query_node->as<QueryNode &>();
/** It is unsafe to call resolveQuery on already resolved query node, because during identifier resolution process
* we replace identifiers with expressions without aliases, also at the end of resolveQuery all aliases from all nodes will be removed.
* For subsequent resolveQuery executions it is possible to have wrong projection header, because for nodes
* with aliases projection name is alias.
*
* If for client it is necessary to resolve query node after clone, client must clear projection columns from query node before resolve.
*/
if (query_node_typed.isResolved())
return;
if (query_node_typed.isCTE())
ctes_in_resolve_process.insert(query_node_typed.getCTEName());
@ -5485,16 +5545,13 @@ void QueryAnalyzer::resolveQuery(const QueryTreeNodePtr & query_node, Identifier
*/
scope.use_identifier_lookup_to_result_cache = false;
if (query_node_typed.getJoinTree())
{
TableExpressionsAliasVisitor table_expressions_visitor(scope);
table_expressions_visitor.visit(query_node_typed.getJoinTree());
TableExpressionsAliasVisitor table_expressions_visitor(scope);
table_expressions_visitor.visit(query_node_typed.getJoinTree());
initializeQueryJoinTreeNode(query_node_typed.getJoinTree(), scope);
scope.aliases.alias_name_to_table_expression_node.clear();
initializeQueryJoinTreeNode(query_node_typed.getJoinTree(), scope);
scope.aliases.alias_name_to_table_expression_node.clear();
resolveQueryJoinTreeNode(query_node_typed.getJoinTree(), scope, visitor);
}
resolveQueryJoinTreeNode(query_node_typed.getJoinTree(), scope, visitor);
if (!scope.group_by_use_nulls)
scope.use_identifier_lookup_to_result_cache = true;
@ -5712,6 +5769,9 @@ void QueryAnalyzer::resolveUnion(const QueryTreeNodePtr & union_node, Identifier
{
auto & union_node_typed = union_node->as<UnionNode &>();
if (union_node_typed.isResolved())
return;
if (union_node_typed.isCTE())
ctes_in_resolve_process.insert(union_node_typed.getCTEName());

4
src/Analyzer/Resolve/QueryAnalyzer.h
View File

@ -220,8 +220,12 @@ private:
ProjectionNames resolveSortNodeList(QueryTreeNodePtr & sort_node_list, IdentifierResolveScope & scope);
void validateSortingKeyType(const DataTypePtr & sorting_key_type, const IdentifierResolveScope & scope) const;
void resolveGroupByNode(QueryNode & query_node_typed, IdentifierResolveScope & scope);
void validateGroupByKeyType(const DataTypePtr & group_by_key_type, const IdentifierResolveScope & scope) const;
void resolveInterpolateColumnsNodeList(QueryTreeNodePtr & interpolate_node_list, IdentifierResolveScope & scope);
void resolveWindowNodeList(QueryTreeNodePtr & window_node_list, IdentifierResolveScope & scope);

21
src/Analyzer/UnionNode.cpp
View File

@ -35,6 +35,7 @@ namespace ErrorCodes
{
extern const int TYPE_MISMATCH;
extern const int BAD_ARGUMENTS;
extern const int LOGICAL_ERROR;
}
UnionNode::UnionNode(ContextMutablePtr context_, SelectUnionMode union_mode_)
@ -50,6 +51,26 @@ UnionNode::UnionNode(ContextMutablePtr context_, SelectUnionMode union_mode_)
children[queries_child_index] = std::make_shared<ListNode>();
}
bool UnionNode::isResolved() const
{
for (const auto & query_node : getQueries().getNodes())
{
bool is_resolved = false;
if (auto * query_node_typed = query_node->as<QueryNode>())
is_resolved = query_node_typed->isResolved();
else if (auto * union_node_typed = query_node->as<UnionNode>())
is_resolved = union_node_typed->isResolved();
else
throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected query tree node type in UNION node");
if (!is_resolved)
return false;
}
return true;
}
NamesAndTypes UnionNode::computeProjectionColumns() const
{
if (recursive_cte_table)

3
src/Analyzer/UnionNode.h
View File

@ -163,6 +163,9 @@ public:
return children[queries_child_index];
}
/// Returns true if union node is resolved, false otherwise
bool isResolved() const;
/// Compute union node projection columns
NamesAndTypes computeProjectionColumns() const;

13
src/Common/HashTable/HashTable.h
View File

@ -658,16 +658,11 @@ protected:
{
if (!std::is_trivially_destructible_v<Cell>)
{
for (iterator it = begin(), it_end = end(); it != it_end; ++it)
for (iterator it = begin(), it_end = end(); it != it_end;)
{
it.ptr->~Cell();
/// In case of poison_in_dtor=1 it will be poisoned,
/// but it maybe used later, during iteration.
///
/// NOTE, that technically this is UB [1], but OK for now.
///
/// [1]: https://github.com/google/sanitizers/issues/854#issuecomment-329661378
__msan_unpoison(it.ptr, sizeof(*it.ptr));
auto ptr = it.ptr;
++it;
ptr->~Cell();
}
/// Everything had been destroyed in the loop above, reset the flag

10
src/Core/Settings.cpp
View File

@ -873,6 +873,12 @@ In CREATE TABLE statement allows specifying Variant type with similar variant ty
)", 0) \
DECLARE(Bool, allow_suspicious_primary_key, false, R"(
Allow suspicious `PRIMARY KEY`/`ORDER BY` for MergeTree (i.e. SimpleAggregateFunction).
)", 0) \
DECLARE(Bool, allow_suspicious_types_in_group_by, false, R"(
Allows or restricts using [Variant](../../sql-reference/data-types/variant.md) and [Dynamic](../../sql-reference/data-types/dynamic.md) types in GROUP BY keys.
)", 0) \
DECLARE(Bool, allow_suspicious_types_in_order_by, false, R"(
Allows or restricts using [Variant](../../sql-reference/data-types/variant.md) and [Dynamic](../../sql-reference/data-types/dynamic.md) types in ORDER BY keys.
)", 0) \
DECLARE(Bool, compile_expressions, false, R"(
Compile some scalar functions and operators to native code. Due to a bug in the LLVM compiler infrastructure, on AArch64 machines, it is known to lead to a nullptr dereference and, consequently, server crash. Do not enable this setting.
@ -2863,7 +2869,7 @@ Limit on size of multipart/form-data content. This setting cannot be parsed from
DECLARE(Bool, calculate_text_stack_trace, true, R"(
Calculate text stack trace in case of exceptions during query execution. This is the default. It requires symbol lookups that may slow down fuzzing tests when a huge amount of wrong queries are executed. In normal cases, you should not disable this option.
)", 0) \
DECLARE(Bool, enable_job_stack_trace, false, R"(
DECLARE(Bool, enable_job_stack_trace, true, R"(
Output stack trace of a job creator when job results in exception
)", 0) \
DECLARE(Bool, allow_ddl, true, R"(
@ -5859,7 +5865,7 @@ Experimental data deduplication for SELECT queries based on part UUIDs
// Please add settings related to formats in Core/FormatFactorySettings.h, move obsolete settings to OBSOLETE_SETTINGS and obsolete format settings to OBSOLETE_FORMAT_SETTINGS.
#define OBSOLETE_SETTINGS(M, ALIAS) \
/** Obsolete settings that do nothing but left for compatibility reasons. Remove each one after half a year of obsolescence. */ \
/** Obsolete settings which are kept around for compatibility reasons. They have no effect anymore. */ \
MAKE_OBSOLETE(M, Bool, update_insert_deduplication_token_in_dependent_materialized_views, 0) \
MAKE_OBSOLETE(M, UInt64, max_memory_usage_for_all_queries, 0) \
MAKE_OBSOLETE(M, UInt64, multiple_joins_rewriter_version, 0) \

3
src/Core/SettingsChangesHistory.cpp
View File

@ -64,6 +64,8 @@ static std::initializer_list<std::pair<ClickHouseVersion, SettingsChangesHistory
},
{"24.11",
{
{"allow_suspicious_types_in_group_by", true, false, "Don't allow Variant/Dynamic types in GROUP BY by default"},
{"allow_suspicious_types_in_order_by", true, false, "Don't allow Variant/Dynamic types in ORDER BY by default"},
{"distributed_cache_discard_connection_if_unread_data", true, true, "New setting"},
{"filesystem_cache_enable_background_download_for_metadata_files_in_packed_storage", true, true, "New setting"},
{"filesystem_cache_enable_background_download_during_fetch", true, true, "New setting"},
@ -78,6 +80,7 @@ static std::initializer_list<std::pair<ClickHouseVersion, SettingsChangesHistory
},
{"24.10",
{
{"enable_job_stack_trace", false, true, "Enable by default collecting stack traces from job's scheduling."},
{"query_metric_log_interval", 0, -1, "New setting."},
{"enforce_strict_identifier_format", false, false, "New setting."},
{"enable_parsing_to_custom_serialization", false, true, "New setting"},

9
src/DataTypes/DataTypeObject.cpp
View File

@ -233,6 +233,15 @@ MutableColumnPtr DataTypeObject::createColumn() const
return ColumnObject::create(std::move(typed_path_columns), max_dynamic_paths, max_dynamic_types);
}
void DataTypeObject::forEachChild(const ChildCallback & callback) const
{
for (const auto & [path, type] : typed_paths)
{
callback(*type);
type->forEachChild(callback);
}
}
namespace
{

2
src/DataTypes/DataTypeObject.h
View File

@ -50,6 +50,8 @@ public:
bool equals(const IDataType & rhs) const override;
void forEachChild(const ChildCallback &) const override;
bool hasDynamicSubcolumnsData() const override { return true; }
std::unique_ptr<SubstreamData> getDynamicSubcolumnData(std::string_view subcolumn_name, const SubstreamData & data, bool throw_if_null) const override;

2
src/Databases/enableAllExperimentalSettings.cpp
View File

@ -32,6 +32,8 @@ void enableAllExperimentalSettings(ContextMutablePtr context)
context->setSetting("allow_suspicious_low_cardinality_types", 1);
context->setSetting("allow_suspicious_fixed_string_types", 1);
context->setSetting("allow_suspicious_types_in_group_by", 1);
context->setSetting("allow_suspicious_types_in_order_by", 1);
context->setSetting("allow_suspicious_indices", 1);
context->setSetting("allow_suspicious_codecs", 1);
context->setSetting("allow_hyperscan", 1);

58
src/Interpreters/ExpressionAnalyzer.cpp
View File

@ -105,6 +105,8 @@ namespace Setting
extern const SettingsBool query_plan_aggregation_in_order;
extern const SettingsBool query_plan_read_in_order;
extern const SettingsUInt64 use_index_for_in_with_subqueries_max_values;
extern const SettingsBool allow_suspicious_types_in_group_by;
extern const SettingsBool allow_suspicious_types_in_order_by;
}
@ -118,6 +120,7 @@ namespace ErrorCodes
extern const int NOT_IMPLEMENTED;
extern const int UNKNOWN_IDENTIFIER;
extern const int UNKNOWN_TYPE_OF_AST_NODE;
extern const int ILLEGAL_COLUMN;
}
namespace
@ -1368,6 +1371,7 @@ bool SelectQueryExpressionAnalyzer::appendGroupBy(ExpressionActionsChain & chain
ExpressionActionsChain::Step & step = chain.lastStep(columns_after_join);
ASTs asts = select_query->groupBy()->children;
NameSet group_by_keys;
if (select_query->group_by_with_grouping_sets)
{
for (const auto & ast : asts)
@ -1375,6 +1379,7 @@ bool SelectQueryExpressionAnalyzer::appendGroupBy(ExpressionActionsChain & chain
for (const auto & ast_element : ast->children)
{
step.addRequiredOutput(ast_element->getColumnName());
group_by_keys.insert(ast_element->getColumnName());
getRootActions(ast_element, only_types, step.actions()->dag);
}
}
@ -1384,10 +1389,17 @@ bool SelectQueryExpressionAnalyzer::appendGroupBy(ExpressionActionsChain & chain
for (const auto & ast : asts)
{
step.addRequiredOutput(ast->getColumnName());
group_by_keys.insert(ast->getColumnName());
getRootActions(ast, only_types, step.actions()->dag);
}
}
for (const auto & result_column : step.getResultColumns())
{
if (group_by_keys.contains(result_column.name))
validateGroupByKeyType(result_column.type);
}
if (optimize_aggregation_in_order)
{
for (auto & child : asts)
@ -1402,6 +1414,26 @@ bool SelectQueryExpressionAnalyzer::appendGroupBy(ExpressionActionsChain & chain
return true;
}
void SelectQueryExpressionAnalyzer::validateGroupByKeyType(const DB::DataTypePtr & key_type) const
{
if (getContext()->getSettingsRef()[Setting::allow_suspicious_types_in_group_by])
return;
auto check = [](const IDataType & type)
{
if (isDynamic(type) || isVariant(type))
throw Exception(
ErrorCodes::ILLEGAL_COLUMN,
"Data types Variant/Dynamic are not allowed in GROUP BY keys, because it can lead to unexpected results. "
"Consider using a subcolumn with a specific data type instead (for example 'column.Int64' or 'json.some.path.:Int64' if "
"its a JSON path subcolumn) or casting this column to a specific data type. "
"Set setting allow_suspicious_types_in_group_by = 1 in order to allow it");
};
check(*key_type);
key_type->forEachChild(check);
}
void SelectQueryExpressionAnalyzer::appendAggregateFunctionsArguments(ExpressionActionsChain & chain, bool only_types)
{
const auto * select_query = getAggregatingQuery();
@ -1599,6 +1631,12 @@ ActionsAndProjectInputsFlagPtr SelectQueryExpressionAnalyzer::appendOrderBy(
with_fill = true;
}
for (const auto & result_column : step.getResultColumns())
{
if (order_by_keys.contains(result_column.name))
validateOrderByKeyType(result_column.type);
}
if (auto interpolate_list = select_query->interpolate())
{
@ -1664,6 +1702,26 @@ ActionsAndProjectInputsFlagPtr SelectQueryExpressionAnalyzer::appendOrderBy(
return actions;
}
void SelectQueryExpressionAnalyzer::validateOrderByKeyType(const DataTypePtr & key_type) const
{
if (getContext()->getSettingsRef()[Setting::allow_suspicious_types_in_order_by])
return;
auto check = [](const IDataType & type)
{
if (isDynamic(type) || isVariant(type))
throw Exception(
ErrorCodes::ILLEGAL_COLUMN,
"Data types Variant/Dynamic are not allowed in ORDER BY keys, because it can lead to unexpected results. "
"Consider using a subcolumn with a specific data type instead (for example 'column.Int64' or 'json.some.path.:Int64' if "
"its a JSON path subcolumn) or casting this column to a specific data type. "
"Set setting allow_suspicious_types_in_order_by = 1 in order to allow it");
};
check(*key_type);
key_type->forEachChild(check);
}
bool SelectQueryExpressionAnalyzer::appendLimitBy(ExpressionActionsChain & chain, bool only_types)
{
const auto * select_query = getSelectQuery();

2
src/Interpreters/ExpressionAnalyzer.h
View File

@ -396,6 +396,7 @@ private:
ActionsAndProjectInputsFlagPtr appendPrewhere(ExpressionActionsChain & chain, bool only_types);
bool appendWhere(ExpressionActionsChain & chain, bool only_types);
bool appendGroupBy(ExpressionActionsChain & chain, bool only_types, bool optimize_aggregation_in_order, ManyExpressionActions &);
void validateGroupByKeyType(const DataTypePtr & key_type) const;
void appendAggregateFunctionsArguments(ExpressionActionsChain & chain, bool only_types);
void appendWindowFunctionsArguments(ExpressionActionsChain & chain, bool only_types);
@ -408,6 +409,7 @@ private:
bool appendHaving(ExpressionActionsChain & chain, bool only_types);
/// appendSelect
ActionsAndProjectInputsFlagPtr appendOrderBy(ExpressionActionsChain & chain, bool only_types, bool optimize_read_in_order, ManyExpressionActions &);
void validateOrderByKeyType(const DataTypePtr & key_type) const;
bool appendLimitBy(ExpressionActionsChain & chain, bool only_types);
/// appendProjectResult
};

5
src/Processors/Formats/Impl/ParquetMetadataInputFormat.cpp
View File

@ -92,8 +92,9 @@ static NamesAndTypesList getHeaderForParquetMetadata()
std::make_shared<DataTypeNullable>(std::make_shared<DataTypeString>()),
std::make_shared<DataTypeNullable>(std::make_shared<DataTypeString>())},
Names{"num_values", "null_count", "distinct_count", "min", "max"}),
DataTypeFactory::instance().get("Bool"),
},
Names{"name", "path", "total_compressed_size", "total_uncompressed_size", "have_statistics", "statistics"}))},
Names{"name", "path", "total_compressed_size", "total_uncompressed_size", "have_statistics", "statistics", "have_bloom_filter"}))},
Names{"num_columns", "num_rows", "total_uncompressed_size", "total_compressed_size", "columns"}))},
};
return names_and_types;
@ -350,6 +351,8 @@ void ParquetMetadataInputFormat::fillColumnChunksMetadata(const std::unique_ptr<
fillColumnStatistics(column_chunk_metadata->statistics(), tuple_column.getColumn(5), row_group_metadata->schema()->Column(column_i)->type_length());
else
tuple_column.getColumn(5).insertDefault();
bool have_bloom_filter = column_chunk_metadata->bloom_filter_offset().has_value();
assert_cast<ColumnUInt8 &>(tuple_column.getColumn(6)).insertValue(have_bloom_filter);
}
array_column.getOffsets().push_back(tuple_column.size());
}

2
src/Storages/AlterCommands.cpp
View File

@ -1488,7 +1488,7 @@ void AlterCommands::validate(const StoragePtr & table, ContextPtr context) const
if (command.to_remove == AlterCommand::RemoveProperty::CODEC && column_from_table.codec == nullptr)
throw Exception(
ErrorCodes::BAD_ARGUMENTS,
"Column {} doesn't have TTL, cannot remove it",
"Column {} doesn't have CODEC, cannot remove it",
backQuote(column_name));
if (command.to_remove == AlterCommand::RemoveProperty::COMMENT && column_from_table.comment.empty())
throw Exception(

12
src/Storages/KeyDescription.cpp
View File

@ -151,6 +151,18 @@ KeyDescription KeyDescription::getSortingKeyFromAST(
throw Exception(ErrorCodes::DATA_TYPE_CANNOT_BE_USED_IN_KEY,
"Column {} with type {} is not allowed in key expression, it's not comparable",
backQuote(result.sample_block.getByPosition(i).name), result.data_types.back()->getName());
auto check = [&](const IDataType & type)
{
if (isDynamic(type) || isVariant(type))
throw Exception(
ErrorCodes::DATA_TYPE_CANNOT_BE_USED_IN_KEY,
"Column with type Variant/Dynamic is not allowed in key expression. Consider using a subcolumn with a specific data "
"type instead (for example 'column.Int64' or 'json.some.path.:Int64' if its a JSON path subcolumn) or casting this column to a specific data type");
};
check(*result.data_types.back());
result.data_types.back()->forEachChild(check);
}
return result;

1
tests/queries/0_stateless/01825_new_type_json_10.sql
View File

@ -1,6 +1,7 @@
-- Tags: no-fasttest
SET allow_experimental_json_type = 1;
SET allow_suspicious_types_in_order_by = 1;
DROP TABLE IF EXISTS t_json_10;
CREATE TABLE t_json_10 (o JSON) ENGINE = Memory;

6
tests/queries/0_stateless/01825_new_type_json_11.sh
View File

@ -57,8 +57,8 @@ $CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(obj)) as
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(obj.key_1[]))) as path FROM t_json_11 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(arrayJoin(obj.key_1[].key_3[])))) as path FROM t_json_11 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(arrayJoin(arrayJoin(obj.key_1[].key_3[].key_4[]))))) as path FROM t_json_11 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT obj FROM t_json_11 ORDER BY obj.id FORMAT JSONEachRow"
$CLICKHOUSE_CLIENT -q "SELECT obj.key_1[].key_3 FROM t_json_11 ORDER BY obj.id FORMAT JSONEachRow"
$CLICKHOUSE_CLIENT -q "SELECT obj.key_1[].key_3[].key_4[].key_5, obj.key_1[].key_3[].key_7 FROM t_json_11 ORDER BY obj.id"
$CLICKHOUSE_CLIENT -q "SELECT obj FROM t_json_11 ORDER BY obj.id FORMAT JSONEachRow" --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "SELECT obj.key_1[].key_3 FROM t_json_11 ORDER BY obj.id FORMAT JSONEachRow" --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "SELECT obj.key_1[].key_3[].key_4[].key_5, obj.key_1[].key_3[].key_7 FROM t_json_11 ORDER BY obj.id" --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "DROP TABLE t_json_11;"

4
tests/queries/0_stateless/01825_new_type_json_12.sh
View File

@ -47,8 +47,8 @@ $CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(obj)) as
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(obj.key_0[]))) as path FROM t_json_12 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(arrayJoin(obj.key_0[].key_1[])))) as path FROM t_json_12 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(arrayJoin(arrayJoin(obj.key_0[].key_1[].key_3[]))))) as path FROM t_json_12 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT obj FROM t_json_12 ORDER BY obj.id FORMAT JSONEachRow" --output_format_json_named_tuples_as_objects 1
$CLICKHOUSE_CLIENT -q "SELECT obj FROM t_json_12 ORDER BY obj.id FORMAT JSONEachRow" --output_format_json_named_tuples_as_objects 1 --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "SELECT obj.key_0[].key_1[].key_3[].key_4, obj.key_0[].key_1[].key_3[].key_5, \
obj.key_0[].key_1[].key_3[].key_6, obj.key_0[].key_1[].key_3[].key_7 FROM t_json_12 ORDER BY obj.id"
obj.key_0[].key_1[].key_3[].key_6, obj.key_0[].key_1[].key_3[].key_7 FROM t_json_12 ORDER BY obj.id" --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "DROP TABLE t_json_12;"

4
tests/queries/0_stateless/01825_new_type_json_13.sh
View File

@ -39,12 +39,12 @@ EOF
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(obj)) as path FROM t_json_13 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(obj.key1[]))) as path FROM t_json_13 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT obj FROM t_json_13 ORDER BY obj.id FORMAT JSONEachRow" --output_format_json_named_tuples_as_objects 1
$CLICKHOUSE_CLIENT -q "SELECT obj FROM t_json_13 ORDER BY obj.id FORMAT JSONEachRow" --output_format_json_named_tuples_as_objects 1 --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "SELECT \
obj.key_1.key_2.key_3.key_8, \
obj.key_1.key_2.key_3.key_4.key_5, \
obj.key_1.key_2.key_3.key_4.key_6, \
obj.key_1.key_2.key_3.key_4.key_7 \
FROM t_json_13 ORDER BY obj.id"
FROM t_json_13 ORDER BY obj.id" --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "DROP TABLE t_json_13;"

2
tests/queries/0_stateless/01825_new_type_json_6.sh
View File

@ -54,6 +54,6 @@ EOF
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(data)) as path FROM t_json_6 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(data.out[]))) as path FROM t_json_6 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(arrayJoin(arrayJoin(data.out[].outputs[])))) as path FROM t_json_6 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT data.key, data.out[].type, data.out[].value, data.out[].outputs[].index, data.out[].outputs[].n FROM t_json_6 ORDER BY data.key"
$CLICKHOUSE_CLIENT -q "SELECT data.key, data.out[].type, data.out[].value, data.out[].outputs[].index, data.out[].outputs[].n FROM t_json_6 ORDER BY data.key" --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "DROP TABLE t_json_6;"

2
tests/queries/0_stateless/01825_new_type_json_7.sh
View File

@ -25,6 +25,6 @@ cat <<EOF | $CLICKHOUSE_CLIENT -q "INSERT INTO t_json_7 FORMAT JSONAsObject"
EOF
$CLICKHOUSE_CLIENT -q "SELECT DISTINCT arrayJoin(JSONAllPathsWithTypes(data)) as path FROM t_json_7 order by path;"
$CLICKHOUSE_CLIENT -q "SELECT data.key, data.categories FROM t_json_7 ORDER BY data.key"
$CLICKHOUSE_CLIENT -q "SELECT data.key, data.categories FROM t_json_7 ORDER BY data.key" --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "DROP TABLE t_json_7;"

4
tests/queries/0_stateless/01825_new_type_json_ghdata.sh
View File

@ -16,12 +16,12 @@ ${CLICKHOUSE_CLIENT} -q "SELECT count() FROM ghdata WHERE NOT ignore(*)"
${CLICKHOUSE_CLIENT} -q \
"SELECT data.repo.name, count() AS stars FROM ghdata \
WHERE data.type = 'WatchEvent' GROUP BY data.repo.name ORDER BY stars DESC, data.repo.name LIMIT 5"
WHERE data.type = 'WatchEvent' GROUP BY data.repo.name ORDER BY stars DESC, data.repo.name LIMIT 5" --allow_suspicious_types_in_order_by 1 --allow_suspicious_types_in_group_by 1
${CLICKHOUSE_CLIENT} --enable_analyzer=1 -q \
"SELECT data.payload.commits[].author.name AS name, count() AS c FROM ghdata \
ARRAY JOIN data.payload.commits[].author.name \
GROUP BY name ORDER BY c DESC, name LIMIT 5"
GROUP BY name ORDER BY c DESC, name LIMIT 5" --allow_suspicious_types_in_order_by 1 --allow_suspicious_types_in_group_by 1
${CLICKHOUSE_CLIENT} -q "SELECT max(data.payload.pull_request.assignees[].size0) FROM ghdata"

3
tests/queries/0_stateless/01825_new_type_json_in_array.sql
View File

@ -2,6 +2,9 @@
SET allow_experimental_json_type = 1;
SET allow_experimental_analyzer = 1;
SET allow_suspicious_types_in_order_by = 1;
SET allow_suspicious_types_in_group_by = 1;
DROP TABLE IF EXISTS t_json_array;
CREATE TABLE t_json_array (id UInt32, arr Array(JSON)) ENGINE = MergeTree ORDER BY id;

2
tests/queries/0_stateless/01825_new_type_json_insert_select.sql
View File

@ -1,6 +1,8 @@
-- Tags: no-fasttest
SET allow_experimental_json_type = 1;
SET allow_suspicious_types_in_order_by = 1;
SET allow_suspicious_types_in_order_by = 1;
DROP TABLE IF EXISTS type_json_src;
DROP TABLE IF EXISTS type_json_dst;

2
tests/queries/0_stateless/02421_new_type_json_async_insert.sh
View File

@ -17,5 +17,5 @@ $CLICKHOUSE_CLIENT --async_insert=1 --wait_for_async_insert=1 -q 'INSERT INTO t_
wait
$CLICKHOUSE_CLIENT -q "SELECT data.k1 FROM t_json_async_insert ORDER BY data.k1"
$CLICKHOUSE_CLIENT -q "SELECT data.k1 FROM t_json_async_insert ORDER BY data.k1" --allow_suspicious_types_in_order_by 1
$CLICKHOUSE_CLIENT -q "DROP TABLE IF EXISTS t_json_async_insert"

2
tests/queries/0_stateless/02494_query_cache_normalize_ast.sql
View File

@ -7,7 +7,7 @@ SYSTEM DROP QUERY CACHE;
-- Run query whose result gets cached in the query cache.
-- Besides "use_query_cache", pass two more knobs (one QC-specific knob and one non-QC-specific knob). We just care
-- *that* they are passed and not about their effect.
SELECT 1 SETTINGS use_query_cache = true, query_cache_store_results_of_queries_with_nondeterministic_functions = true, max_threads = 16;
SELECT 1 SETTINGS use_query_cache = true, query_cache_nondeterministic_function_handling = 'save', max_threads = 16;
-- Check that entry in QC exists
SELECT COUNT(*) FROM system.query_cache;

70
tests/queries/0_stateless/02718_parquet_metadata_format.reference
View File

@ -78,7 +78,8 @@
"distinct_count": null,
"min": "0",
"max": "999"
}
},
"have_bloom_filter": false
},
{
"name": "str",
@ -92,7 +93,8 @@
"distinct_count": null,
"min": "Hello0",
"max": "Hello999"
}
},
"have_bloom_filter": false
},
{
"name": "mod",
@ -106,7 +108,8 @@
"distinct_count": null,
"min": "0",
"max": "8"
}
},
"have_bloom_filter": false
}
]
},
@ -128,7 +131,8 @@
"distinct_count": null,
"min": "0",
"max": "999"
}
},
"have_bloom_filter": false
},
{
"name": "str",
@ -142,7 +146,8 @@
"distinct_count": null,
"min": "Hello0",
"max": "Hello999"
}
},
"have_bloom_filter": false
},
{
"name": "mod",
@ -156,7 +161,8 @@
"distinct_count": null,
"min": "0",
"max": "8"
}
},
"have_bloom_filter": false
}
]
}
@ -223,3 +229,55 @@
}
1
1
{
"num_columns": "1",
"num_rows": "5",
"num_row_groups": "1",
"format_version": "1.0",
"metadata_size": "267",
"total_uncompressed_size": "105",
"total_compressed_size": "128",
"columns": [
{
"name": "ipv6",
"path": "ipv6",
"max_definition_level": "0",
"max_repetition_level": "0",
"physical_type": "FIXED_LEN_BYTE_ARRAY",
"logical_type": "None",
"compression": "GZIP",
"total_uncompressed_size": "105",
"total_compressed_size": "128",
"space_saved": "-21.9%",
"encodings": [
"PLAIN",
"BIT_PACKED"
]
}
],
"row_groups": [
{
"num_columns": "1",
"num_rows": "5",
"total_uncompressed_size": "105",
"total_compressed_size": "128",
"columns": [
{
"name": "ipv6",
"path": "ipv6",
"total_compressed_size": "128",
"total_uncompressed_size": "105",
"have_statistics": true,
"statistics": {
"num_values": "5",
"null_count": "0",
"distinct_count": null,
"min": "27 32 150 125 17 250 66 31 157 44 75 218 51 50 19 144 ",
"max": "154 31 90 141 15 7 68 47 190 29 121 145 188 162 234 154 "
},
"have_bloom_filter": true
}
]
}
]
}

1
tests/queries/0_stateless/02718_parquet_metadata_format.sh
View File

@ -17,3 +17,4 @@ $CLICKHOUSE_LOCAL -q "select some_column from file('$CURDIR/data_parquet/02718_d
$CLICKHOUSE_LOCAL -q "select num_columns from file('$CURDIR/data_parquet/02718_data.parquet', ParquetMetadata, 'num_columns Array(UInt32)')" 2>&1 | grep -c "BAD_ARGUMENTS"
$CLICKHOUSE_LOCAL -q "select * from file('$CURDIR/data_parquet/ipv6_bloom_filter.gz.parquet', ParquetMetadata) format JSONEachRow" | python3 -m json.tool

1
tests/queries/0_stateless/02989_variant_comparison.sql
View File

@ -1,4 +1,5 @@
set allow_experimental_variant_type=1;
set allow_suspicious_types_in_order_by=1;
create table test (v1 Variant(String, UInt64, Array(UInt32)), v2 Variant(String, UInt64, Array(UInt32))) engine=Memory;

1
tests/queries/0_stateless/03035_dynamic_sorting.sql
View File

@ -1,4 +1,5 @@
set allow_experimental_dynamic_type = 1;
set allow_suspicious_types_in_order_by=1;
drop table if exists test;
create table test (d1 Dynamic(max_types=2), d2 Dynamic(max_types=2)) engine=Memory;

1
tests/queries/0_stateless/03036_dynamic_read_shared_subcolumns_small.sql.j2
View File

@ -1,6 +1,7 @@
set allow_experimental_variant_type = 1;
set use_variant_as_common_type = 1;
set allow_experimental_dynamic_type = 1;
set allow_suspicious_types_in_order_by = 1;
drop table if exists test;

1
tests/queries/0_stateless/03036_dynamic_read_subcolumns_small.sql.j2
View File

@ -1,6 +1,7 @@
set allow_experimental_variant_type = 1;
set use_variant_as_common_type = 1;
set allow_experimental_dynamic_type = 1;
set allow_suspicious_types_in_order_by = 1;
drop table if exists test;

4
tests/queries/0_stateless/03096_variant_in_primary_key.reference
View File

@ -1,4 +0,0 @@
1 str_1
1 str_2
1 1
1 2

7
tests/queries/0_stateless/03096_variant_in_primary_key.sql
View File

@ -1,7 +0,0 @@
set allow_experimental_variant_type=1;
drop table if exists test;
create table test (id UInt64, v Variant(UInt64, String)) engine=MergeTree order by (id, v);
insert into test values (1, 1), (1, 'str_1'), (1, 2), (1, 'str_2');
select * from test;
drop table test;

1
tests/queries/0_stateless/03150_dynamic_type_mv_insert.sql
View File

@ -1,4 +1,5 @@
SET allow_experimental_dynamic_type=1;
SET allow_suspicious_types_in_order_by=1;
DROP TABLE IF EXISTS null_table;
CREATE TABLE null_table

5
tests/queries/0_stateless/03151_dynamic_type_scale_max_types.sql
View File

@ -1,5 +1,6 @@
SET allow_experimental_dynamic_type=1;
set min_compress_block_size = 585572, max_compress_block_size = 373374, max_block_size = 60768, max_joined_block_size_rows = 18966, max_insert_threads = 5, max_threads = 50, max_read_buffer_size = 708232, connect_timeout_with_failover_ms = 2000, connect_timeout_with_failover_secure_ms = 3000, idle_connection_timeout = 36000, use_uncompressed_cache = true, stream_like_engine_allow_direct_select = true, replication_wait_for_inactive_replica_timeout = 30, compile_aggregate_expressions = false, min_count_to_compile_aggregate_expression = 0, compile_sort_description = false, group_by_two_level_threshold = 1000000, group_by_two_level_threshold_bytes = 12610083, enable_memory_bound_merging_of_aggregation_results = false, min_chunk_bytes_for_parallel_parsing = 18769830, merge_tree_coarse_index_granularity = 12, min_bytes_to_use_direct_io = 10737418240, min_bytes_to_use_mmap_io = 10737418240, log_queries = true, insert_quorum_timeout = 60000, merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability = 0.05000000074505806, http_response_buffer_size = 294986, fsync_metadata = true, http_send_timeout = 60., http_receive_timeout = 60., opentelemetry_start_trace_probability = 0.10000000149011612, max_bytes_before_external_group_by = 1, max_bytes_before_external_sort = 10737418240, max_bytes_before_remerge_sort = 1326536545, max_untracked_memory = 1048576, memory_profiler_step = 1048576, log_comment = '03151_dynamic_type_scale_max_types.sql', send_logs_level = 'fatal', prefer_localhost_replica = false, optimize_read_in_order = false, optimize_aggregation_in_order = true, aggregation_in_order_max_block_bytes = 27069500, read_in_order_two_level_merge_threshold = 75, allow_introspection_functions = true, database_atomic_wait_for_drop_and_detach_synchronously = true, remote_filesystem_read_method = 'read', local_filesystem_read_prefetch = true, remote_filesystem_read_prefetch = false, merge_tree_compact_parts_min_granules_to_multibuffer_read = 119, async_insert_busy_timeout_max_ms = 5000, read_from_filesystem_cache_if_exists_otherwise_bypass_cache = true, filesystem_cache_segments_batch_size = 10, use_page_cache_for_disks_without_file_cache = true, page_cache_inject_eviction = true, allow_prefetched_read_pool_for_remote_filesystem = false, filesystem_prefetch_step_marks = 50, filesystem_prefetch_min_bytes_for_single_read_task = 16777216, filesystem_prefetch_max_memory_usage = 134217728, filesystem_prefetches_limit = 10, optimize_sorting_by_input_stream_properties = false, allow_experimental_dynamic_type = true, session_timezone = 'Africa/Khartoum', prefer_warmed_unmerged_parts_seconds = 2;
SET allow_experimental_dynamic_type = 1;
SET allow_suspicious_types_in_order_by = 1;
SET optimize_read_in_order = 1;
drop table if exists to_table;

1
tests/queries/0_stateless/03158_dynamic_type_from_variant.sql
View File

@ -1,5 +1,6 @@
SET allow_experimental_dynamic_type=1;
SET allow_experimental_variant_type=1;
SET allow_suspicious_types_in_order_by=1;
CREATE TABLE test_variable (v Variant(String, UInt32, IPv6, Bool, DateTime64)) ENGINE = Memory;
CREATE TABLE test_dynamic (d Dynamic) ENGINE = Memory;

2
tests/queries/0_stateless/03159_dynamic_type_all_types.sql
View File

@ -3,7 +3,7 @@
SET allow_experimental_dynamic_type=1;
SET allow_experimental_variant_type=1;
SET allow_suspicious_low_cardinality_types=1;
SET allow_suspicious_types_in_order_by=1;
CREATE TABLE t (d Dynamic(max_types=254)) ENGINE = Memory;
-- Integer types: signed and unsigned integers (UInt8, UInt16, UInt32, UInt64, UInt128, UInt256, Int8, Int16, Int32, Int64, Int128, Int256)

1
tests/queries/0_stateless/03162_dynamic_type_nested.sql
View File

@ -1,4 +1,5 @@
SET allow_experimental_dynamic_type=1;
SET allow_suspicious_types_in_order_by=1;
CREATE TABLE t (d Dynamic) ENGINE = Memory;

1
tests/queries/0_stateless/03163_dynamic_as_supertype.sql
View File

@ -1,4 +1,5 @@
SET allow_experimental_dynamic_type=1;
SET allow_suspicious_types_in_order_by=1;
SELECT if(number % 2, number::Dynamic(max_types=3), ('str_' || toString(number))::Dynamic(max_types=2)) AS d, toTypeName(d), dynamicType(d) FROM numbers(4);
CREATE TABLE dynamic_test_1 (d Dynamic(max_types=3)) ENGINE = Memory;
INSERT INTO dynamic_test_1 VALUES ('str_1'), (42::UInt64);

1
tests/queries/0_stateless/03214_json_typed_dynamic_path.sql
View File

@ -1,6 +1,7 @@
-- Tags: no-fasttest
set allow_experimental_json_type = 1;
set allow_experimental_dynamic_type = 1;
drop table if exists test;
create table test (json JSON(a Dynamic)) engine=MergeTree order by tuple() settings min_rows_for_wide_part=1, min_bytes_for_wide_part=1;
insert into test select '{"a" : 42}';

1
tests/queries/0_stateless/03228_dynamic_serializations_uninitialized_value.sql
View File

@ -1,4 +1,5 @@
set allow_experimental_dynamic_type=1;
set allow_suspicious_types_in_group_by=1;
set cast_keep_nullable=1;
SELECT toFixedString('str', 3), 3, CAST(if(1 = 0, toInt8(3), NULL), 'Int32') AS x from numbers(10) GROUP BY GROUPING SETS ((CAST(toInt32(1), 'Int32')), ('str', 3), (CAST(toFixedString('str', 3), 'Dynamic')), (CAST(toFixedString(toFixedString('str', 3), 3), 'Dynamic')));

1
tests/queries/0_stateless/03231_dynamic_incomplete_type_insert_bug.sql
View File

@ -1,4 +1,5 @@
SET allow_experimental_dynamic_type = 1;
SET allow_suspicious_types_in_order_by = 1;
DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (c0 Array(Dynamic)) ENGINE = MergeTree() ORDER BY tuple();
INSERT INTO t1 (c0) VALUES ([]);

10
tests/queries/0_stateless/03231_dynamic_not_safe_primary_key.sql
View File

@ -1,10 +0,0 @@
SET allow_experimental_dynamic_type = 1;
DROP TABLE IF EXISTS t0;
DROP TABLE IF EXISTS t1;
CREATE TABLE t0 (c0 Int) ENGINE = AggregatingMergeTree() ORDER BY (c0);
CREATE TABLE t1 (c0 Array(Dynamic), c1 Int) ENGINE = MergeTree() ORDER BY (c0);
INSERT INTO t1 (c0, c1) VALUES ([18446717433683171873], 13623876564923702671), ([-4], 6111684076076982207);
SELECT 1 FROM t0 FINAL JOIN t1 ON TRUE;
DROP TABLE t0;
DROP TABLE t1;

2
tests/queries/0_stateless/03231_dynamic_uniq_group_by.sql
View File

@ -1,4 +1,6 @@
set allow_experimental_dynamic_type = 1;
set allow_suspicious_types_in_group_by = 1;
set allow_suspicious_types_in_order_by = 1;
drop table if exists test;
create table test (d Dynamic(max_types=2)) engine=Memory;
insert into test values (42), ('Hello'), ([1,2,3]), ('2020-01-01');

184
tests/queries/0_stateless/03231_dynamic_variant_in_order_by_group_by.reference Normal file
View File

@ -0,0 +1,184 @@
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
[0]
[1]
[2]
[3]
[4]
{'str':0}
{'str':1}
{'str':2}
{'str':3}
{'str':4}
0
1
2
3
4
\N
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
[0]
[1]
[2]
[3]
[4]
{'str':0}
{'str':1}
{'str':2}
{'str':3}
{'str':4}
0
1
2
3
4
\N
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
[0]
[1]
[2]
[3]
[4]
{'str':0}
{'str':1}
{'str':2}
{'str':3}
{'str':4}
0
1
2
3
4
\N
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
0
1
2
3
4
[0]
[1]
[2]
[3]
[4]
{'str':0}
{'str':1}
{'str':2}
{'str':3}
{'str':4}
0
1
2
3
4
\N

166
tests/queries/0_stateless/03231_dynamic_variant_in_order_by_group_by.sql Normal file
View File

@ -0,0 +1,166 @@
set allow_experimental_variant_type=1;
set allow_experimental_dynamic_type=1;
drop table if exists test;
create table test (d Dynamic) engine=MergeTree order by d; -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=MergeTree order by tuple(d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=MergeTree order by array(d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=MergeTree order by map('str', d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=MergeTree order by tuple() primary key d; -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=MergeTree order by tuple() partition by d; -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=MergeTree order by tuple() partition by tuple(d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=MergeTree order by tuple() partition by array(d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=MergeTree order by tuple() partition by map('str', d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by d; -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by tuple(d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by array(d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by map('str', d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by tuple() primary key d; -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by tuple() partition by d; -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by tuple() partition by tuple(d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by tuple() partition by array(d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Variant(UInt64)) engine=MergeTree order by tuple() partition by map('str', d); -- {serverError DATA_TYPE_CANNOT_BE_USED_IN_KEY}
create table test (d Dynamic) engine=Memory;
insert into test select * from numbers(5);
set allow_experimental_analyzer=1;
set allow_suspicious_types_in_group_by=1;
set allow_suspicious_types_in_order_by=0;
select * from test order by d; -- {serverError ILLEGAL_COLUMN}
select * from test order by tuple(d); -- {serverError ILLEGAL_COLUMN}
select * from test order by array(d); -- {serverError ILLEGAL_COLUMN}
select * from test order by map('str', d); -- {serverError ILLEGAL_COLUMN}
set allow_suspicious_types_in_group_by=0;
set allow_suspicious_types_in_order_by=1;
select * from test group by d; -- {serverError ILLEGAL_COLUMN}
select * from test group by tuple(d); -- {serverError ILLEGAL_COLUMN}
select array(d) from test group by array(d); -- {serverError ILLEGAL_COLUMN}
select map('str', d) from test group by map('str', d); -- {serverError ILLEGAL_COLUMN}
select * from test group by grouping sets ((d), ('str')); -- {serverError ILLEGAL_COLUMN}
set allow_suspicious_types_in_group_by=1;
set allow_suspicious_types_in_order_by=1;
select * from test order by d;
select * from test order by tuple(d);
select * from test order by array(d);
select * from test order by map('str', d);
select * from test group by d order by all;
select * from test group by tuple(d) order by all;
select array(d) from test group by array(d) order by all;
select map('str', d) from test group by map('str', d) order by all;
select * from test group by grouping sets ((d), ('str')) order by all;
set allow_experimental_analyzer=0;
set allow_suspicious_types_in_group_by=1;
set allow_suspicious_types_in_order_by=0;
select * from test order by d; -- {serverError ILLEGAL_COLUMN}
select * from test order by tuple(d); -- {serverError ILLEGAL_COLUMN}
select * from test order by array(d); -- {serverError ILLEGAL_COLUMN}
select * from test order by map('str', d); -- {serverError ILLEGAL_COLUMN}
set allow_suspicious_types_in_group_by=0;
set allow_suspicious_types_in_order_by=1;
select * from test group by d; -- {serverError ILLEGAL_COLUMN}
select * from test group by tuple(d); -- {serverError ILLEGAL_COLUMN}
select array(d) from test group by array(d); -- {serverError ILLEGAL_COLUMN}
select map('str', d) from test group by map('str', d); -- {serverError ILLEGAL_COLUMN}
select * from test group by grouping sets ((d), ('str')); -- {serverError ILLEGAL_COLUMN}
set allow_suspicious_types_in_group_by=1;
set allow_suspicious_types_in_order_by=1;
select * from test order by d;
select * from test order by tuple(d);
select * from test order by array(d);
select * from test order by map('str', d);
select * from test group by d order by all;
select * from test group by tuple(d) order by all;
select array(d) from test group by array(d) order by all;
select map('str', d) from test group by map('str', d) order by all;
select * from test group by grouping sets ((d), ('str')) order by all;
drop table test;
create table test (d Variant(UInt64)) engine=Memory;
insert into test select * from numbers(5);
set allow_experimental_analyzer=1;
set allow_suspicious_types_in_group_by=1;
set allow_suspicious_types_in_order_by=0;
select * from test order by d; -- {serverError ILLEGAL_COLUMN}
select * from test order by tuple(d); -- {serverError ILLEGAL_COLUMN}
select * from test order by array(d); -- {serverError ILLEGAL_COLUMN}
select * from test order by map('str', d); -- {serverError ILLEGAL_COLUMN}
set allow_suspicious_types_in_group_by=0;
set allow_suspicious_types_in_order_by=1;
select * from test group by d; -- {serverError ILLEGAL_COLUMN}
select * from test group by tuple(d); -- {serverError ILLEGAL_COLUMN}
select array(d) from test group by array(d); -- {serverError ILLEGAL_COLUMN}
select map('str', d) from test group by map('str', d); -- {serverError ILLEGAL_COLUMN}
select * from test group by grouping sets ((d), ('str')); -- {serverError ILLEGAL_COLUMN}
set allow_suspicious_types_in_group_by=1;
set allow_suspicious_types_in_order_by=1;
select * from test order by d;
select * from test order by tuple(d);
select * from test order by array(d);
select * from test order by map('str', d);
select * from test group by d order by all;
select * from test group by tuple(d) order by all;
select array(d) from test group by array(d) order by all;
select map('str', d) from test group by map('str', d) order by all;
select * from test group by grouping sets ((d), ('str')) order by all;
set allow_experimental_analyzer=0;
set allow_suspicious_types_in_group_by=1;
set allow_suspicious_types_in_order_by=0;
select * from test order by d; -- {serverError ILLEGAL_COLUMN}
select * from test order by tuple(d); -- {serverError ILLEGAL_COLUMN}
select * from test order by array(d); -- {serverError ILLEGAL_COLUMN}
select * from test order by map('str', d); -- {serverError ILLEGAL_COLUMN}
set allow_suspicious_types_in_group_by=0;
set allow_suspicious_types_in_order_by=1;
select * from test group by d; -- {serverError ILLEGAL_COLUMN}
select * from test group by tuple(d); -- {serverError ILLEGAL_COLUMN}
select array(d) from test group by array(d); -- {serverError ILLEGAL_COLUMN}
select map('str', d) from test group by map('str', d); -- {serverError ILLEGAL_COLUMN}
select * from test group by grouping sets ((d), ('str')); -- {serverError ILLEGAL_COLUMN}
set allow_suspicious_types_in_group_by=1;
set allow_suspicious_types_in_order_by=1;
select * from test order by d;
select * from test order by tuple(d);
select * from test order by array(d);
select * from test order by map('str', d);
select * from test group by d order by all;
select * from test group by tuple(d) order by all;
select array(d) from test group by array(d) order by all;
select map('str', d) from test group by map('str', d) order by all;
select * from test group by grouping sets ((d), ('str')) order by all;
drop table test;

0
tests/queries/0_stateless/03231_dynamic_not_safe_primary_key.reference → tests/queries/0_stateless/03261_json_hints_types_check.reference
View File

9
tests/queries/0_stateless/03261_json_hints_types_check.sql Normal file
View File

@ -0,0 +1,9 @@
set allow_experimental_json_type=1;
set allow_experimental_variant_type=0;
set allow_experimental_object_type=0;
select '{}'::JSON(a LowCardinality(Int128)); -- {serverError SUSPICIOUS_TYPE_FOR_LOW_CARDINALITY}
select '{}'::JSON(a FixedString(100000)); -- {serverError ILLEGAL_COLUMN}
select '{}'::JSON(a Variant(Int32)); -- {serverError ILLEGAL_COLUMN}
select '{}'::JSON(a Object('json')); -- {serverError ILLEGAL_COLUMN}

0
tests/queries/0_stateless/03261_variant_permutation_bug.reference Normal file
View File

6
tests/queries/0_stateless/03261_variant_permutation_bug.sql Normal file
View File

@ -0,0 +1,6 @@
set allow_experimental_variant_type=1;
create table test (x UInt64, d Variant(UInt64)) engine=Memory;
insert into test select number, null from numbers(200000);
select d from test order by d::String limit 32213 format Null;
drop table test;

1
tests/queries/0_stateless/03262_analyzer_materialized_view_in_with_cte.reference Normal file
View File

@ -0,0 +1 @@
1 2 \N test

63
tests/queries/0_stateless/03262_analyzer_materialized_view_in_with_cte.sql Normal file
View File

@ -0,0 +1,63 @@
SET allow_experimental_analyzer = 1;
DROP TABLE IF EXISTS mv_test;
DROP TABLE IF EXISTS mv_test_target;
DROP VIEW IF EXISTS mv_test_mv;
CREATE TABLE mv_test
(
`id` UInt64,
`ref_id` UInt64,
`final_id` Nullable(UInt64),
`display` String
)
ENGINE = Log;
CREATE TABLE mv_test_target
(
`id` UInt64,
`ref_id` UInt64,
`final_id` Nullable(UInt64),
`display` String
)
ENGINE = Log;
CREATE MATERIALIZED VIEW mv_test_mv TO mv_test_target
(
`id` UInt64,
`ref_id` UInt64,
`final_id` Nullable(UInt64),
`display` String
)
AS WITH
tester AS
(
SELECT
id,
ref_id,
final_id,
display
FROM mv_test
),
id_set AS
(
SELECT
display,
max(id) AS max_id
FROM mv_test
GROUP BY display
)
SELECT *
FROM tester
WHERE id IN (
SELECT max_id
FROM id_set
);
INSERT INTO mv_test ( id, ref_id, display) values ( 1, 2, 'test');
SELECT * FROM mv_test_target;
DROP VIEW mv_test_mv;
DROP TABLE mv_test_target;
DROP TABLE mv_test;

1
tests/queries/0_stateless/03263_analyzer_materialized_view_cte_nested.reference Normal file
View File

@ -0,0 +1 @@
2

19
tests/queries/0_stateless/03263_analyzer_materialized_view_cte_nested.sql Normal file
View File

@ -0,0 +1,19 @@
SET allow_experimental_analyzer = 1;
DROP TABLE IF EXISTS test_table;
DROP VIEW IF EXISTS test_mv;
CREATE TABLE test_table ENGINE = MergeTree ORDER BY tuple() AS SELECT 1 as col1;
CREATE MATERIALIZED VIEW test_mv ENGINE = MergeTree ORDER BY tuple() AS
WITH
subquery_on_source AS (SELECT col1 AS aliased FROM test_table),
output AS (SELECT * FROM test_table WHERE col1 IN (SELECT aliased FROM subquery_on_source))
SELECT * FROM output;
INSERT INTO test_table VALUES (2);
SELECT * FROM test_mv;
DROP VIEW test_mv;
DROP TABLE test_table;