Merge 7aadca13ba into 44b4bd38b9

Enable cluster table functions for DataLake Storages

docs/en/sql-reference/table-functions/deltalake.md

@@ -49,4 +49,4 @@ LIMIT 2
 **See Also**
 
 - [DeltaLake engine](/docs/en/engines/table-engines/integrations/deltalake.md)
-
+- [DeltaLake cluster table function](/docs/en/sql-reference/table-functions/deltalakeCluster.md)

docs/en/sql-reference/table-functions/deltalakeCluster.md

@@ -0,0 +1,30 @@
+---
+slug: /en/sql-reference/table-functions/deltalakeCluster
+sidebar_position: 46
+sidebar_label: deltaLakeCluster
+title: "deltaLakeCluster Table Function"
+---
+This is an extension to the [deltaLake](/docs/en/sql-reference/table-functions/deltalake.md) table function.
+
+Allows processing files from [Delta Lake](https://github.com/delta-io/delta) tables in Amazon S3 in parallel from many nodes in a specified cluster. On initiator it creates a connection to all nodes in the cluster and dispatches each file dynamically. On the worker node it asks the initiator about the next task to process and processes it. This is repeated until all tasks are finished.
+
+**Syntax**
+
+``` sql
+deltaLakeCluster(cluster_name, url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression])
+```
+
+**Arguments**
+
+- `cluster_name` — Name of a cluster that is used to build a set of addresses and connection parameters to remote and local servers.
+
+- Description of all other arguments coincides with description of arguments in equivalent [deltaLake](/docs/en/sql-reference/table-functions/deltalake.md) table function.
+
+**Returned value**
+
+A table with the specified structure for reading data from cluster in the specified Delta Lake table in S3.
+
+**See Also**
+
+- [deltaLake engine](/docs/en/engines/table-engines/integrations/deltalake.md)
+- [deltaLake table function](/docs/en/sql-reference/table-functions/deltalake.md)

docs/en/sql-reference/table-functions/hudi.md

@@ -29,4 +29,4 @@ A table with the specified structure for reading data in the specified Hudi tabl
 **See Also**
 
 - [Hudi engine](/docs/en/engines/table-engines/integrations/hudi.md)
-
+- [Hudi cluster table function](/docs/en/sql-reference/table-functions/hudiCluster.md)

docs/en/sql-reference/table-functions/hudiCluster.md

@@ -0,0 +1,30 @@
+---
+slug: /en/sql-reference/table-functions/hudiCluster
+sidebar_position: 86
+sidebar_label: hudiCluster
+title: "hudiCluster Table Function"
+---
+This is an extension to the [hudi](/docs/en/sql-reference/table-functions/hudi.md) table function.
+
+Allows processing files from Apache [Hudi](https://hudi.apache.org/) tables in Amazon S3 in parallel from many nodes in a specified cluster. On initiator it creates a connection to all nodes in the cluster and dispatches each file dynamically. On the worker node it asks the initiator about the next task to process and processes it. This is repeated until all tasks are finished.
+
+**Syntax**
+
+``` sql
+hudiCluster(cluster_name, url [,aws_access_key_id, aws_secret_access_key] [,format] [,structure] [,compression])
+```
+
+**Arguments**
+
+- `cluster_name` — Name of a cluster that is used to build a set of addresses and connection parameters to remote and local servers.
+
+- Description of all other arguments coincides with description of arguments in equivalent [hudi](/docs/en/sql-reference/table-functions/hudi.md) table function.
+
+**Returned value**
+
+A table with the specified structure for reading data from cluster in the specified Hudi table in S3.
+
+**See Also**
+
+- [Hudi engine](/docs/en/engines/table-engines/integrations/hudi.md)
+- [Hudi table function](/docs/en/sql-reference/table-functions/hudi.md)

docs/en/sql-reference/table-functions/iceberg.md

@@ -72,3 +72,4 @@ Table function `iceberg` is an alias to `icebergS3` now.
 **See Also**
 
 - [Iceberg engine](/docs/en/engines/table-engines/integrations/iceberg.md)
+- [Iceberg cluster table function](/docs/en/sql-reference/table-functions/icebergCluster.md)

docs/en/sql-reference/table-functions/icebergCluster.md

@@ -0,0 +1,43 @@
+---
+slug: /en/sql-reference/table-functions/icebergCluster
+sidebar_position: 91
+sidebar_label: icebergCluster
+title: "icebergCluster Table Function"
+---
+This is an extension to the [iceberg](/docs/en/sql-reference/table-functions/iceberg.md) table function.
+
+Allows processing files from Apache [Iceberg](https://iceberg.apache.org/) in parallel from many nodes in a specified cluster. On initiator it creates a connection to all nodes in the cluster and dispatches each file dynamically. On the worker node it asks the initiator about the next task to process and processes it. This is repeated until all tasks are finished.
+
+**Syntax**
+
+``` sql
+icebergS3Cluster(cluster_name, url [, NOSIGN | access_key_id, secret_access_key, [session_token]] [,format] [,compression_method])
+icebergS3Cluster(cluster_name, named_collection[, option=value [,..]])
+
+icebergAzureCluster(cluster_name, connection_string|storage_account_url, container_name, blobpath, [,account_name], [,account_key] [,format] [,compression_method])
+icebergAzureCluster(cluster_name, named_collection[, option=value [,..]])
+
+icebergHDFSCluster(cluster_name, path_to_table, [,format] [,compression_method])
+icebergHDFSCluster(cluster_name, named_collection[, option=value [,..]])
+```
+
+**Arguments**
+
+- `cluster_name` — Name of a cluster that is used to build a set of addresses and connection parameters to remote and local servers.
+
+- Description of all other arguments coincides with description of arguments in equivalent [iceberg](/docs/en/sql-reference/table-functions/iceberg.md) table function.
+
+**Returned value**
+
+A table with the specified structure for reading data from cluster in the specified Iceberg table.
+
+**Examples**
+
+```sql
+SELECT * FROM icebergS3Cluster('cluster_simple', 'http://test.s3.amazonaws.com/clickhouse-bucket/test_table', 'test', 'test')
+```
+
+**See Also**
+
+- [Iceberg engine](/docs/en/engines/table-engines/integrations/iceberg.md)
+- [Iceberg table function](/docs/en/sql-reference/table-functions/iceberg.md)

src/DataTypes/ObjectUtils.cpp

@@ -521,7 +521,7 @@ DataTypePtr createConcreteEmptyDynamicColumn(const DataTypePtr & type_in_storage
     throw Exception(ErrorCodes::EXPERIMENTAL_FEATURE_ERROR, "Type {} unexpectedly has dynamic columns", type_in_storage->getName());
 }
 
-bool hasDynamicSubcolumns(const ColumnsDescription & columns)
+bool hasDynamicSubcolumnsDeprecated(const ColumnsDescription & columns)
 {
     return std::any_of(columns.begin(), columns.end(),
         [](const auto & column)

src/DataTypes/ObjectUtils.h

@@ -63,7 +63,7 @@ DataTypePtr createConcreteEmptyDynamicColumn(const DataTypePtr & type_in_storage
 void extendObjectColumns(NamesAndTypesList & columns_list, const ColumnsDescription & object_columns, bool with_subcolumns);
 
 /// Checks whether @columns contain any column with dynamic subcolumns.
-bool hasDynamicSubcolumns(const ColumnsDescription & columns);
+bool hasDynamicSubcolumnsDeprecated(const ColumnsDescription & columns);
 
 /// Updates types of objects in @object_columns inplace
 /// according to types in new_columns.

src/Formats/FormatSettings.h

@@ -220,7 +220,7 @@ struct FormatSettings
         bool escape_forward_slashes = true;
         bool read_named_tuples_as_objects = false;
         bool use_string_type_for_ambiguous_paths_in_named_tuples_inference_from_objects = false;
-        bool write_named_tuples_as_objects = false;
+        bool write_named_tuples_as_objects = true;
         bool skip_null_value_in_named_tuples = false;
         bool defaults_for_missing_elements_in_named_tuple = false;
         bool ignore_unknown_keys_in_named_tuple = false;

src/Functions/FunctionsConversion.cpp

@@ -4108,6 +4108,7 @@ private:
                 ColumnStringHelpers::WriteHelper write_helper(assert_cast<ColumnString &>(*json_string), input_rows_count);
                 auto & write_buffer = write_helper.getWriteBuffer();
                 FormatSettings format_settings = context ? getFormatSettings(context) : FormatSettings{};
+                format_settings.json.quote_64bit_integers = false;
                 auto serialization = arguments[0].type->getDefaultSerialization();
                 for (size_t i = 0; i < input_rows_count; ++i)
                 {

src/Functions/array/arrayWithConstant.cpp

@@ -62,16 +62,17 @@ public:
         for (size_t i = 0; i < num_rows; ++i)
         {
             auto array_size = col_num->getInt(i);
+            auto element_size = col_value->byteSizeAt(i);
 
             if (unlikely(array_size < 0))
                 throw Exception(ErrorCodes::TOO_LARGE_ARRAY_SIZE, "Array size {} cannot be negative: while executing function {}", array_size, getName());
 
             Int64 estimated_size = 0;
-            if (unlikely(common::mulOverflow(array_size, col_value->byteSize(), estimated_size)))
-                throw Exception(ErrorCodes::TOO_LARGE_ARRAY_SIZE, "Array size {} with element size {} bytes is too large: while executing function {}", array_size, col_value->byteSize(), getName());
+            if (unlikely(common::mulOverflow(array_size, element_size, estimated_size)))
+                throw Exception(ErrorCodes::TOO_LARGE_ARRAY_SIZE, "Array size {} with element size {} bytes is too large: while executing function {}", array_size, element_size, getName());
 
             if (unlikely(estimated_size > max_array_size_in_columns_bytes))
-                throw Exception(ErrorCodes::TOO_LARGE_ARRAY_SIZE, "Array size {} with element size {} bytes is too large: while executing function {}", array_size, col_value->byteSize(), getName());
+                throw Exception(ErrorCodes::TOO_LARGE_ARRAY_SIZE, "Array size {} with element size {} bytes is too large: while executing function {}", array_size, element_size, getName());
 
             offset += array_size;
 

src/Interpreters/ClusterProxy/executeQuery.cpp

@@ -755,7 +755,7 @@ void executeQueryWithParallelReplicasCustomKey(
     }
 
     ColumnsDescriptionByShardNum columns_object;
-    if (hasDynamicSubcolumns(columns))
+    if (hasDynamicSubcolumnsDeprecated(columns))
         columns_object = getExtendedObjectsOfRemoteTables(*query_info.cluster, storage_id, columns, context);
 
     ClusterProxy::SelectStreamFactory select_stream_factory

src/Interpreters/InterpreterCreateQuery.cpp

@@ -1881,7 +1881,7 @@ bool InterpreterCreateQuery::doCreateTable(ASTCreateQuery & create,
 
     validateVirtualColumns(*res);
 
-    if (!res->supportsDynamicSubcolumnsDeprecated() && hasDynamicSubcolumns(res->getInMemoryMetadataPtr()->getColumns()) && mode <= LoadingStrictnessLevel::CREATE)
+    if (!res->supportsDynamicSubcolumnsDeprecated() && hasDynamicSubcolumnsDeprecated(res->getInMemoryMetadataPtr()->getColumns()) && mode <= LoadingStrictnessLevel::CREATE)
     {
         throw Exception(ErrorCodes::ILLEGAL_COLUMN,
             "Cannot create table with column of type Object, "

src/Storages/AlterCommands.cpp

@@ -1450,9 +1450,8 @@ void AlterCommands::validate(const StoragePtr & table, ContextPtr context) const
                 const auto old_data_type = all_columns.getColumn(options, column_name).type;
 
                 bool new_type_has_deprecated_object = command.data_type->hasDynamicSubcolumnsDeprecated();
-                bool old_type_has_deprecated_object = old_data_type->hasDynamicSubcolumnsDeprecated();
 
-                if (new_type_has_deprecated_object || old_type_has_deprecated_object)
+                if (new_type_has_deprecated_object)
                     throw Exception(
                         ErrorCodes::BAD_ARGUMENTS,
                         "The change of data type {} of column {} to {} is not allowed. It has known bugs",

src/Storages/MergeTree/MergeTreeData.cpp

@@ -4104,7 +4104,7 @@ void MergeTreeData::checkPartDynamicColumns(MutableDataPartPtr & part, DataParts
     const auto & columns = metadata_snapshot->getColumns();
     auto virtuals = getVirtualsPtr();
 
-    if (!hasDynamicSubcolumns(columns))
+    if (!hasDynamicSubcolumnsDeprecated(columns))
         return;
 
     const auto & part_columns = part->getColumns();
@@ -8754,8 +8754,11 @@ void MergeTreeData::resetObjectColumnsFromActiveParts(const DataPartsLock & /*lo
 {
     auto metadata_snapshot = getInMemoryMetadataPtr();
     const auto & columns = metadata_snapshot->getColumns();
-    if (!hasDynamicSubcolumns(columns))
+    if (!hasDynamicSubcolumnsDeprecated(columns))
+    {
+        object_columns = ColumnsDescription{};
         return;
+    }
 
     auto range = getDataPartsStateRange(DataPartState::Active);
     object_columns = getConcreteObjectColumns(range, columns);
@@ -8765,7 +8768,7 @@ void MergeTreeData::updateObjectColumns(const DataPartPtr & part, const DataPart
 {
     auto metadata_snapshot = getInMemoryMetadataPtr();
     const auto & columns = metadata_snapshot->getColumns();
-    if (!hasDynamicSubcolumns(columns))
+    if (!hasDynamicSubcolumnsDeprecated(columns))
         return;
 
     DB::updateObjectColumns(object_columns, columns, part->getColumns());

src/Storages/MergeTree/StorageFromMergeTreeDataPart.cpp

@@ -50,7 +50,7 @@ StorageSnapshotPtr
 StorageFromMergeTreeDataPart::getStorageSnapshot(const StorageMetadataPtr & metadata_snapshot, ContextPtr /*query_context*/) const
 {
     const auto & storage_columns = metadata_snapshot->getColumns();
-    if (!hasDynamicSubcolumns(storage_columns))
+    if (!hasDynamicSubcolumnsDeprecated(storage_columns))
         return std::make_shared<StorageSnapshot>(*this, metadata_snapshot);
 
     auto data_parts = storage.getDataPartsVectorForInternalUsage();

src/Storages/StorageDistributed.cpp

@@ -686,7 +686,7 @@ std::optional<QueryProcessingStage::Enum> StorageDistributed::getOptimizedQueryP
 
 static bool requiresObjectColumns(const ColumnsDescription & all_columns, ASTPtr query)
 {
-    if (!hasDynamicSubcolumns(all_columns))
+    if (!hasDynamicSubcolumnsDeprecated(all_columns))
         return false;
 
     if (!query)

src/Storages/StorageMemory.cpp

@@ -180,7 +180,7 @@ StorageSnapshotPtr StorageMemory::getStorageSnapshot(const StorageMetadataPtr &
     /// rows and bytes counters into the MultiVersion-ed struct, then everything would be consistent.
     snapshot_data->rows_approx = total_size_rows.load(std::memory_order_relaxed);
 
-    if (!hasDynamicSubcolumns(metadata_snapshot->getColumns()))
+    if (!hasDynamicSubcolumnsDeprecated(metadata_snapshot->getColumns()))
         return std::make_shared<StorageSnapshot>(*this, metadata_snapshot, ColumnsDescription{}, std::move(snapshot_data));
 
     auto object_columns = getConcreteObjectColumns(

src/TableFunctions/TableFunctionObjectStorage.cpp

@@ -226,6 +226,26 @@ template class TableFunctionObjectStorage<HDFSClusterDefinition, StorageHDFSConf
 #endif
 template class TableFunctionObjectStorage<LocalDefinition, StorageLocalConfiguration>;
 
+#if USE_AVRO && USE_AWS_S3
+template class TableFunctionObjectStorage<IcebergS3ClusterDefinition, StorageS3IcebergConfiguration>;
+#endif
+
+#if USE_AVRO && USE_AZURE_BLOB_STORAGE
+template class TableFunctionObjectStorage<IcebergAzureClusterDefinition, StorageAzureIcebergConfiguration>;
+#endif
+
+#if USE_AVRO && USE_HDFS
+template class TableFunctionObjectStorage<IcebergHDFSClusterDefinition, StorageHDFSIcebergConfiguration>;
+#endif
+
+#if USE_PARQUET && USE_AWS_S3
+template class TableFunctionObjectStorage<DeltaLakeClusterDefinition, StorageS3DeltaLakeConfiguration>;
+#endif
+
+#if USE_AWS_S3
+template class TableFunctionObjectStorage<HudiClusterDefinition, StorageS3HudiConfiguration>;
+#endif
+
 #if USE_AVRO
 void registerTableFunctionIceberg(TableFunctionFactory & factory)
 {

src/TableFunctions/TableFunctionObjectStorageCluster.cpp

@@ -96,7 +96,7 @@ void registerTableFunctionObjectStorageCluster(TableFunctionFactory & factory)
     {
         .documentation = {
             .description=R"(The table function can be used to read the data stored on HDFS in parallel for many nodes in a specified cluster.)",
-            .examples{{"HDFSCluster", "SELECT * FROM HDFSCluster(cluster_name, uri, format)", ""}}},
+            .examples{{"HDFSCluster", "SELECT * FROM HDFSCluster(cluster, uri, format)", ""}}},
             .allow_readonly = false
         }
     );
@@ -105,15 +105,77 @@ void registerTableFunctionObjectStorageCluster(TableFunctionFactory & factory)
     UNUSED(factory);
 }
 
+
+#if USE_AVRO
+void registerTableFunctionIcebergCluster(TableFunctionFactory & factory)
+{
+    UNUSED(factory);
+
 #if USE_AWS_S3
-template class TableFunctionObjectStorageCluster<S3ClusterDefinition, StorageS3Configuration>;
+    factory.registerFunction<TableFunctionIcebergS3Cluster>(
+        {.documentation
+         = {.description = R"(The table function can be used to read the Iceberg table stored on S3 object store in parallel for many nodes in a specified cluster.)",
+            .examples{{"icebergS3Cluster", "SELECT * FROM icebergS3Cluster(cluster, url, [, NOSIGN | access_key_id, secret_access_key, [session_token]], format, [,compression])", ""}},
+            .categories{"DataLake"}},
+         .allow_readonly = false});
 #endif
 
 #if USE_AZURE_BLOB_STORAGE
-template class TableFunctionObjectStorageCluster<AzureClusterDefinition, StorageAzureConfiguration>;
+    factory.registerFunction<TableFunctionIcebergAzureCluster>(
+        {.documentation
+         = {.description = R"(The table function can be used to read the Iceberg table stored on Azure object store in parallel for many nodes in a specified cluster.)",
+            .examples{{"icebergAzureCluster", "SELECT * FROM icebergAzureCluster(cluster, connection_string|storage_account_url, container_name, blobpath, [account_name, account_key, format, compression])", ""}},
+            .categories{"DataLake"}},
+         .allow_readonly = false});
 #endif
 
 #if USE_HDFS
-template class TableFunctionObjectStorageCluster<HDFSClusterDefinition, StorageHDFSConfiguration>;
+    factory.registerFunction<TableFunctionIcebergHDFSCluster>(
+        {.documentation
+         = {.description = R"(The table function can be used to read the Iceberg table stored on HDFS virtual filesystem in parallel for many nodes in a specified cluster.)",
+            .examples{{"icebergHDFSCluster", "SELECT * FROM icebergHDFSCluster(cluster, uri, [format], [structure], [compression_method])", ""}},
+            .categories{"DataLake"}},
+         .allow_readonly = false});
 #endif
 }
+#endif
+
+#if USE_AWS_S3
+#if USE_PARQUET
+void registerTableFunctionDeltaLakeCluster(TableFunctionFactory & factory)
+{
+    factory.registerFunction<TableFunctionDeltaLakeCluster>(
+        {.documentation
+         = {.description = R"(The table function can be used to read the DeltaLake table stored on object store in parallel for many nodes in a specified cluster.)",
+            .examples{{"deltaLakeCluster", "SELECT * FROM deltaLakeCluster(cluster, url, access_key_id, secret_access_key)", ""}},
+            .categories{"DataLake"}},
+         .allow_readonly = false});
+}
+#endif
+
+void registerTableFunctionHudiCluster(TableFunctionFactory & factory)
+{
+    factory.registerFunction<TableFunctionHudiCluster>(
+        {.documentation
+         = {.description = R"(The table function can be used to read the Hudi table stored on object store in parallel for many nodes in a specified cluster.)",
+            .examples{{"hudiCluster", "SELECT * FROM hudiCluster(cluster, url, access_key_id, secret_access_key)", ""}},
+            .categories{"DataLake"}},
+         .allow_readonly = false});
+}
+#endif
+
+void registerDataLakeClusterTableFunctions(TableFunctionFactory & factory)
+{
+    UNUSED(factory);
+#if USE_AVRO
+    registerTableFunctionIcebergCluster(factory);
+#endif
+#if USE_AWS_S3
+#if USE_PARQUET
+    registerTableFunctionDeltaLakeCluster(factory);
+#endif
+    registerTableFunctionHudiCluster(factory);
+#endif
+}
+
+}

src/TableFunctions/TableFunctionObjectStorageCluster.h

@@ -33,6 +33,36 @@ struct HDFSClusterDefinition
     static constexpr auto storage_type_name = "HDFSCluster";
 };
 
+struct IcebergS3ClusterDefinition
+{
+    static constexpr auto name = "icebergS3Cluster";
+    static constexpr auto storage_type_name = "IcebergS3Cluster";
+};
+
+struct IcebergAzureClusterDefinition
+{
+    static constexpr auto name = "icebergAzureCluster";
+    static constexpr auto storage_type_name = "IcebergAzureCluster";
+};
+
+struct IcebergHDFSClusterDefinition
+{
+    static constexpr auto name = "icebergHDFSCluster";
+    static constexpr auto storage_type_name = "IcebergHDFSCluster";
+};
+
+struct DeltaLakeClusterDefinition
+{
+    static constexpr auto name = "deltaLakeCluster";
+    static constexpr auto storage_type_name = "DeltaLakeS3Cluster";
+};
+
+struct HudiClusterDefinition
+{
+    static constexpr auto name = "hudiCluster";
+    static constexpr auto storage_type_name = "HudiS3Cluster";
+};
+
 /**
 * Class implementing s3/hdfs/azureBlobStorageCluster(...) table functions,
 * which allow to process many files from S3/HDFS/Azure blob storage on a specific cluster.
@@ -79,4 +109,25 @@ using TableFunctionAzureBlobCluster = TableFunctionObjectStorageCluster<AzureClu
 #if USE_HDFS
 using TableFunctionHDFSCluster = TableFunctionObjectStorageCluster<HDFSClusterDefinition, StorageHDFSConfiguration>;
 #endif
+
+#if USE_AVRO && USE_AWS_S3
+using TableFunctionIcebergS3Cluster = TableFunctionObjectStorageCluster<IcebergS3ClusterDefinition, StorageS3IcebergConfiguration>;
+#endif
+
+#if USE_AVRO && USE_AZURE_BLOB_STORAGE
+using TableFunctionIcebergAzureCluster = TableFunctionObjectStorageCluster<IcebergAzureClusterDefinition, StorageAzureIcebergConfiguration>;
+#endif
+
+#if USE_AVRO && USE_HDFS
+using TableFunctionIcebergHDFSCluster = TableFunctionObjectStorageCluster<IcebergHDFSClusterDefinition, StorageHDFSIcebergConfiguration>;
+#endif
+
+#if USE_AWS_S3 && USE_PARQUET
+using TableFunctionDeltaLakeCluster = TableFunctionObjectStorageCluster<DeltaLakeClusterDefinition, StorageS3DeltaLakeConfiguration>;
+#endif
+
+#if USE_AWS_S3
+using TableFunctionHudiCluster = TableFunctionObjectStorageCluster<HudiClusterDefinition, StorageS3HudiConfiguration>;
+#endif
+
 }

src/TableFunctions/registerTableFunctions.cpp

@@ -66,6 +66,7 @@ void registerTableFunctions(bool use_legacy_mongodb_integration [[maybe_unused]]
     registerTableFunctionObjectStorage(factory);
     registerTableFunctionObjectStorageCluster(factory);
     registerDataLakeTableFunctions(factory);
+    registerDataLakeClusterTableFunctions(factory);
 }
 
 }

src/TableFunctions/registerTableFunctions.h

@@ -70,6 +70,7 @@ void registerTableFunctionExplain(TableFunctionFactory & factory);
 void registerTableFunctionObjectStorage(TableFunctionFactory & factory);
 void registerTableFunctionObjectStorageCluster(TableFunctionFactory & factory);
 void registerDataLakeTableFunctions(TableFunctionFactory & factory);
+void registerDataLakeClusterTableFunctions(TableFunctionFactory & factory);
 
 void registerTableFunctionTimeSeries(TableFunctionFactory & factory);
 

tests/integration/test_storage_iceberg/configs/config.d/cluster.xml

@@ -0,0 +1,20 @@
+<clickhouse>
+    <remote_servers>
+        <cluster_simple>
+            <shard>
+                <replica>
+                    <host>node1</host>
+                    <port>9000</port>
+                </replica>
+                <replica>
+                    <host>node2</host>
+                    <port>9000</port>
+                </replica>
+                <replica>
+                    <host>node3</host>
+                    <port>9000</port>
+                </replica>
+            </shard>
+        </cluster_simple>
+    </remote_servers>
+</clickhouse>

tests/integration/test_storage_iceberg/configs/config.d/query_log.xml

@@ -0,0 +1,6 @@
+<clickhouse>
+    <query_log>
+      <database>system</database>
+      <table>query_log</table>
+    </query_log>
+</clickhouse>

tests/integration/test_storage_iceberg/test.py

@@ -73,14 +73,38 @@ def started_cluster():
         cluster.add_instance(
             "node1",
             main_configs=[
+                "configs/config.d/query_log.xml",
+                "configs/config.d/cluster.xml",
                 "configs/config.d/named_collections.xml",
                 "configs/config.d/filesystem_caches.xml",
             ],
             user_configs=["configs/users.d/users.xml"],
             with_minio=True,
             with_azurite=True,
-            stay_alive=True,
             with_hdfs=with_hdfs,
+            stay_alive=True,
+        )
+        cluster.add_instance(
+            "node2",
+            main_configs=[
+                "configs/config.d/query_log.xml",
+                "configs/config.d/cluster.xml",
+                "configs/config.d/named_collections.xml",
+                "configs/config.d/filesystem_caches.xml",
+            ],
+            user_configs=["configs/users.d/users.xml"],
+            stay_alive=True,
+        )
+        cluster.add_instance(
+            "node3",
+            main_configs=[
+                "configs/config.d/query_log.xml",
+                "configs/config.d/cluster.xml",
+                "configs/config.d/named_collections.xml",
+                "configs/config.d/filesystem_caches.xml",
+            ],
+            user_configs=["configs/users.d/users.xml"],
+            stay_alive=True,
         )
 
         logging.info("Starting cluster...")
@@ -182,6 +206,7 @@ def get_creation_expression(
     cluster,
     format="Parquet",
     table_function=False,
+    run_on_cluster=False,
     **kwargs,
 ):
     if storage_type == "s3":
@@ -189,7 +214,11 @@ def get_creation_expression(
             bucket = kwargs["bucket"]
         else:
             bucket = cluster.minio_bucket
-        print(bucket)
+
+        if run_on_cluster:
+            assert table_function
+            return f"icebergS3Cluster('cluster_simple', s3, filename = 'iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/')"
+        else:
             if table_function:
                 return f"icebergS3(s3, filename = 'iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/')"
             else:
@@ -197,7 +226,14 @@ def get_creation_expression(
                     DROP TABLE IF EXISTS {table_name};
                     CREATE TABLE {table_name}
                     ENGINE=IcebergS3(s3, filename = 'iceberg_data/default/{table_name}/', format={format}, url = 'http://minio1:9001/{bucket}/')"""
+
     elif storage_type == "azure":
+        if run_on_cluster:
+            assert table_function
+            return f"""
+                icebergAzureCluster('cluster_simple', azure, container = '{cluster.azure_container_name}', storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/iceberg_data/default/{table_name}/', format={format})
+            """
+        else:
             if table_function:
                 return f"""
                     icebergAzure(azure, container = '{cluster.azure_container_name}', storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/iceberg_data/default/{table_name}/', format={format})
@@ -207,7 +243,14 @@ def get_creation_expression(
                     DROP TABLE IF EXISTS {table_name};
                     CREATE TABLE {table_name}
                     ENGINE=IcebergAzure(azure, container = {cluster.azure_container_name}, storage_account_url = '{cluster.env_variables["AZURITE_STORAGE_ACCOUNT_URL"]}', blob_path = '/iceberg_data/default/{table_name}/', format={format})"""
+
     elif storage_type == "hdfs":
+        if run_on_cluster:
+            assert table_function
+            return f"""
+                icebergHDFSCluster('cluster_simple', hdfs, filename= 'iceberg_data/default/{table_name}/', format={format}, url = 'hdfs://hdfs1:9000/')
+            """
+        else:
             if table_function:
                 return f"""
                     icebergHDFS(hdfs, filename= 'iceberg_data/default/{table_name}/', format={format}, url = 'hdfs://hdfs1:9000/')
@@ -217,7 +260,10 @@ def get_creation_expression(
                     DROP TABLE IF EXISTS {table_name};
                     CREATE TABLE {table_name}
                     ENGINE=IcebergHDFS(hdfs, filename = 'iceberg_data/default/{table_name}/', format={format}, url = 'hdfs://hdfs1:9000/');"""
+
     elif storage_type == "local":
+        assert not run_on_cluster
+
         if table_function:
             return f"""
                 icebergLocal(local, path = '/iceberg_data/default/{table_name}/', format={format})
@@ -227,6 +273,7 @@ def get_creation_expression(
                 DROP TABLE IF EXISTS {table_name};
                 CREATE TABLE {table_name}
                 ENGINE=IcebergLocal(local, path = '/iceberg_data/default/{table_name}/', format={format});"""
+
     else:
         raise Exception(f"Unknown iceberg storage type: {storage_type}")
 
@@ -492,6 +539,108 @@ def test_types(started_cluster, format_version, storage_type):
     )
 
 
+@pytest.mark.parametrize("format_version", ["1", "2"])
+@pytest.mark.parametrize("storage_type", ["s3", "azure", "hdfs"])
+def test_cluster_table_function(started_cluster, format_version, storage_type):
+    if is_arm() and storage_type == "hdfs":
+        pytest.skip("Disabled test IcebergHDFS for aarch64")
+
+    instance = started_cluster.instances["node1"]
+    spark = started_cluster.spark_session
+
+    TABLE_NAME = (
+        "test_iceberg_cluster_"
+        + format_version
+        + "_"
+        + storage_type
+        + "_"
+        + get_uuid_str()
+    )
+
+    def add_df(mode):
+        write_iceberg_from_df(
+            spark,
+            generate_data(spark, 0, 100),
+            TABLE_NAME,
+            mode=mode,
+            format_version=format_version,
+        )
+
+        files = default_upload_directory(
+            started_cluster,
+            storage_type,
+            f"/iceberg_data/default/{TABLE_NAME}/",
+            f"/iceberg_data/default/{TABLE_NAME}/",
+        )
+
+        logging.info(f"Adding another dataframe. result files: {files}")
+
+        return files
+
+    files = add_df(mode="overwrite")
+    for i in range(1, len(started_cluster.instances)):
+        files = add_df(mode="append")
+
+    logging.info(f"Setup complete. files: {files}")
+    assert len(files) == 5 + 4 * (len(started_cluster.instances) - 1)
+
+    clusters = instance.query(f"SELECT * FROM system.clusters")
+    logging.info(f"Clusters setup: {clusters}")
+
+    # Regular Query only node1
+    table_function_expr = get_creation_expression(
+        storage_type, TABLE_NAME, started_cluster, table_function=True
+    )
+    select_regular = (
+        instance.query(f"SELECT * FROM {table_function_expr}").strip().split()
+    )
+
+    # Cluster Query with node1 as coordinator
+    table_function_expr_cluster = get_creation_expression(
+        storage_type,
+        TABLE_NAME,
+        started_cluster,
+        table_function=True,
+        run_on_cluster=True,
+    )
+    select_cluster = (
+        instance.query(f"SELECT * FROM {table_function_expr_cluster}").strip().split()
+    )
+
+    # Simple size check
+    assert len(select_regular) == 600
+    assert len(select_cluster) == 600
+
+    # Actual check
+    assert select_cluster == select_regular
+
+    # Check query_log
+    for replica in started_cluster.instances.values():
+        replica.query("SYSTEM FLUSH LOGS")
+
+    for node_name, replica in started_cluster.instances.items():
+        cluster_secondary_queries = (
+            replica.query(
+                f"""
+                SELECT query, type, is_initial_query, read_rows, read_bytes FROM system.query_log
+                WHERE
+                    type = 'QueryStart' AND
+                    positionCaseInsensitive(query, '{storage_type}Cluster') != 0 AND
+                    position(query, '{TABLE_NAME}') != 0 AND
+                    position(query, 'system.query_log') = 0 AND
+                    NOT is_initial_query
+            """
+            )
+            .strip()
+            .split("\n")
+        )
+
+        logging.info(
+            f"[{node_name}] cluster_secondary_queries: {cluster_secondary_queries}"
+        )
+        assert len(cluster_secondary_queries) == 1
+
+
 @pytest.mark.parametrize("format_version", ["1", "2"])
 @pytest.mark.parametrize("storage_type", ["s3", "azure", "hdfs", "local"])
 def test_delete_files(started_cluster, format_version, storage_type):

tests/queries/0_stateless/02725_object_column_alter.sql

@@ -3,13 +3,7 @@
 SET allow_experimental_object_type=1;
 
 DROP TABLE IF EXISTS t_to;
-DROP TABLE IF EXISTS t_from;
-
 CREATE TABLE t_to (id UInt64, value Nullable(String)) ENGINE MergeTree() ORDER BY id;
-CREATE TABLE t_from (id UInt64, value Object('json')) ENGINE MergeTree() ORDER BY id;
-
 ALTER TABLE t_to MODIFY COLUMN value Object('json'); -- { serverError BAD_ARGUMENTS }
-ALTER TABLE t_from MODIFY COLUMN value Nullable(String); -- { serverError BAD_ARGUMENTS }
-
 DROP TABLE t_to;
-DROP TABLE t_from;
+

tests/queries/0_stateless/03216_arrayWithConstant_limits.sql

@@ -1,3 +1,6 @@
 SELECT arrayWithConstant(96142475, ['qMUF']); -- { serverError TOO_LARGE_ARRAY_SIZE }
 SELECT arrayWithConstant(100000000, materialize([[[[[[[[[['Hello, world!']]]]]]]]]])); -- { serverError TOO_LARGE_ARRAY_SIZE }
 SELECT length(arrayWithConstant(10000000, materialize([[[[[[[[[['Hello world']]]]]]]]]])));
+
+CREATE TEMPORARY TABLE args (value Array(Int)) ENGINE=Memory AS SELECT [1, 1, 1, 1] as value FROM numbers(1, 100);
+SELECT length(arrayWithConstant(1000000, value)) FROM args FORMAT NULL;

tests/queries/0_stateless/03261_tuple_map_object_to_json_cast.reference

@@ -1,20 +1,20 @@
 Map to JSON
 {"a":"0","b":"1970-01-01","c":[],"d":[{"e":"0"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a":"1","b":"1970-01-02","c":["0"],"d":[{"e":"1"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a":"2","b":"1970-01-03","c":["0","1"],"d":[{"e":"2"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a":"3","b":"1970-01-04","c":["0","1","2"],"d":[{"e":"3"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a":"4","b":"1970-01-05","c":["0","1","2","3"],"d":[{"e":"4"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a":"1","b":"1970-01-02","c":["0"],"d":[{"e":"1"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(Int64))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a":"2","b":"1970-01-03","c":["0","1"],"d":[{"e":"2"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(Int64))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a":"3","b":"1970-01-04","c":["0","1","2"],"d":[{"e":"3"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(Int64))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a":"4","b":"1970-01-05","c":["0","1","2","3"],"d":[{"e":"4"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(Int64))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
 {"a0":"0","b0":"1970-01-01","c0":[],"d0":[{"e0":"0"}]}	{'a0':'Int64','b0':'Date','c0':'Array(Nullable(String))','d0':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a1":"1","b1":"1970-01-02","c1":["0"],"d1":[{"e1":"1"}]}	{'a1':'Int64','b1':'Date','c1':'Array(Nullable(String))','d1':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a2":"2","b2":"1970-01-03","c2":["0","1"],"d2":[{"e2":"2"}]}	{'a2':'Int64','b2':'Date','c2':'Array(Nullable(String))','d2':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a0":"3","b0":"1970-01-04","c0":["0","1","2"],"d0":[{"e0":"3"}]}	{'a0':'Int64','b0':'Date','c0':'Array(Nullable(String))','d0':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a1":"4","b1":"1970-01-05","c1":["0","1","2","3"],"d1":[{"e1":"4"}]}	{'a1':'Int64','b1':'Date','c1':'Array(Nullable(String))','d1':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a1":"1","b1":"1970-01-02","c1":["0"],"d1":[{"e1":"1"}]}	{'a1':'Int64','b1':'Date','c1':'Array(Nullable(Int64))','d1':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a2":"2","b2":"1970-01-03","c2":["0","1"],"d2":[{"e2":"2"}]}	{'a2':'Int64','b2':'Date','c2':'Array(Nullable(Int64))','d2':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a0":"3","b0":"1970-01-04","c0":["0","1","2"],"d0":[{"e0":"3"}]}	{'a0':'Int64','b0':'Date','c0':'Array(Nullable(Int64))','d0':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a1":"4","b1":"1970-01-05","c1":["0","1","2","3"],"d1":[{"e1":"4"}]}	{'a1':'Int64','b1':'Date','c1':'Array(Nullable(Int64))','d1':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
 Tuple to JSON
 {"a":"0","b":"1970-01-01","c":[],"d":[{"e":"0"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a":"1","b":"1970-01-02","c":["0"],"d":[{"e":"1"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a":"2","b":"1970-01-03","c":["0","1"],"d":[{"e":"2"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a":"3","b":"1970-01-04","c":["0","1","2"],"d":[{"e":"3"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
-{"a":"4","b":"1970-01-05","c":["0","1","2","3"],"d":[{"e":"4"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a":"1","b":"1970-01-02","c":["0"],"d":[{"e":"1"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(Int64))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a":"2","b":"1970-01-03","c":["0","1"],"d":[{"e":"2"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(Int64))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a":"3","b":"1970-01-04","c":["0","1","2"],"d":[{"e":"3"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(Int64))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
+{"a":"4","b":"1970-01-05","c":["0","1","2","3"],"d":[{"e":"4"}]}	{'a':'Int64','b':'Date','c':'Array(Nullable(Int64))','d':'Array(JSON(max_dynamic_types=16, max_dynamic_paths=256))'}
 Object to JSON
 {"a":"0","b":"1970-01-01","c":[],"d":{"e":["0"]}}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d.e':'Array(Nullable(Int64))'}
 {"a":"1","b":"1970-01-02","c":["0"],"d":{"e":["1"]}}	{'a':'Int64','b':'Date','c':'Array(Nullable(String))','d.e':'Array(Nullable(Int64))'}

tests/queries/0_stateless/03270_object_to_json_alter.reference

@@ -0,0 +1,132 @@
+{'a0':['Int64'],'a1':['Int64'],'a10':['Int64'],'a11':['Int64'],'a12':['Int64'],'a13':['Int64'],'a14':['Int64'],'a15':['Int64'],'a16':['Int64'],'a17':['Int64'],'a18':['Int64'],'a19':['Int64'],'a2':['Int64'],'a20':['Int64'],'a21':['Int64'],'a22':['Int64'],'a23':['Int64'],'a24':['Int64'],'a25':['Int64'],'a26':['Int64'],'a27':['Int64'],'a28':['Int64'],'a29':['Int64'],'a3':['Int64'],'a30':['Int64'],'a31':['Int64'],'a32':['Int64'],'a33':['Int64'],'a34':['Int64'],'a35':['Int64'],'a36':['Int64'],'a37':['Int64'],'a38':['Int64'],'a39':['Int64'],'a4':['Int64'],'a40':['Int64'],'a41':['Int64'],'a42':['Int64'],'a43':['Int64'],'a44':['Int64'],'a45':['Int64'],'a46':['Int64'],'a47':['Int64'],'a48':['Int64'],'a49':['Int64'],'a5':['Int64'],'a50':['Int64'],'a51':['Int64'],'a52':['Int64'],'a53':['Int64'],'a54':['Int64'],'a55':['Int64'],'a56':['Int64'],'a57':['Int64'],'a58':['Int64'],'a59':['Int64'],'a6':['Int64'],'a60':['Int64'],'a61':['Int64'],'a62':['Int64'],'a63':['Int64'],'a64':['Int64'],'a65':['Int64'],'a66':['Int64'],'a67':['Int64'],'a68':['Int64'],'a69':['Int64'],'a7':['Int64'],'a70':['Int64'],'a71':['Int64'],'a72':['Int64'],'a73':['Int64'],'a74':['Int64'],'a75':['Int64'],'a76':['Int64'],'a77':['Int64'],'a78':['Int64'],'a79':['Int64'],'a8':['Int64'],'a80':['Int64'],'a81':['Int64'],'a82':['Int64'],'a83':['Int64'],'a84':['Int64'],'a85':['Int64'],'a86':['Int64'],'a87':['Int64'],'a88':['Int64'],'a89':['Int64'],'a9':['Int64'],'a90':['Int64'],'a91':['Int64'],'a92':['Int64'],'a93':['Int64'],'a94':['Int64'],'a95':['Int64'],'a96':['Int64'],'a97':['Int64'],'a98':['Int64'],'a99':['Int64']}
+0
+100
+200
+300
+400
+500
+600
+700
+800
+900
+0
+99
+199
+299
+399
+499
+599
+699
+799
+899
+999
+{'a0':['Int64'],'a1':['Int64'],'a10':['Int64'],'a11':['Int64'],'a12':['Int64'],'a13':['Int64'],'a14':['Int64'],'a15':['Int64'],'a16':['Int64'],'a17':['Int64'],'a18':['Int64'],'a19':['Int64'],'a2':['Int64'],'a20':['Int64'],'a21':['Int64'],'a22':['Int64'],'a23':['Int64'],'a24':['Int64'],'a25':['Int64'],'a26':['Int64'],'a27':['Int64'],'a28':['Int64'],'a29':['Int64'],'a3':['Int64'],'a30':['Int64'],'a31':['Int64'],'a32':['Int64'],'a33':['Int64'],'a34':['Int64'],'a35':['Int64'],'a36':['Int64'],'a37':['Int64'],'a38':['Int64'],'a39':['Int64'],'a4':['Int64'],'a40':['Int64'],'a41':['Int64'],'a42':['Int64'],'a43':['Int64'],'a44':['Int64'],'a45':['Int64'],'a46':['Int64'],'a47':['Int64'],'a48':['Int64'],'a49':['Int64'],'a5':['Int64'],'a50':['Int64'],'a51':['Int64'],'a52':['Int64'],'a53':['Int64'],'a54':['Int64'],'a55':['Int64'],'a56':['Int64'],'a57':['Int64'],'a58':['Int64'],'a59':['Int64'],'a6':['Int64'],'a60':['Int64'],'a61':['Int64'],'a62':['Int64'],'a63':['Int64'],'a64':['Int64'],'a65':['Int64'],'a66':['Int64'],'a67':['Int64'],'a68':['Int64'],'a69':['Int64'],'a7':['Int64'],'a70':['Int64'],'a71':['Int64'],'a72':['Int64'],'a73':['Int64'],'a74':['Int64'],'a75':['Int64'],'a76':['Int64'],'a77':['Int64'],'a78':['Int64'],'a79':['Int64'],'a8':['Int64'],'a80':['Int64'],'a81':['Int64'],'a82':['Int64'],'a83':['Int64'],'a84':['Int64'],'a85':['Int64'],'a86':['Int64'],'a87':['Int64'],'a88':['Int64'],'a89':['Int64'],'a9':['Int64'],'a90':['Int64'],'a91':['Int64'],'a92':['Int64'],'a93':['Int64'],'a94':['Int64'],'a95':['Int64'],'a96':['Int64'],'a97':['Int64'],'a98':['Int64'],'a99':['Int64']}
+0
+100
+200
+300
+400
+500
+600
+700
+800
+900
+0
+99
+199
+299
+399
+499
+599
+699
+799
+899
+999
+{'a0':['Int64'],'a1':['Int64'],'a10':['Int64'],'a11':['Int64'],'a12':['Int64'],'a13':['Int64'],'a14':['Int64'],'a15':['Int64'],'a16':['Int64'],'a17':['Int64'],'a18':['Int64'],'a19':['Int64'],'a2':['Int64'],'a20':['Int64'],'a21':['Int64'],'a22':['Int64'],'a23':['Int64'],'a24':['Int64'],'a25':['Int64'],'a26':['Int64'],'a27':['Int64'],'a28':['Int64'],'a29':['Int64'],'a3':['Int64'],'a30':['Int64'],'a31':['Int64'],'a32':['Int64'],'a33':['Int64'],'a34':['Int64'],'a35':['Int64'],'a36':['Int64'],'a37':['Int64'],'a38':['Int64'],'a39':['Int64'],'a4':['Int64'],'a40':['Int64'],'a41':['Int64'],'a42':['Int64'],'a43':['Int64'],'a44':['Int64'],'a45':['Int64'],'a46':['Int64'],'a47':['Int64'],'a48':['Int64'],'a49':['Int64'],'a5':['Int64'],'a50':['Int64'],'a51':['Int64'],'a52':['Int64'],'a53':['Int64'],'a54':['Int64'],'a55':['Int64'],'a56':['Int64'],'a57':['Int64'],'a58':['Int64'],'a59':['Int64'],'a6':['Int64'],'a60':['Int64'],'a61':['Int64'],'a62':['Int64'],'a63':['Int64'],'a64':['Int64'],'a65':['Int64'],'a66':['Int64'],'a67':['Int64'],'a68':['Int64'],'a69':['Int64'],'a7':['Int64'],'a70':['Int64'],'a71':['Int64'],'a72':['Int64'],'a73':['Int64'],'a74':['Int64'],'a75':['Int64'],'a76':['Int64'],'a77':['Int64'],'a78':['Int64'],'a79':['Int64'],'a8':['Int64'],'a80':['Int64'],'a81':['Int64'],'a82':['Int64'],'a83':['Int64'],'a84':['Int64'],'a85':['Int64'],'a86':['Int64'],'a87':['Int64'],'a88':['Int64'],'a89':['Int64'],'a9':['Int64'],'a90':['Int64'],'a91':['Int64'],'a92':['Int64'],'a93':['Int64'],'a94':['Int64'],'a95':['Int64'],'a96':['Int64'],'a97':['Int64'],'a98':['Int64'],'a99':['Int64']}
+0
+100
+200
+300
+400
+500
+600
+700
+800
+900
+0
+99
+199
+299
+399
+499
+599
+699
+799
+899
+999
+{'a0':['Int64'],'a1':['Int64'],'a10':['Int64'],'a11':['Int64'],'a12':['Int64'],'a13':['Int64'],'a14':['Int64'],'a15':['Int64'],'a16':['Int64'],'a17':['Int64'],'a18':['Int64'],'a19':['Int64'],'a2':['Int64'],'a20':['Int64'],'a21':['Int64'],'a22':['Int64'],'a23':['Int64'],'a24':['Int64'],'a25':['Int64'],'a26':['Int64'],'a27':['Int64'],'a28':['Int64'],'a29':['Int64'],'a3':['Int64'],'a30':['Int64'],'a31':['Int64'],'a32':['Int64'],'a33':['Int64'],'a34':['Int64'],'a35':['Int64'],'a36':['Int64'],'a37':['Int64'],'a38':['Int64'],'a39':['Int64'],'a4':['Int64'],'a40':['Int64'],'a41':['Int64'],'a42':['Int64'],'a43':['Int64'],'a44':['Int64'],'a45':['Int64'],'a46':['Int64'],'a47':['Int64'],'a48':['Int64'],'a49':['Int64'],'a5':['Int64'],'a50':['Int64'],'a51':['Int64'],'a52':['Int64'],'a53':['Int64'],'a54':['Int64'],'a55':['Int64'],'a56':['Int64'],'a57':['Int64'],'a58':['Int64'],'a59':['Int64'],'a6':['Int64'],'a60':['Int64'],'a61':['Int64'],'a62':['Int64'],'a63':['Int64'],'a64':['Int64'],'a65':['Int64'],'a66':['Int64'],'a67':['Int64'],'a68':['Int64'],'a69':['Int64'],'a7':['Int64'],'a70':['Int64'],'a71':['Int64'],'a72':['Int64'],'a73':['Int64'],'a74':['Int64'],'a75':['Int64'],'a76':['Int64'],'a77':['Int64'],'a78':['Int64'],'a79':['Int64'],'a8':['Int64'],'a80':['Int64'],'a81':['Int64'],'a82':['Int64'],'a83':['Int64'],'a84':['Int64'],'a85':['Int64'],'a86':['Int64'],'a87':['Int64'],'a88':['Int64'],'a89':['Int64'],'a9':['Int64'],'a90':['Int64'],'a91':['Int64'],'a92':['Int64'],'a93':['Int64'],'a94':['Int64'],'a95':['Int64'],'a96':['Int64'],'a97':['Int64'],'a98':['Int64'],'a99':['Int64']}
+0
+100
+200
+300
+400
+500
+600
+700
+800
+900
+0
+99
+199
+299
+399
+499
+599
+699
+799
+899
+999
+{'a0':['Int64'],'a1':['Int64'],'a10':['Int64'],'a11':['Int64'],'a12':['Int64'],'a13':['Int64'],'a14':['Int64'],'a15':['Int64'],'a16':['Int64'],'a17':['Int64'],'a18':['Int64'],'a19':['Int64'],'a2':['Int64'],'a20':['Int64'],'a21':['Int64'],'a22':['Int64'],'a23':['Int64'],'a24':['Int64'],'a25':['Int64'],'a26':['Int64'],'a27':['Int64'],'a28':['Int64'],'a29':['Int64'],'a3':['Int64'],'a30':['Int64'],'a31':['Int64'],'a32':['Int64'],'a33':['Int64'],'a34':['Int64'],'a35':['Int64'],'a36':['Int64'],'a37':['Int64'],'a38':['Int64'],'a39':['Int64'],'a4':['Int64'],'a40':['Int64'],'a41':['Int64'],'a42':['Int64'],'a43':['Int64'],'a44':['Int64'],'a45':['Int64'],'a46':['Int64'],'a47':['Int64'],'a48':['Int64'],'a49':['Int64'],'a5':['Int64'],'a50':['Int64'],'a51':['Int64'],'a52':['Int64'],'a53':['Int64'],'a54':['Int64'],'a55':['Int64'],'a56':['Int64'],'a57':['Int64'],'a58':['Int64'],'a59':['Int64'],'a6':['Int64'],'a60':['Int64'],'a61':['Int64'],'a62':['Int64'],'a63':['Int64'],'a64':['Int64'],'a65':['Int64'],'a66':['Int64'],'a67':['Int64'],'a68':['Int64'],'a69':['Int64'],'a7':['Int64'],'a70':['Int64'],'a71':['Int64'],'a72':['Int64'],'a73':['Int64'],'a74':['Int64'],'a75':['Int64'],'a76':['Int64'],'a77':['Int64'],'a78':['Int64'],'a79':['Int64'],'a8':['Int64'],'a80':['Int64'],'a81':['Int64'],'a82':['Int64'],'a83':['Int64'],'a84':['Int64'],'a85':['Int64'],'a86':['Int64'],'a87':['Int64'],'a88':['Int64'],'a89':['Int64'],'a9':['Int64'],'a90':['Int64'],'a91':['Int64'],'a92':['Int64'],'a93':['Int64'],'a94':['Int64'],'a95':['Int64'],'a96':['Int64'],'a97':['Int64'],'a98':['Int64'],'a99':['Int64']}
+0
+100
+200
+300
+400
+500
+600
+700
+800
+900
+0
+99
+199
+299
+399
+499
+599
+699
+799
+899
+999
+{'a0':['Int64'],'a1':['Int64'],'a10':['Int64'],'a11':['Int64'],'a12':['Int64'],'a13':['Int64'],'a14':['Int64'],'a15':['Int64'],'a16':['Int64'],'a17':['Int64'],'a18':['Int64'],'a19':['Int64'],'a2':['Int64'],'a20':['Int64'],'a21':['Int64'],'a22':['Int64'],'a23':['Int64'],'a24':['Int64'],'a25':['Int64'],'a26':['Int64'],'a27':['Int64'],'a28':['Int64'],'a29':['Int64'],'a3':['Int64'],'a30':['Int64'],'a31':['Int64'],'a32':['Int64'],'a33':['Int64'],'a34':['Int64'],'a35':['Int64'],'a36':['Int64'],'a37':['Int64'],'a38':['Int64'],'a39':['Int64'],'a4':['Int64'],'a40':['Int64'],'a41':['Int64'],'a42':['Int64'],'a43':['Int64'],'a44':['Int64'],'a45':['Int64'],'a46':['Int64'],'a47':['Int64'],'a48':['Int64'],'a49':['Int64'],'a5':['Int64'],'a50':['Int64'],'a51':['Int64'],'a52':['Int64'],'a53':['Int64'],'a54':['Int64'],'a55':['Int64'],'a56':['Int64'],'a57':['Int64'],'a58':['Int64'],'a59':['Int64'],'a6':['Int64'],'a60':['Int64'],'a61':['Int64'],'a62':['Int64'],'a63':['Int64'],'a64':['Int64'],'a65':['Int64'],'a66':['Int64'],'a67':['Int64'],'a68':['Int64'],'a69':['Int64'],'a7':['Int64'],'a70':['Int64'],'a71':['Int64'],'a72':['Int64'],'a73':['Int64'],'a74':['Int64'],'a75':['Int64'],'a76':['Int64'],'a77':['Int64'],'a78':['Int64'],'a79':['Int64'],'a8':['Int64'],'a80':['Int64'],'a81':['Int64'],'a82':['Int64'],'a83':['Int64'],'a84':['Int64'],'a85':['Int64'],'a86':['Int64'],'a87':['Int64'],'a88':['Int64'],'a89':['Int64'],'a9':['Int64'],'a90':['Int64'],'a91':['Int64'],'a92':['Int64'],'a93':['Int64'],'a94':['Int64'],'a95':['Int64'],'a96':['Int64'],'a97':['Int64'],'a98':['Int64'],'a99':['Int64']}
+0
+100
+200
+300
+400
+500
+600
+700
+800
+900
+0
+99
+199
+299
+399
+499
+599
+699
+799
+899
+999

tests/queries/0_stateless/03270_object_to_json_alter.sql

@@ -0,0 +1,56 @@
+set allow_experimental_object_type = 1;
+set allow_experimental_json_type = 1;
+set max_block_size = 100;
+set max_insert_block_size = 100;
+set min_insert_block_size_rows = 100;
+set output_format_json_quote_64bit_integers = 0;
+
+drop table if exists test;
+
+create table test (json Object('json')) engine=Memory;
+insert into test select toJSONString(map('a' || number % 100, number)) from numbers(1000);
+alter table test modify column json JSON;
+select distinctJSONPathsAndTypes(json) from test;
+select distinct json.a0 from test order by json.a0.:Int64;
+select distinct json.a99 from test order by json.a99.:Int64;
+drop table test;
+
+create table test (json Object('json')) engine=Memory;
+insert into test select toJSONString(map('a' || number % 100, number)) from numbers(1000);
+alter table test modify column json JSON(max_dynamic_paths=10);
+select distinctJSONPathsAndTypes(json) from test;
+select distinct json.a0 from test order by json.a0.:Int64;
+select distinct json.a99 from test order by json.a99.:Int64;
+drop table test;
+
+create table test (json Object('json')) engine=MergeTree order by tuple() settings min_rows_for_wide_part=10000000, min_bytes_for_wide_part=100000000;
+insert into test select toJSONString(map('a' || number % 100, number)) from numbers(1000);
+alter table test modify column json JSON;
+select distinctJSONPathsAndTypes(json) from test;
+select distinct json.a0 from test order by json.a0.:Int64;
+select distinct json.a99 from test order by json.a99.:Int64;
+drop table test;
+
+create table test (json Object('json')) engine=MergeTree order by tuple() settings min_rows_for_wide_part=10000000, min_bytes_for_wide_part=100000000;
+insert into test select toJSONString(map('a' || number % 100, number)) from numbers(1000);
+alter table test modify column json JSON(max_dynamic_paths=10);
+select distinctJSONPathsAndTypes(json) from test;
+select distinct json.a0 from test order by json.a0.:Int64;
+select distinct json.a99 from test order by json.a99.:Int64;
+drop table test;
+
+create table test (json Object('json')) engine=MergeTree order by tuple() settings min_rows_for_wide_part=1, min_bytes_for_wide_part=1;
+insert into test select toJSONString(map('a' || number % 100, number)) from numbers(1000);
+alter table test modify column json JSON();
+select distinctJSONPathsAndTypes(json) from test;
+select distinct json.a0 from test order by json.a0.:Int64;
+select distinct json.a99 from test order by json.a99.:Int64;
+drop table test;
+
+create table test (json Object('json')) engine=MergeTree order by tuple() settings min_rows_for_wide_part=1, min_bytes_for_wide_part=1;
+insert into test select toJSONString(map('a' || number % 100, number)) from numbers(1000);
+alter table test modify column json JSON(max_dynamic_paths=10);
+select distinctJSONPathsAndTypes(json) from test;
+select distinct json.a0 from test order by json.a0.:Int64;
+select distinct json.a99 from test order by json.a99.:Int64;
+drop table test;

tests/queries/0_stateless/03271_ghdata_object_to_json_alter.reference

@@ -0,0 +1,12 @@
+5000
+leonardomso/33-js-concepts	3
+ytdl-org/youtube-dl	3
+Bogdanp/neko	2
+bminossi/AllVideoPocsFromHackerOne	2
+disclose/diodata	2
+Commit	182
+chipeo345	119
+phanwi346	114
+Nicholas Piggin	95
+direwolf-github	49
+2

tests/queries/0_stateless/03271_ghdata_object_to_json_alter.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Tags: no-fasttest, long
+
+CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+# shellcheck source=../shell_config.sh
+. "$CUR_DIR"/../shell_config.sh
+
+${CLICKHOUSE_CLIENT} -q "DROP TABLE IF EXISTS ghdata"
+${CLICKHOUSE_CLIENT} -q "CREATE TABLE ghdata (data Object('json')) ENGINE = MergeTree ORDER BY tuple() SETTINGS index_granularity = 8192, index_granularity_bytes = '10Mi'" --allow_experimental_object_type
+
+cat $CUR_DIR/data_json/ghdata_sample.json | ${CLICKHOUSE_CLIENT} \
+  --max_memory_usage 10G --query "INSERT INTO ghdata FORMAT JSONAsObject"
+
+${CLICKHOUSE_CLIENT} -q "ALTER TABLE ghdata MODIFY column data JSON SETTINGS mutations_sync=1" --allow_experimental_json_type 1
+
+${CLICKHOUSE_CLIENT} -q "SELECT count() FROM ghdata WHERE NOT ignore(*)"
+
+${CLICKHOUSE_CLIENT} -q \
+"SELECT data.repo.name, count() AS stars FROM ghdata \
+    WHERE data.type = 'WatchEvent' GROUP BY data.repo.name ORDER BY stars DESC, data.repo.name LIMIT 5" --allow_suspicious_types_in_group_by=1 --allow_suspicious_types_in_order_by=1
+
+${CLICKHOUSE_CLIENT} --enable_analyzer=1 -q \
+"SELECT data.payload.commits[].author.name AS name, count() AS c FROM ghdata \
+    ARRAY JOIN data.payload.commits[].author.name \
+    GROUP BY name ORDER BY c DESC, name LIMIT 5" --allow_suspicious_types_in_group_by=1 --allow_suspicious_types_in_order_by=1
+
+${CLICKHOUSE_CLIENT} -q "SELECT max(data.payload.pull_request.assignees[].size0) FROM ghdata"
+
+${CLICKHOUSE_CLIENT} -q "DROP TABLE IF EXISTS ghdata"

tests/queries/0_stateless/03272_ghdata_object_to_json_alter_shared_data.reference

@@ -0,0 +1,12 @@
+5000
+leonardomso/33-js-concepts	3
+ytdl-org/youtube-dl	3
+Bogdanp/neko	2
+bminossi/AllVideoPocsFromHackerOne	2
+disclose/diodata	2
+Commit	182
+chipeo345	119
+phanwi346	114
+Nicholas Piggin	95
+direwolf-github	49
+2

tests/queries/0_stateless/03272_ghdata_object_to_json_alter_shared_data.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Tags: no-fasttest, no-msan, no-tsan, no-asan, long
+
+CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+# shellcheck source=../shell_config.sh
+. "$CUR_DIR"/../shell_config.sh
+
+${CLICKHOUSE_CLIENT} -q "DROP TABLE IF EXISTS ghdata"
+${CLICKHOUSE_CLIENT} -q "CREATE TABLE ghdata (data Object('json')) ENGINE = MergeTree ORDER BY tuple() SETTINGS index_granularity = 8192, index_granularity_bytes = '10Mi'" --allow_experimental_object_type=1
+
+cat $CUR_DIR/data_json/ghdata_sample.json | ${CLICKHOUSE_CLIENT} \
+  --max_memory_usage 10G --query "INSERT INTO ghdata FORMAT JSONAsObject"
+
+${CLICKHOUSE_CLIENT} -q "ALTER TABLE ghdata MODIFY column data JSON(max_dynamic_paths=8) SETTINGS mutations_sync=1" --allow_experimental_json_type 1
+
+${CLICKHOUSE_CLIENT} -q "SELECT count() FROM ghdata WHERE NOT ignore(*)"
+
+${CLICKHOUSE_CLIENT} -q \
+"SELECT data.repo.name, count() AS stars FROM ghdata \
+    WHERE data.type = 'WatchEvent' GROUP BY data.repo.name ORDER BY stars DESC, data.repo.name LIMIT 5" --allow_suspicious_types_in_group_by=1 --allow_suspicious_types_in_order_by=1
+
+${CLICKHOUSE_CLIENT} --enable_analyzer=1 -q \
+"SELECT data.payload.commits[].author.name AS name, count() AS c FROM ghdata \
+    ARRAY JOIN data.payload.commits[].author.name \
+    GROUP BY name ORDER BY c DESC, name LIMIT 5" --allow_suspicious_types_in_group_by=1 --allow_suspicious_types_in_order_by=1
+
+${CLICKHOUSE_CLIENT} -q "SELECT max(data.payload.pull_request.assignees[].size0) FROM ghdata"
+
+${CLICKHOUSE_CLIENT} -q "DROP TABLE IF EXISTS ghdata"

utils/check-style/aspell-ignore/en/aspell-dict.txt

@@ -244,7 +244,10 @@ Deduplication
 DefaultTableEngine
 DelayedInserts
 DeliveryTag
+Deltalake
 DeltaLake
+deltalakeCluster
+deltaLakeCluster
 Denormalize
 DestroyAggregatesThreads
 DestroyAggregatesThreadsActive
@@ -377,10 +380,15 @@ Homebrew's
 HorizontalDivide
 Hostname
 HouseOps
+hudi
 Hudi
+hudiCluster
+HudiCluster
 HyperLogLog
 Hypot
 IANA
+icebergCluster
+IcebergCluster
 IDE
 IDEs
 IDNA