#pragma once #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include namespace DB { namespace ErrorCodes { extern const int DICTIONARIES_WAS_NOT_LOADED; extern const int UNSUPPORTED_METHOD; extern const int UNKNOWN_TYPE; extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; extern const int TYPE_MISMATCH; } /** Functions that use plug-ins (external) dictionaries. * * Get the value of the attribute of the specified type. * dictGetType(dictionary, attribute, id), * Type - placeholder for the type name, any numeric and string types are currently supported. * The type must match the actual attribute type with which it was declared in the dictionary structure. * * Get an array of identifiers, consisting of the source and parents chain. * dictGetHierarchy(dictionary, id). * * Is the first identifier the child of the second. * dictIsIn(dictionary, child_id, parent_id). */ class FunctionDictHas final : public IFunction { public: static constexpr auto name = "dictHas"; static FunctionPtr create(const Context & context) { return std::make_shared(context.getExternalDictionaries()); } FunctionDictHas(const ExternalDictionaries & dictionaries) : dictionaries(dictionaries) {} String getName() const override { return name; } private: size_t getNumberOfArguments() const override { return 2; } bool useDefaultImplementationForConstants() const final { return true; } ColumnNumbers getArgumentsThatAreAlwaysConstant() const final { return {0}; } DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override { if (!arguments[0]->isString()) throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!checkDataType(arguments[1].get()) && !checkDataType(arguments[1].get())) throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() + ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; return std::make_shared(); } bool isDeterministic() override { return false; } void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override { const auto dict_name_col = checkAndGetColumnConst(block.getByPosition(arguments[0]).column.get()); if (!dict_name_col) throw Exception{"First argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; /** Do not require existence of the dictionary if the function is called for empty block. * This is needed to allow successful query analysis on a server, * that is the initiator of a distributed query, * in the case when the function will be invoked for real data only at the remote servers. * This feature is controversial and implemented specially * for backward compatibility with the case in Yandex Banner System. */ if (input_rows_count== 0) { auto & elem = block.getByPosition(result); elem.column = elem.type->createColumn(); return; } auto dict = dictionaries.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); if (!executeDispatchSimple(block, arguments, result, dict_ptr) && !executeDispatchSimple(block, arguments, result, dict_ptr) && !executeDispatchSimple(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr)) throw Exception{"Unsupported dictionary type " + dict_ptr->getTypeName(), ErrorCodes::UNKNOWN_TYPE}; } template bool executeDispatchSimple( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; const auto id_col_untyped = block.getByPosition(arguments[1]).column.get(); if (const auto id_col = checkAndGetColumn(id_col_untyped)) { const auto & ids = id_col->getData(); auto out = ColumnUInt8::create(ext::size(ids)); dict->has(ids, out->getData()); block.getByPosition(result).column = std::move(out); } else throw Exception{"Second argument of function " + getName() + " must be UInt64", ErrorCodes::ILLEGAL_COLUMN}; return true; } template bool executeDispatchComplex( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; const ColumnWithTypeAndName & key_col_with_type = block.getByPosition(arguments[1]); const ColumnPtr & key_col = key_col_with_type.column; if (checkColumn(key_col.get())) { const auto & key_columns = static_cast(*key_col).getColumns(); const auto & key_types = static_cast(*key_col_with_type.type).getElements(); auto out = ColumnUInt8::create(key_col_with_type.column->size()); dict->has(key_columns, key_types, out->getData()); block.getByPosition(result).column = std::move(out); } else throw Exception{"Second argument of function " + getName() + " must be " + dict->getKeyDescription(), ErrorCodes::TYPE_MISMATCH}; return true; } const ExternalDictionaries & dictionaries; }; static bool isDictGetFunctionInjective(const ExternalDictionaries & dictionaries, const Block & sample_block) { if (sample_block.columns() != 3 && sample_block.columns() != 4) throw Exception{"Function dictGet... takes 3 or 4 arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; const auto dict_name_col = checkAndGetColumnConst(sample_block.getByPosition(0).column.get()); if (!dict_name_col) throw Exception{"First argument of function dictGet... must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; const auto attr_name_col = checkAndGetColumnConst(sample_block.getByPosition(1).column.get()); if (!attr_name_col) throw Exception{"Second argument of function dictGet... must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; return dictionaries.getDictionary(dict_name_col->getValue())->isInjective(attr_name_col->getValue()); } class FunctionDictGetString final : public IFunction { public: static constexpr auto name = "dictGetString"; static FunctionPtr create(const Context & context) { return std::make_shared(context.getExternalDictionaries()); } FunctionDictGetString(const ExternalDictionaries & dictionaries) : dictionaries(dictionaries) {} String getName() const override { return name; } private: bool isVariadic() const override { return true; } size_t getNumberOfArguments() const override { return 0; } bool useDefaultImplementationForConstants() const final { return true; } ColumnNumbers getArgumentsThatAreAlwaysConstant() const final { return {0, 1}; } bool isInjective(const Block & sample_block) override { return isDictGetFunctionInjective(dictionaries, sample_block); } DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override { if (arguments.size() != 3 && arguments.size() != 4) throw Exception{"Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size()) + ", should be 3 or 4.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; if (!arguments[0]->isString()) { throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; } if (!arguments[1]->isString()) { throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; } if (!checkDataType(arguments[2].get()) && !checkDataType(arguments[2].get())) { throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName() + ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; } /// This is for the case of range dictionaries. if (arguments.size() == 4 && !checkDataType(arguments[3].get())) { throw Exception{"Illegal type " + arguments[3]->getName() + " of fourth argument of function " + getName() + ", must be Date.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; } return std::make_shared(); } bool isDeterministic() override { return false; } void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override { const auto dict_name_col = checkAndGetColumnConst(block.getByPosition(arguments[0]).column.get()); if (!dict_name_col) throw Exception{"First argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; if (input_rows_count == 0) { auto & elem = block.getByPosition(result); elem.column = elem.type->createColumn(); return; } auto dict = dictionaries.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); if (!executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchRange(block, arguments, result, dict_ptr)) throw Exception{"Unsupported dictionary type " + dict_ptr->getTypeName(), ErrorCodes::UNKNOWN_TYPE}; } template bool executeDispatch( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; if (arguments.size() != 3) throw Exception{"Function " + getName() + " for dictionary of type " + dict->getTypeName() + " requires exactly 3 arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const auto id_col_untyped = block.getByPosition(arguments[2]).column.get(); if (const auto id_col = checkAndGetColumn(id_col_untyped)) { auto out = ColumnString::create(); dict->getString(attr_name, id_col->getData(), out.get()); block.getByPosition(result).column = std::move(out); } else throw Exception{"Third argument of function " + getName() + " must be UInt64", ErrorCodes::ILLEGAL_COLUMN}; return true; } template bool executeDispatchComplex( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; if (arguments.size() != 3) throw Exception{"Function " + getName() + " for dictionary of type " + dict->getTypeName() + " requires exactly 3 arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const ColumnWithTypeAndName & key_col_with_type = block.getByPosition(arguments[2]); ColumnPtr key_col = key_col_with_type.column; /// Functions in external dictionaries only support full-value (not constant) columns with keys. if (ColumnPtr key_col_materialized = key_col_with_type.column->convertToFullColumnIfConst()) key_col = key_col_materialized; if (checkColumn(key_col.get())) { const auto & key_columns = static_cast(*key_col).getColumns(); const auto & key_types = static_cast(*key_col_with_type.type).getElements(); auto out = ColumnString::create(); dict->getString(attr_name, key_columns, key_types, out.get()); block.getByPosition(result).column = std::move(out); } else throw Exception{"Third argument of function " + getName() + " must be " + dict->getKeyDescription(), ErrorCodes::TYPE_MISMATCH}; return true; } template bool executeDispatchRange( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; if (arguments.size() != 4) throw Exception{"Function " + getName() + " for dictionary of type " + dict->getTypeName() + " requires exactly 4 arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const auto id_col_untyped = block.getByPosition(arguments[2]).column.get(); const auto date_col_untyped = block.getByPosition(arguments[3]).column.get(); if (const auto id_col = checkAndGetColumn(id_col_untyped)) executeRange(block, result, dict, attr_name, id_col, date_col_untyped); else if (const auto id_col = checkAndGetColumnConst>(id_col_untyped)) executeRange(block, result, dict, attr_name, id_col, date_col_untyped); else { throw Exception{"Third argument of function " + getName() + " must be UInt64", ErrorCodes::ILLEGAL_COLUMN}; } return true; } template void executeRange( Block & block, const size_t result, const DictionaryType * dictionary, const std::string & attr_name, const ColumnUInt64 * id_col, const IColumn * date_col_untyped) { if (const auto date_col = checkAndGetColumn(date_col_untyped)) { auto out = ColumnString::create(); dictionary->getString(attr_name, id_col->getData(), date_col->getData(), out.get()); block.getByPosition(result).column = std::move(out); } else if (const auto date_col = checkAndGetColumnConst>(date_col_untyped)) { auto out = ColumnString::create(); const PaddedPODArray dates(id_col->size(), date_col->getValue()); dictionary->getString(attr_name, id_col->getData(), dates, out.get()); block.getByPosition(result).column = std::move(out); } else { throw Exception{"Fourth argument of function " + getName() + " must be Date", ErrorCodes::ILLEGAL_COLUMN}; } } template void executeRange( Block & block, const size_t result, const DictionaryType * dictionary, const std::string & attr_name, const ColumnConst * id_col, const IColumn * date_col_untyped) { if (const auto date_col = checkAndGetColumn(date_col_untyped)) { auto out = ColumnString::create(); const PaddedPODArray ids(date_col->size(), id_col->getValue()); dictionary->getString(attr_name, ids, date_col->getData(), out.get()); block.getByPosition(result).column = std::move(out); } else if (const auto date_col = checkAndGetColumnConst>(date_col_untyped)) { const PaddedPODArray ids(1, id_col->getValue()); const PaddedPODArray dates(1, date_col->getValue()); auto out = ColumnString::create(); dictionary->getString(attr_name, ids, dates, out.get()); block.getByPosition(result).column = DataTypeString().createColumnConst(id_col->size(), out->getDataAt(0).toString()); } else { throw Exception{"Fourth argument of function " + getName() + " must be Date", ErrorCodes::ILLEGAL_COLUMN}; } } const ExternalDictionaries & dictionaries; }; class FunctionDictGetStringOrDefault final : public IFunction { public: static constexpr auto name = "dictGetStringOrDefault"; static FunctionPtr create(const Context & context) { return std::make_shared(context.getExternalDictionaries()); } FunctionDictGetStringOrDefault(const ExternalDictionaries & dictionaries) : dictionaries(dictionaries) {} String getName() const override { return name; } private: size_t getNumberOfArguments() const override { return 4; } bool useDefaultImplementationForConstants() const final { return true; } ColumnNumbers getArgumentsThatAreAlwaysConstant() const final { return {0, 1}; } DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override { if (!arguments[0]->isString()) throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!arguments[1]->isString()) throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!checkDataType(arguments[2].get()) && !checkDataType(arguments[2].get())) { throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName() + ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; } if (!arguments[3]->isString()) throw Exception{"Illegal type " + arguments[3]->getName() + " of fourth argument of function " + getName() + ", must be String.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; return std::make_shared(); } bool isDeterministic() override { return false; } void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override { const auto dict_name_col = checkAndGetColumnConst(block.getByPosition(arguments[0]).column.get()); if (!dict_name_col) throw Exception{"First argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; if (input_rows_count == 0) { auto & elem = block.getByPosition(result); elem.column = elem.type->createColumn(); return; } auto dict = dictionaries.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); if (!executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr)) throw Exception{"Unsupported dictionary type " + dict_ptr->getTypeName(), ErrorCodes::UNKNOWN_TYPE}; } template bool executeDispatch( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const auto id_col_untyped = block.getByPosition(arguments[2]).column.get(); if (const auto id_col = checkAndGetColumn(id_col_untyped)) executeDispatch(block, arguments, result, dict, attr_name, id_col); else if (const auto id_col = checkAndGetColumnConst>(id_col_untyped)) executeDispatch(block, arguments, result, dict, attr_name, id_col); else throw Exception{"Third argument of function " + getName() + " must be UInt64", ErrorCodes::ILLEGAL_COLUMN}; return true; } template void executeDispatch( Block & block, const ColumnNumbers & arguments, const size_t result, const DictionaryType * dictionary, const std::string & attr_name, const ColumnUInt64 * id_col) { const auto default_col_untyped = block.getByPosition(arguments[3]).column.get(); if (const auto default_col = checkAndGetColumn(default_col_untyped)) { /// vector ids, vector defaults auto out = ColumnString::create(); const auto & ids = id_col->getData(); dictionary->getString(attr_name, ids, default_col, out.get()); block.getByPosition(result).column = std::move(out); } else if (const auto default_col = checkAndGetColumnConstStringOrFixedString(default_col_untyped)) { /// vector ids, const defaults auto out = ColumnString::create(); const auto & ids = id_col->getData(); String def = default_col->getValue(); dictionary->getString(attr_name, ids, def, out.get()); block.getByPosition(result).column = std::move(out); } else throw Exception{"Fourth argument of function " + getName() + " must be String", ErrorCodes::ILLEGAL_COLUMN}; } template void executeDispatch( Block & block, const ColumnNumbers & arguments, const size_t result, const DictionaryType * dictionary, const std::string & attr_name, const ColumnConst * id_col) { const auto default_col_untyped = block.getByPosition(arguments[3]).column.get(); if (const auto default_col = checkAndGetColumn(default_col_untyped)) { /// const ids, vector defaults /// @todo avoid materialization const PaddedPODArray ids(id_col->size(), id_col->getValue()); auto out = ColumnString::create(); dictionary->getString(attr_name, ids, default_col, out.get()); block.getByPosition(result).column = std::move(out); } else if (const auto default_col = checkAndGetColumnConstStringOrFixedString(default_col_untyped)) { /// const ids, const defaults const PaddedPODArray ids(1, id_col->getValue()); auto out = ColumnString::create(); String def = default_col->getValue(); dictionary->getString(attr_name, ids, def, out.get()); block.getByPosition(result).column = DataTypeString().createColumnConst(id_col->size(), out->getDataAt(0).toString()); } else throw Exception{"Fourth argument of function " + getName() + " must be String", ErrorCodes::ILLEGAL_COLUMN}; } template bool executeDispatchComplex( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const ColumnWithTypeAndName & key_col_with_type = block.getByPosition(arguments[2]); ColumnPtr key_col = key_col_with_type.column; /// Functions in external dictionaries only support full-value (not constant) columns with keys. if (ColumnPtr key_col_materialized = key_col_with_type.column->convertToFullColumnIfConst()) key_col = key_col_materialized; const auto & key_columns = typeid_cast(*key_col).getColumns(); const auto & key_types = static_cast(*key_col_with_type.type).getElements(); auto out = ColumnString::create(); const auto default_col_untyped = block.getByPosition(arguments[3]).column.get(); if (const auto default_col = checkAndGetColumn(default_col_untyped)) { dict->getString(attr_name, key_columns, key_types, default_col, out.get()); } else if (const auto default_col = checkAndGetColumnConstStringOrFixedString(default_col_untyped)) { String def = default_col->getValue(); dict->getString(attr_name, key_columns, key_types, def, out.get()); } else throw Exception{"Fourth argument of function " + getName() + " must be String", ErrorCodes::ILLEGAL_COLUMN}; block.getByPosition(result).column = std::move(out); return true; } const ExternalDictionaries & dictionaries; }; template struct DictGetTraits; #define DECLARE_DICT_GET_TRAITS(TYPE, DATA_TYPE) \ template <> struct DictGetTraits\ {\ template \ static void get(\ const DictionaryType * dict, const std::string & name, const PaddedPODArray & ids,\ PaddedPODArray & out)\ {\ dict->get##TYPE(name, ids, out);\ }\ template \ static void get(\ const DictionaryType * dict, const std::string & name, const Columns & key_columns,\ const DataTypes & key_types, PaddedPODArray & out)\ {\ dict->get##TYPE(name, key_columns, key_types, out);\ }\ template \ static void get(\ const DictionaryType * dict, const std::string & name, const PaddedPODArray & ids,\ const PaddedPODArray & dates, PaddedPODArray & out)\ {\ dict->get##TYPE(name, ids, dates, out);\ }\ template \ static void getOrDefault(\ const DictionaryType * dict, const std::string & name, const PaddedPODArray & ids,\ const DefaultsType & def, PaddedPODArray & out)\ {\ dict->get##TYPE(name, ids, def, out);\ }\ template \ static void getOrDefault(\ const DictionaryType * dict, const std::string & name, const Columns & key_columns,\ const DataTypes & key_types, const DefaultsType & def, PaddedPODArray & out)\ {\ dict->get##TYPE(name, key_columns, key_types, def, out);\ }\ }; DECLARE_DICT_GET_TRAITS(UInt8, DataTypeUInt8) DECLARE_DICT_GET_TRAITS(UInt16, DataTypeUInt16) DECLARE_DICT_GET_TRAITS(UInt32, DataTypeUInt32) DECLARE_DICT_GET_TRAITS(UInt64, DataTypeUInt64) DECLARE_DICT_GET_TRAITS(Int8, DataTypeInt8) DECLARE_DICT_GET_TRAITS(Int16, DataTypeInt16) DECLARE_DICT_GET_TRAITS(Int32, DataTypeInt32) DECLARE_DICT_GET_TRAITS(Int64, DataTypeInt64) DECLARE_DICT_GET_TRAITS(Float32, DataTypeFloat32) DECLARE_DICT_GET_TRAITS(Float64, DataTypeFloat64) DECLARE_DICT_GET_TRAITS(UInt16, DataTypeDate) DECLARE_DICT_GET_TRAITS(UInt32, DataTypeDateTime) DECLARE_DICT_GET_TRAITS(UInt128, DataTypeUUID) #undef DECLARE_DICT_GET_TRAITS template class FunctionDictGet final : public IFunction { using Type = typename DataType::FieldType; public: static constexpr auto name = Name::name; static FunctionPtr create(const Context & context) { return std::make_shared(context.getExternalDictionaries()); } FunctionDictGet(const ExternalDictionaries & dictionaries) : dictionaries(dictionaries) {} String getName() const override { return name; } private: bool isVariadic() const override { return true; } size_t getNumberOfArguments() const override { return 0; } bool useDefaultImplementationForConstants() const final { return true; } ColumnNumbers getArgumentsThatAreAlwaysConstant() const final { return {0, 1}; } bool isInjective(const Block & sample_block) override { return isDictGetFunctionInjective(dictionaries, sample_block); } DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override { if (arguments.size() != 3 && arguments.size() != 4) throw Exception{"Function " + getName() + " takes 3 or 4 arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; if (!arguments[0]->isString()) throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!arguments[1]->isString()) throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!checkDataType(arguments[2].get()) && !checkDataType(arguments[2].get())) throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName() + ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (arguments.size() == 4 && !checkDataType(arguments[3].get())) throw Exception{"Illegal type " + arguments[3]->getName() + " of fourth argument of function " + getName() + ", must be Date.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; return std::make_shared(); } bool isDeterministic() override { return false; } void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override { const auto dict_name_col = checkAndGetColumnConst(block.getByPosition(arguments[0]).column.get()); if (!dict_name_col) throw Exception{"First argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; if (input_rows_count == 0) { auto & elem = block.getByPosition(result); elem.column = elem.type->createColumn(); return; } auto dict = dictionaries.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); if (!executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchRange(block, arguments, result, dict_ptr)) throw Exception{"Unsupported dictionary type " + dict_ptr->getTypeName(), ErrorCodes::UNKNOWN_TYPE}; } template bool executeDispatch(Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; if (arguments.size() != 3) throw Exception{"Function " + getName() + " for dictionary of type " + dict->getTypeName() + " requires exactly 3 arguments.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const auto id_col_untyped = block.getByPosition(arguments[2]).column.get(); if (const auto id_col = checkAndGetColumn(id_col_untyped)) { auto out = ColumnVector::create(id_col->size()); const auto & ids = id_col->getData(); auto & data = out->getData(); DictGetTraits::get(dict, attr_name, ids, data); block.getByPosition(result).column = std::move(out); } else if (const auto id_col = checkAndGetColumnConst>(id_col_untyped)) { const PaddedPODArray ids(1, id_col->getValue()); PaddedPODArray data(1); DictGetTraits::get(dict, attr_name, ids, data); block.getByPosition(result).column = DataTypeNumber().createColumnConst(id_col->size(), toField(data.front())); } else throw Exception{"Third argument of function " + getName() + " must be UInt64", ErrorCodes::ILLEGAL_COLUMN}; return true; } template bool executeDispatchComplex( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; if (arguments.size() != 3) throw Exception{"Function " + getName() + " for dictionary of type " + dict->getTypeName() + " requires exactly 3 arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const ColumnWithTypeAndName & key_col_with_type = block.getByPosition(arguments[2]); ColumnPtr key_col = key_col_with_type.column; /// Functions in external dictionaries only support full-value (not constant) columns with keys. if (ColumnPtr key_col_materialized = key_col_with_type.column->convertToFullColumnIfConst()) key_col = key_col_materialized; if (checkColumn(key_col.get())) { const auto & key_columns = static_cast(*key_col).getColumns(); const auto & key_types = static_cast(*key_col_with_type.type).getElements(); auto out = ColumnVector::create(key_columns.front()->size()); auto & data = out->getData(); DictGetTraits::get(dict, attr_name, key_columns, key_types, data); block.getByPosition(result).column = std::move(out); } else throw Exception{"Third argument of function " + getName() + " must be " + dict->getKeyDescription(), ErrorCodes::TYPE_MISMATCH}; return true; } template bool executeDispatchRange( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; if (arguments.size() != 4) throw Exception{"Function " + getName() + " for dictionary of type " + dict->getTypeName() + " requires exactly 4 arguments", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH}; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const auto id_col_untyped = block.getByPosition(arguments[2]).column.get(); const auto date_col_untyped = block.getByPosition(arguments[3]).column.get(); if (const auto id_col = checkAndGetColumn(id_col_untyped)) executeRange(block, result, dict, attr_name, id_col, date_col_untyped); else if (const auto id_col = checkAndGetColumnConst>(id_col_untyped)) executeRange(block, result, dict, attr_name, id_col, date_col_untyped); else throw Exception{"Third argument of function " + getName() + " must be UInt64", ErrorCodes::ILLEGAL_COLUMN}; return true; } template void executeRange( Block & block, const size_t result, const DictionaryType * dictionary, const std::string & attr_name, const ColumnUInt64 * id_col, const IColumn * date_col_untyped) { if (const auto date_col = checkAndGetColumn(date_col_untyped)) { const auto size = id_col->size(); const auto & ids = id_col->getData(); const auto & dates = date_col->getData(); auto out = ColumnVector::create(size); auto & data = out->getData(); DictGetTraits::get(dictionary, attr_name, ids, dates, data); block.getByPosition(result).column = std::move(out); } else if (const auto date_col = checkAndGetColumnConst>(date_col_untyped)) { const auto size = id_col->size(); const auto & ids = id_col->getData(); const PaddedPODArray dates(size, date_col->getValue()); auto out = ColumnVector::create(size); auto & data = out->getData(); DictGetTraits::get(dictionary, attr_name, ids, dates, data); block.getByPosition(result).column = std::move(out); } else throw Exception{"Fourth argument of function " + getName() + " must be Date", ErrorCodes::ILLEGAL_COLUMN}; } template void executeRange( Block & block, const size_t result, const DictionaryType * dictionary, const std::string & attr_name, const ColumnConst * id_col, const IColumn * date_col_untyped) { if (const auto date_col = checkAndGetColumn(date_col_untyped)) { const auto size = date_col->size(); const PaddedPODArray ids(size, id_col->getValue()); const auto & dates = date_col->getData(); auto out = ColumnVector::create(size); auto & data = out->getData(); DictGetTraits::get(dictionary, attr_name, ids, dates, data); block.getByPosition(result).column = std::move(out); } else if (const auto date_col = checkAndGetColumnConst>(date_col_untyped)) { const PaddedPODArray ids(1, id_col->getValue()); const PaddedPODArray dates(1, date_col->getValue()); PaddedPODArray data(1); DictGetTraits::get(dictionary, attr_name, ids, dates, data); block.getByPosition(result).column = DataTypeNumber().createColumnConst(id_col->size(), toField(data.front())); } else throw Exception{"Fourth argument of function " + getName() + " must be Date", ErrorCodes::ILLEGAL_COLUMN}; } const ExternalDictionaries & dictionaries; }; struct NameDictGetUInt8 { static constexpr auto name = "dictGetUInt8"; }; struct NameDictGetUInt16 { static constexpr auto name = "dictGetUInt16"; }; struct NameDictGetUInt32 { static constexpr auto name = "dictGetUInt32"; }; struct NameDictGetUInt64 { static constexpr auto name = "dictGetUInt64"; }; struct NameDictGetInt8 { static constexpr auto name = "dictGetInt8"; }; struct NameDictGetInt16 { static constexpr auto name = "dictGetInt16"; }; struct NameDictGetInt32 { static constexpr auto name = "dictGetInt32"; }; struct NameDictGetInt64 { static constexpr auto name = "dictGetInt64"; }; struct NameDictGetFloat32 { static constexpr auto name = "dictGetFloat32"; }; struct NameDictGetFloat64 { static constexpr auto name = "dictGetFloat64"; }; struct NameDictGetDate { static constexpr auto name = "dictGetDate"; }; struct NameDictGetDateTime { static constexpr auto name = "dictGetDateTime"; }; struct NameDictGetUUID { static constexpr auto name = "dictGetUUID"; }; using FunctionDictGetUInt8 = FunctionDictGet; using FunctionDictGetUInt16 = FunctionDictGet; using FunctionDictGetUInt32 = FunctionDictGet; using FunctionDictGetUInt64 = FunctionDictGet; using FunctionDictGetInt8 = FunctionDictGet; using FunctionDictGetInt16 = FunctionDictGet; using FunctionDictGetInt32 = FunctionDictGet; using FunctionDictGetInt64 = FunctionDictGet; using FunctionDictGetFloat32 = FunctionDictGet; using FunctionDictGetFloat64 = FunctionDictGet; using FunctionDictGetDate = FunctionDictGet; using FunctionDictGetDateTime = FunctionDictGet; using FunctionDictGetUUID = FunctionDictGet; template class FunctionDictGetOrDefault final : public IFunction { using Type = typename DataType::FieldType; public: static constexpr auto name = Name::name; static FunctionPtr create(const Context & context) { return std::make_shared(context.getExternalDictionaries()); } FunctionDictGetOrDefault(const ExternalDictionaries & dictionaries) : dictionaries(dictionaries) {} String getName() const override { return name; } private: size_t getNumberOfArguments() const override { return 4; } bool useDefaultImplementationForConstants() const final { return true; } ColumnNumbers getArgumentsThatAreAlwaysConstant() const final { return {0, 1}; } DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override { if (!arguments[0]->isString()) throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!arguments[1]->isString()) throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!checkDataType(arguments[2].get()) && !checkDataType(arguments[2].get())) throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName() + ", must be UInt64 or tuple(...).", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!checkDataType(arguments[3].get())) throw Exception{"Illegal type " + arguments[3]->getName() + " of fourth argument of function " + getName() + ", must be " + String(DataType{}.getFamilyName()) + ".", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; return std::make_shared(); } bool isDeterministic() override { return false; } void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override { const auto dict_name_col = checkAndGetColumnConst(block.getByPosition(arguments[0]).column.get()); if (!dict_name_col) throw Exception{"First argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; if (input_rows_count == 0) { auto & elem = block.getByPosition(result); elem.column = elem.type->createColumn(); return; } auto dict = dictionaries.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); if (!executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr) && !executeDispatchComplex(block, arguments, result, dict_ptr)) throw Exception{"Unsupported dictionary type " + dict_ptr->getTypeName(), ErrorCodes::UNKNOWN_TYPE}; } template bool executeDispatch(Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const auto id_col_untyped = block.getByPosition(arguments[2]).column.get(); if (const auto id_col = checkAndGetColumn(id_col_untyped)) executeDispatch(block, arguments, result, dict, attr_name, id_col); else if (const auto id_col = checkAndGetColumnConst>(id_col_untyped)) executeDispatch(block, arguments, result, dict, attr_name, id_col); else throw Exception{"Third argument of function " + getName() + " must be UInt64", ErrorCodes::ILLEGAL_COLUMN}; return true; } template void executeDispatch( Block & block, const ColumnNumbers & arguments, const size_t result, const DictionaryType * dictionary, const std::string & attr_name, const ColumnUInt64 * id_col) { const auto default_col_untyped = block.getByPosition(arguments[3]).column.get(); if (const auto default_col = checkAndGetColumn>(default_col_untyped)) { /// vector ids, vector defaults auto out = ColumnVector::create(id_col->size()); const auto & ids = id_col->getData(); auto & data = out->getData(); const auto & defs = default_col->getData(); DictGetTraits::getOrDefault(dictionary, attr_name, ids, defs, data); block.getByPosition(result).column = std::move(out); } else if (const auto default_col = checkAndGetColumnConst>(default_col_untyped)) { /// vector ids, const defaults auto out = ColumnVector::create(id_col->size()); const auto & ids = id_col->getData(); auto & data = out->getData(); const auto def = default_col->template getValue(); DictGetTraits::getOrDefault(dictionary, attr_name, ids, def, data); block.getByPosition(result).column = std::move(out); } else throw Exception{"Fourth argument of function " + getName() + " must be " + String(DataType{}.getFamilyName()), ErrorCodes::ILLEGAL_COLUMN}; } template void executeDispatch( Block & block, const ColumnNumbers & arguments, const size_t result, const DictionaryType * dictionary, const std::string & attr_name, const ColumnConst * id_col) { const auto default_col_untyped = block.getByPosition(arguments[3]).column.get(); if (const auto default_col = checkAndGetColumn>(default_col_untyped)) { /// const ids, vector defaults /// @todo avoid materialization const PaddedPODArray ids(id_col->size(), id_col->getValue()); auto out = ColumnVector::create(id_col->size()); auto & data = out->getData(); const auto & defs = default_col->getData(); DictGetTraits::getOrDefault(dictionary, attr_name, ids, defs, data); block.getByPosition(result).column = std::move(out); } else if (const auto default_col = checkAndGetColumnConst>(default_col_untyped)) { /// const ids, const defaults const PaddedPODArray ids(1, id_col->getValue()); PaddedPODArray data(1); const auto & def = default_col->template getValue(); DictGetTraits::getOrDefault(dictionary, attr_name, ids, def, data); block.getByPosition(result).column = DataTypeNumber().createColumnConst(id_col->size(), toField(data.front())); } else throw Exception{"Fourth argument of function " + getName() + " must be " + String(DataType{}.getFamilyName()), ErrorCodes::ILLEGAL_COLUMN}; } template bool executeDispatchComplex( Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; const auto attr_name_col = checkAndGetColumnConst(block.getByPosition(arguments[1]).column.get()); if (!attr_name_col) throw Exception{"Second argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; String attr_name = attr_name_col->getValue(); const ColumnWithTypeAndName & key_col_with_type = block.getByPosition(arguments[2]); ColumnPtr key_col = key_col_with_type.column; /// Functions in external dictionaries only support full-value (not constant) columns with keys. if (ColumnPtr key_col_materialized = key_col_with_type.column->convertToFullColumnIfConst()) key_col = key_col_materialized; const auto & key_columns = typeid_cast(*key_col).getColumns(); const auto & key_types = static_cast(*key_col_with_type.type).getElements(); /// @todo detect when all key columns are constant const auto rows = key_col->size(); auto out = ColumnVector::create(rows); auto & data = out->getData(); const auto default_col_untyped = block.getByPosition(arguments[3]).column.get(); if (const auto default_col = checkAndGetColumn>(default_col_untyped)) { /// const defaults const auto & defs = default_col->getData(); DictGetTraits::getOrDefault(dict, attr_name, key_columns, key_types, defs, data); } else if (const auto default_col = checkAndGetColumnConst>(default_col_untyped)) { const auto def = default_col->template getValue(); DictGetTraits::getOrDefault(dict, attr_name, key_columns, key_types, def, data); } else throw Exception{"Fourth argument of function " + getName() + " must be " + String(DataType{}.getFamilyName()), ErrorCodes::ILLEGAL_COLUMN}; block.getByPosition(result).column = std::move(out); return true; } const ExternalDictionaries & dictionaries; }; struct NameDictGetUInt8OrDefault { static constexpr auto name = "dictGetUInt8OrDefault"; }; struct NameDictGetUInt16OrDefault { static constexpr auto name = "dictGetUInt16OrDefault"; }; struct NameDictGetUInt32OrDefault { static constexpr auto name = "dictGetUInt32OrDefault"; }; struct NameDictGetUInt64OrDefault { static constexpr auto name = "dictGetUInt64OrDefault"; }; struct NameDictGetInt8OrDefault { static constexpr auto name = "dictGetInt8OrDefault"; }; struct NameDictGetInt16OrDefault { static constexpr auto name = "dictGetInt16OrDefault"; }; struct NameDictGetInt32OrDefault { static constexpr auto name = "dictGetInt32OrDefault"; }; struct NameDictGetInt64OrDefault { static constexpr auto name = "dictGetInt64OrDefault"; }; struct NameDictGetFloat32OrDefault { static constexpr auto name = "dictGetFloat32OrDefault"; }; struct NameDictGetFloat64OrDefault { static constexpr auto name = "dictGetFloat64OrDefault"; }; struct NameDictGetDateOrDefault { static constexpr auto name = "dictGetDateOrDefault"; }; struct NameDictGetDateTimeOrDefault { static constexpr auto name = "dictGetDateTimeOrDefault"; }; struct NameDictGetUUIDOrDefault { static constexpr auto name = "dictGetUUIDOrDefault"; }; using FunctionDictGetUInt8OrDefault = FunctionDictGetOrDefault; using FunctionDictGetUInt16OrDefault = FunctionDictGetOrDefault; using FunctionDictGetUInt32OrDefault = FunctionDictGetOrDefault; using FunctionDictGetUInt64OrDefault = FunctionDictGetOrDefault; using FunctionDictGetInt8OrDefault = FunctionDictGetOrDefault; using FunctionDictGetInt16OrDefault = FunctionDictGetOrDefault; using FunctionDictGetInt32OrDefault = FunctionDictGetOrDefault; using FunctionDictGetInt64OrDefault = FunctionDictGetOrDefault; using FunctionDictGetFloat32OrDefault = FunctionDictGetOrDefault; using FunctionDictGetFloat64OrDefault = FunctionDictGetOrDefault; using FunctionDictGetDateOrDefault = FunctionDictGetOrDefault; using FunctionDictGetDateTimeOrDefault = FunctionDictGetOrDefault; using FunctionDictGetUUIDOrDefault = FunctionDictGetOrDefault; /// Functions to work with hierarchies. class FunctionDictGetHierarchy final : public IFunction { public: static constexpr auto name = "dictGetHierarchy"; static FunctionPtr create(const Context & context) { return std::make_shared(context.getExternalDictionaries()); } FunctionDictGetHierarchy(const ExternalDictionaries & dictionaries) : dictionaries(dictionaries) {} String getName() const override { return name; } private: size_t getNumberOfArguments() const override { return 2; } bool isInjective(const Block & /*sample_block*/) override { return true; } bool useDefaultImplementationForConstants() const final { return true; } ColumnNumbers getArgumentsThatAreAlwaysConstant() const final { return {0}; } DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override { if (!arguments[0]->isString()) throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!checkDataType(arguments[1].get())) throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() + ", must be UInt64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; return std::make_shared(std::make_shared()); } bool isDeterministic() override { return false; } void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override { const auto dict_name_col = checkAndGetColumnConst(block.getByPosition(arguments[0]).column.get()); if (!dict_name_col) throw Exception{"First argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; if (input_rows_count == 0) { auto & elem = block.getByPosition(result); elem.column = elem.type->createColumn(); return; } auto dict = dictionaries.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); if (!executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr)) throw Exception{"Unsupported dictionary type " + dict_ptr->getTypeName(), ErrorCodes::UNKNOWN_TYPE}; } template bool executeDispatch(Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; if (!dict->hasHierarchy()) throw Exception{"Dictionary does not have a hierarchy", ErrorCodes::UNSUPPORTED_METHOD}; const auto get_hierarchies = [&] (const PaddedPODArray & in, PaddedPODArray & out, PaddedPODArray & offsets) { const auto size = in.size(); /// copy of `in` array auto in_array = std::make_unique>(std::begin(in), std::end(in)); /// used for storing and handling result of ::toParent call auto out_array = std::make_unique>(size); /// resulting hierarchies std::vector> hierarchies(size); /// TODO Bad code, poor performance. /// total number of non-zero elements, used for allocating all the required memory upfront size_t total_count = 0; while (true) { auto all_zeroes = true; /// erase zeroed identifiers, store non-zeroed ones for (const auto i : ext::range(0, size)) { const auto id = (*in_array)[i]; if (0 == id) continue; auto & hierarchy = hierarchies[i]; /// Checking for loop if (std::find(std::begin(hierarchy), std::end(hierarchy), id) != std::end(hierarchy)) continue; all_zeroes = false; /// place id at it's corresponding place hierarchy.push_back(id); ++total_count; } if (all_zeroes) break; /// translate all non-zero identifiers at once dict->toParent(*in_array, *out_array); /// we're going to use the `in_array` from this iteration as `out_array` on the next one std::swap(in_array, out_array); } out.reserve(total_count); offsets.resize(size); for (const auto i : ext::range(0, size)) { const auto & ids = hierarchies[i]; out.insert_assume_reserved(std::begin(ids), std::end(ids)); offsets[i] = out.size(); } }; const auto id_col_untyped = block.getByPosition(arguments[1]).column.get(); if (const auto id_col = checkAndGetColumn(id_col_untyped)) { const auto & in = id_col->getData(); auto backend = ColumnUInt64::create(); auto offsets = ColumnArray::ColumnOffsets::create(); get_hierarchies(in, backend->getData(), offsets->getData()); block.getByPosition(result).column = ColumnArray::create(std::move(backend), std::move(offsets)); } else if (const auto id_col = checkAndGetColumnConst>(id_col_untyped)) { const PaddedPODArray in(1, id_col->getValue()); auto backend = ColumnUInt64::create(); auto offsets = ColumnArray::ColumnOffsets::create(); get_hierarchies(in, backend->getData(), offsets->getData()); auto array = ColumnArray::create(std::move(backend), std::move(offsets)); block.getByPosition(result).column = block.getByPosition(result).type->createColumnConst(id_col->size(), (*array)[0].get()); } else throw Exception{"Second argument of function " + getName() + " must be UInt64", ErrorCodes::ILLEGAL_COLUMN}; return true; } const ExternalDictionaries & dictionaries; }; class FunctionDictIsIn final : public IFunction { public: static constexpr auto name = "dictIsIn"; static FunctionPtr create(const Context & context) { return std::make_shared(context.getExternalDictionaries()); } FunctionDictIsIn(const ExternalDictionaries & dictionaries) : dictionaries(dictionaries) {} String getName() const override { return name; } private: size_t getNumberOfArguments() const override { return 3; } bool useDefaultImplementationForConstants() const final { return true; } ColumnNumbers getArgumentsThatAreAlwaysConstant() const final { return {0}; } DataTypePtr getReturnTypeImpl(const DataTypes & arguments) const override { if (!arguments[0]->isString()) throw Exception{"Illegal type " + arguments[0]->getName() + " of first argument of function " + getName() + ", expected a string.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!checkDataType(arguments[1].get())) throw Exception{"Illegal type " + arguments[1]->getName() + " of second argument of function " + getName() + ", must be UInt64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; if (!checkDataType(arguments[2].get())) throw Exception{"Illegal type " + arguments[2]->getName() + " of third argument of function " + getName() + ", must be UInt64.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; return std::make_shared(); } bool isDeterministic() override { return false; } void executeImpl(Block & block, const ColumnNumbers & arguments, size_t result, size_t input_rows_count) override { const auto dict_name_col = checkAndGetColumnConst(block.getByPosition(arguments[0]).column.get()); if (!dict_name_col) throw Exception{"First argument of function " + getName() + " must be a constant string", ErrorCodes::ILLEGAL_COLUMN}; if (input_rows_count == 0) { auto & elem = block.getByPosition(result); elem.column = elem.type->createColumn(); return; } auto dict = dictionaries.getDictionary(dict_name_col->getValue()); const auto dict_ptr = dict.get(); if (!executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr) && !executeDispatch(block, arguments, result, dict_ptr)) throw Exception{"Unsupported dictionary type " + dict_ptr->getTypeName(), ErrorCodes::UNKNOWN_TYPE}; } template bool executeDispatch(Block & block, const ColumnNumbers & arguments, const size_t result, const IDictionaryBase * dictionary) { const auto dict = typeid_cast(dictionary); if (!dict) return false; if (!dict->hasHierarchy()) throw Exception{"Dictionary does not have a hierarchy", ErrorCodes::UNSUPPORTED_METHOD}; const auto child_id_col_untyped = block.getByPosition(arguments[1]).column.get(); const auto ancestor_id_col_untyped = block.getByPosition(arguments[2]).column.get(); if (const auto child_id_col = checkAndGetColumn(child_id_col_untyped)) execute(block, result, dict, child_id_col, ancestor_id_col_untyped); else if (const auto child_id_col = checkAndGetColumnConst>(child_id_col_untyped)) execute(block, result, dict, child_id_col, ancestor_id_col_untyped); else throw Exception{"Illegal column " + child_id_col_untyped->getName() + " of second argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN}; return true; } template bool execute(Block & block, const size_t result, const DictionaryType * dictionary, const ColumnUInt64 * child_id_col, const IColumn * ancestor_id_col_untyped) { if (const auto ancestor_id_col = checkAndGetColumn(ancestor_id_col_untyped)) { auto out = ColumnUInt8::create(); const auto & child_ids = child_id_col->getData(); const auto & ancestor_ids = ancestor_id_col->getData(); auto & data = out->getData(); const auto size = child_id_col->size(); data.resize(size); dictionary->isInVectorVector(child_ids, ancestor_ids, data); block.getByPosition(result).column = std::move(out); } else if (const auto ancestor_id_col = checkAndGetColumnConst>(ancestor_id_col_untyped)) { auto out = ColumnUInt8::create(); const auto & child_ids = child_id_col->getData(); const auto ancestor_id = ancestor_id_col->getValue(); auto & data = out->getData(); const auto size = child_id_col->size(); data.resize(size); dictionary->isInVectorConstant(child_ids, ancestor_id, data); block.getByPosition(result).column = std::move(out); } else { throw Exception{"Illegal column " + ancestor_id_col_untyped->getName() + " of third argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN}; } return true; } template bool execute(Block & block, const size_t result, const DictionaryType * dictionary, const ColumnConst * child_id_col, const IColumn * ancestor_id_col_untyped) { if (const auto ancestor_id_col = checkAndGetColumn(ancestor_id_col_untyped)) { auto out = ColumnUInt8::create(); const auto child_id = child_id_col->getValue(); const auto & ancestor_ids = ancestor_id_col->getData(); auto & data = out->getData(); const auto size = child_id_col->size(); data.resize(size); dictionary->isInConstantVector(child_id, ancestor_ids, data); block.getByPosition(result).column = std::move(out); } else if (const auto ancestor_id_col = checkAndGetColumnConst>(ancestor_id_col_untyped)) { const auto child_id = child_id_col->getValue(); const auto ancestor_id = ancestor_id_col->getValue(); UInt8 res = 0; dictionary->isInConstantConstant(child_id, ancestor_id, res); block.getByPosition(result).column = DataTypeUInt8().createColumnConst(child_id_col->size(), UInt64(res)); } else throw Exception{"Illegal column " + ancestor_id_col_untyped->getName() + " of third argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN}; return true; } const ExternalDictionaries & dictionaries; }; };