#pragma once #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include namespace DB { /** Функции по работе с массивами: * * array(с1, с2, ...) - создать массив из констант. * arrayElement(arr, i) - получить элемент массива по индексу. * Индекс начинается с 1. Также индекс может быть отрицательным - тогда он считается с конца массива. * has(arr, x) - есть ли в массиве элемент x. * indexOf(arr, x) - возвращает индекс элемента x (начиная с 1), если он есть в массиве, или 0, если его нет. * arrayEnumerate(arr) - возаращает массив [1,2,3,..., length(arr)] * arrayEnumerateUniq(arr) - возаращает массив, параллельный данному, где для каждого элемента указано, * какой он по счету среди элементов с таким значением. * Например: arrayEnumerateUniq([10, 20, 10, 30]) = [1, 1, 2, 1] */ class FunctionArray : public IFunction { public: static constexpr auto name = "array"; static IFunction * create(const Context & context) { return new FunctionArray; } private: /// Получить имя функции. String getName() const { return name; } template bool checkRightType(DataTypePtr left, DataTypePtr right, DataTypePtr & type_res) const { if (typeid_cast(&*right)) { typedef typename NumberTraits::ResultOfIf::Type ResultType; type_res = DataTypeFromFieldTypeOrError::getDataType(); if (!type_res) throw Exception("Arguments of function " + getName() + " are not upscalable to a common type without loss of precision.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); return true; } return false; } template bool checkLeftType(DataTypePtr left, DataTypePtr right, DataTypePtr & type_res) const { if (typeid_cast(&*left)) { if ( checkRightType(left, right, type_res) || checkRightType(left, right, type_res) || checkRightType(left, right, type_res) || checkRightType(left, right, type_res) || checkRightType(left, right, type_res) || checkRightType(left, right, type_res) || checkRightType(left, right, type_res) || checkRightType(left, right, type_res) || checkRightType(left, right, type_res) || checkRightType(left, right, type_res)) return true; else throw Exception("Illegal type " + right->getName() + " as argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } return false; } template bool tryAddField(DataTypePtr type_res, const Field & f, Array & arr) const { if (typeid_cast(&*type_res)) { arr.push_back(apply_visitor(FieldVisitorConvertToNumber(), f)); return true; } return false; } bool addField(DataTypePtr type_res, const Field & f, Array & arr) const { /// Иначе необходимо if ( tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) || tryAddField(type_res, f, arr) ) return true; else throw Exception("Illegal result type " + type_res->getName() + " of function " + getName(), ErrorCodes::LOGICAL_ERROR); } DataTypePtr getLeastCommonType(DataTypePtr left, DataTypePtr right) const { DataTypePtr type_res; if (!( checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res) || checkLeftType(left, right, type_res))) throw Exception("Internal error: unexpected type " + left->getName() + " as argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); return type_res; } public: /// Получить тип результата по типам аргументов. Если функция неприменима для данных аргументов - кинуть исключение. DataTypePtr getReturnType(const DataTypes & arguments) const { if (arguments.empty()) throw Exception("Function array requires at least one argument.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); DataTypePtr result_type = arguments[0]; if (result_type->behavesAsNumber()) { /// Если тип числовой, пробуем выделить наименьший общий тип for (size_t i = 1, size = arguments.size(); i < size; ++i) result_type = getLeastCommonType(result_type, arguments[i]); } else { /// Иначе все аргументы должны быть одинаковыми for (size_t i = 1, size = arguments.size(); i < size; ++i) if (arguments[i]->getName() != arguments[0]->getName()) throw Exception("Arguments for function array must have same type or behave as number.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } return new DataTypeArray(result_type); } /// Выполнить функцию над блоком. void execute(Block & block, const ColumnNumbers & arguments, size_t result) { /// Все аргументы должны быть константами. for (size_t i = 0, size = arguments.size(); i < size; ++i) if (!block.getByPosition(arguments[i]).column->isConst()) throw Exception("Arguments for function array must be constant.", ErrorCodes::ILLEGAL_COLUMN); DataTypePtr result_type = block.getByPosition(arguments[0]).type; if (result_type->behavesAsNumber()) { /// Если тип числовой, вычисляем наименьший общий тип for (size_t i = 1, size = arguments.size(); i < size; ++i) result_type = getLeastCommonType(result_type, block.getByPosition(arguments[i]).type); } Array arr; for (size_t i = 0, size = arguments.size(); i < size; ++i) if (block.getByPosition(arguments[i]).type->getName() == result_type->getName()) /// Если элемент такого же типа как результат, просто добавляем его в ответ arr.push_back((*block.getByPosition(arguments[i]).column)[0]); else /// Иначе необходимо привести его к типу результата addField(result_type, (*block.getByPosition(arguments[i]).column)[0], arr); block.getByPosition(result).column = new ColumnConstArray(block.getByPosition(arguments[0]).column->size(), arr, new DataTypeArray(result_type)); } }; template struct ArrayElementNumImpl { /** Процедура для константного идекса * Если negative = false - передаётся индекс с начала массива, начиная с нуля. * Если negative = true - передаётся индекс с конца массива, начиная с нуля. */ template static void vectorConst( const PODArray & data, const ColumnArray::Offsets_t & offsets, const ColumnArray::Offset_t index, PODArray & result) { size_t size = offsets.size(); result.resize(size); ColumnArray::Offset_t current_offset = 0; for (size_t i = 0; i < size; ++i) { size_t array_size = offsets[i] - current_offset; if (index < array_size) result[i] = !negative ? data[current_offset + index] : data[offsets[i] - index - 1]; else result[i] = T(); current_offset = offsets[i]; } } /** Процедура для неконстантного идекса * index_type - тип данных идекса */ template static void vector( const PODArray & data, const ColumnArray::Offsets_t & offsets, const ColumnVector & index, PODArray & result) { size_t size = offsets.size(); result.resize(size); ColumnArray::Offset_t current_offset = 0; for (size_t i = 0; i < size; ++i) { size_t array_size = offsets[i] - current_offset; if (index[i].getType() == Field::Types::UInt64) { UInt64 cur_id = safeGet(index[i]); if (cur_id > 0 && cur_id <= array_size) result[i] = data[current_offset + cur_id - 1]; else result[i] = T(); } else if (index[i].getType() == Field::Types::Int64) { Int64 cur_id = safeGet(index[i]); if (cur_id > 0 && static_cast(cur_id) <= array_size) result[i] = data[current_offset + cur_id - 1]; else if (cur_id < 0 && static_cast(-cur_id) <= array_size) result[i] = data[offsets[i] + cur_id]; else result[i] = T(); } else throw Exception("Illegal type of array index", ErrorCodes::LOGICAL_ERROR); current_offset = offsets[i]; } } }; struct ArrayElementStringImpl { /** Процедура для константного идекса * Если negative = false - передаётся индекс с начала массива, начиная с нуля. * Если negative = true - передаётся индекс с конца массива, начиная с нуля. */ template static void vectorConst( const ColumnString::Chars_t & data, const ColumnArray::Offsets_t & offsets, const ColumnString::Offsets_t & string_offsets, const ColumnArray::Offset_t index, ColumnString::Chars_t & result_data, ColumnArray::Offsets_t & result_offsets) { size_t size = offsets.size(); result_offsets.resize(size); result_data.reserve(data.size()); ColumnArray::Offset_t current_offset = 0; ColumnArray::Offset_t current_result_offset = 0; for (size_t i = 0; i < size; ++i) { size_t array_size = offsets[i] - current_offset; if (index < array_size) { size_t adjusted_index = !negative ? index : (array_size - index - 1); ColumnArray::Offset_t string_pos = current_offset == 0 && adjusted_index == 0 ? 0 : string_offsets[current_offset + adjusted_index - 1]; ColumnArray::Offset_t string_size = string_offsets[current_offset + adjusted_index] - string_pos; result_data.resize(current_result_offset + string_size); memcpy(&result_data[current_result_offset], &data[string_pos], string_size); current_result_offset += string_size; result_offsets[i] = current_result_offset; } else { /// Вставим пустую строку. result_data.resize(current_result_offset + 1); result_data[current_result_offset] = 0; current_result_offset += 1; result_offsets[i] = current_result_offset; } current_offset = offsets[i]; } } /** Процедура для неконстантного идекса * index_type - тип данных идекса */ template static void vector( const ColumnString::Chars_t & data, const ColumnArray::Offsets_t & offsets, const ColumnString::Offsets_t & string_offsets, const ColumnVector & index, ColumnString::Chars_t & result_data, ColumnArray::Offsets_t & result_offsets) { size_t size = offsets.size(); result_offsets.resize(size); result_data.reserve(data.size()); ColumnArray::Offset_t current_offset = 0; ColumnArray::Offset_t current_result_offset = 0; for (size_t i = 0; i < size; ++i) { size_t array_size = offsets[i] - current_offset; size_t adjusted_index; if (index[i].getType() == Field::Types::UInt64) { UInt64 cur_id = safeGet(index[i]); if (cur_id > 0 && cur_id <= array_size) adjusted_index = cur_id - 1; else adjusted_index = array_size; /// Индекс не вписывается в рамки массива, заменяем заведомо слишком большим } else if (index[i].getType() == Field::Types::Int64) { Int64 cur_id = safeGet(index[i]); if (cur_id > 0 && static_cast(cur_id) <= array_size) adjusted_index = cur_id - 1; else if (cur_id < 0 && static_cast(-cur_id) <= array_size) adjusted_index = array_size + cur_id; else adjusted_index = array_size; /// Индекс не вписывается в рамки массива, заменяем слишком большим } else throw Exception("Illegal type of array index", ErrorCodes::LOGICAL_ERROR); if (adjusted_index < array_size) { ColumnArray::Offset_t string_pos = current_offset == 0 && adjusted_index == 0 ? 0 : string_offsets[current_offset + adjusted_index - 1]; ColumnArray::Offset_t string_size = string_offsets[current_offset + adjusted_index] - string_pos; result_data.resize(current_result_offset + string_size); memcpy(&result_data[current_result_offset], &data[string_pos], string_size); current_result_offset += string_size; result_offsets[i] = current_result_offset; } else { /// Вставим пустую строку. result_data.resize(current_result_offset + 1); result_data[current_result_offset] = 0; current_result_offset += 1; result_offsets[i] = current_result_offset; } current_offset = offsets[i]; } } }; class FunctionArrayElement : public IFunction { public: static constexpr auto name = "arrayElement"; static IFunction * create(const Context & context) { return new FunctionArrayElement; } private: template bool executeNumberConst(Block & block, const ColumnNumbers & arguments, size_t result, const Field & index) { const ColumnArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const ColumnVector * col_nested = typeid_cast *>(&col_array->getData()); if (!col_nested) return false; ColumnVector * col_res = new ColumnVector; block.getByPosition(result).column = col_res; if (index.getType() == Field::Types::UInt64) ArrayElementNumImpl::template vectorConst(col_nested->getData(), col_array->getOffsets(), safeGet(index) - 1, col_res->getData()); else if (index.getType() == Field::Types::Int64) ArrayElementNumImpl::template vectorConst(col_nested->getData(), col_array->getOffsets(), -safeGet(index) - 1, col_res->getData()); else throw Exception("Illegal type of array index", ErrorCodes::LOGICAL_ERROR); return true; } template bool executeNumber(Block & block, const ColumnNumbers & arguments, size_t result, const ColumnVector & index) { const ColumnArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const ColumnVector * col_nested = typeid_cast *>(&col_array->getData()); if (!col_nested) return false; ColumnVector * col_res = new ColumnVector; block.getByPosition(result).column = col_res; ArrayElementNumImpl::template vector(col_nested->getData(), col_array->getOffsets(), index, col_res->getData()); return true; } bool executeStringConst(Block & block, const ColumnNumbers & arguments, size_t result, const Field & index) { const ColumnArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const ColumnString * col_nested = typeid_cast(&col_array->getData()); if (!col_nested) return false; ColumnString * col_res = new ColumnString; block.getByPosition(result).column = col_res; if (index.getType() == Field::Types::UInt64) ArrayElementStringImpl::vectorConst( col_nested->getChars(), col_array->getOffsets(), col_nested->getOffsets(), safeGet(index) - 1, col_res->getChars(), col_res->getOffsets()); else if (index.getType() == Field::Types::Int64) ArrayElementStringImpl::vectorConst( col_nested->getChars(), col_array->getOffsets(), col_nested->getOffsets(), -safeGet(index) - 1, col_res->getChars(), col_res->getOffsets()); else throw Exception("Illegal type of array index", ErrorCodes::LOGICAL_ERROR); return true; } template bool executeString(Block & block, const ColumnNumbers & arguments, size_t result, const ColumnVector & index) { const ColumnArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const ColumnString * col_nested = typeid_cast(&col_array->getData()); if (!col_nested) return false; ColumnString * col_res = new ColumnString; block.getByPosition(result).column = col_res; ArrayElementStringImpl::vector( col_nested->getChars(), col_array->getOffsets(), col_nested->getOffsets(), index, col_res->getChars(), col_res->getOffsets()); return true; } bool executeConstConst(Block & block, const ColumnNumbers & arguments, size_t result, const Field & index) { const ColumnConstArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const DB::Array & array = col_array->getData(); size_t array_size = array.size(); size_t real_index = 0; if (index.getType() == Field::Types::UInt64) real_index = safeGet(index) - 1; else if (index.getType() == Field::Types::Int64) real_index = array_size + safeGet(index); else throw Exception("Illegal type of array index", ErrorCodes::LOGICAL_ERROR); Field value = col_array->getData().at(real_index); block.getByPosition(result).column = block.getByPosition(result).type->createConstColumn( block.rowsInFirstColumn(), value); return true; } template bool executeConst(Block & block, const ColumnNumbers & arguments, size_t result, const ColumnVector & index) { const ColumnConstArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const DB::Array & array = col_array->getData(); size_t array_size = array.size(); block.getByPosition(result).column = block.getByPosition(result).type->createColumn(); for (size_t i = 0; i < col_array->size(); ++i) { if (index[i].getType() == Field::Types::UInt64) { UInt64 cur_id = safeGet(index[i]); if (cur_id > 0 && cur_id <= array_size) block.getByPosition(result).column->insert(array[cur_id - 1]); else block.getByPosition(result).column->insertDefault(); } else if (index[i].getType() == Field::Types::Int64) { Int64 cur_id = safeGet(index[i]); if (cur_id > 0 && static_cast(cur_id) <= array_size) block.getByPosition(result).column->insert(array[cur_id - 1]); else if (cur_id < 0 && static_cast(-cur_id) <= array_size) block.getByPosition(result).column->insert(array[array_size + cur_id]); else block.getByPosition(result).column->insertDefault(); } else throw Exception("Illegal type of array index", ErrorCodes::LOGICAL_ERROR); } return true; } template bool executeArgument(Block & block, const ColumnNumbers & arguments, size_t result) { const ColumnVector * index = typeid_cast *> (&*block.getByPosition(arguments[1]).column); if (!index) return false; if (!( executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeNumber (block, arguments, result, *index) || executeConst (block, arguments, result, *index) || executeString (block, arguments, result, *index))) throw Exception("Illegal column " + block.getByPosition(arguments[0]).column->getName() + " of first argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN); return true; } public: /// Получить имя функции. String getName() const { return name; } /// Получить типы результата по типам аргументов. Если функция неприменима для данных аргументов - кинуть исключение. DataTypePtr getReturnType(const DataTypes & arguments) const { if (arguments.size() != 2) throw Exception("Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size()) + ", should be 2.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); const DataTypeArray * array_type = typeid_cast(&*arguments[0]); if (!array_type) throw Exception("First argument for function " + getName() + " must be array.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); if (!arguments[1]->isNumeric() || (0 != arguments[1]->getName().compare(0, 4, "UInt") && 0 != arguments[1]->getName().compare(0, 3, "Int"))) throw Exception("Second argument for function " + getName() + " must have UInt or Int type.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); return array_type->getNestedType(); } /// Выполнить функцию над блоком. void execute(Block & block, const ColumnNumbers & arguments, size_t result) { if (!block.getByPosition(arguments[1]).column->isConst()) { if (!( executeArgument (block, arguments, result) || executeArgument (block, arguments, result) || executeArgument (block, arguments, result) || executeArgument (block, arguments, result) || executeArgument (block, arguments, result) || executeArgument (block, arguments, result) || executeArgument (block, arguments, result) || executeArgument (block, arguments, result))) throw Exception("Second argument for function " + getName() + " must must have UInt or Int type.", ErrorCodes::ILLEGAL_COLUMN); } else { Field index = (*block.getByPosition(arguments[1]).column)[0]; if (index == UInt64(0)) throw Exception("Array indices is 1-based", ErrorCodes::ZERO_ARRAY_OR_TUPLE_INDEX); if (!( executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeNumberConst (block, arguments, result, index) || executeConstConst (block, arguments, result, index) || executeStringConst (block, arguments, result, index))) throw Exception("Illegal column " + block.getByPosition(arguments[0]).column->getName() + " of first argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN); } } }; /// Для has. struct IndexToOne { typedef UInt8 ResultType; static inline bool apply(size_t j, ResultType & current) { current = 1; return false; } }; /// Для indexOf. struct IndexIdentity { typedef UInt64 ResultType; /// Индекс возвращается начиная с единицы. static inline bool apply(size_t j, ResultType & current) { current = j + 1; return false; } }; /// Для countEqual. struct IndexCount { typedef UInt32 ResultType; static inline bool apply(size_t j, ResultType & current) { ++current; return true; } }; template struct ArrayIndexNumImpl { static void vector( const PODArray & data, const ColumnArray::Offsets_t & offsets, const T value, PODArray & result) { size_t size = offsets.size(); result.resize(size); ColumnArray::Offset_t current_offset = 0; for (size_t i = 0; i < size; ++i) { size_t array_size = offsets[i] - current_offset; typename IndexConv::ResultType current = 0; for (size_t j = 0; j < array_size; ++j) { if (data[current_offset + j] == value) { if (!IndexConv::apply(j, current)) break; } } result[i] = current; current_offset = offsets[i]; } } }; template struct ArrayIndexStringImpl { static void vector( const ColumnString::Chars_t & data, const ColumnArray::Offsets_t & offsets, const ColumnString::Offsets_t & string_offsets, const String & value, PODArray & result) { size_t size = offsets.size(); size_t value_size = value.size(); result.resize(size); ColumnArray::Offset_t current_offset = 0; for (size_t i = 0; i < size; ++i) { size_t array_size = offsets[i] - current_offset; typename IndexConv::ResultType current = 0; for (size_t j = 0; j < array_size; ++j) { ColumnArray::Offset_t string_pos = current_offset == 0 && j == 0 ? 0 : string_offsets[current_offset + j - 1]; ColumnArray::Offset_t string_size = string_offsets[current_offset + j] - string_pos; if (string_size == value_size + 1 && 0 == memcmp(value.data(), &data[string_pos], value_size)) { if (!IndexConv::apply(j, current)) break; } } result[i] = current; current_offset = offsets[i]; } } }; template class FunctionArrayIndex : public IFunction { public: static constexpr auto name = Name::name; static IFunction * create(const Context & context) { return new FunctionArrayIndex; } private: typedef ColumnVector ResultColumnType; template bool executeNumber(Block & block, const ColumnNumbers & arguments, size_t result, const Field & value) { const ColumnArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const ColumnVector * col_nested = typeid_cast *>(&col_array->getData()); if (!col_nested) return false; ResultColumnType * col_res = new ResultColumnType; block.getByPosition(result).column = col_res; ArrayIndexNumImpl::vector( col_nested->getData(), col_array->getOffsets(), safeGet::Type>(value), col_res->getData()); return true; } bool executeString(Block & block, const ColumnNumbers & arguments, size_t result, const Field & value) { const ColumnArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const ColumnString * col_nested = typeid_cast(&col_array->getData()); if (!col_nested) return false; ResultColumnType * col_res = new ResultColumnType; block.getByPosition(result).column = col_res; ArrayIndexStringImpl::vector( col_nested->getChars(), col_array->getOffsets(), col_nested->getOffsets(), safeGet(value), col_res->getData()); return true; } bool executeConst(Block & block, const ColumnNumbers & arguments, size_t result, const Field & value) { const ColumnConstArray * col_array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!col_array) return false; const Array & arr = col_array->getData(); size_t i = 0; size_t size = arr.size(); typename IndexConv::ResultType current = 0; for (; i < size; ++i) { if (arr[i] == value) { if (!IndexConv::apply(i, current)) break; } } block.getByPosition(result).column = block.getByPosition(result).type->createConstColumn( block.rowsInFirstColumn(), static_cast::Type>(current)); return true; } public: /// Получить имя функции. String getName() const { return name; } /// Получить типы результата по типам аргументов. Если функция неприменима для данных аргументов - кинуть исключение. DataTypePtr getReturnType(const DataTypes & arguments) const { if (arguments.size() != 2) throw Exception("Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size()) + ", should be 2.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); const DataTypeArray * array_type = typeid_cast(&*arguments[0]); if (!array_type) throw Exception("First argument for function " + getName() + " must be array.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); if (array_type->getNestedType()->getName() != arguments[1]->getName()) throw Exception("Type of array elements and second argument for function " + getName() + " must be same." " Passed: " + arguments[0]->getName() + " and " + arguments[1]->getName() + ".", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); return new typename DataTypeFromFieldType::Type; } /// Выполнить функцию над блоком. void execute(Block & block, const ColumnNumbers & arguments, size_t result) { if (!block.getByPosition(arguments[1]).column->isConst()) throw Exception("Second argument for function " + getName() + " must be constant.", ErrorCodes::ILLEGAL_COLUMN); Field value = (*block.getByPosition(arguments[1]).column)[0]; if (!( executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeNumber (block, arguments, result, value) || executeConst (block, arguments, result, value) || executeString (block, arguments, result, value))) throw Exception("Illegal column " + block.getByPosition(arguments[0]).column->getName() + " of first argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN); } }; class FunctionArrayEnumerate : public IFunction { public: static constexpr auto name = "arrayEnumerate"; static IFunction * create (const Context & context) { return new FunctionArrayEnumerate; } /// Получить имя функции. String getName() const { return name; } /// Получить типы результата по типам аргументов. Если функция неприменима для данных аргументов - кинуть исключение. DataTypePtr getReturnType(const DataTypes & arguments) const { if (arguments.size() != 1) throw Exception("Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size()) + ", should be 1.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); const DataTypeArray * array_type = typeid_cast(&*arguments[0]); if (!array_type) throw Exception("First argument for function " + getName() + " must be array.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); return new DataTypeArray(new DataTypeUInt32); } /// Выполнить функцию над блоком. void execute(Block & block, const ColumnNumbers & arguments, size_t result) { if (const ColumnArray * array = typeid_cast(&*block.getByPosition(arguments[0]).column)) { const ColumnArray::Offsets_t & offsets = array->getOffsets(); ColumnVector * res_nested = new ColumnVector; ColumnArray * res_array = new ColumnArray(res_nested, array->getOffsetsColumn()); block.getByPosition(result).column = res_array; ColumnVector::Container_t & res_values = res_nested->getData(); res_values.resize(array->getData().size()); size_t prev_off = 0; for (size_t i = 0; i < offsets.size(); ++i) { size_t off = offsets[i]; for (size_t j = prev_off; j < off; ++j) { res_values[j] = j - prev_off + 1; } prev_off = off; } } else if (const ColumnConstArray * array = typeid_cast(&*block.getByPosition(arguments[0]).column)) { const Array & values = array->getData(); Array res_values(values.size()); for (size_t i = 0; i < values.size(); ++i) { res_values[i] = i + 1; } ColumnConstArray * res_array = new ColumnConstArray(array->size(), res_values, new DataTypeArray(new DataTypeUInt32)); block.getByPosition(result).column = res_array; } else { throw Exception("Illegal column " + block.getByPosition(arguments[0]).column->getName() + " of first argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN); } } }; class FunctionArrayEnumerateUniq : public IFunction { public: static constexpr auto name = "arrayEnumerateUniq"; static IFunction * create(const Context & context) { return new FunctionArrayEnumerateUniq; } /// Получить имя функции. String getName() const { return name; } /// Получить типы результата по типам аргументов. Если функция неприменима для данных аргументов - кинуть исключение. DataTypePtr getReturnType(const DataTypes & arguments) const { if (arguments.size() == 0) throw Exception("Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size()) + ", should be at least 1.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); for (size_t i = 0; i < arguments.size(); ++i) { const DataTypeArray * array_type = typeid_cast(&*arguments[i]); if (!array_type) throw Exception("All arguments for function " + getName() + " must be arrays; argument " + toString(i + 1) + " isn't.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); } return new DataTypeArray(new DataTypeUInt32); } /// Выполнить функцию над блоком. void execute(Block & block, const ColumnNumbers & arguments, size_t result) { if (arguments.size() == 1 && executeConst(block, arguments, result)) return; Columns array_columns(arguments.size()); const ColumnArray::Offsets_t * offsets = nullptr; ConstColumnPlainPtrs data_columns(arguments.size()); for (size_t i = 0; i < arguments.size(); ++i) { ColumnPtr array_ptr = block.getByPosition(arguments[i]).column; const ColumnArray * array = typeid_cast(&*array_ptr); if (!array) { const ColumnConstArray * const_array = typeid_cast(&*block.getByPosition(arguments[i]).column); if (!const_array) throw Exception("Illegal column " + block.getByPosition(arguments[i]).column->getName() + " of " + toString(i + 1) + "-th argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN); array_ptr = const_array->convertToFullColumn(); array = typeid_cast(&*array_ptr); } array_columns[i] = array_ptr; const ColumnArray::Offsets_t & offsets_i = array->getOffsets(); if (!i) offsets = &offsets_i; else if (offsets_i != *offsets) throw Exception("Lengths of all arrays passsed to " + getName() + " must be equal.", ErrorCodes::SIZES_OF_ARRAYS_DOESNT_MATCH); data_columns[i] = &array->getData(); } const ColumnArray * first_array = typeid_cast(&*array_columns[0]); ColumnVector * res_nested = new ColumnVector; ColumnArray * res_array = new ColumnArray(res_nested, first_array->getOffsetsColumn()); block.getByPosition(result).column = res_array; ColumnVector::Container_t & res_values = res_nested->getData(); if (!offsets->empty()) res_values.resize(offsets->back()); if (arguments.size() == 1) { if (!( executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeNumber (first_array, res_values) || executeString (first_array, res_values))) throw Exception("Illegal column " + block.getByPosition(arguments[0]).column->getName() + " of first argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN); } else { if (!execute128bit(*offsets, data_columns, res_values)) executeHashed(*offsets, data_columns, res_values); } } private: /// Изначально выделить кусок памяти для 512 элементов. static constexpr size_t INITIAL_SIZE_DEGREE = 9; template bool executeNumber(const ColumnArray * array, ColumnVector::Container_t & res_values) { const ColumnVector * nested = typeid_cast *>(&*array->getDataPtr()); if (!nested) return false; const ColumnArray::Offsets_t & offsets = array->getOffsets(); const typename ColumnVector::Container_t & values = nested->getData(); typedef ClearableHashMap, HashTableGrower, HashTableAllocatorWithStackMemory<(1 << INITIAL_SIZE_DEGREE) * sizeof(T)> > ValuesToIndices; ValuesToIndices indices; size_t prev_off = 0; for (size_t i = 0; i < offsets.size(); ++i) { indices.clear(); size_t off = offsets[i]; for (size_t j = prev_off; j < off; ++j) { res_values[j] = ++indices[values[j]]; } prev_off = off; } return true; } bool executeString(const ColumnArray * array, ColumnVector::Container_t & res_values) { const ColumnString * nested = typeid_cast(&*array->getDataPtr()); if (!nested) return false; const ColumnArray::Offsets_t & offsets = array->getOffsets(); size_t prev_off = 0; typedef ClearableHashMap, HashTableAllocatorWithStackMemory<(1 << INITIAL_SIZE_DEGREE) * sizeof(StringRef)> > ValuesToIndices; ValuesToIndices indices; for (size_t i = 0; i < offsets.size(); ++i) { indices.clear(); size_t off = offsets[i]; for (size_t j = prev_off; j < off; ++j) { res_values[j] = ++indices[nested->getDataAt(j)]; } prev_off = off; } return true; } bool executeConst(Block & block, const ColumnNumbers & arguments, size_t result) { const ColumnConstArray * array = typeid_cast(&*block.getByPosition(arguments[0]).column); if (!array) return false; const Array & values = array->getData(); Array res_values(values.size()); std::map indices; for (size_t i = 0; i < values.size(); ++i) { res_values[i] = static_cast(++indices[values[i]]); } ColumnConstArray * res_array = new ColumnConstArray(array->size(), res_values, new DataTypeArray(new DataTypeUInt32)); block.getByPosition(result).column = res_array; return true; } bool execute128bit( const ColumnArray::Offsets_t & offsets, const ConstColumnPlainPtrs & columns, ColumnVector::Container_t & res_values) { size_t count = columns.size(); size_t keys_bytes = 0; Sizes key_sizes(count); for (size_t j = 0; j < count; ++j) { if (!columns[j]->isFixed()) return false; key_sizes[j] = columns[j]->sizeOfField(); keys_bytes += key_sizes[j]; } if (keys_bytes > 16) return false; typedef ClearableHashMap, HashTableAllocatorWithStackMemory<(1 << INITIAL_SIZE_DEGREE) * sizeof(UInt128)> > ValuesToIndices; ValuesToIndices indices; size_t prev_off = 0; for (size_t i = 0; i < offsets.size(); ++i) { indices.clear(); size_t off = offsets[i]; for (size_t j = prev_off; j < off; ++j) { res_values[j] = ++indices[pack128(j, count, columns, key_sizes)]; } prev_off = off; } return true; } void executeHashed( const ColumnArray::Offsets_t & offsets, const ConstColumnPlainPtrs & columns, ColumnVector::Container_t & res_values) { size_t count = columns.size(); typedef ClearableHashMap, HashTableAllocatorWithStackMemory<(1 << INITIAL_SIZE_DEGREE) * sizeof(UInt128)> > ValuesToIndices; ValuesToIndices indices; size_t prev_off = 0; for (size_t i = 0; i < offsets.size(); ++i) { indices.clear(); size_t off = offsets[i]; for (size_t j = prev_off; j < off; ++j) { res_values[j] = ++indices[hash128(j, count, columns)]; } prev_off = off; } } }; template struct TypeToColumnType { using ColumnType = ColumnVector; }; template <> struct TypeToColumnType { using ColumnType = ColumnString; }; template struct DataTypeToName : TypeName { }; template <> struct DataTypeToName { static std::string get() { return "Date"; } }; template <> struct DataTypeToName { static std::string get() { return "DateTime"; } }; template struct FunctionEmptyArray : public IFunction { static constexpr auto base_name = "emptyArray"; static const String name; static IFunction * create(const Context & context) { return new FunctionEmptyArray; } private: String getName() const { return name; } DataTypePtr getReturnType(const DataTypes & arguments) const { if (arguments.size() != 0) throw Exception("Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size()) + ", should be 0.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); return new DataTypeArray{new DataType{}}; } void execute(Block & block, const ColumnNumbers & arguments, size_t result) { using UnderlyingColumnType = typename TypeToColumnType::ColumnType; block.getByPosition(result).column = new ColumnArray{ new UnderlyingColumnType, new ColumnArray::ColumnOffsets_t{block.rowsInFirstColumn(), 0} }; } }; template const String FunctionEmptyArray::name = FunctionEmptyArray::base_name + DataTypeToName::get(); class FunctionRange : public IFunction { public: static constexpr auto max_elements = 100000000; static constexpr auto name = "range"; static IFunction * create(const Context &) { return new FunctionRange; } private: String getName() const override { return name; } DataTypePtr getReturnType(const DataTypes & arguments) const override { if (arguments.size() != 1) throw Exception{ "Number of arguments for function " + getName() + " doesn't match: passed " + toString(arguments.size()) + ", should be 1.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH }; const auto arg = arguments.front().get(); if (!typeid_cast(arg) && !typeid_cast(arg) && !typeid_cast(arg) & !typeid_cast(arg)) { throw Exception{ "Illegal type " + arg->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT }; } return new DataTypeArray{arg->clone()}; } template bool execute(Block & block, const IColumn * const arg, const size_t result) { if (const auto in = typeid_cast *>(arg)) { const auto & in_data = in->getData(); const auto total_values = std::accumulate(std::begin(in_data), std::end(in_data), std::size_t{}, std::plus{}); if (total_values > max_elements) throw Exception{ "A call to function " + getName() + " would produce " + std::to_string(total_values) + " array elements, which is greater than the allowed maximum of " + std::to_string(max_elements), ErrorCodes::ARGUMENT_OUT_OF_BOUND }; const auto data_col = new ColumnVector{total_values}; const auto out = new ColumnArray{ data_col, new ColumnArray::ColumnOffsets_t{in->size()} }; block.getByPosition(result).column = out; auto & out_data = data_col->getData(); auto & out_offsets = out->getOffsets(); IColumn::Offset_t offset{}; for (const auto i : ext::range(0, in->size())) { std::copy(ext::make_range_iterator(T{}), ext::make_range_iterator(in_data[i]), &out_data[offset]); offset += in_data[i]; out_offsets[i] = offset; } return true; } else if (const auto in = typeid_cast *>(arg)) { const auto & in_data = in->getData(); const std::size_t total_values = in->size() * in_data; if (total_values > max_elements) throw Exception{ "A call to function " + getName() + " would produce " + std::to_string(total_values) + " array elements, which is greater than the allowed maximum of " + std::to_string(max_elements), ErrorCodes::ARGUMENT_OUT_OF_BOUND }; const auto data_col = new ColumnVector{total_values}; const auto out = new ColumnArray{ data_col, new ColumnArray::ColumnOffsets_t{in->size()} }; block.getByPosition(result).column = out; auto & out_data = data_col->getData(); auto & out_offsets = out->getOffsets(); IColumn::Offset_t offset{}; for (const auto i : ext::range(0, in->size())) { std::copy(ext::make_range_iterator(T{}), ext::make_range_iterator(in_data), &out_data[offset]); offset += in_data; out_offsets[i] = offset; } return true; } return false; } void execute(Block & block, const ColumnNumbers & arguments, const size_t result) override { const auto col = block.getByPosition(arguments[0]).column.get(); if (!execute(block, col, result) && !execute(block, col, result) && !execute(block, col, result) && !execute(block, col, result)) { throw Exception{ "Illegal column " + col->getName() + " of argument of function " + getName(), ErrorCodes::ILLEGAL_COLUMN }; } } }; struct NameHas { static constexpr auto name = "has"; }; struct NameIndexOf { static constexpr auto name = "indexOf"; }; struct NameCountEqual { static constexpr auto name = "countEqual"; }; typedef FunctionArrayIndex FunctionHas; typedef FunctionArrayIndex FunctionIndexOf; typedef FunctionArrayIndex FunctionCountEqual; using FunctionEmptyArrayUInt8 = FunctionEmptyArray; using FunctionEmptyArrayUInt16 = FunctionEmptyArray; using FunctionEmptyArrayUInt32 = FunctionEmptyArray; using FunctionEmptyArrayUInt64 = FunctionEmptyArray; using FunctionEmptyArrayInt8 = FunctionEmptyArray; using FunctionEmptyArrayInt16 = FunctionEmptyArray; using FunctionEmptyArrayInt32 = FunctionEmptyArray; using FunctionEmptyArrayInt64 = FunctionEmptyArray; using FunctionEmptyArrayFloat32 = FunctionEmptyArray; using FunctionEmptyArrayFloat64 = FunctionEmptyArray; using FunctionEmptyArrayDate = FunctionEmptyArray; using FunctionEmptyArrayDateTime = FunctionEmptyArray; using FunctionEmptyArrayString = FunctionEmptyArray; }