#pragma once #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include namespace DB { /// uniq struct AggregateFunctionUniqUniquesHashSetData { using Set = UniquesHashSet>; Set set; static String getName() { return "uniq"; } }; /// For a function that takes multiple arguments. Such a function pre-hashes them in advance, so TrivialHash is used here. struct AggregateFunctionUniqUniquesHashSetDataForVariadic { using Set = UniquesHashSet; Set set; static String getName() { return "uniq"; } }; /// uniqHLL12 template struct AggregateFunctionUniqHLL12Data { using Set = HyperLogLogWithSmallSetOptimization; Set set; static String getName() { return "uniqHLL12"; } }; template <> struct AggregateFunctionUniqHLL12Data { using Set = HyperLogLogWithSmallSetOptimization; Set set; static String getName() { return "uniqHLL12"; } }; template <> struct AggregateFunctionUniqHLL12Data { using Set = HyperLogLogWithSmallSetOptimization; Set set; static String getName() { return "uniqHLL12"; } }; struct AggregateFunctionUniqHLL12DataForVariadic { using Set = HyperLogLogWithSmallSetOptimization; Set set; static String getName() { return "uniqHLL12"; } }; /// uniqExact template struct AggregateFunctionUniqExactData { using Key = T; /// When creating, the hash table must be small. using Set = HashSet< Key, HashCRC32, HashTableGrower<4>, HashTableAllocatorWithStackMemory>; Set set; static String getName() { return "uniqExact"; } }; /// For rows, we put the SipHash values (128 bits) into the hash table. template <> struct AggregateFunctionUniqExactData { using Key = UInt128; /// When creating, the hash table must be small. using Set = HashSet< Key, UInt128TrivialHash, HashTableGrower<3>, HashTableAllocatorWithStackMemory>; Set set; static String getName() { return "uniqExact"; } }; namespace detail { /** Hash function for uniq. */ template struct AggregateFunctionUniqTraits { static UInt64 hash(T x) { if constexpr (std::is_same_v) { return sipHash64(x); } else if constexpr (std::is_same_v || std::is_same_v) { return ext::bit_cast(x); } else if constexpr (sizeof(T) <= sizeof(UInt64)) return x; else return DefaultHash64(x); } }; /** The structure for the delegation work to add one element to the `uniq` aggregate functions. * Used for partial specialization to add strings. */ template struct OneAdder { static void ALWAYS_INLINE add(Data & data, const IColumn & column, size_t row_num) { if constexpr (std::is_same_v || std::is_same_v>) { if constexpr (!std::is_same_v) { const auto & value = assert_cast &>(column).getElement(row_num); data.set.insert(AggregateFunctionUniqTraits::hash(value)); } else { StringRef value = column.getDataAt(row_num); data.set.insert(CityHash_v1_0_2::CityHash64(value.data, value.size)); } } else if constexpr (std::is_same_v>) { if constexpr (!std::is_same_v) { data.set.insert(assert_cast &>(column).getData()[row_num]); } else { StringRef value = column.getDataAt(row_num); UInt128 key; SipHash hash; hash.update(value.data, value.size); hash.get128(key.low, key.high); data.set.insert(key); } } } }; } /// Calculates the number of different values approximately or exactly. template class AggregateFunctionUniq final : public IAggregateFunctionDataHelper> { public: AggregateFunctionUniq(const DataTypes & argument_types_) : IAggregateFunctionDataHelper>(argument_types_, {}) {} String getName() const override { return Data::getName(); } DataTypePtr getReturnType() const override { return std::make_shared(); } bool allocatesMemoryInArena() const override { return false; } /// ALWAYS_INLINE is required to have better code layout for uniqHLL12 function void ALWAYS_INLINE add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena *) const override { detail::OneAdder::add(this->data(place), *columns[0], row_num); } void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena *) const override { this->data(place).set.merge(this->data(rhs).set); } void serialize(ConstAggregateDataPtr __restrict place, WriteBuffer & buf) const override { this->data(place).set.write(buf); } void deserialize(AggregateDataPtr __restrict place, ReadBuffer & buf, Arena *) const override { this->data(place).set.read(buf); } void insertResultInto(AggregateDataPtr __restrict place, IColumn & to, Arena *) const override { assert_cast(to).getData().push_back(this->data(place).set.size()); } }; /** For multiple arguments. To compute, hashes them. * You can pass multiple arguments as is; You can also pass one argument - a tuple. * But (for the possibility of efficient implementation), you can not pass several arguments, among which there are tuples. */ template class AggregateFunctionUniqVariadic final : public IAggregateFunctionDataHelper> { private: size_t num_args = 0; public: AggregateFunctionUniqVariadic(const DataTypes & arguments) : IAggregateFunctionDataHelper>(arguments, {}) { if (argument_is_tuple) num_args = typeid_cast(*arguments[0]).getElements().size(); else num_args = arguments.size(); } String getName() const override { return Data::getName(); } DataTypePtr getReturnType() const override { return std::make_shared(); } bool allocatesMemoryInArena() const override { return false; } void add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena *) const override { this->data(place).set.insert(typename Data::Set::value_type( UniqVariadicHash::apply(num_args, columns, row_num))); } void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena *) const override { this->data(place).set.merge(this->data(rhs).set); } void serialize(ConstAggregateDataPtr __restrict place, WriteBuffer & buf) const override { this->data(place).set.write(buf); } void deserialize(AggregateDataPtr __restrict place, ReadBuffer & buf, Arena *) const override { this->data(place).set.read(buf); } void insertResultInto(AggregateDataPtr __restrict place, IColumn & to, Arena *) const override { assert_cast(to).getData().push_back(this->data(place).set.size()); } }; }