Example of such cases:
- SELECT GROUP BY LIMIT
- SELECT GROUP BY with subsequent MEMORY_LIMIT_EXCEEDED error
And it should be two-level aggregation, since otherwise there will be
only one hashtable which will be cleared correctly, only if you have
two-level GROUP BY some of hashtables will not be cleared since nobody
consume rows.
Before this patch:
09:39.015292 [ 3070801 ] {609a0610-e377-4132-9cf3-f49454cf3c96} <Information> executeQuery: Read 1000000 rows, 7.63 MiB in 0.707 sec., 1413826 rows/sec., 10.79 MiB/sec.
09:39.015348 [ 3070801 ] {609a0610-e377-4132-9cf3-f49454cf3c96} <Debug> MemoryTracker: Peak memory usage (for query): 51.93 MiB.
09:39.015942 [ 3070801 ] {} <Trace> Aggregator: Destroying aggregate states <-- **problem**
09:39.017057 [ 3070801 ] {} <Trace> Aggregator: Destroying aggregate states <--
09:39.017961 [ 3070801 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 51.93 MiB.
09:39.018029 [ 3070801 ] {} <Information> TCPHandler: Processed in 0.711 sec.
After this patch:
16:24.544030 [ 3087333 ] {79da208a-b3c0-48d4-9943-c974a3cbb6ea} <Information> executeQuery: Read 1000000 rows, 7.63 MiB in 0.599 sec., 1670199 rows/sec., 12.74 MiB/sec.
16:24.544084 [ 3087333 ] {79da208a-b3c0-48d4-9943-c974a3cbb6ea} <Debug> MemoryTracker: Peak memory usage (for query): 72.11 MiB.
16:24.544398 [ 3087333 ] {79da208a-b3c0-48d4-9943-c974a3cbb6ea} <Trace> Aggregator: Destroying aggregate states
16:24.545485 [ 3087333 ] {79da208a-b3c0-48d4-9943-c974a3cbb6ea} <Trace> Aggregator: Destroying aggregate states
16:24.547053 [ 3087333 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 72.11 MiB.
16:24.547093 [ 3087333 ] {} <Information> TCPHandler: Processed in 0.603 sec.
There are lots of places where HashTable is used with
AllocatorWithStackMemory, but the size of allocator inline memory is set
incorrectly, and it's less than the initial HashTable buffer size.
Because of this, HashTable is always allocated on the heap, and the
inline memory becomes a useless dead weight.
For PODArray, we have previously added a helper template that makes sure
these values are in sync, so there was only one such discrepancy left,
in the unit test.
