Amos Bird
3707da4fbf
Better addBatchArray aggregator
2019-11-11 16:56:11 +08:00
alexey-milovidov
1d910c5071
Merge pull request #7608 from amosbird/batchreduce
...
Optimize arrayReduce, -Array and -State combinators
2019-11-10 22:10:03 +03:00
Amos Bird
544af39d23
Vectorize arrayReduce and -Array combinator
...
Also devirtualize -State combinator.
2019-11-07 11:53:47 +08:00
Amos Bird
fbfbe161ec
Unify hash tables interface.
2019-11-06 00:08:04 +08:00
Amos Bird
a864447802
Introduce String Hash Map.
...
It speeds up aggregation over short string keys. Use it as a default
aggregation method for string keys.
2019-10-21 16:27:35 +03:00
chertus
c46cc3ff1a
refactoring: remove copy-paste
2019-10-01 21:51:33 +03:00
Nikolai Kochetov
1319168a3a
Merge branch 'master' into processors-2
2019-09-12 16:28:51 +03:00
Amos Bird
8f28a36c17
Use internal iteration over hash tables in Aggregator.
...
For compound hash tables such as the future StringHashMap, an
iterator-based API might be inefficient for iterating over a table or
for merging two tables, because:
1) the key has to be converted to a general format from a from a
component-specific format, which may differ between the components;
2) the information about the component of the compound hash table to
which the value belongs is lost, and has to be recalculated if the
value is reinserted.
A more efficient approach is to use internal iteration, that is,
map-like functions, which avoids unnecessary conversions when iterating,
and allows to use an efficient component-wise approach when merging.
2019-09-10 21:52:53 +03:00
Alexander Kuzmenkov
160e0faf77
Do not use iterator::getValue() in Aggregator.
...
Use separate key and "mapped" value references instead. This is
important for hash tables that do not store the key/"mapped" pair
directly, and cannot provide this interface without some runtime
overhead.
2019-09-10 20:12:30 +03:00
Nikolai Kochetov
52305cefbc
Merge branch 'merging-aggregated-processors' into processors-2
2019-09-06 15:21:17 +03:00
Nikolai Kochetov
d10b7d7a2b
Added ConvertingAggregatedToChunksTransform.
2019-09-06 15:19:59 +03:00
Nikolai Kochetov
bf0c0ed4f1
Merge branch 'master' into processors-2
2019-09-02 16:02:09 +03:00
Nikolai Kochetov
300dc80194
Update Aggregator interface.
2019-08-31 11:58:16 +03:00
alexey-milovidov
e2b05f6f9c
Merge pull request #6700 from yandex/aku/shm-5417-formatting
...
Formatting changes for StringHashMap PR #5417 .
2019-08-28 18:30:59 +03:00
Alexander Kuzmenkov
0b5a7f5f4a
Disable consecutive key optimization for UInt8/16 LowCardinality
...
columns.
A follow-up to #6298 , this is as well a part of StringHashMap PR #5417
by Amos Bird.
2019-08-28 14:05:52 +03:00
Alexander Kuzmenkov
24a8755b82
Formatting changes for StringHashMap PR #5417 .
2019-08-28 13:52:14 +03:00
Weiqing Xu
2fbcd607f3
check free space when use external sort/aggerator
2019-08-27 08:23:07 +08:00
Alexey Milovidov
cdd6dca517
Remove Compiler
2019-08-24 00:10:26 +03:00
Alexey Milovidov
8966520801
Added "assert_cast" function as a replacement for static_cast for columns
2019-08-21 05:28:04 +03:00
Alexey Milovidov
2570907f44
Removed useless parameter
2019-08-12 00:45:18 +03:00
Alexey Milovidov
011e50cae6
Removed useless function arguments
2019-08-11 02:27:25 +03:00
Alexey Milovidov
efa51a6cd9
Batch aggregation (experimental)
2019-08-11 01:36:55 +03:00
alexey-milovidov
349ad17436
Merge pull request #6298 from yandex/aku/uint816-nocache
...
Disable consecutive key optimization for UInt8/16.
2019-08-03 06:23:03 +03:00
Alexander Kuzmenkov
d808fafa8f
Disable consecutive key optimization for UInt8/16.
...
These types use a FixedHashMap for aggregation, which makes lookup
almost free, so we don't have to cache the last lookup result.
This is a part of StringHashMap PR #5417 by Amos Bird.
2019-08-02 18:37:32 +03:00
Alexander Kuzmenkov
011150fa8a
Make PairNoInit a simple struct.
2019-08-01 19:54:33 +03:00
Max Akhmedov
5e9ea2a905
Remove spaces from empty line
2019-07-28 20:46:48 +03:00
Max Akhmedov
f632a0b241
Fix build under gcc-8.2
2019-07-28 18:12:07 +03:00
Nikolai Kochetov
209793c981
Merged with master.
2019-07-08 16:00:54 +03:00
chertus
8932dc9c23
minpr perf improvement
2019-07-02 15:31:20 +03:00
chertus
0c634129da
add IColumn.isNullable()
2019-07-01 14:44:19 +03:00
chertus
db7c8748e9
remove IColumn.isColumnNullable()
2019-06-27 21:50:20 +03:00
chertus
65c0d23efa
better casts to ColumnNullable
2019-06-26 20:20:33 +03:00
Nikolai Kochetov
f9b29bfdd2
Merged with master.
2019-06-25 20:00:54 +03:00
Ivan Lezhankin
0821d62516
Merge ITableDeclaration into IStorage.
...
Also a couple of massive include-refactorings with better forwarding
2019-05-17 17:34:25 +03:00
Nikolai Kochetov
808d0f3b08
Merged with master.
2019-05-14 13:31:19 +03:00
Alexey Milovidov
58d7d1e451
Resolved some of PVS-Studio warnings
2019-04-22 19:07:09 +03:00
Nikolai Kochetov
5e39ec610c
Merged with master
2019-04-03 14:21:38 +03:00
Nikolai Kochetov
6976b65b2d
Added MergingAggregatedTransform.
2019-04-03 13:13:25 +03:00
Amos Bird
26ab5dd7a7
A Proper lookup table that uses HashTable's API
...
This is the first step of allowing heterogeneous cells in hash tables.
performance test results are
```
1. HashMap<UInt16, UInt8, TrivialHash, HashTableFixedGrower<16>>;
2. NewLookupMap<UInt16, UInt8>
ResolutionWidth 30000 1 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................223550276.46
ResolutionWidth 30000 2 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................248772721.24
Best: 2 - 24877272124
ResolutionWidth 100000 1 ..........................................................................................................................................................................................................................................................238498413.99
ResolutionWidth 100000 2 ..........................................................................................................................................................................................................................................................261808889.98
Best: 2 - 26180888998
ResolutionWidth 300000 1 ...................................................................................239307348.81
ResolutionWidth 300000 2 ...................................................................................257592761.30
Best: 2 - 25759276130
ResolutionWidth 1000000 1 .........................240144759.26
ResolutionWidth 1000000 2 .........................257093531.91
Best: 2 - 25709353191
ResolutionWidth 5000000 1 .....241573260.35
ResolutionWidth 5000000 2 .....259314162.79
Best: 2 - 25931416279
ResolutionDepth 30000 1 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................217108119.84
ResolutionDepth 30000 2 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................249459504.41
Best: 2 - 24945950441
ResolutionDepth 100000 1 ..........................................................................................................................................................................................................................................................229065162.17
ResolutionDepth 100000 2 ..........................................................................................................................................................................................................................................................253769105.64
Best: 2 - 25376910564
ResolutionDepth 300000 1 ...................................................................................233079225.18
ResolutionDepth 300000 2 ...................................................................................256316273.78
Best: 2 - 25631627378
ResolutionDepth 1000000 1 .........................234184633.51
ResolutionDepth 1000000 2 .........................261100491.57
Best: 2 - 26110049157
ResolutionDepth 5000000 1 .....233118795.66
ResolutionDepth 5000000 2 .....252436160.41
Best: 2 - 25243616041
```
2019-03-01 16:47:13 +08:00
Nikolai Kochetov
cf40a1538b
Fix size() and empty() for AggregationDataWithNullKey. #4223
2019-02-13 19:54:55 +03:00
Nikolai Kochetov
2c2932e185
Merged with master
2019-01-29 15:25:19 +03:00
Nikolai Kochetov
6fce028b56
Refactor ColumnsHashing.
2019-01-21 13:39:53 +03:00
Nikolai Kochetov
d207498573
Added ColumnsHashing
2019-01-21 13:39:24 +03:00
Alexey Milovidov
abcd5a2a49
Attempt to implemnt global thread pool #4018
2019-01-11 22:12:36 +03:00
Alexey Milovidov
06a8c3f02d
Added some warnings from clang's -Weverything
2019-01-05 06:33:22 +03:00
Alexey Milovidov
1d4701b0bc
Added some warnings from clang's -Weverything
2019-01-04 15:10:00 +03:00
Alexey Milovidov
e33e5150b7
Miscellaneous [#CLICKHOUSE-2]
2019-01-02 09:44:36 +03:00
Alexey Milovidov
b109ec8c77
Fixed UB #3920
2018-12-27 23:24:37 +03:00
Alexey Milovidov
220e568894
Fixed UB #3920
2018-12-27 23:10:37 +03:00
Alexey Milovidov
29ffa46838
Less indirection #3920
2018-12-27 23:02:51 +03:00
