thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-18 04:12:19 +00:00
Code Issues Packages Projects Releases Wiki Activity
34,799 Commits 9,324 Branches 1,783 Tags 2.2 GiB
d1bfacae4b
Commit Graph

115 Commits

Author SHA1 Message Date
Azat Khuzhin
88bfb788a9 Add ability to use multiple disks/volumes for temporary data 
This patch adds <tmp_policy> config directive, that will define the
policy to use for storing temporary files, if it is not set (default)
the <tmp_path> will be used.

Also tmp_policy has some limitations:
- move_factor              is ignored
- keep_free_space_bytes    is ignored
- max_data_part_size_bytes is ignored
- must have exactly one volume
 2020-01-23 20:31:01 +03:00
Nikolai Kochetov
c2a054ac70 Add cancell flag to MergingAggrgated. 2020-01-13 12:56:21 +03:00
Amos Bird
3707da4fbf
Better addBatchArray aggregator 2019-11-11 16:56:11 +08:00
alexey-milovidov
1d910c5071
Merge pull request #7608 from amosbird/batchreduce 
Optimize arrayReduce, -Array and -State combinators
 2019-11-10 22:10:03 +03:00
Amos Bird
544af39d23
Vectorize arrayReduce and -Array combinator 
Also devirtualize -State combinator.
 2019-11-07 11:53:47 +08:00
Amos Bird
fbfbe161ec
Unify hash tables interface. 2019-11-06 00:08:04 +08:00
Amos Bird
a864447802 Introduce String Hash Map. 
It speeds up aggregation over short string keys. Use it as a default
aggregation method for string keys.
 2019-10-21 16:27:35 +03:00
chertus
c46cc3ff1a refactoring: remove copy-paste 2019-10-01 21:51:33 +03:00
Nikolai Kochetov
1319168a3a Merge branch 'master' into processors-2 2019-09-12 16:28:51 +03:00
Amos Bird
8f28a36c17 Use internal iteration over hash tables in Aggregator. 
For compound hash tables such as the future StringHashMap, an
iterator-based API might be inefficient for iterating over a table or
for merging two tables, because:

1) the key has to be converted to a general format from a from a
   component-specific format, which may differ between the components;

2) the information about the component of the compound hash table to
   which the value belongs is lost, and has to be recalculated if the
   value is reinserted.

A more efficient approach is to use internal iteration, that is,
map-like functions, which avoids unnecessary conversions when iterating,
and allows to use an efficient component-wise approach when merging.
 2019-09-10 21:52:53 +03:00
Alexander Kuzmenkov
160e0faf77 Do not use iterator::getValue() in Aggregator. 
Use separate key and "mapped" value references instead. This is
important for hash tables that do not store the key/"mapped" pair
directly, and cannot provide this interface without some runtime
overhead.
 2019-09-10 20:12:30 +03:00
Nikolai Kochetov
52305cefbc Merge branch 'merging-aggregated-processors' into processors-2 2019-09-06 15:21:17 +03:00
Nikolai Kochetov
d10b7d7a2b Added ConvertingAggregatedToChunksTransform. 2019-09-06 15:19:59 +03:00
Nikolai Kochetov
bf0c0ed4f1 Merge branch 'master' into processors-2 2019-09-02 16:02:09 +03:00
Nikolai Kochetov
300dc80194 Update Aggregator interface. 2019-08-31 11:58:16 +03:00
alexey-milovidov
e2b05f6f9c
Merge pull request #6700 from yandex/aku/shm-5417-formatting 
Formatting changes for StringHashMap PR #5417.
 2019-08-28 18:30:59 +03:00
Alexander Kuzmenkov
0b5a7f5f4a Disable consecutive key optimization for UInt8/16 LowCardinality 
columns.

A follow-up to #6298, this is as well a part of StringHashMap PR #5417
by Amos Bird.
 2019-08-28 14:05:52 +03:00
Alexander Kuzmenkov
24a8755b82 Formatting changes for StringHashMap PR #5417. 2019-08-28 13:52:14 +03:00
Weiqing Xu
2fbcd607f3 check free space when use external sort/aggerator 2019-08-27 08:23:07 +08:00
Alexey Milovidov
cdd6dca517 Remove Compiler 2019-08-24 00:10:26 +03:00
Alexey Milovidov
8966520801 Added "assert_cast" function as a replacement for static_cast for columns 2019-08-21 05:28:04 +03:00
Alexey Milovidov
2570907f44 Removed useless parameter 2019-08-12 00:45:18 +03:00
Alexey Milovidov
011e50cae6 Removed useless function arguments 2019-08-11 02:27:25 +03:00
Alexey Milovidov
efa51a6cd9 Batch aggregation (experimental) 2019-08-11 01:36:55 +03:00
alexey-milovidov
349ad17436
Merge pull request #6298 from yandex/aku/uint816-nocache 
Disable consecutive key optimization for UInt8/16.
 2019-08-03 06:23:03 +03:00
Alexander Kuzmenkov
d808fafa8f Disable consecutive key optimization for UInt8/16. 
These types use a FixedHashMap for aggregation, which makes lookup
almost free, so we don't have to cache the last lookup result.

This is a part of StringHashMap PR #5417 by Amos Bird.
 2019-08-02 18:37:32 +03:00
Alexander Kuzmenkov
011150fa8a Make PairNoInit a simple struct. 2019-08-01 19:54:33 +03:00
Max Akhmedov
5e9ea2a905
Remove spaces from empty line 2019-07-28 20:46:48 +03:00
Max Akhmedov
f632a0b241
Fix build under gcc-8.2 2019-07-28 18:12:07 +03:00
Nikolai Kochetov
209793c981 Merged with master. 2019-07-08 16:00:54 +03:00
chertus
8932dc9c23 minpr perf improvement 2019-07-02 15:31:20 +03:00
chertus
0c634129da add IColumn.isNullable() 2019-07-01 14:44:19 +03:00
chertus
db7c8748e9 remove IColumn.isColumnNullable() 2019-06-27 21:50:20 +03:00
chertus
65c0d23efa better casts to ColumnNullable 2019-06-26 20:20:33 +03:00
Nikolai Kochetov
f9b29bfdd2 Merged with master. 2019-06-25 20:00:54 +03:00
Ivan Lezhankin
0821d62516 Merge ITableDeclaration into IStorage. 
Also a couple of massive include-refactorings with better forwarding
 2019-05-17 17:34:25 +03:00
Nikolai Kochetov
808d0f3b08 Merged with master. 2019-05-14 13:31:19 +03:00
Alexey Milovidov
58d7d1e451 Resolved some of PVS-Studio warnings 2019-04-22 19:07:09 +03:00
Nikolai Kochetov
5e39ec610c Merged with master 2019-04-03 14:21:38 +03:00
Nikolai Kochetov
6976b65b2d Added MergingAggregatedTransform. 2019-04-03 13:13:25 +03:00
Amos Bird
26ab5dd7a7 A Proper lookup table that uses HashTable's API 
This is the first step of allowing heterogeneous cells in hash tables.

performance test results are

```

1. HashMap<UInt16, UInt8, TrivialHash, HashTableFixedGrower<16>>;
2. NewLookupMap<UInt16, UInt8>

ResolutionWidth 30000 1 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................223550276.46
ResolutionWidth 30000 2 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................248772721.24
Best: 2 - 24877272124

ResolutionWidth 100000 1 ..........................................................................................................................................................................................................................................................238498413.99
ResolutionWidth 100000 2 ..........................................................................................................................................................................................................................................................261808889.98
Best: 2 - 26180888998

ResolutionWidth 300000 1 ...................................................................................239307348.81
ResolutionWidth 300000 2 ...................................................................................257592761.30
Best: 2 - 25759276130

ResolutionWidth 1000000 1 .........................240144759.26
ResolutionWidth 1000000 2 .........................257093531.91
Best: 2 - 25709353191

ResolutionWidth 5000000 1 .....241573260.35
ResolutionWidth 5000000 2 .....259314162.79
Best: 2 - 25931416279

ResolutionDepth 30000 1 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................217108119.84
ResolutionDepth 30000 2 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................249459504.41
Best: 2 - 24945950441

ResolutionDepth 100000 1 ..........................................................................................................................................................................................................................................................229065162.17
ResolutionDepth 100000 2 ..........................................................................................................................................................................................................................................................253769105.64
Best: 2 - 25376910564

ResolutionDepth 300000 1 ...................................................................................233079225.18
ResolutionDepth 300000 2 ...................................................................................256316273.78
Best: 2 - 25631627378

ResolutionDepth 1000000 1 .........................234184633.51
ResolutionDepth 1000000 2 .........................261100491.57
Best: 2 - 26110049157

ResolutionDepth 5000000 1 .....233118795.66
ResolutionDepth 5000000 2 .....252436160.41
Best: 2 - 25243616041

```
 2019-03-01 16:47:13 +08:00
Nikolai Kochetov
cf40a1538b Fix size() and empty() for AggregationDataWithNullKey. #4223 2019-02-13 19:54:55 +03:00
Nikolai Kochetov
2c2932e185 Merged with master 2019-01-29 15:25:19 +03:00
Nikolai Kochetov
6fce028b56 Refactor ColumnsHashing. 2019-01-21 13:39:53 +03:00
Nikolai Kochetov
d207498573 Added ColumnsHashing 2019-01-21 13:39:24 +03:00
Alexey Milovidov
abcd5a2a49 Attempt to implemnt global thread pool #4018 2019-01-11 22:12:36 +03:00
Alexey Milovidov
06a8c3f02d Added some warnings from clang's -Weverything 2019-01-05 06:33:22 +03:00
Alexey Milovidov
1d4701b0bc Added some warnings from clang's -Weverything 2019-01-04 15:10:00 +03:00
Alexey Milovidov
e33e5150b7 Miscellaneous [#CLICKHOUSE-2] 2019-01-02 09:44:36 +03:00
Alexey Milovidov
b109ec8c77 Fixed UB #3920 2018-12-27 23:24:37 +03:00