Alexey Milovidov
3602126a1e
Remove unused ErrorCodes
2020-02-25 17:35:37 +03:00
Vitaly Baranov
1fa64a2a86
Use separate variables for the database and name in dictionaries.
2019-12-29 23:48:10 +07:00
Amos Bird
fbfbe161ec
Unify hash tables interface.
2019-11-06 00:08:04 +08:00
alesapin
10f69eef03
Add tests for non standart dictionaries and fix bugs
2019-10-21 19:05:45 +03:00
alesapin
08681ac210
Preparation for AST configurations for external dictionaries
2019-09-30 19:12:08 +03:00
Alexander Kuzmenkov
f3bde19b74
Do not use iterators in find() and emplace() methods of hash tables.
...
Instead, these methods return a pointer to the required data as they are
stored inside the hash table. The caller uses overloaded functions to
get the key and "mapped" values from this pointer. Such an interface
avoids the need for constructing iterator-like wrapper objects, which is
especially important for compound hash tables such as the future
StringHashMap.
2019-09-24 17:44:35 +03:00
Azat Khuzhin
52f14c1ead
Update HashedDictionary.cpp (fixed rename in return type for SFINAE)
2019-09-22 10:39:50 +03:00
alexey-milovidov
9afea6f0b3
Update HashedDictionary.cpp
2019-09-22 05:04:49 +03:00
Azat Khuzhin
420089c301
Add new dictionary layout (sparse_hashed) that is more memory efficient
...
With this new layout, sparsehash will be used over default HashMap,
sparsehash is more memory efficient but it is also slower.
So in a nutshell:
- HashMap uses ~2x more memory then sparse_hash_map
- HashMap ~2-2.5x faster then sparse_hash_map
(tested on lots of input, and the most close to production was
dictionary with 600KK hashes and UInt16 as value)
TODO:
- fix allocated memory calculation
- getBufferSizeInBytes/getBufferSizeInCells interface
- benchmarks
v0: replace HashMap with google::sparse_hash_map
v2: use google::sparse_hash_map only when <sparse> isset to true
v3: replace attributes with different layout
v4: use ch hash over std::hash
2019-09-21 02:22:40 +03:00
kreuzerkrieg
112fc71276
adding -Wshadow for GCC
2019-08-09 23:58:16 +03:00
Vitaly Baranov
6d563c51e6
Remove unnecessary try-catch in the constructors of classes derived from IExternalLoadable.
2019-07-17 11:39:36 +03:00
chertus
49e12eaa60
remove unused specializations in dictionaries
2019-05-28 22:30:01 +03:00
alexey-milovidov
c5d18dfde1
Merge pull request #5446 from 4ertus2/some
...
Disable implicit type conversions in dictGet functions (it's not working)
2019-05-28 20:34:13 +03:00
chertus
0d77a96b3c
disable implicit type conversions in dictGet functions
2019-05-28 17:04:00 +03:00
Azat Khuzhin
0275d62523
Fix element_count for hashed dictionary (do not include duplicates)
2019-05-28 11:25:31 +03:00
Amos Bird
26ab5dd7a7
A Proper lookup table that uses HashTable's API
...
This is the first step of allowing heterogeneous cells in hash tables.
performance test results are
```
1. HashMap<UInt16, UInt8, TrivialHash, HashTableFixedGrower<16>>;
2. NewLookupMap<UInt16, UInt8>
ResolutionWidth 30000 1 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................223550276.46
ResolutionWidth 30000 2 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................248772721.24
Best: 2 - 24877272124
ResolutionWidth 100000 1 ..........................................................................................................................................................................................................................................................238498413.99
ResolutionWidth 100000 2 ..........................................................................................................................................................................................................................................................261808889.98
Best: 2 - 26180888998
ResolutionWidth 300000 1 ...................................................................................239307348.81
ResolutionWidth 300000 2 ...................................................................................257592761.30
Best: 2 - 25759276130
ResolutionWidth 1000000 1 .........................240144759.26
ResolutionWidth 1000000 2 .........................257093531.91
Best: 2 - 25709353191
ResolutionWidth 5000000 1 .....241573260.35
ResolutionWidth 5000000 2 .....259314162.79
Best: 2 - 25931416279
ResolutionDepth 30000 1 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................217108119.84
ResolutionDepth 30000 2 .................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................249459504.41
Best: 2 - 24945950441
ResolutionDepth 100000 1 ..........................................................................................................................................................................................................................................................229065162.17
ResolutionDepth 100000 2 ..........................................................................................................................................................................................................................................................253769105.64
Best: 2 - 25376910564
ResolutionDepth 300000 1 ...................................................................................233079225.18
ResolutionDepth 300000 2 ...................................................................................256316273.78
Best: 2 - 25631627378
ResolutionDepth 1000000 1 .........................234184633.51
ResolutionDepth 1000000 2 .........................261100491.57
Best: 2 - 26110049157
ResolutionDepth 5000000 1 .....233118795.66
ResolutionDepth 5000000 2 .....252436160.41
Best: 2 - 25243616041
```
2019-03-01 16:47:13 +08:00
Vitaly Baranov
86b204d1c7
IDictionary::getBlockInputStream() function, "max_block_size" parameter: change type UInt64 -> size_t.
2019-02-19 14:49:03 +03:00
Alexey Milovidov
58a6accee5
Updated test #4246
2019-02-10 19:55:12 +03:00
Alexey Milovidov
c7b95b5175
Miscellaneous changes for PVS-Studio
2019-01-20 02:27:52 +03:00
proller
5e89894b98
Fix style
2018-12-10 18:50:58 +03:00
proller
c9e98c8deb
clang-format of dbms/src/Dictionaries/*
2018-12-10 18:25:45 +03:00
proller
dd2371e071
CLICKHOUSE-4137 DictionaryFactory, DictionarySourceFactory ( #3653 )
...
* Split ComplexKeyCacheDictionary to faster compile (part2)
* Dictionaries as lib WIP
* wip
* clean
* Fix build with old capnp
* fix
* wip
* fixes
* fix
* clean
* clean
* clean
* wip
* wip
* wip
* flat
* wip
* cache
* clean
* wip
* faster
* fix style
* fixes
* clean
* clean
* Split CacheDictionary.cpp for faster compile
* fix
* fix
* Less memory usage while compiling
* missing file
* format
* Update registerDictionaries.h
* clean
2018-11-28 14:37:12 +03:00
Alexey Milovidov
78e72a1962
Little better #3624
2018-11-20 23:09:20 +03:00
Alexey Milovidov
69e57187d2
Fixed error; removed unique_ptr #3565
2018-11-13 22:43:17 +03:00
Alexey Milovidov
4b0645cb25
Fixed error in #3565
2018-11-13 21:41:07 +03:00
chertus
f21c4c898e
Decimal dictionaries support [CLICKHOUSE-4045]
2018-10-08 22:45:17 +03:00
Alexey Milovidov
c7b1a40296
Fixed test [#CLICKHOUSE-2]
2018-09-09 05:23:24 +03:00
Alexey Milovidov
f584344184
Miscellaneous [#CLICKHOUSE-3939]
2018-09-06 05:24:35 +03:00
Alexey Milovidov
d8809eb2a1
Whitespaces [#CLICKHOUSE-3939]
2018-09-06 04:57:55 +03:00
Alexey Milovidov
f5adbceed2
Non significant changes according to clang's -Weverything, part 3 [#CLICKHOUSE-2]
2018-06-03 23:39:06 +03:00
Alexey Milovidov
956128a67a
Fixed style a bit [#CLICKHOUSE-2]
2018-05-07 05:01:11 +03:00
Nikolai Kochetov
a399c17f01
&& - qualified IColumn::mutate
2018-03-20 17:01:00 +03:00
Arsen Hakobyan
7a77b8abf0
Improvements done according to requested changes
2018-02-15 17:08:23 +04:00
Arsen Hakobyan
89ca08147b
Fixes memory leak in hash dictionary
...
Fixes case when keys might not be loaded for complex key hash dictionary
2018-01-24 16:28:51 +04:00
Arsen Hakobyan
7658665737
Add external dictionary update feature
...
feature is implemented for DICT TYPES THAT SUPPORT
and for sources SOURCE TYPES THAT ARE SUPPORTED
to use the feature add <update_field>...</...> to dictionary sourcei config.
If the field is skipped or used with unsupported dictionary types,
all data is updated every time if the field is provided,
value of last update time is passed to the source
with the expectation that only records that were updated after
provided time will be passed to the dictionary
2018-01-15 16:44:39 +04:00
Alexey Milovidov
0bfea2f52c
Miscellaneous [#CLICKHOUSe-2]
2018-01-10 03:04:08 +03:00
Alexey Milovidov
431da20d99
Better [#CLICKHOUSE-2].
2017-12-01 23:21:35 +03:00
Иванов Евгений
48636ac601
Added support UUID type for dictionaries
2017-11-15 23:31:53 +03:00
alexey-milovidov
d4d266fa18
ColumnConst unification ( #1011 )
...
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* Fixed error in ColumnArray::replicateGeneric [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
* ColumnConst: unification (incomplete) [#CLICKHOUSE-3150].
2017-07-21 09:35:58 +03:00
Nikolai Kochetov
bff77d1bae
merged with master
2017-06-08 15:00:11 +03:00
Alexey Milovidov
9c5ddce06d
Get rid of .hpp files [#CLICKHOUSE-2].
2017-06-06 20:18:32 +03:00
Nikolai Kochetov
7c736b12ef
added raw serialization functions for IColumn; changed key serialization in ComplexKeyCacheDictionary; style fixes
2017-05-29 20:26:45 +03:00
Nikolai Kochetov
5fb5397941
added max_block_size for DictionaryBlockImputStreamBase
2017-05-26 19:08:56 +03:00
Nikolai Kochetov
1604ce138a
changed key storage format for complex key dictionaries
2017-05-04 21:14:23 +03:00
Nikolai Kochetov
79564054a9
added RangeDictionaryBlockInputStream
2017-04-28 21:33:31 +03:00
Alexey Milovidov
310736cd7e
Moved headers and sources to same place [#CLICKHOUSE-3].
2017-04-01 12:22:42 +03:00
Alexey Milovidov
137ad95929
Changed tabs to spaces in code [#CLICKHOUSE-3].
2017-04-01 11:35:09 +03:00
Alexey Milovidov
e6ba252507
Hierarchical dictionaries performance: continued [#CLICKHOUSE-2144].
2017-03-26 03:42:01 +03:00
Alexey Milovidov
06b2404e6a
Merge branch 'master' into hierarchical-dictionaries-performance
...
Conflicts:
dbms/include/DB/Functions/FunctionsDictionaries.h
2017-01-17 03:31:31 +03:00
Alexey Milovidov
2b4f3b5d34
Renamed method (less confusion) [#METR-2944].
2017-01-02 23:12:12 +03:00