thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-13 18:02:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
109,356 Commits 9,328 Branches 1,783 Tags 2.1 GiB
81b30021db
Commit Graph

150 Commits

Author SHA1 Message Date
Azat Khuzhin
4fa2ae76bc Fix memory leak in AggregatingInOrderTransform 
Reproducer:

    # NOTE: we need clickhouse from 33957 since right now LSan is broken due to getauxval().
    $ url=https://s3.amazonaws.com/clickhouse-builds/33957/e04b862673644d313712607a0078f5d1c48b5377/package_asan/clickhouse
    $ wget $url -o clickhouse-asan
    $ chmod +x clickhouse-asan
    $ ./clickhouse-asan server &

    $ ./clickhouse-asan client
    :) create table data (key Int, value String) engine=MergeTree() order by key
    :) insert into data select number%5, toString(number) from numbers(10e6)

    # usually it is enough one query, benchmark is just for stability of the results
    # note, that if the exception was not happen from AggregatingInOrderTransform then add --continue_on_errors and wait
    $ ./clickhouse-asan benchmark --query 'select key, uniqCombined64(value), groupArray(value) from data group by key' --optimize_aggregation_in_order=1 --memory_tracker_fault_probability=0.01, max_untracked_memory='2Mi'

LSan report:

    ==24595==ERROR: LeakSanitizer: detected memory leaks

    Direct leak of 3932160 byte(s) in 6 object(s) allocated from:
        0 0xcadba93 in realloc ()
        1 0xcc108d9 in Allocator<false, false>::realloc() obj-x86_64-linux-gnu/../src/Common/Allocator.h:134:30
        2 0xde19eae in void DB::PODArrayBase<>::realloc<DB::Arena*&>(unsigned long, DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h:161:25
        3 0xde5f039 in void DB::PODArrayBase<>::reserveForNextSize<DB::Arena*&>(DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h
        4 0xde5f039 in void DB::PODArray<>::push_back<>(DB::GroupArrayNodeString*&, DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h:432:19
        5 0xde5f039 in DB::GroupArrayGeneralImpl<>::add() const obj-x86_64-linux-gnu/../src/AggregateFunctions/AggregateFunctionGroupArray.h:465:31
        6 0xde5f039 in DB::IAggregateFunctionHelper<>::addBatchSinglePlaceFromInterval() const obj-x86_64-linux-gnu/../src/AggregateFunctions/IAggregateFunction.h:481:53
        7 0x299df134 in DB::Aggregator::executeOnIntervalWithoutKeyImpl() obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:869:31
        8 0x2ca75f7d in DB::AggregatingInOrderTransform::consume() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingInOrderTransform.cpp:124:13

    ...

    SUMMARY: AddressSanitizer: 4523184 byte(s) leaked in 12 allocation(s).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-09 09:23:56 +03:00
Dmitry Novik
1095814bbf Cleanup code 2022-01-11 11:26:13 +00:00
fanzhou
83e9e5d0e5 some changes 2022-01-11 11:26:12 +00:00
MaxTheHuman
e6bd807f60 grouping sets development 2022-01-11 11:26:10 +00:00
MaxTheHuman
4d1b354b5f grouping sets development 2022-01-11 11:26:10 +00:00
MaxTheHuman
abe09324c1 grouping sets development 2022-01-11 11:26:10 +00:00
MaxTheHuman
3195d600c5 feat grouping-sets: initial changes 2022-01-11 11:26:10 +00:00
alexey-milovidov
0a55fa3dc2
Revert "Grouping sets dev" 2021-12-25 20:30:31 +03:00
alexey-milovidov
6b97af4c63
Merge pull request #26869 from taylor12805/grouping-sets-dev 
Grouping sets dev
 2021-12-17 20:50:15 +03:00
Dmitry Novik
56a3f4a000 Cleanup code 2021-12-14 22:15:14 +03:00
fanzhou
43db4594ba some changes 2021-11-29 19:35:33 +03:00
MaxTheHuman
ddd1799743 grouping sets development 2021-11-26 22:11:34 +03:00
MaxTheHuman
d2258decf5 grouping sets development 2021-11-26 21:50:03 +03:00
MaxTheHuman
e60d1dd818 grouping sets development 2021-11-26 21:38:44 +03:00
MaxTheHuman
2bd07ef338 feat grouping-sets: initial changes 2021-11-26 20:24:35 +03:00
Anton Popov
d50137013c Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-01 16:55:53 +03:00
Nikolai Kochetov
a92dc0a826 Update obsolete comments. 2021-10-19 12:58:10 +03:00
Anton Popov
d71ffc355a Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-18 15:18:22 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
Anton Popov
7aa6068fb2 Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-14 19:44:08 +03:00
Nikolai Kochetov
ab28c6c855 Remove BlockInputStream interfaces. 2021-10-14 13:25:43 +03:00
Nikolai Kochetov
c6bce1a4cf Update Native. 2021-10-08 20:21:19 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Anton Popov
eef436fe22 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-16 18:07:42 +03:00
Amos Bird
91293c7449
Fix crash on exception with projection aggregate 2021-09-09 10:43:56 +08:00
Anton Popov
c3c3a06078 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-20 01:45:38 +03:00
Anton Popov
16ed0f6ed4 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-02 17:55:17 +03:00
Maksim Kita
3a6b37691a Compile aggregate functions without key 2021-07-27 19:50:57 +03:00
Maksim Kita
1fea19846b Compile aggregate functions profile events fix 2021-07-23 00:43:31 +03:00
Anton Popov
2b0ec88f5a disable jit aggregation for sparse columns 2021-07-20 20:02:41 +03:00
Anton Popov
14168b11f2 Merge remote-tracking branch 'upstream/master' into HEAD 2021-07-07 17:05:11 +03:00
Maksim Kita
325b54f623 Aggregator compile only part of aggregate functions 2021-07-01 22:56:36 +03:00
Maksim Kita
da8c957167 Aggregator added CompiledExpressionCache 2021-07-01 22:56:36 +03:00
Maksim Kita
d24d3ae992 Added second variant of compilation 2021-07-01 22:56:36 +03:00
Maksim Kita
1e2f22a183 Aggregator compile part of aggregate functions 2021-07-01 22:56:36 +03:00
Maksim Kita
a5ef0067b8 Compile AggregateFunctionIf 2021-07-01 22:56:35 +03:00
Maksim Kita
9b71b1040a Aggregate functions update compile interface 2021-07-01 22:56:35 +03:00
Maksim Kita
3fe559b31f Compile aggregate functions 2021-07-01 22:56:35 +03:00
Anton Popov
567043113c Merge remote-tracking branch 'upstream/master' into HEAD 2021-06-21 01:36:06 +03:00
Alexey Milovidov
885ce194e0 Making fundamentals correct 2021-06-07 00:49:55 +03:00
Anton Popov
3e92c7f61a Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-25 21:45:19 +03:00
Amos Bird
07b1be5a76
Fix distributed processing when using projection 2021-05-16 22:40:06 +08:00
Anton Popov
d8df0903b9 Merge remote-tracking branch 'upstream/master' into HEAD 2021-05-14 23:38:16 +03:00
Nikolai Kochetov
3296c9292f
Try to merge projectons faster. 2021-05-11 18:12:26 +08:00
Alexey Milovidov
9f74565948 PVS-Studio 2021-05-03 01:42:01 +03:00
Alexey Milovidov
1fda866881 Merge branch 'master' into normalize-bigint 2021-05-02 21:24:45 +03:00
Nikolai Kochetov
c508868570
Merge pull request #23082 from ClickHouse/explain-output 
JSON output for EXPLAIN
 2021-04-28 15:39:06 +03:00
Maksim Kita
cac4a85286 Aggregator remove unused code 2021-04-26 01:17:24 +03:00
Alexey Milovidov
60be962dca Merging with master 2021-04-25 07:08:57 +03:00
Alexey Milovidov
8f01af62d9 Merge branch 'master' into normalize-bigint 2021-04-25 06:57:44 +03:00
Nikolai Kochetov
9b86c19836 Use JSONBuilder for explain json result. 2021-04-14 17:07:56 +03:00
Anton Popov
d46958a8d2 Merge remote-tracking branch 'upstream/master' into HEAD 2021-04-06 00:54:49 +03:00
Anton Popov
a000d6154c fix clang-tidy 2021-04-02 15:10:49 +03:00
Anton Popov
56320bcbfd minor refactoring 2021-04-02 01:16:54 +03:00
Anton Popov
88cd775f6a Merge remote-tracking branch 'upstream/master' into HEAD 2021-04-02 00:14:03 +03:00
Anton Popov
372a1b1fe7 Merge remote-tracking branch 'upstream/master' into HEAD 2021-03-29 19:57:49 +03:00
Anton Popov
577d571300 ColumnSparse: initial implementation 2021-03-29 19:54:24 +03:00
Anton Popov
5de2b8512f fix destruction of aggregate states 2021-03-23 01:12:14 +03:00
Anton Popov
6f7c8894e9 fix bugs in aggregation by primary key 2021-03-18 23:57:42 +03:00
Nikolai Kochetov
66d5968cbc Batch aggregation for overflow rows. 2021-03-18 12:31:14 +03:00
Nikolai Kochetov
4c8c45b163 Refactor 2021-03-10 14:00:24 +03:00
Nikolai Kochetov
545528917f Fix tests. 2021-02-26 13:17:00 +03:00
Nikolai Kochetov
ce365e2c19 Fix tests. 2021-02-26 11:52:50 +03:00
Alexey Milovidov
5bdc570046 Improve performance of GROUP BY multiple fixed size keys 2021-02-14 01:56:04 +03:00
Alexey Milovidov
093108bf81 Normalize BigInt implementation 2021-01-27 03:54:57 +03:00
feng lv
beb0fec8bb mirror changes in code and comment 2021-01-22 09:13:22 +00:00
Anton Popov
857e2ef557 remove unused code 2020-12-23 21:03:41 +03:00
Anton Popov
c8cdaba180 Merge remote-tracking branch 'upstream/master' into remove-unused-code-1 2020-12-23 20:38:29 +03:00
Anton Popov
204a20fe29 remove unused code 2020-12-23 20:34:00 +03:00
Anton Popov
57a1642035 fix race in aggregation with combinator distinct 2020-12-22 14:30:29 +03:00
Alexey Milovidov
4ecfe8fef4 Remove unused headers 2020-12-08 02:10:22 +03:00
Amos Bird
8ca4d6634e
better fixed string group by support 2020-09-20 15:20:09 +08:00
Artem Zuikov
d304d522cc
Refactoring: rename types to allow change bigint library (#14219) 2020-09-01 12:54:50 +03:00
Alexey Milovidov
460bda7759 Use stored size for UInt16 case 2020-08-02 01:11:00 +03:00
Alexey Milovidov
f1363403b7 Avoid increment of size in the inner loop of aggregation by UInt8/UInt16 2020-08-02 01:11:00 +03:00
alexey-milovidov
626b709085
Merge pull request #13055 from ClickHouse/fixed-hash-map-implicit-zero 
Slightly improve the inner loop of aggregation by UInt8/UInt16 keys
 2020-08-02 01:09:00 +03:00
Alexey Milovidov
490a8ba3d4 Slightly improve inner loop of aggregation by UInt8/UInt16 keys 2020-07-29 04:50:12 +03:00
Gleb Novikov
aac97957ce Fixed integration test_tmp_policy 2020-07-26 21:27:36 +03:00
Gleb Novikov
7f5b6fba78 Generic volume is coming... 
1. SingleDiskVolume for temporary volumes
2. Generic VolumePtr in StoragePolicies
3. Removed max_data_part_size in system.storage_policies, added volume_type
 2020-07-26 21:27:36 +03:00
Nikolai Kochetov
6ed2908e2a Rename ident to indent. 2020-07-07 22:51:32 +03:00
Nikolai Kochetov
5b1596cdb2 Refactor some code. 2020-06-27 17:02:24 +03:00
Nikolai Kochetov
893a75bcf7 Merge branch 'master' into initial-explain 2020-06-27 12:45:40 +03:00
Nikolai Kochetov
d88e19b69d Added actions setting for Explain query. 2020-06-27 12:31:06 +03:00
Anton Popov
14e09e5650 Merge remote-tracking branch 'upstream/master' into HEAD 2020-06-25 14:59:15 +03:00
Nikolai Kochetov
b7bed87a7f Use QueryPlan in InterpreterSelectQuery [part 3]. 2020-06-18 21:29:33 +03:00
Anton Popov
88b325dcdc rework distinct combinator 2020-06-17 22:36:27 +03:00
Alexey Milovidov
58e513f59e Corrected implementation for the case without key 2020-06-08 07:23:56 +03:00
Anton Popov
5c42408add
Merge pull request #9113 from dimarub2000/group_by_in_order_optimization 
[WIP] Optimization of GROUP BY with respect to table sorting key.
 2020-06-06 14:25:59 +03:00
Nikolai Kochetov
3a0acb00c1 Remove some streams. 2020-06-01 19:31:06 +03:00
Alexey Milovidov
25f941020b Remove namespace pollution 2020-05-31 00:57:37 +03:00
Dmitry
a597e62d3b nested_columns_holder is not destoyed 2020-05-15 22:27:18 +03:00
Dmitry
c4295d97f9 Merge branch 'master' of github.com:yandex/ClickHouse into group_by_in_order_optimization 2020-05-12 17:57:32 +03:00
Dmitry
3ec80b5531 now aggregates in parallel 2020-05-12 17:50:13 +03:00
Dmitry
015a3555c6 small fixes 2020-05-08 16:28:18 +03:00
Dmitry
465dfe47fc fixed faults on LC 2020-05-07 23:13:51 +03:00
Dmitry
0883dd67d3 removed some code duplication 2020-05-07 17:54:15 +03:00
Dmitry
e98c23a8cf read in order enabled 2020-05-05 17:35:23 +03:00
Gleb Novikov
c637d99e07 Volumes and storages refactoring: 
1. Moved Volume to separate file
  2. Created IVolume interface and implemented current behaviour in implementation of new interface — VolumeJBOD
  3. Replaced all old volume usages with new VolumeJBOD. Where it is unnecessary to have JBOD — left just IVolume.
  4. Removed old Volume completely
  5. Moved StoragePolicy to separated files
  6. Moved DiskSelector to separated files
  7. Removed DiskSpaceMonitor file
 2020-05-04 23:15:38 +03:00
Dmitry
25cd4ce132 Merge branch 'master' of github.com:yandex/ClickHouse into group_by_in_order_optimization 2020-04-18 12:56:34 +03:00
Ivan Lezhankin
06446b4f08 dbms/ → src/ 2020-04-03 18:14:31 +03:00