thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-25 09:02:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
156,696 Commits 9,170 Branches 1,781 Tags 2.1 GiB
e2c2e67c7b
Commit Graph

1360 Commits

Author SHA1 Message Date
vdimir
737cff7e57 Remove whole join_set_filter.xml, will resubmit 2023-07-03 17:00:20 +02:00
vdimir
9ea5d929a5 Update tests/performance/join_set_filter.xml 2023-07-03 17:00:20 +02:00
vdimir
ebd7ecb230 Remove unstable queries from performance/join_set_filter 2023-07-03 17:00:20 +02:00
Igor Nikonov
35bc97e5f9 Merge remote-tracking branch 'origin/master' into remove-perf-test-duplicate-order-by-and-distinct 2023-06-16 20:56:56 +00:00
Azat Khuzhin
5caa3a9e80 Adjust min_insert_block_size_rows for materialized_view_parallelize_output_from_storages 
Otherwise it is too slow for perf tests on CI [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/50214/e287ec50920c7cadabea6ec19ef14b353345ac93/performance_comparison_[3_4]/report.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-06-14 19:11:23 +03:00
Azat Khuzhin
3e419730c3 Disable parallelize_output_from_storages for processing MATERIALIZED VIEWs 
Adding more processors for parallelize_output_from_storages is not a
costless operation (I've experienced some issues in production because
of this), and it is not easy to fix in a normal way, so let's disable it
for now.

Before this patch:
- INSERT INTO input SELECT * FROM numbers(10e6) SETTINGS parallelize_output_from_storages=1, min_insert_block_size_rows=1000
  0 rows in set. Elapsed: 3.648 sec. Processed 20.00 million rows, 120.00 MB (5.48 million rows/s., 32.90 MB/s.)

- INSERT INTO input SELECT * FROM numbers(10e6) SETTINGS parallelize_output_from_storages=0, min_insert_block_size_rows=1000
  0 rows in set. Elapsed: 1.851 sec. Processed 20.00 million rows, 120.00 MB (10.80 million rows/s., 64.82 MB/s.)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-06-14 19:11:23 +03:00
Igor Nikonov
79f53f428b
Merge branch 'master' into remove-perf-test-duplicate-order-by-and-distinct 2023-06-13 13:45:36 +02:00
flynn
92c87dedad
Add parallel state merge for some other combinator except If (#50413) 
* Add parallel state merge for some other combinator except If

* add test

* update test
 2023-06-08 00:41:32 +02:00
flynn
f616314f8b fix typo 2023-05-29 02:22:13 +00:00
flynn
05783f99cd update test 2023-05-28 14:17:59 +00:00
flynn
ec82c657eb Parallel merge of uniqExactIf states 2023-05-28 06:04:23 +00:00
Azat Khuzhin
2996b38606 Add ability to configure maximum load factor for the HASHED/SPARSE_HASHED layout 
As it turns out, HashMap/PackedHashMap works great even with max load
factor of 0.99. By "great" I mean it least it works faster then
google sparsehash, and not to mention it's friendliness to the memory
allocator (it has zero fragmentation since it works with a continuious
memory region, in comparison to the sparsehash that doing lots of
realloc, which jemalloc does not like, due to it's slabs).

Here is a table of different setups:

settings                         | load (sec) | read (sec) | read (million rows/s) | bytes_allocated | RSS
-                                | -          | -          | -                     | -               | -
HASHED upstream                  | -          | -          | -                     | -               | 35GiB
SPARSE_HASHED upstream           | -          | -          | -                     | -               | 26GiB
-                                | -          | -          | -                     | -               | -
sparse_hash_map glibc hashbench  | -          | -          | -                     | -               | 17.5GiB
sparse_hash_map packed allocator | 101.878    | 231.48     | 4.32                  | -               | 17.7GiB
PackedHashMap 0.5                | 15.514     | 42.35      | 23.61                 | 20GiB           | 22GiB
hashed 0.95                      | 34.903     | 115.615    | 8.65                  | 16GiB           | 18.7GiB
**PackedHashMap 0.95**           | **93.6**   | **19.883** | **10.68**             | **10GiB**       | **12.8GiB**
PackedHashMap 0.99               | 26.113     | 83.6       | 11.96                 | 10GiB           | 12.3GiB

As it shows, PackedHashMap with 0.95 max_load_factor, eats 2.6x less
memory then SPARSE_HASHED in upstream, and it also 2x faster for read!

v2: fix grower
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-19 06:07:21 +02:00
Azat Khuzhin
7b5d156cc5 Optimize SPARSE_HASHED layout (by using PackedHashMap) 
In case you want dictionary optimized for memory, SPARSE_HASHED is not
always gives you what you need.

Consider the following example <UInt64, UInt16> as <Key, Value>, but
this pair will also have a 6 byte padding (on amd64), so this is almost
40% of space wastage.

And because of this padding, even google::sparse_hash_map, does not make
picture better, in fact, sparse_hash_map is not very friendly to memory
allocators (especially jemalloc).

Here are some numbers for dictionary with 1e9 elements and UInt64 as
key, and UInt16 as value:

settings                         | load (sec) | read (sec) | read (million rows/s) | bytes_allocated | RSS
HASHED upstream                  | -          | -          | -                     | -               | 35GiB
SPARSE_HASHED upstream           | -          | -          | -                     | -               | 26GiB
-                                | -          | -          | -                     | -               | -
sparse_hash_map glibc hashbench  | -          | -          | -                     | -               | 17.5GiB
sparse_hash_map packed allocator | 101.878    | 231.48     | 4.32                  | -               | 17.7GiB
PackedHashMap                    | 15.514     | 42.35      | 23.61                 | 20GiB           | 22GiB

As you can see PackedHashMap looks way more better then HASHED, and
even better then SPARSE_HASHED, but slightly worse then sparse_hash_map
with packed allocator (it is done with a custom patch to google
sparse_hash_map).

v2: rebase on top of bucket_count fix
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-05-19 06:07:21 +02:00
Lirikl
39a505e3f3 init 2023-05-11 22:54:00 +03:00
lgbo-ustc
a07359fbe8 enable used flags's reinit only when the hash talbe rehash 2023-05-11 11:06:13 +08:00
Alexey Milovidov
8a6e07f0ea Make projections production-ready 2023-05-10 03:35:13 +02:00
Alexey Milovidov
f449df85b6 Deprecate in-memory parts 2023-05-03 00:31:09 +02:00
Alexey Milovidov
c279516ac1
Merge branch 'master' into parallel-reading-from-file 2023-04-10 08:02:50 +03:00
Igor Nikonov
8fdc2b3326 Perf test 2023-04-07 20:06:11 +00:00
Anton Popov
10d2b1330b add perf test 2023-04-04 21:29:52 +00:00
Anton Popov
1e79245b94 add tests 2023-03-28 17:20:05 +00:00
Ongkong
d9c7bc1859
Fix ASOF LEFT JOIN performance degradation (#47544) 2023-03-18 23:53:00 +01:00
LiuNeng
d4c5ab9dcd
Optimize one nullable key aggregate performance (#45772) 2023-03-02 21:01:52 +01:00
Igor Nikonov
2f7aa8849b
Merge branch 'master' into remove-perf-test-duplicate-order-by-and-distinct 2023-03-02 20:48:28 +01:00
Igor Nikonov
548d79c2e8 Remove perf test duplicate_order_by_and_distinct.xml 2023-03-02 12:31:09 +00:00
Alexander Gololobov
f64d08bd5c Enable lightweight delete support by default 2023-03-01 19:35:55 +01:00
Nikita Taranov
ab44740efb
Enable perf tests added in #45364 (#46623) 2023-02-28 00:26:11 +01:00
Alexey Milovidov
17992b178a
Merge pull request #45364 from nickitat/aggr_partitions_independently 
Add option to aggregate partitions independently
 2023-02-19 17:44:18 +03:00
Alexey Milovidov
417158f59f
Merge branch 'master' into lower_upper 2023-02-19 04:05:10 +03:00
Nikita Taranov
f70044f34b Merge branch 'master' into aggr_partitions_independently 2023-02-18 13:19:05 +00:00
Alexey Milovidov
6e0dab71ed
Merge pull request #46188 from bigo-sg/rewrite_array_exists 
Rewrite array exists to has
 2023-02-12 05:53:22 +03:00
Alexey Milovidov
786aa069e1
Merge pull request #46187 from ClickHouse/speed-up-count-digits 
Speed up `countDigits`
 2023-02-10 07:41:12 +03:00
taiyang-li
b83ad6bb81 add perf test 2023-02-09 12:30:50 +08:00
Alexey Milovidov
9a86d0087c Add performance test 2023-02-09 04:52:33 +01:00
Alexey Milovidov
66043eec24
Merge branch 'master' into decimal-performance 2023-02-09 04:59:37 +03:00
Igor Nikonov
72c393e7c4
Merge pull request #46014 from ClickHouse/inorder-optimization-update-sorting-properties 
Update sorting properties after reading in order applied
 2023-02-08 10:19:47 +01:00
Alexey Milovidov
a2df6e950e Whitespace 2023-02-08 03:38:23 +01:00
Alexey Milovidov
168fbc9d7b Add a test 2023-02-08 02:17:23 +01:00
李扬
444373679a
Merge branch 'master' into improve_decimal 2023-02-06 13:08:51 +08:00
Igor Nikonov
089a0009ad Polishing 
+ try to stabilize distinct in order perf test
 2023-02-05 13:38:20 +00:00
Nikita Taranov
b983b363f8 Merge branch 'master' into aggr_partitions_independently 2023-02-04 18:24:31 +00:00
李扬
ad6f39389d
Update tests/performance/column_array_filter.xml 
Co-authored-by: Alexander Gololobov <440544+davenger@users.noreply.github.com>
 2023-02-04 18:49:13 +08:00
Nikita Mikhaylov
33877b5e00
Parallel replicas. Part [2] (#43772) 2023-02-03 14:34:18 +01:00
taiyang-li
36a98a1628 add performance tests 2023-02-02 20:16:16 +08:00
Nikita Taranov
e7ca90adab fix perf test 2023-01-30 17:11:56 +00:00
Nikita Taranov
ac77808133 fix perf test 2023-01-30 17:11:56 +00:00
Nikita Taranov
52fe7edbd9 better key analysis 2023-01-30 17:11:56 +00:00
Nikita Taranov
2057db68a2 cosmetics 2023-01-30 17:10:45 +00:00
Nikita Taranov
1d45cce03c support for aggr in order 2023-01-30 17:10:45 +00:00
Nikita Taranov
a2c9aeb7c9 stash 2023-01-30 17:10:45 +00:00
taiyang-li
d25740da83 change as request 2023-01-30 16:13:12 +08:00
Alexey Milovidov
bc2f454522
Merge branch 'master' into block-non-float-gorilla-v2 2023-01-28 03:30:12 +03:00
Igor Nikonov
300f78df96
Merge pull request #45567 from ClickHouse/enable-remove-redundant-sorting 
Enable query_plan_remove_redundant_sorting optimization by default
 2023-01-27 19:14:36 +01:00
Igor Nikonov
41b94b4954 Enable query_plan_remove_redundant_sorting optimization by default 2023-01-24 13:38:21 +00:00
Robert Schulze
97d1bed114
Merge branch 'master' into improve_week_day 2023-01-21 20:40:33 +01:00
Robert Schulze
e6167d6b36
Deprecate Gorilla compression of non-float columns 
Reasons:

1. The original Gorilla paper proposed a compression schema for pairs of
   time stamps and double-precision FP values. ClickHouse's Gorilla
   codec only implements compression of the latter and it does not
   impose any data type restrictions.
   - Data types != Float* or (U)Int* (e.g. Decimal, Point etc.) are
     definitely not supposed to be used with Gorilla.
   - (U)Int* types are debatable. The paper only considers
     integers-stored-as-FP-values, a practical use case for which
     Gorilla works well. Standalone integers are not considered which
     makes them at least suspicious.

2. Achieve consistency with FPC, another specialized floating-point
   timeseries codec, which rejects non-float data.

3. On practical datasets, ZSTD is often "good enough" (**) so it should
   be okay to disincentive non-ZSTD codecs a little bit. If needed,
   Delta and DoubleDelta codecs are viable alternative for slowly
   changing (time-series-like) integer sequences.

Since on-prem and hosted users may still have Gorilla-compressed
non-float data, this combination is only deprecated for now. No warning
or error will be emitted. Users are encouraged to migrate
Gorilla-compressed non-float data to an alternative codec. It is planned
to treat Gorilla-compressed non-float columns as "suspicious" six months
after this commit (i.e. in v23.6). Even then, it will still be possible
to set "allow_suspicious_codecs = true" and read and write
Gorilla-compressed non-float data.

(*) Sec. 4.1.2, "Gorilla restricts the value element in its tuple to a
    double floating point type.", https://doi.org/10.14778/2824032.2824078

(**) https://clickhouse.com/blog/optimize-clickhouse-codecs-compression-schema
 2023-01-20 17:31:16 +00:00
Igor Nikonov
7ed8fec94f
Revert "Remove redundant sorting" 2023-01-18 18:38:25 +01:00
Igor Nikonov
72066846cf
Merge pull request #43905 from ClickHouse/igor/remove_redundant_order_by 
Remove redundant sorting
 2023-01-18 13:25:03 +01:00
Igor Nikonov
0cfa08df7a Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-17 16:28:17 +00:00
Alexander Tokmakov
df75c24f01
Revert "Disallow Gorilla codec on non-float columns" 2023-01-16 19:14:28 +03:00
Igor Nikonov
a34991cb65 Merge remote-tracking branch 'origin/master' into igor/remove_redundant_order_by 2023-01-16 12:14:02 +00:00
Robert Schulze
bd41c74ddf
Various test, code and docs fixups 2023-01-15 13:47:34 +00:00
Robert Schulze
7023d68536
Fix codecs_int_*.xml 2023-01-15 13:31:45 +00:00
Azat Khuzhin
925fd2c33a tests/performance: do not use scientific notation in hashed_dictionary_sharded 
v2: fix few mistakes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-13 13:39:26 +01:00
Azat Khuzhin
345c422e28 Add ability to load hashed dictionaries using multiple threads 
Right now dictionaries (here I will talk about only
HASHED/SPARSE_HASHED/COMPLEX_KEY_HASHED/COMPLEX_KEY_SPARSE_HASHED)
can load data only in one thread, since it uses one hash table that
cannot be filled from multiple threads.

And in case you have very big dictionary (i.e. 10e9 elements), it can
take a awhile to load them, especially for SPARSE_HASHED variants (and
if you have such amount of elements there, you are likely use
SPARSE_HASHED, since it requires less memory), in my env it takes ~4
hours, which is enormous amount of time.

So this patch add support of shards for dictionaries, number of shards
determine how much hash tables will use this dictionary, also, and which
is more important, how much threads it can use to load the data.

And with 16 threads this works 2x faster, not perfect though, see the
follow up patches in this series.

v0: PARTITION BY
v1: SHARDS 1
v2: SHARDS(1)
v3: tried optimized mod - logical and, but it does not gain even 10%
v4: tried squashing more (max_block_size * shards), but it does not gain even 10% either
v5: move SHARDS into layout parameters (unknown simply ignored)
v6: tune params for perf tests (to avoid too long queries)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-13 13:39:25 +01:00
Nikolai Kochetov
30310df5be
Merge branch 'master' into logical-optimizer-lowcardinality 2023-01-12 18:51:05 +01:00
Nikita Taranov
006fdd32d4
Apply preallocation optimisation more carefully (#44455) 
* impl

* add perf test

* fix

* review fixes
 2023-01-09 13:30:48 +01:00
Igor Nikonov
2187bdd4cc Disable diagnostics 
+ cleanup
+ disable optimization in sort performance test since it removes sorting
  at all
 2023-01-06 17:00:05 +00:00
Nikolay Degterinsky
dfe93b5d82
Merge pull request #42284 from Algunenano/perf_experiment 
Performance experiment
 2022-12-30 03:14:22 +01:00
Alexey Milovidov
79f2e747e4 Remove QuestDB (flaky test) 2022-12-28 12:42:14 +01:00
Raúl Marín
fc1fa82a39
Merge branch 'master' into perf_experiment 2022-12-27 10:51:58 +01:00
Raúl Marín
45d27f461b
Merge branch 'master' into perf_experiment 2022-12-20 09:07:48 +00:00
Kruglov Pavel
37df9b9990
Merge branch 'master' into refactor-schema-inference 2022-12-16 19:13:15 +01:00
Azat Khuzhin
53bac4de71 tests/perf: fix dependency check during DROP 
CI [1]:

    DB::Exception: Cannot drop or rename default.hierarchical_dictionary_source_table, because some tables depend on it: default.hierarchical_hashed_array_dictionary, default.hierarchical_flat_dictionary, default.hierarchical_hashed_dictionary. Stack trace:

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/44256/8e67a361a8f14abec6717af09ee997eb25151685/performance_comparison_[1/4]/report.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-12-16 15:15:15 +01:00
Nikolay Degterinsky
9b6d31b95d
Merge branch 'master' into perf_experiment 2022-12-13 17:15:07 +01:00
avogar
7375a7d429 Refactor and improve schema inference for text formats 2022-12-07 21:19:27 +00:00
Guo Wangyang
b86686b3f8
Merge branch 'master' into logical-optimizer-lowcardinality 2022-12-07 13:33:25 +08:00
Maksim Kita
1cdc7ab62a
Merge pull request #43556 from Algunenano/interpretation_benchmark 
Add benchmark for query interpretation with JOINs
 2022-12-01 22:53:02 +03:00
Vladimir C
53dc70a2d0
Merge pull request #38191 from BigRedEye/grace_hash_join 
Closes https://github.com/ClickHouse/ClickHouse/issues/11596
 2022-11-30 17:01:00 +01:00
Nikolai Kochetov
51439e2c19
Merge pull request #43260 from ClickHouse/read-from-mt-in-io-pool 
Read from MergeTree in I/O pool
 2022-11-29 12:09:03 +01:00
Nikolai Kochetov
d9fc13b230
Update async_remote_read.xml 2022-11-28 14:00:49 +01:00
Nikita Taranov
8ed5cfc265
Memory bound merging for distributed aggregation in order (#40879) 
* impl

* fix style

* make executeQueryWithParallelReplicas similar to executeQuery

* impl for parallel replicas

* cleaner code for remote sorting properties

* update test

* fix

* handle when nodes of old versions participate

* small fixes

* temporary enable for testing

* fix after merge

* Revert "temporary enable for testing"

This reverts commit cce7f8884c.

* review fixes

* add bc test

* Update src/Core/Settings.h
 2022-11-28 00:41:31 +01:00
Nikita Taranov
d1c258cf20
Add xxh3 hash function (#43411) 
* impl

* try fix

* add docs

* add test

* rm unused file

* excellent
 2022-11-26 00:14:08 +01:00
Nikolai Kochetov
4632e7c644 Add max_streams_for_merge_tree_reading setting. 2022-11-25 17:14:22 +00:00
Nikolai Kochetov
dfd3976040
Update async_remote_read.xml 2022-11-25 14:53:45 +01:00
Igor Nikonov
236e7e3989 Small fixes 2022-11-25 12:04:12 +00:00
Igor Nikonov
20e67b7140 Merge remote-tracking branch 'origin/master' into HEAD 2022-11-24 13:10:37 +00:00
Nikolai Kochetov
e79c91947a
Update async_remote_read.xml 2022-11-24 12:35:02 +01:00
Raúl Marín
e910648c5d Add benchmark for query interpretation with JOINs 2022-11-23 13:15:35 +01:00
Raúl Marín
ed0c174c0c Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-11-21 11:02:31 +01:00
Guo Wangyang
7d6ff90e34
Merge branch 'master' into logical-optimizer-lowcardinality 2022-11-20 09:56:50 +08:00
Nikolai Kochetov
5da1d893fd
Merge branch 'master' into read-from-mt-in-io-pool 2022-11-18 21:10:45 +01:00
Nikita Taranov
7beb58b0cf
Optimize merge of uniqExact without_key (#43072) 
* impl for uniqExact

* rm unused (read|write)Text methods

* fix style

* small fixes

* impl for variadic uniqExact

* refactor

* fix style

* more agressive inlining

* disable if max_threads=1

* small improvements

* review fixes

* Revert "rm unused (read|write)Text methods"

This reverts commit a7e7480584.

* encapsulate is_able_to_parallelize_merge in Data

* encapsulate is_exact & argument_is_tuple in Data
 2022-11-17 13:19:02 +01:00
Kruglov Pavel
1b68f605a2
Merge pull request #42761 from AlfVII/fix-slow-json-extract-with-low-cardinality 
Fixed slowness in JSONExtract with LowCardinality(String) tuples
 2022-11-17 12:49:18 +01:00
Raúl Marín
97d6fc3071 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-11-17 11:48:46 +01:00
Nikolai Kochetov
10f449c6c1 Add a query to perftest. 2022-11-15 18:08:03 +00:00
李扬
1de5bb2392
Add function canonicalRand (#43124) 
* add function canonicalRand

* add perf test

* revert rand.xml
 2022-11-15 00:27:19 +01:00
Wangyang Guo
887779e8d8 Add perftest: low_cardinality_query 2022-11-08 17:19:18 +08:00
Kruglov Pavel
e9a01a1946
Merge branch 'master' into fix-slow-json-extract-with-low-cardinality 2022-11-04 11:13:46 +01:00
Nikolay Degterinsky
30ad1a6826
Merge branch 'master' into perf_experiment 2022-11-03 02:18:21 +03:00
vdimir
6a4247ca32
Merge branch 'master' into grace_hash_join 2022-10-31 09:54:37 +00:00
Alfonso Martinez
9e33b13737 Merge remote-tracking branch 'upstream/master' into fix-slow-json-extract-with-low-cardinality 2022-10-31 08:46:55 +01:00
avogar
fe0aea2e3a Support parallel parsing for LineAsString input format 2022-10-28 21:56:09 +00:00
Alfonso Martinez
c37b154254 Added reverted files and fixes for failing fuzzer tests 2022-10-28 12:37:59 +02:00
Raúl Marín
891484b462 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-27 13:17:07 +02:00
Vladimir C
31e8f92cd9
Merge pull request #42664 from ClickHouse/vdimir/followup-42274 2022-10-27 12:20:46 +02:00
vdimir
14d0f6457b
Add tests and doc for some url-related functions 2022-10-26 10:52:57 +00:00
Raúl Marín
9395f77421 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-26 11:46:17 +02:00
Raúl Marín
6e0a9452e7 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-25 15:25:06 +02:00
Anton Popov
eed21ad4ca
Revert "Low cardinality cases moved to the function for its corresponding type" 2022-10-25 01:30:32 +02:00
vdimir
adb63a5583
Merge branch 'master' into grace_hash_join 2022-10-17 12:32:56 +00:00
Raúl Marín
46616d341c Make explain_ast even larger 2022-10-14 14:06:44 +02:00
AlfVII
5b2703c412
Merge branch 'master' into fix-slow-json-extract-with-low-cardinality 2022-10-11 13:28:07 +02:00
vdimir
ff55c369bc
Merge branch 'tmp-data-followup' 2022-10-05 18:10:05 +00:00
BoloniniD
9dd15998c7 Print nicer exception if BLAKE3 is unavailable 2022-10-05 00:11:41 +03:00
Sergei Trifonov
a592150ae7
Merge branch 'master' into fix-slow-json-extract-with-low-cardinality 2022-10-03 18:10:07 +02:00
Vitaly Baranov
65c61877c7
Merge pull request #33435 from BoloniniD/BLAKE3 
Integrating Rust code into ClickHouse
 2022-10-03 15:25:06 +02:00
Anton Popov
77eacfbbe0
Update bitmap_array_element.xml 2022-10-03 14:56:34 +02:00
BoloniniD
f5c57cd4a8 Fix test queries 2022-10-03 00:20:44 +03:00
flynn
7109aff2f0 fix style 2022-09-30 18:04:51 +08:00
flynn
1f51a86285 add test 2022-09-30 18:04:51 +08:00
vdimir
7ebc297f4c
Merge branch 'master' into pr/BigRedEye/38191 2022-09-30 09:40:47 +00:00
BoloniniD
55c79230b3 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-09-29 23:53:25 +03:00
Sergei Trifonov
976804b5db
Merge branch 'master' into fix-slow-json-extract-with-low-cardinality 2022-09-26 12:48:49 +02:00
Alfonso Martinez
9cb74c7807 Low cardinality cases moved to the function for its corresponding type 2022-09-23 14:12:37 +02:00
Igor Nikonov
8c93a9adda Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-22 07:40:14 +00:00
Nikita Taranov
930d050b55
fix (#41648) 2022-09-21 19:04:03 +02:00
Nikita Taranov
100c055510
Prefetching in aggregation (#39304) 
* impl

* stash

* clean up

* do not apply when HT is small

* make branch static

* also in merge

* do not hardcode look ahead value

* fix

* apply to methods with cheap key calculation

* more tests

* silence tidy

* fix build

* support HashMethodKeysFixed

* apply during merge only for cheap

* stash

* fixes

* rename method

* add feature flag

* cache prefetch threshold value

* fix

* fix

* Update HashMap.h

* fix typo

* 256KB as default l2 size

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2022-09-21 18:59:07 +02:00
BoloniniD
55fcb98f29 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-09-19 21:53:14 +03:00
Igor Nikonov
aca810ba62 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-19 18:34:38 +00:00
Kruglov Pavel
4c3194eefe
Merge pull request #41286 from azat/utf8-fix 
Do not allow invalid sequences influence other rows in lowerUTF8/upperUTF8
 2022-09-19 14:07:10 +02:00
Azat Khuzhin
bd54a6c45d tests: add perf test for lowerUTF8()/upperUTF8() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-17 11:16:45 +02:00
BoloniniD
452ef4435b Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-09-16 20:05:56 +03:00
Nikita Taranov
ee31be4286 impl 2022-09-16 15:41:15 +02:00
Igor Nikonov
eeecaf7a31 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-16 10:30:52 +00:00
Raúl Marín
c3ff66bd9d
Implement batch processing for aggregate functions with multiple nullable arguments (#41058) 
* Implement batch processing for aggregate functions with multiple nullable arguments

* Fix broken perf test

* Improve filter handling in addBatchSinglePlace with nullable arguments

* Fix detecting the Null filter usage
 2022-09-15 23:51:38 +02:00
Raúl Marín
6dac509739
Speed up reading uniqState (#41089) 
* Speed up reading UniquesHashSet

* Improve uniq serialization tests
 2022-09-15 23:41:15 +02:00
Igor Nikonov
8a4806e8c0 Fix test 
- remove perfomance queries which can be unstable
 2022-09-15 10:53:42 +00:00
BoloniniD
e8bcbcd016
Merge branch 'master' into BLAKE3 2022-09-09 11:48:31 +03:00
vdimir
6d4b6c452a
Merge branch 'master' into grace_hash_join 2022-09-07 08:00:14 +00:00
Nikita Taranov
7c4f42d014
Skip empty literals in lz4 decompression (#40142) 2022-09-06 13:58:26 +02:00
Alexey Milovidov
193cd1b3b2
Merge pull request #39138 from nickitat/control_block_size_in_aggregator 
Control block size in aggregator
 2022-09-04 04:51:00 +03:00
vdimir
e21763e759
remove new setting from join_set_filter.xml 2022-08-29 09:49:13 +00:00
vdimir
470dcff89c
Add tests/performance/join_set_filter.xml 2022-08-29 09:49:11 +00:00
Alexey Milovidov
ab91c99495
Merge branch 'master' into control_block_size_in_aggregator 2022-08-20 21:28:27 +03:00
Kruglov Pavel
b67cb9e378
Merge pull request #40173 from Avogar/arrow-dict 
Improve and fix dictionaries in Arrow format
 2022-08-18 20:54:55 +02:00
Igor Nikonov
46ed4f6cdf
Merge pull request #38719 from ClickHouse/skipping_sorting_step 
SortingStep: deduce way to sort based on input stream sort description
 2022-08-17 12:58:11 +02:00
Nikita Taranov
63bc894a42 more parallelism 2022-08-16 18:56:22 +02:00
Alexander Tokmakov
6fd4d2cfb3 Revert "tests/performance: cover sparse_hashed dictionary (#40027)" 
This reverts commit 6a30c23252.
 2022-08-16 15:32:50 +03:00
avogar
c8571f82f9 Fix performance test 2022-08-15 11:41:03 +00:00
Kruglov Pavel
ac85676d84
Update arrow_format.xml 2022-08-15 00:10:08 +02:00
avogar
398576e9c9 Improve and fix dictionaries in Arrow format 2022-08-12 18:56:21 +00:00
Igor Nikonov
75f6fcfa70 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-08-11 12:35:55 +00:00
Azat Khuzhin
6a30c23252
tests/performance: cover sparse_hashed dictionary (#40027) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-10 21:48:00 +02:00
BoloniniD
b161773f71 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-08-02 20:25:25 +03:00
Igor Nikonov
7f0adb5eb0 Merge remote-tracking branch 'origin/master' into skipping_sorting_step 2022-07-31 07:07:36 +00:00
Alexey Milovidov
c9e6850306
Merge pull request #39325 from azat/perf-parallel_mv-fix 
tests/performance: improve parallel_mv test
 2022-07-31 02:51:38 +03:00
Alexey Milovidov
36e6500e54
Merge branch 'master' into BLAKE3 2022-07-30 23:14:05 +03:00
Anton Popov
1547c010b9
Merge pull request #39432 from ClickHouse/distinct_sorted_chunk_perf_impr 
DISTINCT in order: perf improvement
 2022-07-27 14:17:58 +02:00
Alexander Gololobov
460950ecdc
Merge branch 'master' into feature/sql-standard-delete 2022-07-24 21:27:22 +02:00
Alexander Gololobov
594195451e Cleanups 2022-07-24 12:21:18 +02:00
Igor Nikonov
739ff34c6e Add some tests, still not sure about optimize_memory_usage option 2022-07-22 22:48:26 +00:00
Igor Nikonov
7db5d54820 Adopt to the case when not all columns in distinct are part of sorting 
description
 2022-07-21 21:04:58 +00:00
Igor Nikonov
122a1123b2 - disable the worst case for distinct in order in perf test for now 
+ functional test for query with the worst perfomance
+ debug logging in DistinctStep
 2022-07-21 15:03:19 +00:00
Igor Nikonov
ac116324b2 rename and fix perf test 2022-07-19 21:21:39 +00:00
Igor Nikonov
c74600d282
Merge branch 'master' into skipping_sorting_step 2022-07-19 18:59:36 +02:00
Igor Nikonov
1fe83cc8d8 optimize_sorting_for_input_stream setting and perf tests 2022-07-19 16:58:15 +00:00
Alexander Gololobov
f31788ed2a Perf test for read after deleting many rows 2022-07-18 20:08:09 +02:00
Azat Khuzhin
cf1a5baa23 tests/performance: improve parallel_mv test 
Right now it is possible for parallel_mv to fail [1] due to exceeding 15
seconds limit.

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/39183/ad6b50b087086fef8aa6f0f72b3a42f014266763/performance_comparison_aarch64_[4/4]/report.html

Let's try to really disable MERGES and see.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-18 16:23:17 +03:00
Igor Nikonov
828f3711d2 Update perf tests (to test perf test) 2022-07-17 20:40:58 +00:00
Igor Nikonov
6f224b026a Perf test. Code polishing 2022-07-15 21:54:57 +00:00
Kruglov Pavel
3436fcfda6
Update tests/performance/low_cardinality_argument.xml 
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
 2022-07-15 18:24:44 +02:00
Kruglov Pavel
9c443038c7
Update low_cardinality_argument.xml 2022-07-14 18:28:25 +02:00
Kruglov Pavel
1f7fe10313
Update low_cardinality_argument.xml 2022-07-14 12:54:14 +02:00
avogar
390b1ac2f7 Improve isNullable/isConstant/isNull/isNotNull performance for LowCardinality argument 2022-07-13 17:56:34 +00:00
Igor Nikonov
16d2319a8d SortingStep: type of sorting is deduced based on input stream sorting description in during transformation 
+ perf test
 2022-07-11 20:59:38 +00:00
Igor Nikonov
1f46f48d7d Fix: remove heeavy performance tests, introduced within this PR 2022-07-07 07:57:05 +00:00
Igor Nikonov
a20a15ff30 Tests 
+ check that EXPLAIN SYNTAX return the same result for ordinary ORDER BY and ORDER BY tuple
+ performance
 2022-07-06 22:27:53 +00:00
Alexander Gololobov
612e836e60
Merge pull request #38740 from ClickHouse/array_norm_vectorize 
Improved vectorized execution of main loop for array norm/distance
 2022-07-04 10:19:57 +02:00
Alexey Milovidov
c711012399
Merge pull request #38731 from azat/views-max_insert_threads 
Fix number of threads for pushing to views
 2022-07-04 07:43:26 +03:00
Azat Khuzhin
4ae7db8369 Fix max_insert_threads while pushing to views 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-03 15:14:05 +03:00
Alexander Gololobov
ca2829188d Perf test for norm/distance with long arrays of floats 2022-07-03 08:01:49 +02:00
mergify[bot]
12f5250e86
Merge branch 'master' into dictinct_in_order_optimization 2022-07-01 22:51:35 +00:00
Igor Nikonov
9ef8ff5a31 Addressing review comments 2022-07-01 22:50:00 +00:00
Igor Nikonov
488ee75fc4 + use DistinctSorted for final distinct step 
+ fix performance tests
 2022-06-30 13:03:39 +00:00
Anton Popov
7c721578c7
Merge pull request #38320 from CurtizJ/dynamic-columns-16 
Improve performace of insertion to columns of type JSON
 2022-06-30 14:18:20 +02:00
Igor Nikonov
d435532c68 Adapt range search algorithm to high cardinality case 
+ range search done in steps of some number of rows.
  Controled by new
  setting `distinct_in_order_range_search_step`. By default 0, i.e.
  whole chunk
+ before start binary search, linear probing is done on each step (32
  rows currently)
 2022-06-29 23:30:35 +00:00
mergify[bot]
36139eacd7
Merge branch 'master' into dictinct_in_order_optimization 2022-06-29 13:37:16 +00:00
Igor Nikonov
3627c6ff36 Perf tests with high cardinality 2022-06-29 13:13:39 +00:00
Alexander Tokmakov
ceb66ade4b
Merge pull request #38335 from ClickHouse/deprecate_ordinary_database 
Deprecate Ordinary database and old *MergeTree syntax
 2022-06-29 13:42:59 +03:00
Nikita Taranov
f5d26572df
Quick fix for aggregation pipeline (#38295) 2022-06-29 01:16:30 +02:00
Anton Popov
58c8facebb minor fixes 2022-06-28 14:21:21 +00:00
BoloniniD
6ddcec0906
Merge branch 'master' into BLAKE3 2022-06-28 16:53:06 +03:00
Alexander Tokmakov
31dcc7634e Merge branch 'master' into deprecate_ordinary_database 2022-06-24 18:16:07 +02:00
Alexander Tokmakov
0d304f7b8c fix tests 2022-06-23 21:19:07 +02:00
mergify[bot]
234f0c6399
Merge branch 'master' into revert-35914-FIPS_compliance 2022-06-23 12:06:17 +00:00
Anton Popov
3e62d0fb8c fix test 2022-06-23 11:31:39 +00:00
Alexander Tokmakov
f00e6b5a7a deprecate old MergeTree syntax 2022-06-23 11:24:54 +02:00
Sergey Skvortsov
202a2fd709 feat: Add grace hash join perf tests 2022-06-23 08:44:26 +03:00
Anton Popov
52db1b35a1 improve performace of insertion to columns of type JSON 2022-06-22 17:45:51 +00:00
Nikita Taranov
41ba0118b5
Bring back #36396 (#38110) 
* Revert "Revert "More parallel execution for queries with `FINAL` (#36396)""

This reverts commit 5bfb15262c.

* fix tests

* fix review suggestions

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-06-22 15:05:07 +02:00
Alexey Milovidov
5855668514 Remove trash 2022-06-22 06:23:35 +02:00
Alexey Milovidov
0cf88e0950
Revert "ClickHouse's boringssl module updated to the official version of the FIPS compliant." 2022-06-18 23:16:18 +03:00
Antonio Andelic
f72e509b3b
Merge pull request #38052 from amosbird/join_regression_fix 
Fix significant join performance regression
 2022-06-17 19:55:33 +02:00
Robert Schulze
a0d936cc9f
Small follow-up for FPC codec 
- add paper reference + doxygen

- remove endianness handling (ClickHouse assumes little endian)

- documentation

- other minor stuff
 2022-06-15 14:21:28 +02:00
mergify[bot]
2cb9579234
Merge branch 'master' into join_regression_fix 2022-06-15 11:53:42 +00:00
Nikita Taranov
c8afeafe0e
More parallel execution for queries with FINAL (#36396) 2022-06-15 12:44:20 +02:00
Robert Schulze
9794098ebb
Merge pull request #37553 from koloshmet/fpc_codec 
FPC Codec for floating point data
 2022-06-15 12:03:41 +02:00
Maksim Kita
dc2e117cce UnaryLogicalFunctions improve performance using dynamic dispatch 2022-06-14 17:30:11 +02:00
Amos Bird
9a6e6ccfaf
Fix significant join performance regression 2022-06-14 21:14:18 +08:00
Maksim Kita
daa128f378 Fixed performance tests 2022-06-13 13:31:02 +02:00
Maksim Kita
1247ba1b01 Hierarchical dictionaries performance test fix 2022-06-13 12:31:39 +02:00
Mikhail Guzov
092a00d95a
Merge branch 'ClickHouse:master' into fpc_codec 2022-06-11 21:24:06 +03:00
Maksim Kita
3a0e7b662c
Merge pull request #37954 from kitaisreal/normalize-utf8-performance-tests-fix 
Normalize UTF8 performance test fix
 2022-06-11 15:23:06 +02:00
mergify[bot]
a44590ea84
Merge branch 'master' into normalize-utf8-performance-tests-fix 2022-06-09 14:33:29 +00:00
Maksim Kita
5009374036 Normalize UTF8 performance test fix 2022-06-09 15:35:53 +02:00
BoloniniD
b05ee41d25 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-06-06 16:03:10 +03:00
Nikita Taranov
0a9d8398d8 impl 2022-06-04 19:14:38 +00:00
Robert Schulze
b3b0716b32
Merge pull request #37544 from ClickHouse/cached_patterns 
Cache compiled regexps when evaluating non-const needles
 2022-06-01 19:55:25 +02:00
Robert Schulze
81318e07d6
Try to fix performance test results 2022-06-01 11:53:37 +02:00
BoloniniD
dd8aefdf1e Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-06-01 11:46:55 +03:00
Anton Popov
20e319d67a
Merge pull request #37666 from CurtizJ/optimize-coalesce 
Optimize function `COALESCE` with two arguments
 2022-05-31 23:48:13 +02:00
Anton Popov
30f8eb800a optimize function coalesce with two arguments 2022-05-30 22:29:35 +00:00
Robert Schulze
ad12adc31c
Measure and rework internal re2 caching 
This commit is based on local benchmarks of ClickHouse's re2 caching.

Question 1: -----------------------------------------------------------
Is pattern caching useful for queries with const LIKE/REGEX
patterns? E.g. SELECT LIKE(col_haystack, '%HelloWorld') FROM T;

The short answer is: no. Runtime is (unsurprisingly) dominated by
pattern evaluation + other stuff going on in queries, but definitely not
pattern compilation. For space reasons, I omit details of the local
experiments.

(Side note: the current caching scheme is unbounded in size which poses
a DoS risk (think of multi-tenancy). This risk is more pronounced when
unbounded caching is used with non-const patterns ..., see next
question)

Question 2: -----------------------------------------------------------
Is pattern caching useful for queries with non-const LIKE/REGEX
patterns? E.g. SELECT LIKE(col_haystack, col_needle) FROM T;

I benchmarked five caching strategies:
1. no caching as a baseline (= recompile for each row)
2. unbounded cache (= threadsafe global hash-map)
3. LRU cache (= threadsafe global hash-map + LRU queue)
4. lightweight local cache 1 (= not threadsafe local hashmap with
   collision list which grows to a certain size (here: 10 elements) and
   afterwards never changes)
5. lightweight local cache 2 (not threadsafe local hashmap without
   collision list in which a collision replaces the stored element, idea
   by Alexey)

... using a haystack of 2 mio strings and
A). 2 mio distinct simple patterns
B). 10 simple patterns
C)  2 mio distinct complex patterns
D)  10 complex patterns

Fo A) and C), caching does not help but these queries still allow to
judge the static overhead of caching on query runtimes.

B) and D) are extreme but common cases in practice. They include
queries like "SELECT ... WHERE LIKE (col_haystack, flag ? '%pattern1%' :
'%pattern2%'). Caching should help significantly.

Because LIKE patterns are internally translated to re2 expressions, I
show only measurements for MATCH queries.

Results in sec, averaged over on multiple measurements;

1.A): 2.12
  B): 1.68
  C): 9.75
  D): 9.45

2.A): 2.17
  B): 1.73
  C): 9.78
  D): 9.47

3.A): 9.8
  B): 0.63
  C): 31.8
  D): 0.98

4.A): 2.14
  B): 0.29
  C): 9.82
  D): 0.41

5.A) 2.12 / 2.15 / 2.26
  B) 1.51 / 0.43 / 0.30
  C) 9.97 / 9.88 / 10.13
  D) 5.70 / 0.42 / 0.43
(10/100/1000 buckets, resp. 10/1/0.1% collision rate)

Evaluation:

1. This is the baseline. It was surprised that complex patterns (C, D)
   slow down the queries so badly compared to simple patterns (A, B).
   The runtime includes evaluation costs, but as caching only helps with
   compilation, and looking at 4.D and 5.D, compilation makes up over 90%
   of the runtime!

2. No speedup compared to 1, probably due to locking overhead. The cache
   is unbounded, and in experiments with data sets > 2 mio rows, 2. is
   the only scheme to throw OOM exceptions which is not acceptable.

3. Unique patterns (A and C) lead to thrashing of the LRU cache and very
   bad runtimes due to LRU queue maintenance and locking. Works pretty
   well however with few distinct patterns (B and D).

4. This scheme is tailored to queries B and D where it performs pretty
   good. More importantly, the caching is lightweight enough to not
   deteriorate performance on datasets A and C.

5. After some tuning of the hash map size, 100 buckets seem optimal to
   be in the same ballpark with 10 distinct patterns as 4. Performance
   also does not deteriorate on A and C compared to the baseline.
   Unlike 4., this scheme behaves LRU-like and can adjust to changing
   pattern distributions.

As a conclusion, this commit implementes two things:

1. Based on Q1, pattern search with const needle no longer uses
   caching. This applies to LIKE and MATCH + a few (exotic) other SQL
   functions. The code for the unbounded caching was removed.

2. Based on Q2, pattern search with non-const needles now use method 5.
 2022-05-30 20:00:35 +02:00
Alexey Milovidov
9e3242f186
Merge pull request #37617 from CurtizJ/aggregation-sparse-columns 
Better performance with sparse columns in aggregate functions
 2022-05-29 09:36:07 +03:00
Anton Popov
c39d95e2e6 add perf test 2022-05-28 12:56:38 +00:00
Alexey Milovidov
86afa3a245
Merge pull request #37502 from ClickHouse/array_norm_dist_fixes 
Renamed arrayXXNorm/arrayXXDistance functions to XXNorm/XXDistance and fixed some overflow cases
 2022-05-27 00:56:29 +03:00
koloshmet
7e69779575 added fpc codec to float perftest 2022-05-26 22:32:56 +03:00
Maksim Kita
3a92e61827
Merge pull request #37148 from kitaisreal/dictionary-get-descendants-performance-improvement 
Dictionary getDescendants performance improvement
 2022-05-26 12:29:17 +02:00
Maksim Kita
bee3c30f66
Merge pull request #37524 from kitaisreal/geo-distance-functions-improve-performance 
Geo distance functions improve performance
 2022-05-25 22:40:40 +02:00
Alexander Gololobov
168b47d0ad Use same norm and distance function names for tuples and arrays 2022-05-25 22:39:59 +02:00
Maksim Kita
45da28ecae Improve performance of geo distance functions 2022-05-25 14:22:22 +02:00
Maksim Kita
3c0c322d7c
Merge pull request #37480 from kitaisreal/dynamic-dispatch-infrastructure-improvements 
Dynamic dispatch infrastructure style fixes
 2022-05-24 18:13:53 +02:00
Maksim Kita
e6e4b2826d Dynamic dispatch infrastructure style fixes 2022-05-24 14:25:29 +02:00
Kruglov Pavel
6c9a524f6b
Merge pull request #37192 from Avogar/formats-with-names 
Improve performance and memory usage for select of subset of columns for some formats
 2022-05-24 13:28:14 +02:00
avogar
3651ef93fe Fix performance test 2022-05-23 17:42:13 +00:00
Alexander Gololobov
d0f5551c9f Parameterized with norm kind 2022-05-23 18:27:41 +02:00
Alexander Gololobov
2658a9eeeb Test with max_threads=1 2022-05-23 18:06:07 +02:00
Maksim Kita
94772f9cfc Added performance tests 2022-05-23 14:43:13 +02:00
Alexander Gololobov
70cc27ecac Test with different element types 2022-05-23 14:08:15 +02:00
Alexander Gololobov
7897a5bac7 Perf test for Norm and Distance fuctions for arrays and tuples 2022-05-23 10:18:24 +02:00
avogar
566d1b15fd Merge branch 'master' of github.com:ClickHouse/ClickHouse into formats-with-names 2022-05-20 13:54:52 +00:00
Maksim Kita
beb34e7062 Improve performance of unary arithmetic functions 2022-05-17 13:53:20 +02:00
avogar
cef13c2c02 Allow to skip unknown columns in Native format 2022-05-13 14:27:15 +00:00
avogar
b17fec659a Improve performance and memory usage for select of subset of columns for some formats 2022-05-13 13:51:28 +00:00
Maksim Kita
437d70d4da Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
837f2e8b9c Update performance test 2022-05-11 21:59:51 +02:00
Maksim Kita
ea8ce3140a Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
d85d72e5ad Added performance tests 2022-05-11 21:59:51 +02:00
Alexey Milovidov
5a750d3305 Merge branch 'master' into revert-group-array-sorted 2022-05-05 00:51:01 +02:00
Alexey Milovidov
c95591f294
Merge pull request #36841 from ClickHouse/fix-performance-test-5 
Remove "preconditions" from performance tests (overengineering, unneeded feature)
 2022-05-02 12:22:56 +03:00
Alexey Milovidov
7cc64d40a6 Remove tags as well 2022-05-02 02:35:44 +02:00
Alexey Milovidov
88826e2da5 Remove "preconditions" from performance tests (overengineering, unneeded feature) 2022-05-02 02:33:22 +02:00
Alexey Milovidov
2691261aa2 Fix performance test 2022-05-02 02:21:19 +02:00
Alexey Milovidov
d6c0de0d40 Revert "Merge pull request #34055 from palegre-tiny/groupSortedArray" 
This reverts commit f055d7b692, reversing
changes made to 4ec3c35e14.
 2022-04-30 12:29:23 +02:00
Alexey Milovidov
03dc4f6b83
Merge pull request #36779 from ClickHouse/fix-performance-test-2 
Fix performance test
 2022-04-30 08:56:44 +03:00
Alexey Milovidov
6c75b63953
Merge pull request #35914 from DevTeamBK/FIPS_compliance 
ClickHouse's boringssl module updated to the official version of the FIPS compliant.
 2022-04-29 21:08:51 +03:00
Alexey Milovidov
76d660d6df Fix performance test 2022-04-29 10:27:44 +02:00
Alexey Milovidov
df45c9503d Trim down some tests 2022-04-29 04:33:12 +02:00
Alexey Milovidov
31215f874c Fix slow performance test 2022-04-29 04:29:06 +02:00
Alexey Milovidov
bf100e0a4c Remove "partial merge join" performance test, because we are not interested in the results 2022-04-29 04:13:30 +02:00
Meena Renganathan
bdaf5391cf Merge branch 'master' of https://github.com/DevTeamBK/ClickHouse into FIPS_compliance 2022-04-28 06:15:46 -07:00
Meena Renganathan
98543a9a3f Removed the tests aes-192-cfb128 and aes-256-cfb128 since the latest boringssl modules doesn't support 2022-04-27 07:48:43 -07:00
Alexander Gololobov
3c000b098a
Merge pull request #36638 from nickitat/fix_sorting_step 
Fix SortingStep::updateOutputStream()
 2022-04-26 15:49:49 +02:00
mergify[bot]
d2ac9b2223
Merge branch 'master' into to_start_of_five_minutes 2022-04-25 17:36:38 +00:00
Nikita Taranov
5dc9478bac fix SortingStep::updateOutputStream() 2022-04-25 17:29:14 +00:00
Mikhail f. Shiryaev
f53040b95e
Merge pull request #36559 from ClickHouse/performance-rebalance 
Use just index to split performance tests by group
 2022-04-25 14:15:23 +02:00
Mikhail f. Shiryaev
f3aaff773a
Disable broken performance test 2022-04-25 11:26:05 +02:00
mergify[bot]
705d5af3a0
Merge branch 'master' into to_start_of_five_minutes 2022-04-24 22:24:24 +00:00
Mikhail f. Shiryaev
2aaaf41ee4
Speed-up file_table_function 20 times 2022-04-23 14:33:03 +02:00
Memo
39aadf0975 replaced toStartOfFiveMinute to toStartOfFiveMinutes 2022-04-22 10:49:59 +08:00
BoloniniD
30712ead45
Merge branch 'master' into BLAKE3 2022-04-13 00:45:44 +03:00
Maksim Kita
04089be144 Fix hash_table_sizes_stats performance test 2022-04-12 17:46:16 +02:00
Maksim Kita
3d36698f56 Fix group_by_sundy_li performance test 2022-04-12 17:06:38 +02:00
Maksim Kita
9b332c1e31 Fix early_constant_folding performance test 2022-04-12 17:06:21 +02:00
Maksim Kita
8cf67ed4c0 Fix constant_column_search performance tests 2022-04-12 15:22:14 +02:00
Maksim Kita
7803ecaee5 Fix performance tests 2022-04-12 15:22:14 +02:00
Kruglov Pavel
73adbb4c15
Merge pull request #35986 from amosbird/better-scalar1 
Fix performance regression of scalar query
 2022-04-07 14:07:59 +02:00
Amos Bird
df06f9f974
Fix performance regression of scalar query 2022-04-06 17:50:22 +08:00
Nikolai Kochetov
4479b68980
Merge pull request #35623 from nickitat/function_calculation_after_sorting_and_limit 
Functions calculation after sorting
 2022-04-05 12:09:56 +02:00
Maksim Kita
b160ffd726
Merge pull request #35723 from ClickHouse/array-has-all-sse-avx2-optimizations 
Merging #27653
 2022-04-05 11:09:14 +02:00
Nickita Taranov
4c51329ad6 stash 2022-04-04 14:33:57 +02:00
Maksim Kita
3c472a7897 Simplified hasAll performance tests 2022-04-04 13:34:40 +02:00
Nikolay Degterinsky
f055d7b692
Merge pull request #34055 from palegre-tiny/groupSortedArray 
Add groupSortedArray() function
 2022-03-31 01:20:15 +03:00
Nikita Taranov
30f2a942c5
Predict size of hash table for GROUP BY (#33439) 
* use AggregationMethod ctor with reserve

* add new settings

* add HashTablesStatistics

* support queries with limit

* support distributed and with external aggregation

* add new profile events

* add some tests

* add perf test

* export cache stats through AsynchronousMetrics

* rm redundant trace

* fix style

* fix 02122_parallel_formatting test

* review fixes

* fix 02122_parallel_formatting test

* apply also to two-level HTs

* try simpler strategy

* increase max_size_to_preallocate_for_aggregation for experiment

* fixes

* Revert "increase max_size_to_preallocate_for_aggregation for experiment"

This reverts commit 6cf6f75704.

* fix test

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-03-30 22:47:51 +02:00
Maksim Kita
2742b88e6c
Merge pull request #27653 from ContentSquare/hasAllAny_SIMD 
Implement HasAll specialization for SSE and AVX2
 2022-03-29 16:35:59 +02:00
BoloniniD
2d2420c8a9 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-03-29 11:00:50 +03:00
mergify[bot]
bf5a9dcb7a
Merge branch 'master' into groupSortedArray 2022-03-28 20:57:07 +00:00
Alexey Milovidov
d54138425f Rename yandexConsistentHash to kostikConsistentHash 2022-03-24 02:18:25 +01:00
mergify[bot]
5df84df596
Merge branch 'master' into groupSortedArray 2022-03-21 13:35:06 +00:00
Raúl Marín
90fd425117 Add ASOF performance test with random data 2022-03-17 19:48:25 +01:00
Raúl Marín
e2cec4e65b Merge remote-tracking branch 'blessed/master' into asof_ftw 2022-03-17 16:33:29 +01:00
Maksim Kita
b202130841 Fixed performance tests 2022-03-15 15:43:39 +00:00
Maksim Kita
08bb39d869 Fixed performance tests 2022-03-15 15:43:39 +00:00
Maksim Kita
d49df02074 Fixed performance tests 2022-03-15 15:43:39 +00:00
Maksim Kita
98df85d2b7 Performance tests fix 2022-03-15 15:43:39 +00:00
BoloniniD
253afd032f Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-03-13 21:18:46 +03:00
Maksim Kita
0dd807d19d
Merge pull request #34750 from kitaisreal/merge-tree-improve-insert-performance 
MergeTree improve insert performance
 2022-03-13 13:39:18 +01:00
Alexey Milovidov
0995c63ea1 Adjust timezone in performance tests 2022-03-11 23:49:13 +01:00
Maksim Kita
f1d2f2a9e1 Updated tests 2022-03-11 21:16:25 +00:00
Maksim Kita
d12618cd2e Updated performance tests 2022-03-10 21:45:31 +00:00
Maksim Kita
765cd09d06 MergeTree improve insert performance 2022-03-10 21:45:31 +00:00
BoloniniD
8474babe88 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-03-02 18:27:53 +03:00
mergify[bot]
93b13c0232
Merge branch 'master' into asof_ftw 2022-03-02 13:20:50 +00:00
mergify[bot]
cd6f1d8fa4
Merge branch 'master' into groupSortedArray 2022-02-25 11:45:48 +00:00
Raúl Marín
e0c6014ecd Mention scipy dep in performance bench README 2022-02-20 02:43:28 +01:00
BoloniniD
3f149e88be Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-02-19 20:23:19 +03:00
Raúl Marín
2627c8d437 Add a performance test using ASOF 2022-02-18 17:37:24 +01:00
Maksim Kita
80b0efb367 Performance tests fix H3 2022-02-18 15:57:54 +00:00
Pablo Alegre
9466aafb3c fixup! Add groupSortedArray() function 2022-02-15 14:48:20 +01:00
Anton Popov
5c316ffabe support filtering by sparse columns without convertion to full 2022-02-15 14:30:54 +03:00
mergify[bot]
aab54f4c83
Merge branch 'master' into groupSortedArray 2022-02-14 12:47:48 +00:00
BoloniniD
c8bb6fb38a Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-02-13 12:47:18 +03:00
Maksim Kita
e2c8ba9ab2 Added performance test 2022-02-12 16:05:35 +00:00
Pablo Alegre
1e4b504ae2 fixup! Add groupSortedArray() function 2022-02-10 16:49:28 +01:00
avogar
bfa96463ca Fix possible error 'file_size: Operation not supported' 2022-02-10 09:23:27 +03:00
Maksim Kita
613c9fa3c2
Merge pull request #34339 from kitaisreal/map-populate-series-refactoring 
Function mapPopulateSeries added additional performance test
 2022-02-06 01:19:32 +01:00
Maksim Kita
35235d2d7f Added additional performance test 2022-02-05 16:11:36 +00:00
Maksim Kita
eff16baaf3
Merge pull request #34318 from kitaisreal/map-populate-series-refactoring 
Function mapPopulateSeries refactoring
 2022-02-05 12:51:02 +01:00
Maksim Kita
6e789f98ea Added performance tests 2022-02-04 14:58:55 +00:00
BoloniniD
31e7093b5b Updated Extras repo for BLAKE3 2022-02-03 22:04:28 +03:00
Danila Kutenin
c90b1f7794 Optimize quantilesExact{Low,High} to use nth_element instead of sort 2022-02-03 12:24:33 +00:00
BoloniniD
9b2292ec6c Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-02-02 12:09:01 +03:00
avogar
6229ec530d Fix some perf tests 2022-01-31 21:07:20 +03:00
youenn lebras
c0864e6cd9
Update branch - Merge master 2022-01-31 10:22:25 +01:00
BoloniniD
b04ed43ebd Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-01-30 22:58:04 +03:00
Maksim Kita
f3453024ff
Merge pull request #34060 from amosbird/optimizetupleorderby 
Make ORDER BY tuple almost as fast as ORDER BY columns
 2022-01-29 15:58:09 +01:00
BoloniniD
44591b79ff Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-01-27 19:49:58 +03:00
Amos Bird
faee95b897
Make ORDER BY tuple almost as fast as ORDER BY columns 
We have special optimizations for multiple column ORDER BY: https://github.com/ClickHouse/ClickHouse/pull/10831 . It's beneficial to also apply to tuple columns.

Before:

select * from numbers(300000000) order by (1 - number , number + 1 , number) limit 10;
2.613 sec.

After:

select * from numbers(300000000) order by (1 - number , number + 1 , number) limit 10;
0.755 sec

No tuple:

select * from numbers(300000000) order by 1 - number , number + 1 , number limit 10;
0.755 sec
 2022-01-27 21:42:08 +08:00
Pablo Alegre
e813f6413f Add groupSortedArray() function 2022-01-27 12:22:59 +01:00
BoloniniD
6f078c164d Merge branch 'master' of github.com:ClickHouse/ClickHouse into BLAKE3 2022-01-26 20:11:27 +03:00
Nikolay Degterinsky
5edbae6734 Merge branch 'master' into classification 2022-01-25 10:22:47 +00:00
Maksim Kita
1f72bde52f Fix performance tests 2022-01-19 10:41:58 +00:00
Maksim Kita
6536c3aaee Added performance tests 2022-01-18 13:20:54 +00:00
BoloniniD
b486f9462b Fix the memory leak 2022-01-12 18:28:30 +03:00
BoloniniD
450f0a53b9 Add BLAKE3 to performance tests 2022-01-10 23:12:33 +03:00
Nikolay Degterinsky
2bdee0b0b8 Merge branch 'master' into classification 2021-12-30 02:17:47 +00:00
Nikolay Degterinsky
c5ccefb9df Add tests 2021-12-27 13:43:19 +00:00
Raúl Marín
d9bcb633de Change perf set_index test to check build index 2021-12-23 13:53:36 +01:00
Raúl Marín
eddccb0065 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-21 10:00:17 +01:00
alesapin
6bd7e425c6
Merge pull request #22535 from CurtizJ/sparse-serialization 
Sparse serialization and ColumnSparse
 2021-12-17 15:26:17 +03:00
mergify[bot]
bff0424139
Merge branch 'master' into kill_scalar_github 2021-12-15 09:22:58 +00:00
Anton Popov
16312e7e4a Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-14 18:58:17 +03:00
youenn lebras
c2b761acf2
Add cmake option to enable or not AVX2 instructions 
This reverts commit bca8eca44fe382b6efe80a381d42e6ede8a91fa3.
 2021-12-14 09:45:03 +01:00
youenn lebras
0154eab9cb
Modify performance tests for HasAll, removing Large tests to see if it helps passing CICD 2021-12-13 17:57:33 +01:00
Youenn Lebras
a71944d11d
Add performance tests for HasAll for int{64,16,8} 2021-12-13 17:57:30 +01:00
Youenn Lebras
ade754d444
Fix a bug for avx2 and add performance tests for HasAll<int32> 2021-12-13 17:57:29 +01:00
alesapin
e2a2d1cdc3 Merge branch 'master' into add_performance_tests 2021-12-13 12:01:31 +03:00
Alexey Milovidov
d677a5ab67 Fix perf test 2021-12-12 04:31:56 +03:00
Raúl Marín
f38ddb4f53 Mark affected tests as short 2021-12-10 10:33:08 +01:00