Commit Graph

105675 Commits

Author SHA1 Message Date
avogar
6cb7c4d175 Better commit, mark noexcept 2023-01-13 16:33:11 +00:00
avogar
76c89c6d20 Fix heap-use-after-free in reading from s3 2023-01-13 16:31:30 +00:00
Smita Kulkarni
d132d30707 Addressed review comments - 42648 Support scalar subqueries cache 2023-01-13 17:28:35 +01:00
Alexander Tokmakov
50bb1db9cc
Merge pull request #45251 from ClickHouse/tavplubix-patch-1
Update clickhouse-test
2023-01-13 18:39:00 +03:00
Alexander Tokmakov
6de4837580 fix 2023-01-13 16:07:20 +01:00
Maksim Kita
dc24d831cf
Merge pull request #42970 from ClickHouse/optimize-redundant-function
Implement optimize_redundant_functions_in_order_by on top of QueryTree.
2023-01-13 17:36:56 +03:00
Maksim Kita
05b1b78104
Merge pull request #44013 from kitaisreal/analyzer-aggregate-functions-passes-small-fixes
Analyzer aggregate functions passes small fixes
2023-01-13 17:31:53 +03:00
avogar
abfb6b096f Better exception message 2023-01-13 14:23:30 +00:00
Smita Kulkarni
a0fe26f506 Addressed review comments and updated name to ServerStartupMilliseconds - Record server startup time in ProfileEvents 2023-01-13 14:38:54 +01:00
Alexander Tokmakov
9d5ec474a3
Merge pull request #43998 from evillique/make_system_replicas_parallel
Make `system.replicas` parallel
2023-01-13 16:33:36 +03:00
Alexander Tokmakov
36c282e48e
Update clickhouse-test 2023-01-13 16:29:08 +03:00
Alexander Tokmakov
b88aae9d5c Merge branch 'master' into fix_44496 2023-01-13 14:05:57 +01:00
Smita Kulkarni
cf5cb0da97 Record server startup time in ProfileEvents
Implementation:
* Added ProfileEvents::ServerStartupTime.
* Recorded time from start of main till listening to sockets.
Testing:
* Added a test 02532_profileevents_server_startup_time.sql
2023-01-13 13:47:54 +01:00
Azat Khuzhin
64e3677961 Avoid double hash calculation in HashedDictionary::getShard(StringRef)
Previously it was written this way because getShard() was a simple
module operation.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
2783850f08 Minor review fixes in HashedDictionary
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
6e0a7add93 Completelly exception safe HashedDictionary dtor
Previously there was one (even though very unlikely) case when the dtor
can throw - logging code or ThreadPool::wait.

Just guard the dtor with try/catch and done with it.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
74def83c5d Destroy hashtables for hashed dictionary in parallel only for sharded dict
Since there can be multiple hashtables, since each attribute uses it's
own hashtable.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
1c0e0ea1e4 Disable sharded dictionaries with updatable sources
Support of sharded dictionary for updatable sources is questionable
since:
- sharded dictionary developed for hashed dictionary with a huge number
  of keys
- updatable source requires storing the whole table in memory (due to
  how reload works)
- also it is an open question will it have some benefits from the
  updatable source or not, since using updatable source with a huge
  number of changes in the source does not looks optimal and on the
  other side if there are small amount of changes the you don't need
  sharded dictionary at all

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
0506792790 tests: cover sharded hashed dictionary with update_field
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
c97991fce1 Use shared arena for HashedDictionary::blockToAttributes()
This should decrease number of allocations.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
01b100da61 Use shared arena in ParallelDictionaryLoader::createShardSelector() (and add missing rollback)
This should decrease number of allocations.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
64874824b4 Minor review fixes in HashedDictionary
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
77c1f07636 Make HashedDictionary::~HashedDictionary exception safe
Before it was possible for the desturctor to throw, in case of thread
allocation fails, rewrite it to trySchedule() and do sequential destroy
in this case.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
925fd2c33a tests/performance: do not use scientific notation in hashed_dictionary_sharded
v2: fix few mistakes
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
a3f189e191 Optimize sharded dictionaries with skewed distribution
In case of skewed distribution simple division by module will not give
you good distribution between shards and eventually this can lead to
performance the same as non-sharded dictionary (except for it will
occupy +1 thread for Block::scatter).

But if HashedDictionary::blockToAttributes() will not have calls to
HashedDictionary::getShard() this can be fixed by using a more complex
key-to-shard (getShard()) mapping. And actually you do not need to call
getShard() in blockToAttributes() you can simply use passed shard, and
that's it.

And by wrapping key with intHash64() in getShard() skewed distribution
can be fixed.

Note, that previously I tried similar approach but did not removed
getShard() from blockToAttributes(), that's why it failed.

And now it works almost as fast as with simple createBlockSelector(),
just 13.6% slower (18.75min vs 16.5min, with 16 threads).

Note, that I've also tried to add libdivide for this, but it does not
improves the performance.

I've also tried the approach without scatter, and it works 20% slower
then this one (22.5min VS 18.75min, with 16 threads).

v2: Use intHashCRC32() over intHash64() for HashedDictionary::getShard()
    (with intHash64() it works very slower, almost 2x slower, there was
    18min with 32 threads)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
655a564280 Parallel hash tables destroy for hashed dictionaries
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
99063b152f Allow to configure queue backlog of the parallel hashed dictionary loader
v2: Decrease default parallel_queue_backlog to 10000 (same speed)
v3: Rename parallel_queue_backlog to per_shard_load_backlog
v3: Rename per_shard_load_backlog to shard_load_queue_backlog
v4: Fix documentation
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
79ad81dfdf Implement separate queue for parallel loader of hashed dictionaries
Previous patches in this series has a bottleneck in rehash(). This is
the most slowest operation when insert lots of rows into the hashtable
and eventually all that thread pool sometimes work as the most slowest
thread since we did not have any queue of blocks.

This patch adds such queue and now it scales linearly, so initialy with
1 thread I had ~4 hours for 10e9 elements (UInt64 key, UInt16 value),
after this patch it works in 16 minutes with 16 threads (well actually I
have to use 32 threads because of distribution of data in the source
table).

And now with 16 threads it works 16 times faster.

Also this patch adds more optimal block splitting for the non-complex
dictionaries, and usual block splitting for complex dictionaries.
But anyway this moves the overhead from the loading into the hashtable
threads out to the reader thread, and this is better, since reader does
not uses that much CPU.

v2: fix use-after-free on failed load (add missing wait in dtor)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
5d0fd3cdc4 Remove sharded overhead for non-sharded hashed dictionaries
By adding one more template parameter - HashedDictionary<sharded> (yes,
it is already too much of them, for the template class that has explicit
instantion).

Since perf tests [1] shows 20% slowdown.

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/40003/8f0cf2d6b8a7df511afe901331d5e2c7b06c0b4d/performance_comparison_[1/4]/report.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:26 +01:00
Azat Khuzhin
345c422e28 Add ability to load hashed dictionaries using multiple threads
Right now dictionaries (here I will talk about only
HASHED/SPARSE_HASHED/COMPLEX_KEY_HASHED/COMPLEX_KEY_SPARSE_HASHED)
can load data only in one thread, since it uses one hash table that
cannot be filled from multiple threads.

And in case you have very big dictionary (i.e. 10e9 elements), it can
take a awhile to load them, especially for SPARSE_HASHED variants (and
if you have such amount of elements there, you are likely use
SPARSE_HASHED, since it requires less memory), in my env it takes ~4
hours, which is enormous amount of time.

So this patch add support of shards for dictionaries, number of shards
determine how much hash tables will use this dictionary, also, and which
is more important, how much threads it can use to load the data.

And with 16 threads this works 2x faster, not perfect though, see the
follow up patches in this series.

v0: PARTITION BY
v1: SHARDS 1
v2: SHARDS(1)
v3: tried optimized mod - logical and, but it does not gain even 10%
v4: tried squashing more (max_block_size * shards), but it does not gain even 10% either
v5: move SHARDS into layout parameters (unknown simply ignored)
v6: tune params for perf tests (to avoid too long queries)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:25 +01:00
Azat Khuzhin
c43b1d0c41 tests: remove extra RELOAD from 01509_dictionary_preallocate
This will avoid extra messages on destroy:
- Destroying 1 non empty hash tables (using 1 threads)
- Hash tables destroyed

And actually we cannot wait them in tests, since the query does not
waits until the dictionary will be fully unloaded:

    $ pigz -cd clickhouse-server.log.gz.1 | grep 1511e339-a077-4ee7-808e-0211ece99409 -a
    2022.12.11 18:21:41.069825 [ 102234 ] {1511e339-a077-4ee7-808e-0211ece99409} <Debug> executeQuery: (from [::1]:58964) (comment: 01509_dictionary_preallocate.sh) SYSTEM RELOAD DICTIONARY dict_01509_preallocate (stage: Complete)
    ...
    2022.12.11 18:21:41.072887 [ 7291 ] {1511e339-a077-4ee7-808e-0211ece99409} <Trace> HashedDictionary: Preallocated 10000 elements
    ...
    2022.12.11 18:21:41.076531 [ 7291 ] {1511e339-a077-4ee7-808e-0211ece99409} <Trace> HashedDictionary: Destroying 1 non empty hash tables (using 1 threads)
    2022.12.11 18:21:41.076600 [ 102234 ] {1511e339-a077-4ee7-808e-0211ece99409} <Debug> MemoryTracker: Peak memory usage (for query): 3.05 MiB.
    2022.12.11 18:21:41.076618 [ 102234 ] {1511e339-a077-4ee7-808e-0211ece99409} <Debug> TCPHandler: Processed in 0.007111647 sec.
    2022.12.11 18:21:41.076697 [ 7291 ] {1511e339-a077-4ee7-808e-0211ece99409} <Trace> HashedDictionary: Hash tables destroyed

See, first the TCPHandler finished and only after HashedDictionary dtor.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-13 13:39:25 +01:00
Vladimir C
eefbffcc5b
Merge pull request #45230 from ClickHouse/vdimir/semi_join_null_const_bug 2023-01-13 13:22:57 +01:00
Anton Popov
71188c22ee fix race on 'relative_data_path' 2023-01-13 12:19:41 +00:00
vdimir
f881a82417
Fix viewExplain, add testcases 2023-01-13 12:19:25 +00:00
vdimir
bdb9222736
Support EXPLAIN SYNTAX oneline = 1 2023-01-13 12:18:58 +00:00
Alexander Tokmakov
51d94314d6
Merge pull request #45235 from ClickHouse/more_verbose_logs_about_replication_log_entries
More verbose logs about replication log entries
2023-01-13 15:05:21 +03:00
Alexander Tokmakov
db91c4a749
Merge pull request #45168 from ClickHouse/tavplubix-patch-1
Try to fix flaky test_ttl_move_memory_usage
2023-01-13 15:01:42 +03:00
Maksim Kita
44f4184e11
Merge pull request #44540 from kitaisreal/analyzer-support-distributed
Analyzer support distributed queries processing
2023-01-13 14:45:36 +03:00
Vitaly Baranov
00908dcc6c
Fix http requests without path for AWS. (#45238) 2023-01-13 12:35:39 +01:00
Nikolai Kochetov
6e9dd2af45
Merge pull request #42889 from guowangy/logical-optimizer-lowcardinality
Enable logical optimizer for LowCardinality regardless of short chain
2023-01-13 12:28:57 +01:00
vdimir
023162df1d
fix clang-tidy style 2023-01-13 11:25:07 +00:00
Mikhail f. Shiryaev
9dc9bb2ae6
Mention only approvals from the team members 2023-01-13 12:23:07 +01:00
kssenii
0110b63a95 Better fix 2023-01-13 12:12:27 +01:00
Robert Schulze
9779d034eb
Merge pull request #45144 from ClibMouse/crc-power-fix
Changes to support the CRC32 in PowerPC.
2023-01-13 11:24:18 +01:00
Maksim Kita
296dc5006d Fixed tests 2023-01-13 10:59:26 +01:00
simpleton
45842da72e
Merge branch 'master' into master 2023-01-13 17:42:36 +08:00
Alexander Gololobov
d850225f6b
Merge pull request #45229 from CurtizJ/fix-rare-logical-error
Fix rare logical error: `Too large alignment`
2023-01-13 09:48:28 +01:00
Antonio Andelic
99548c8c15 Merge branch 'master' into fix-crash-kv-store 2023-01-13 08:42:08 +00:00
Antonio Andelic
164b3b5f48 Fix test 2023-01-13 08:41:44 +00:00
taiyang-li
de5474c9f9 optimize match(a, '.*') 2023-01-13 14:55:54 +08:00