Commit Graph

117128 Commits

Author SHA1 Message Date
Nikolay Degterinsky
47480de25a Fix a crash in s3 and s3Cluster functions 2023-06-27 16:38:29 +00:00
Nikolay Degterinsky
ea11774939 Skip the analysis of headers argument with the new analyzer 2023-06-27 11:47:24 +00:00
Nikolay Degterinsky
ac85cdbb84
Fix build 2023-06-21 12:05:44 +02:00
Nikolay Degterinsky
e130a9fdb9
Fix build 2023-06-21 02:33:38 +02:00
Nikolay Degterinsky
575a1a4907 Add header checks to HTTP dictionary source 2023-06-20 13:29:25 +00:00
Nikolay Degterinsky
9a25958be8 Add HTTP header filtering 2023-06-15 13:49:49 +00:00
Kseniia Sumarokova
c15e7b93cb
Merge pull request #50976 from kssenii/fix-data-lakes-too-many-head-requests
Fix data lakes slowness because of synchronous head requests
2023-06-15 13:33:09 +02:00
Dan Roscigno
ee9faa9be1
Merge pull request #50968 from DerekChia/patch-3
Update delete.md
2023-06-15 07:06:49 -04:00
Alexander Tokmakov
6fc9479730
Update settings.md (#51007) 2023-06-15 13:51:42 +03:00
Igor Nikonov
1113a7c524
Merge pull request #50214 from azat/parallelize_output_from_storages-fix
Disable parallelize_output_from_storages for processing MATERIALIZED VIEWs and storages with one block only
2023-06-15 12:48:54 +02:00
Kseniia Sumarokova
b7fbc4dd8e
Merge pull request #50977 from valbok/empty-table-overrides
MaterializedMySQL: Keep parentheses for empty table overrides
2023-06-15 12:48:04 +02:00
Kseniia Sumarokova
58650b3280
Merge pull request #51002 from ClickHouse/Avogar-patch-1
Fix flaky test_seekable_formats
2023-06-15 12:46:04 +02:00
Kseniia Sumarokova
f1f8b302bf
Merge pull request #50934 from azat/tests/fix-query_log
Fix tests sanity checks and avoid dropping system.query_log table
2023-06-15 12:44:23 +02:00
Alexander Tokmakov
f73a3cf255
Merge pull request #50924 from valbok/missing-drops
MaterializedMySQL: Add missing DROP DATABASE for tests
2023-06-15 13:33:28 +03:00
Robert Schulze
946e1a8c6f
Merge pull request #50966 from arenadata/ADQM-934
clickhouse-client: Force that `--password` is specified at most once
2023-06-15 11:14:26 +02:00
Antonio Andelic
c1faf42481
Merge pull request #50967 from baibaichen/feature/fix_build_clang15
fix build issue on clang 15
2023-06-15 10:32:38 +02:00
Kseniia Sumarokova
c786fbf8bd
Add comment 2023-06-15 10:22:02 +02:00
Kseniia Sumarokova
c8619ee6e4
Merge pull request #50974 from kssenii/iceberg-metadata-fix
Fix iceberg V2 optional metadata parsing
2023-06-15 09:31:36 +02:00
Alexey Milovidov
a13db6a5ec
Merge pull request #50958 from den-crane/patch-51
Doc. Note about supported docker version
2023-06-15 01:58:18 +03:00
pufit
e2b9f59e1e
Merge pull request #50965 from ClickHouse/pufit/keeper-client-better-help-message
Fix keeper-client help message
2023-06-14 18:42:26 -04:00
Dan Roscigno
bd6744982b
Merge pull request #51009 from DanRoscigno/update-cl-ru
catch ru changelog up to 2023
2023-06-14 18:41:21 -04:00
Dan Roscigno
1dfbfd9018
Merge pull request #51010 from ClickHouse/DanRoscigno-patch-5
Update gcs.md
2023-06-14 18:40:50 -04:00
Alexey Gerasimchuck
75e6b4475d corrections after review 2023-06-14 22:35:38 +00:00
Alexey Gerasimchuck
a6420ee338
Merge branch 'master' into ADQM-934 2023-06-15 08:25:24 +10:00
Michael Kolupaev
badde0fde2 Print git hash when crashing 2023-06-14 15:22:44 -07:00
Dan Roscigno
e75cb90b7d
Update gcs.md
fix nav

closes https://github.com/ClickHouse/clickhouse-docs/issues/1230
2023-06-14 17:56:34 -04:00
DanRoscigno
005e7d3b27 Chinese also 2023-06-14 17:40:29 -04:00
DanRoscigno
40d4508a72 catch ru changelog up to 2023 2023-06-14 17:32:03 -04:00
Denny Crane
c142ed8a06
Merge branch 'master' into patch-51 2023-06-14 17:39:00 -03:00
Jiebin Sun
fcadb851c8
Maintain per-thread timer_id rather than create/delete frequently (#48778)
* Maintain per-thread timer_id rather than create/delete frequently

The QueryProfiler will frequently create/delete timer_id globally, which
will result in heavy kernel lock contention.
The idea is to maintain thread-local timer_id. Before create the
timer_id, it should check whether there is a timer_id already. And we
could stop the timer by timer_settime() rather than delete the timer_id
with timer_delete().

Apply the patch and run clickbench on latest 65d671b7c7 ClickHouse with
SPR 112 x 2 vCPUs. Query 4, 0, 5, 3, 15, 32 have 17.5%, 14.4%, 8.3%, 7.9%,
7.1%, 5.8% performance gain. The overall geomean has got 2.5%
performance gain.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

* Pack the timer and delete the timer_id when thread terminates

Pack the timer and related methods into the class. Delete the timer_id
when the thread terminates.

According to the issue (ClickHouse#49965),
all of the SSB queries benefit from this optimization, some have even got
improved by ~30% and the overall QPS could be significantly improved by ~18%.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

* Update src/Common/QueryProfiler.cpp

Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>

* Update src/Common/QueryProfiler.cpp

Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>

* Fix the review issue of QueryProfiler Timer from PR
https://github.com/ClickHouse/ClickHouse/pull/48778.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

* Update src/Common/QueryProfiler.cpp

Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>

* Add two separate CurrentMetrics for created and active timers
in QueryProfiler.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

---------

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
Co-authored-by: Nikita Taranov <nikita.taranov@clickhouse.com>
2023-06-14 21:22:09 +02:00
Igor Nikonov
ea46ba0d05
Merge branch 'master' into parallelize_output_from_storages-fix 2023-06-14 19:54:55 +02:00
Kruglov Pavel
bb7ec69b90
Fix flaky test_seekable_formats 2023-06-14 19:38:03 +02:00
robot-ch-test-poll3
c96f0c9f2f
Merge pull request #50992 from ClickHouse/thomoco-patch-4
Update mysql.md
2023-06-14 19:30:37 +02:00
robot-ch-test-poll3
d36c6ae870
Merge pull request #50988 from ClickHouse/thomoco-patch-3
Update postgresql.md
2023-06-14 19:30:11 +02:00
Sergei Trifonov
170e17c9e8
Merge pull request #50935 from azat/tests/02703_max_local_write_bandwidth 2023-06-14 18:54:49 +02:00
Thom O'Connor
fb304c0b05
Update postgresql.md
Fix table definition by adding ORDER BY() statement
2023-06-14 10:39:55 -06:00
Thom O'Connor
8eb84e9a58
Merge branch 'master' into thomoco-patch-3 2023-06-14 10:36:52 -06:00
Kruglov Pavel
7fc58d0171
Merge pull request #50987 from ClickHouse/Avogar-patch-1
Change submodule capnproto to it's fork in ClickHouse
2023-06-14 18:27:22 +02:00
Azat Khuzhin
e9c9db9335 Disable parallelize_output_from_storages for all system tables
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-14 19:11:23 +03:00
Azat Khuzhin
5caa3a9e80 Adjust min_insert_block_size_rows for materialized_view_parallelize_output_from_storages
Otherwise it is too slow for perf tests on CI [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/50214/e287ec50920c7cadabea6ec19ef14b353345ac93/performance_comparison_[3_4]/report.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-14 19:11:23 +03:00
Azat Khuzhin
3c4f638bd7 Disable parallelize_output_from_storages for storages with only one block
$ clickhouse-benchmark -i 10000 -d 0 --parallelize_output_from_storages=0 -q "select * from values('foo')"
    Loaded 1 queries.

    Queries executed: 10000.

    localhost:9000, queries 10000, QPS: 2800.490, RPS: 2800.490, MiB/s: 0.032, result RPS: 2800.490, result MiB/s: 0.032.

    0.000%          0.000 sec.
    10.000%         0.000 sec.
    20.000%         0.000 sec.
    30.000%         0.000 sec.
    40.000%         0.000 sec.
    50.000%         0.000 sec.
    60.000%         0.000 sec.
    70.000%         0.000 sec.
    80.000%         0.000 sec.
    90.000%         0.000 sec.
    95.000%         0.000 sec.
    99.000%         0.001 sec.
    99.900%         0.001 sec.
    99.990%         0.001 sec.

    $ clickhouse-benchmark -i 10000 -d 0 --parallelize_output_from_storages=1 -q "select * from values('foo')"
    Loaded 1 queries.

    Queries executed: 10000.

    localhost:9000, queries 10000, QPS: 1259.805, RPS: 1259.805, MiB/s: 0.014, result RPS: 1259.805, result MiB/s: 0.014.

    0.000%          0.001 sec.
    10.000%         0.001 sec.
    20.000%         0.001 sec.
    30.000%         0.001 sec.
    40.000%         0.001 sec.
    50.000%         0.001 sec.
    60.000%         0.001 sec.
    70.000%         0.001 sec.
    80.000%         0.001 sec.
    90.000%         0.001 sec.
    95.000%         0.001 sec.
    99.000%         0.001 sec.
    99.900%         0.001 sec.
    99.990%         0.003 sec.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-14 19:11:23 +03:00
Azat Khuzhin
3e419730c3 Disable parallelize_output_from_storages for processing MATERIALIZED VIEWs
Adding more processors for parallelize_output_from_storages is not a
costless operation (I've experienced some issues in production because
of this), and it is not easy to fix in a normal way, so let's disable it
for now.

Before this patch:
- INSERT INTO input SELECT * FROM numbers(10e6) SETTINGS parallelize_output_from_storages=1, min_insert_block_size_rows=1000
  0 rows in set. Elapsed: 3.648 sec. Processed 20.00 million rows, 120.00 MB (5.48 million rows/s., 32.90 MB/s.)

- INSERT INTO input SELECT * FROM numbers(10e6) SETTINGS parallelize_output_from_storages=0, min_insert_block_size_rows=1000
  0 rows in set. Elapsed: 1.851 sec. Processed 20.00 million rows, 120.00 MB (10.80 million rows/s., 64.82 MB/s.)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-14 19:11:23 +03:00
Azat Khuzhin
035334fc15 tests: mark 02585_query_status_deadlock as long
CI: https://s3.amazonaws.com/clickhouse-test-reports/50934/bb971fd7b7fd5bc400bbe28d16867c9dc337fb17/stateless_tests_flaky_check__asan_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-14 18:06:36 +02:00
Alexander Tokmakov
ffde5affeb
Merge pull request #50929 from ClickHouse/fix_illegal_column_timezone
Fix 'Illegal column timezone' in stress tests
2023-06-14 17:56:57 +03:00
Thom O'Connor
37245783ab
Update mysql.md
Updating mysql table function with example for copying data from MySQL into a ClickHouse MergeTree table
2023-06-14 08:03:18 -06:00
Thom O'Connor
c49bde8961
Update postgresql.md
Updated PostgreSQL documentation to better show example of using PostgreSQL() table engine for SELECT and INSERT, vs. copying data via postgresql() table function for copying data into ClickHouse
2023-06-14 07:11:28 -06:00
Igor Nikonov
bdeabd848a
Merge pull request #50095 from ClickHouse/test-plan-optimization-analyzer
Test plan optimization analyzer
2023-06-14 14:56:35 +02:00
Kruglov Pavel
85db1eb6fc
Change submodule capnproto to it's fork in ClickHouse 2023-06-14 14:55:11 +02:00
Derek Chia
811f6c1532
Update delete.md 2023-06-14 20:50:47 +08:00
Dmitry Novik
1d88b16830
Merge pull request #50584 from ClickHouse/analyzer-optimizations-on-shards
Analyzer: Do not apply Query Tree optimizations on shards
2023-06-14 14:31:46 +02:00