Commit Graph

5787 Commits

Author SHA1 Message Date
Nikita Taranov
87d6564b9b
Update gtest_async_loader.cpp 2023-05-29 15:50:06 +02:00
Sergei Trifonov
c26b3b36ae
Merge branch 'master' into fix-flaky-async-loader-test 2023-05-29 13:38:35 +02:00
Sergei Trifonov
355f917daa
Update src/Common/tests/gtest_async_loader.cpp 2023-05-29 13:38:14 +02:00
alesapin
6b7e2a0041
Merge pull request #49961 from CheSema/mem-consumption-s3-write
Introduce a limitation of async calls inside s3 write buffer, make a test that memory is bounded even when s3 is slow
2023-05-29 13:29:01 +02:00
serxa
bdb974ab3c Fix flaky AsyncLoader.StaticPriorities unit test 2023-05-29 11:21:44 +00:00
Robert Schulze
47daaad172
Merge pull request #49778 from helifu/master2
Support custom 'order by' expression for system log tables
2023-05-29 10:05:14 +02:00
helifu
03e11b6e16 Remove the duplicate pair of parentheses 2023-05-28 12:59:34 +08:00
Sema Checherinda
b0eff95388 update doc, add profile event WriteBufferFromS3WaitInflightLimitMicroseconds 2023-05-26 18:59:21 +02:00
serxa
c4687a8b93 Unify priorities: rework AsyncLoader 2023-05-26 16:46:59 +00:00
serxa
edffedbabf fix 2023-05-26 15:10:08 +00:00
Sergei Trifonov
84203341ca
Merge branch 'master' into async-loader-workloads 2023-05-26 16:17:56 +02:00
Sergei Trifonov
0d1f2e297b
Unify priorities: lower value means higher priority (#50205) 2023-05-26 15:55:30 +02:00
Sema Checherinda
9603f2b49c
Merge pull request #50190 from ClibMouse/feature/unify-core-id-querying
Implement a uniform way to query processor core IDs
2023-05-26 15:46:39 +02:00
serxa
b8d3e495e5 add pool_id out-of-bound checks 2023-05-25 12:42:19 +00:00
serxa
e3ce2f834a fix style 2023-05-25 12:35:00 +00:00
Sergei Trifonov
2df22d396d
Merge branch 'master' into async-loader-workloads 2023-05-25 14:32:12 +02:00
Sergei Trifonov
78c89da8bb
Update src/Common/AsyncLoader.cpp
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2023-05-25 13:48:03 +02:00
Sergei Trifonov
32cb9931b6
Update src/Common/AsyncLoader.h
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2023-05-25 13:47:56 +02:00
何李夫
e4c8c4cecf
Add zookeeper name in endpoint id (#49780)
* Add zookeeper name in endpoint id

When we migrate a replicated table from one zookeeper cluster to
another (the reason why we migration is that zookeeper's load is
too high), we will create a new table with the same zpath, but it
will fail and the old table will be in trouble.

Here is some infomation:
1.old table:
  CREATE TABLE a1 (`id` UInt64)
  ENGINE = ReplicatedMergeTree('/clickhouse/tables/default/a1/{shard}', '{replica}')
  ORDER BY (id);
2.new table:
  CREATE TABLE a2 (`id` UInt64)
  ENGINE = ReplicatedMergeTree('aux1:/clickhouse/tables/default/a1/{shard}', '{replica}')
  ORDER BY (id);
3.error info:
  <Error> executeQuery: Code: 220. DB::Exception: Duplicate interserver IO endpoint:
          DataPartsExchange:/clickhouse/tables/default/a1/01/replicas/02.
          (DUPLICATE_INTERSERVER_IO_ENDPOINT)
  <Error> InterserverIOHTTPHandler: Code: 221. DB::Exception: No interserver IO endpoint
          named DataPartsExchange:/clickhouse/tables/default/a1/01/replicas/02.
          (NO_SUCH_INTERSERVER_IO_ENDPOINT)

* Revert "Add zookeeper name in endpoint id"

This reverts commit 9deb75b249619b7abdd38e3949ca8b3a76c9df8e.

* Add zookeeper name in endpoint id

When we migrate a replicated table from one zookeeper cluster to
another (the reason why we migration is that zookeeper's load is
too high), we will create a new table with the same zpath, but it
will fail and the old table will be in trouble.

* Fix incompatible with a new setting

* add a test, fix other issues

* Update 02442_auxiliary_zookeeper_endpoint_id.sql

* Update 02735_system_zookeeper_connection.reference

* Update 02735_system_zookeeper_connection.sql

* Update run.sh

* Remove the 'no-fasttest' tag

* Update 02442_auxiliary_zookeeper_endpoint_id.sql

---------

Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-05-25 12:50:14 +03:00
ltrk2
f76f989b53 Implement a uniform way to query processor core IDs 2023-05-24 13:33:05 +00:00
Kruglov Pavel
4689412ab3
Change fields destruction order in AsyncTaskExecutor 2023-05-23 16:14:24 +02:00
Robert Schulze
f4c73e94d2
Merge pull request #49989 from arenadata/ADQM-811
Add support of Date|Date32 arguments to the toUnixTimestamp() function
2023-05-23 08:55:56 +02:00
Kseniia Sumarokova
c6e4db969f
Merge pull request #50028 from kssenii/some-minor-changes
Move some common code to common
2023-05-22 18:25:24 +02:00
Sergei Trifonov
f099e3f41e
Merge pull request #50045 from azat/ddl-fix-opentelemetry
Add proper escaping for DDL OpenTelemetry context serialization
2023-05-22 18:20:17 +02:00
Nikolay Degterinsky
d4b89cb643
Merge pull request #49356 from Ziy1-Tan/vcol
Support for `_path` and `_file` virtual columns for table function `url`.
2023-05-22 18:10:32 +02:00
Sergei Trifonov
0f90bfdd11
Merge pull request #50051 from ClickHouse/add-more-distributed-connection-profile-events
Add more profile events for distributed connections
2023-05-22 13:37:56 +02:00
Sergei Trifonov
7fbfa4b21e
Merge branch 'master' into async-loader-workloads 2023-05-22 13:31:42 +02:00
Victor Krasnov
98aace14ae Add DATE_SECONDS_PER_DAY macro definition to replace the numeric literal 86400 2023-05-22 09:23:23 +00:00
Han Fei
a2c0a65344
Merge pull request #49919 from hanfei1991/hanfei/fix-optimize-regexp-prefix
fix `is_prefix` in OptimizeRegularExpression
2023-05-22 10:50:36 +02:00
Han Fei
a257ff6cf3 address comment 2023-05-22 10:41:22 +02:00
serxa
372601d6df fix 2023-05-21 17:04:49 +00:00
serxa
cd11c25864 fix test + more testing for dynamic prioritization 2023-05-21 17:04:00 +00:00
serxa
c5765e71f9 requeue jobs w/o allocations and spawn workers during prioritization 2023-05-21 17:02:56 +00:00
Sergei Trifonov
e33455ed87
Merge branch 'master' into async-loader-workloads 2023-05-21 16:30:58 +02:00
serxa
128b8e5889 fix tests + add test for dynamic pools 2023-05-21 14:28:16 +00:00
Robert Schulze
2a9ff30a7f
Merge pull request #49380 from azat/dict/hashed-memory
Improve memory usage and speed of SPARSE_HASHED/HASHED dictionaries
2023-05-21 15:46:41 +02:00
serxa
44b1754ccf more profile events 2023-05-21 12:43:47 +00:00
serxa
c56e6a8b80 Add more profile events for distributconnections 2023-05-21 12:15:06 +00:00
Sergei Trifonov
3c002755e2
Merge pull request #50036 from ClickHouse/fix-load-balancing
Load balancing bugfixes
2023-05-21 11:21:55 +02:00
Azat Khuzhin
82054d40a5 Add proper escaping for DDL OpenTelemetry context serialization
Before you was able to break the format by using "\n" or "\t", that will
simply lead to DDL hang, because DDLWorker will simply log the error and
do nothing more:

    <Error> DDLWorker: Cannot parse DDL task query-0000000056: Incorrect task format. Will try to send error status: Code: 27. DB::ParsingException: Cannot parse input: expected '\n' before: 'bar\n1\n'. (CANNOT_PARSE_INPUT_ASSERTION_FAILED) (version 23.5.1.1)

Fix this by adding proper escaping.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-20 13:15:49 +02:00
Azat Khuzhin
52c5fd5cb9 Rewrite OpenTelemetry context serialization for DDL without IO/Operators.h
This is required to switch to escaped versions.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-20 13:14:40 +02:00
Igor Nikonov
af80e29519
Merge branch 'master' into clearable_hash_set_without_zero_storage 2023-05-19 23:36:30 +02:00
Sergei Trifonov
14e8132ac4
Merge branch 'master' into fix-load-balancing 2023-05-19 23:05:27 +02:00
serxa
052d8aca71 limit max_tries value by max_error_cap to avoid unlimited number of retries 2023-05-19 18:13:29 +00:00
serxa
d69c35fcdd fix PoolWithFailover error_count integer overflow 2023-05-19 17:57:00 +00:00
kssenii
791bb6cd4c Fix style check 2023-05-19 17:35:01 +02:00
kssenii
0eab528f9f Move common code 2023-05-19 16:23:56 +02:00
mateng915
5237dd0245
New system table zookeeper connection (#45245)
* Feature: Support new system table to show which zookeeper node be connected

Description:
============
Currently we have no place to check which zk node be connected otherwise using
lsof command. It not convenient

Solution:
=========
Implemented a new system table, system.zookeeper_host when CK Server has zk
this table will show the zk node dir which connected by current CK server

Noted: This table can support multi-zookeeper cluster scenario.

* fixed review comments

* added test case

* update test cases

* remove unused code

* fixed review comments and removed unused code

* updated test cases for print host, port and is_expired

* modify the code comments

* fixed CI Failed

* fixed code style check failure

* updated test cases by added Tags

* update test reference

* update test cases

* added system.zookeeper_connection doc

* Update docs/en/operations/system-tables/zookeeper_connection.md

* Update docs/en/operations/system-tables/zookeeper_connection.md

* Update docs/en/operations/system-tables/zookeeper_connection.md

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-05-19 17:06:43 +03:00
Sergei Trifonov
5db5f6e44b
Merge branch 'master' into fix-throttlers 2023-05-19 14:08:36 +02:00
Azat Khuzhin
01bf041cca Rewrite HashTableGrower{,WithPrecalculation}::set w/o ternary operators
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-19 06:07:21 +02:00