Commit Graph

8367 Commits

Author SHA1 Message Date
Robert Schulze
4cacfa6b43
Merge pull request #57509 from jinjunzh/qatzstd_main
Add Intel-QAT-based `ZSTD_QAT` compression codec
2024-01-10 10:31:39 +01:00
Kseniia Sumarokova
1b517105db
Merge pull request #57323 from valbok/mysql-binlog-client
MaterializedMySQL: Introduce MySQL Binlog Client
2024-01-10 09:08:13 +01:00
Robert Schulze
ced9e93ac6
Merge remote-tracking branch 'rschu1ze/master' into qatzstd_main 2024-01-09 20:36:02 +00:00
Sema Checherinda
7c7e72c4b7
Merge pull request #58573 from ClickHouse/chesema-s3-client-creation
fix and test that S3Clients are reused
2024-01-09 13:28:20 +01:00
Sema Checherinda
6dbf7c967c
Merge pull request #57663 from ClickHouse/CheSema-remote-blobs-naming-2
generate object storage key by template
2024-01-09 13:26:56 +01:00
robot-clickhouse
29173a602e Automatic style fix 2024-01-08 02:39:19 +00:00
Robert Schulze
53965bb9f1
Merge remote-tracking branch 'rschu1ze/master' into qatzstd_main 2024-01-07 14:11:22 +00:00
Sema Checherinda
d5f86f671d fix and test that S3Clients reused 2024-01-07 02:19:06 +01:00
robot-clickhouse
4a473c37ec Automatic style fix 2024-01-06 15:55:20 +00:00
Alexey Milovidov
2751b351cb Fix test 2024-01-06 16:44:38 +01:00
Nikolay Degterinsky
56aa835954
Merge pull request #58444 from bharatnc/ncb/support-hints-for-db-engines
support hints for database engines
2024-01-05 17:12:44 +01:00
Sergei Trifonov
8d6b012223
Merge pull request #57877 from azat/merges-mutations-throttling
Add ability to throttle merges/mutations
2024-01-05 17:08:50 +01:00
Sema Checherinda
2861cc70ba fix style 2024-01-05 13:42:30 +00:00
Sema Checherinda
8ab4aa3423 enable randomization setting 2024-01-05 13:42:30 +00:00
Sema Checherinda
53f36f8b21 object storage key template 2024-01-05 13:42:30 +00:00
Alexander Tokmakov
d2752f4b66
Merge pull request #57603 from evillique/allow-not-resolving-hostnames
Allow avoiding resolving hostnames in DDLWorker
2024-01-04 15:46:28 +01:00
Alexander Tokmakov
972756486a
Merge pull request #58350 from ClickHouse/rdb_dont_wait_inactive
An option to avoid waiting for inactive Replicated database replicas
2024-01-04 14:50:11 +01:00
Alexander Tokmakov
10e3ae9e68 Merge branch 'master' into allow-not-resolving-hostnames 2024-01-04 13:22:15 +01:00
Nikita Taranov
66d2db5283
New parallel replicas coordinator implementation (#57968) 2024-01-04 00:27:04 +01:00
Bharat Nallan Chakravarthy
d63765fb19 Merge upstream/master into ncb/support-hints-for-db-engines 2024-01-03 08:44:11 -08:00
Alexander Tokmakov
b1688b1953 Merge branch 'master' into rdb_dont_wait_inactive 2024-01-03 17:03:01 +01:00
Val Doroshchuk
5c221d123d MaterializedMySQL: Introduce MySQL Binlog Client
One binlog connection for many databases.

Suggesting to disable this feature by default for now. It should be explicitly enabled by SETTINGS use_binlog_client=1.
But if you would permanently enable it in MaterializedMySQLSettings, it should keep old behavior and all tests should pass too.

1. Introduced `IBinlog` and its impl to read the binlog events from socket - `BinlogFromSocket`, or file - `BinlogFromFile`. Based on prev impl of `EventBase` and the same old binlog parsers. It fully keeps BC with old version. Fixed `./check-mysql-binlog` to test new impl.
2. Introduced `BinlogEventsDispatcher`, it reads the event from the source `IBinlog` and sends it to currently attached `IBinlog` instances.
3. Introduced `BinlogClient`, which is used to group a list of `BinlogEventsDispatcher` by MySQL binlog connection which is defined by `user:password@host:port`. All dispatchers with the same binlog position should be merged to one.
4. Introduced `BinlogClientFactory`, which is a singleton and it is used to track all binlogs created over the instance.
5. Introduced `use_binlog_client` setting to `MaterializedMySQL`, which forces to reuse a `BinlogClient` if it already exists in `BinlogClientCatalog` or create new one. By default, it is disabled.
6. Introduced `max_bytes_in_binlog_queue` setting to define the limit of bytes in binlog's queue of events. If bytes in the queue increases this limit, `BinlogEventsDispatcher` will stop reading new events from source `IBinlog` until the space for new events will be freed.
7. Introduced `max_milliseconds_to_wait_in_binlog_queue` setting to define max ms to wait when the max bytes exceeded.
7. Introduced `max_milliseconds_to_wait_in_binlog_queue` setting to define max ms to wait when the max bytes exceeded.
8. Introduced `max_bytes_in_binlog_dispatcher_buffer` setting to define max bytes in the binlog dispatcher's buffer before it is flushed to attached binlogs.
9. Introduced `max_flush_milliseconds_in_binlog_dispatcher` setting to define max milliseconds in the binlog dispatcher's buffer to wait before it is flushed to attached binlogs.
10. Introduced `system.mysql_binlogs` system table, which shows a list of active binlogs.
11. Introduced `UnparsedRowsEvent` and `MYSQL_UNPARSED_ROWS_EVENT`, which defines that an event is not parsed and should be explicitly parsed later.
12. Fixed bug when not possible to apply DDL since syntax error or unsupported SQL.

@larspars is the author of following:
`GTIDSets::contains()`
`ReplicationHelper`
`shouldReconnectOnException()`
2024-01-03 15:26:09 +01:00
Kruglov Pavel
570d1c013b
Merge pull request #57053 from Avogar/better-parsing-exceptions
Better exception messages in input formats
2024-01-03 14:47:51 +01:00
Bharat Nallan Chakravarthy
2da0a30626 add a test 2024-01-02 13:52:52 -08:00
Nikolay Degterinsky
debbeb505d
Merge pull request #58409 from ClickHouse/evillique-patch-1
Fix test_user_valid_until
2024-01-02 17:06:11 +01:00
avogar
5f6a806d1c Fix style 2024-01-02 15:43:44 +00:00
avogar
0844528ec1 Fix tests 2024-01-02 15:22:42 +00:00
Kruglov Pavel
e69bda7f08
Merge pull request #57438 from Avogar/fix-resetting-buffers
Fix working with read buffers in StreamingFormatExecutor
2024-01-02 15:34:35 +01:00
Nikolay Degterinsky
004730fe87
Fix test_user_valid_until 2024-01-02 14:50:28 +01:00
Alexey Milovidov
78523639e0 Remove function 2023-12-30 14:52:54 +01:00
avogar
f20917048e Fix kafka tests 2023-12-29 17:02:46 +00:00
Alexander Tokmakov
bfc10bd234 an option to avoid waiting for inactive Replicated db replicas 2023-12-29 15:19:16 +01:00
Kruglov Pavel
9389ea8745 Fix style 2023-12-28 18:47:57 +00:00
Kruglov Pavel
69f6147499 Fix style 2023-12-28 18:47:57 +00:00
avogar
c6085b3eb9 Fix working with read buffers in StreamingFormatExecutor 2023-12-28 18:47:57 +00:00
Michael Kolupaev
609b2c216c Fix some of the CI 2023-12-28 17:56:05 +00:00
koloshmet
49367186e3 fix fix fix 2023-12-28 17:56:04 +00:00
Kseniia Sumarokova
8e8fd84cb7
Merge pull request #58293 from ClickHouse/fix-s3-queue-test
Fix test_storage_s3_queue/test.py::test_drop_table
2023-12-28 17:18:11 +01:00
Alexander Tokmakov
38fe70c68a
Revert "Refreshable materialized views (takeover)" 2023-12-28 13:12:20 +01:00
kssenii
529a1d67cb Fix 2023-12-28 12:48:56 +01:00
Alexey Milovidov
40a5dbdeba
Merge branch 'master' into mv 2023-12-28 03:16:27 +01:00
Alexey Milovidov
30f6d03b37
Merge pull request #57977 from bianpengyuan/add-s3-read-only-setting
Add a setting to specify s3 disk is read only.
2023-12-28 01:24:50 +01:00
Alexey Milovidov
5406c79c89
Merge pull request #58264 from ClickHouse/fs-cache-better-multi-cache-configuration
Make fs caches with the same path use the same cache objects
2023-12-28 00:10:15 +01:00
Michael Kolupaev
673743e2ac Fix some of the CI 2023-12-27 20:24:55 +00:00
koloshmet
808cb0fa05 fix fix fix 2023-12-27 20:24:54 +00:00
Sema Checherinda
617ee5abaf
Merge pull request #57551 from bharatnc/ncb/async-metrics-pri-key-size
add asynchronous metrics for total primary key size in memory and allocated
2023-12-27 20:24:20 +01:00
Pengyuan Bian
26b553c3b6 format. 2023-12-27 18:27:36 +00:00
kssenii
041c5d4219 Make caches with the same path use the same cache objects 2023-12-27 18:37:43 +01:00
Pengyuan Bian
3dcc1f5448 update. 2023-12-27 07:54:18 +00:00
Pengyuan Bian
2765acfe45 wip test. 2023-12-26 07:50:46 +00:00