thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
132,656 Commits 9,328 Branches 1,783 Tags 2.1 GiB
9cfdff2ddb
Commit Graph

15874 Commits

Author SHA1 Message Date
Raúl Marín
8f998eaa82
Merge pull request #58447 from canhld94/fix_symlink_in_user_files 
Allow users to work with symlinks in user_files_path (again)
 2024-01-08 10:40:36 +01:00
Robert Schulze
6665d23243
Merge pull request #58029 from skyoct/feat/server_settings 
Add `changeable_without_restart` column to `system.server_settings`
 2024-01-08 10:09:51 +01:00
Robert Schulze
c11a2fcb7f
Fix style 2024-01-08 08:50:34 +00:00
Duc Canh Le
149de9aa93 discard fs::canonical return 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-08 03:40:48 +00:00
Robert Schulze
f553b55e3a
Merge remote-tracking branch 'rschu1ze/master' into regex-std-re2 2024-01-07 22:31:35 +00:00
Robert Schulze
8e804487f3
Some fixups 2024-01-07 22:28:08 +00:00
Sema Checherinda
6f626d8294 fix auth_settings.hasUpdates function 2024-01-07 23:00:26 +01:00
Alexander Tokmakov
371d664667
Update MergeTreeData.cpp 2024-01-07 17:27:22 +01:00
Robert Schulze
53965bb9f1
Merge remote-tracking branch 'rschu1ze/master' into qatzstd_main 2024-01-07 14:11:22 +00:00
Robert Schulze
e947ed710b
Hijack and document 'update' map to represent reload status 2024-01-07 13:09:27 +00:00
Sema Checherinda
d5f86f671d fix and test that S3Clients reused 2024-01-07 02:19:06 +01:00
Anton Popov
8c9cf394b3 Merge remote-tracking branch 'upstream/master' into HEAD 2024-01-06 20:58:17 +00:00
Nikolai Kochetov
105789b936
Merge pull request #53705 from ClickHouse/analyzer-execution-names 
Analyzer: always qualify execution names
 2024-01-06 17:39:58 +01:00
Anton Popov
48a7402b34 fix build 2024-01-06 16:39:27 +00:00
Nikolai Kochetov
c2f00bc75a
Merge pull request #58480 from ClickHouse/try-to-remove-pk-analysis-on-ast 
Check if I can remove KeyCondition analysis on AST.
 2024-01-06 12:40:46 +01:00
Anton Popov
f252412915 Merge remote-tracking branch 'upstream/master' into HEAD 2024-01-06 03:08:52 +00:00
Alexey Milovidov
e24ec55451
Merge pull request #58482 from ClickHouse/fix_new_intersecting_parts 
Fix a stupid case of intersecting parts
 2024-01-05 21:53:21 +01:00
Alexander Tokmakov
f94f33c554
Merge pull request #57755 from azat/rmt/ALTER_METADATA-MERGE_PARTS-race 
[RFC] Eliminate possible race between ALTER_METADATA and MERGE_PARTS
 2024-01-05 17:11:57 +01:00
Sergei Trifonov
8d6b012223
Merge pull request #57877 from azat/merges-mutations-throttling 
Add ability to throttle merges/mutations
 2024-01-05 17:08:50 +01:00
Nikolai Kochetov
c6858f8666 Fixing tidy 2024-01-05 15:22:41 +00:00
Nikolai Kochetov
8bcfa888a5 Fixing index hint 2024-01-05 11:50:09 +00:00
Nikolai Kochetov
eff6232418 Merge branch 'master' into try-to-remove-pk-analysis-on-ast 2024-01-05 10:54:46 +00:00
Nikolai Kochetov
15b66031bc Update prepared sets. 2024-01-05 10:53:07 +00:00
Nikolai Kochetov
80c1e59d3f Merge branch 'master' into analyzer-execution-names 2024-01-05 09:31:25 +00:00
Robert Schulze
2166df0640
Merge pull request #57882 from lingtaolf/optimization/BF_support_rg 
Utilize `ngrambf` and `tokenbf` indexes by function `match()`
 2024-01-05 00:00:50 +01:00
Nikolai Kochetov
85c1bb80fd Merge branch 'master' into hdfs-virtuals 2024-01-04 18:00:10 +00:00
Nikolai Kochetov
85ea5c1f82 Remove more code 2024-01-04 17:46:06 +00:00
Robert Schulze
f276a7f9d4
Merge remote-tracking branch 'ClickHouse/master' into optimization/BF_support_rg 2024-01-04 17:15:17 +00:00
Alexander Tokmakov
9f5015737b fix a stupid case of intersecting parts 2024-01-04 17:45:42 +01:00
Nikolai Kochetov
494a32f4e4 Review fixes 2024-01-04 14:41:04 +00:00
Kruglov Pavel
7e6e835e2e
Merge pull request #57520 from Avogar/ignore-mv-with-dropped-target-table 
Ignore MVs with dropped target table during pushing to views
 2024-01-04 15:33:27 +01:00
Duc Canh Le
2e14cfb526 add settings for output compression level and window size 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-04 08:16:00 +00:00
Nikita Taranov
66d2db5283
New parallel replicas coordinator implementation (#57968) 2024-01-04 00:27:04 +01:00
Nikolai Kochetov
7a271f09ed Check if I can remove KeyCondition analysis on AST. 2024-01-03 17:50:46 +00:00
Nikolai Kochetov
d06de83ac1 Fix KeyCondition for file/url/s3 2024-01-03 17:44:28 +00:00
Anton Popov
cf4604bfb6 fix build and tests 2024-01-03 16:59:13 +00:00
Anton Popov
ce1b978a2d Merge remote-tracking branch 'upstream/master' into HEAD 2024-01-03 16:01:44 +00:00
Anton Popov
7dd128f90f Revert "remove projection from StorageSnapshot" 
This reverts commit a01acf5d2a.
 2024-01-03 16:00:50 +00:00
Anton Popov
6b6f395cc7
Merge pull request #57631 from Avogar/cache-for-reading-compact-subcolumns 
Read column once while reading more that one subcolumn from it in Compact parts
 2024-01-03 16:31:54 +01:00
Nikolai Kochetov
a3a24bdb40 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hdfs-virtuals 2024-01-03 15:10:57 +00:00
Nikolai Kochetov
7834519212
Merge pull request #58255 from ClickHouse/filter-virtual-columns-storage-merge 
Refactor StorageMerge virtual columns filtering.
 2024-01-03 16:09:49 +01:00
Robert Schulze
8d4b519bb1
Replace std::regex by re2 2024-01-03 15:06:20 +00:00
Val Doroshchuk
5c221d123d MaterializedMySQL: Introduce MySQL Binlog Client 
One binlog connection for many databases.

Suggesting to disable this feature by default for now. It should be explicitly enabled by SETTINGS use_binlog_client=1.
But if you would permanently enable it in MaterializedMySQLSettings, it should keep old behavior and all tests should pass too.

1. Introduced `IBinlog` and its impl to read the binlog events from socket - `BinlogFromSocket`, or file - `BinlogFromFile`. Based on prev impl of `EventBase` and the same old binlog parsers. It fully keeps BC with old version. Fixed `./check-mysql-binlog` to test new impl.
2. Introduced `BinlogEventsDispatcher`, it reads the event from the source `IBinlog` and sends it to currently attached `IBinlog` instances.
3. Introduced `BinlogClient`, which is used to group a list of `BinlogEventsDispatcher` by MySQL binlog connection which is defined by `user:password@host:port`. All dispatchers with the same binlog position should be merged to one.
4. Introduced `BinlogClientFactory`, which is a singleton and it is used to track all binlogs created over the instance.
5. Introduced `use_binlog_client` setting to `MaterializedMySQL`, which forces to reuse a `BinlogClient` if it already exists in `BinlogClientCatalog` or create new one. By default, it is disabled.
6. Introduced `max_bytes_in_binlog_queue` setting to define the limit of bytes in binlog's queue of events. If bytes in the queue increases this limit, `BinlogEventsDispatcher` will stop reading new events from source `IBinlog` until the space for new events will be freed.
7. Introduced `max_milliseconds_to_wait_in_binlog_queue` setting to define max ms to wait when the max bytes exceeded.
7. Introduced `max_milliseconds_to_wait_in_binlog_queue` setting to define max ms to wait when the max bytes exceeded.
8. Introduced `max_bytes_in_binlog_dispatcher_buffer` setting to define max bytes in the binlog dispatcher's buffer before it is flushed to attached binlogs.
9. Introduced `max_flush_milliseconds_in_binlog_dispatcher` setting to define max milliseconds in the binlog dispatcher's buffer to wait before it is flushed to attached binlogs.
10. Introduced `system.mysql_binlogs` system table, which shows a list of active binlogs.
11. Introduced `UnparsedRowsEvent` and `MYSQL_UNPARSED_ROWS_EVENT`, which defines that an event is not parsed and should be explicitly parsed later.
12. Fixed bug when not possible to apply DDL since syntax error or unsupported SQL.

@larspars is the author of following:
`GTIDSets::contains()`
`ReplicationHelper`
`shouldReconnectOnException()`
 2024-01-03 15:26:09 +01:00
Nikolai Kochetov
9dc1f4d99c
Update StorageMerge.cpp 2024-01-03 14:53:40 +01:00
Kruglov Pavel
c8acc7c2d1
Fix build 2024-01-03 14:44:00 +01:00
Nikolai Kochetov
aee933a437 Merge branch 'master' into hdfs-virtuals 2024-01-03 12:16:57 +00:00
Nikolai Kochetov
d5dcb6661d Review fixes. 2024-01-03 10:12:08 +00:00
Nikolai Kochetov
78776a060c
Update src/Storages/StorageMerge.cpp 
Co-authored-by: Dmitry Novik <n0vik@clickhouse.com>
 2024-01-03 11:06:57 +01:00
Nikolai Kochetov
eeed23b1bc Fix sanitizer assert. 2024-01-03 09:45:25 +00:00
Duc Canh Le
12fda5f309 fix 02771_multidirectory_globs_storage_file 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-03 09:22:29 +00:00
Duc Canh Le
3502245ecb use fs::absolute instead of fs::canonical 
To allow users to work with symlinks in user_files_path

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-03 04:58:26 +00:00
Alexey Milovidov
44432b1ffc Fix build 2024-01-02 21:54:13 +01:00
Alexey Milovidov
deac0b5f3e Merge branch 'master' into insert-quorum-host-name-check 2024-01-02 21:46:12 +01:00
avogar
488e36de46 More fixes 2024-01-02 20:20:39 +00:00
Nikolai Kochetov
9c25cb6692 Cleanup 2024-01-02 18:08:04 +00:00
Nikolai Kochetov
4f99a8bc1f Remove more unused code. 2024-01-02 17:54:20 +00:00
Nikolai Kochetov
1b20ce5162 Cleanup 2024-01-02 17:50:06 +00:00
Nikolai Kochetov
c808b03e55 Remove unneeded code 2024-01-02 17:27:33 +00:00
Nikolai Kochetov
8936c8376a Use predicate in getTaskIteratorExtension. 2024-01-02 17:14:16 +00:00
avogar
c5ba97f0cd Fix tests 2024-01-02 16:38:45 +00:00
Kruglov Pavel
f2dfe8bdda
Fix build 2024-01-02 16:42:17 +01:00
Nikolai Kochetov
3e3fed1cbe Add reading step to URL 2024-01-02 15:18:13 +00:00
Nikolai Kochetov
0f76967f97 Add reading step to Azure. 2024-01-02 13:46:08 +00:00
Alexander Tokmakov
d13abac912
Merge pull request #58333 from ClickHouse/fix_no_such_key_detached_part 
Fix lost blobs after dropping a replica with broken detached parts
 2024-01-02 14:21:55 +01:00
Nikolai Kochetov
b7cc6d4615 Fixing tests. 2024-01-02 13:08:04 +00:00
Duc Canh Le
d623702378 Merge branch 'master' into final_no_copy 
Resolve conflicts

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-02 06:07:49 +00:00
Bharat Nallan Chakravarthy
00b5e28d6a Merge upstream/master into ncb/system-database-engines 2023-12-30 23:10:48 -08:00
Bharat Nallan Chakravarthy
3c4ac0ba08 add system.database_engines table 2023-12-30 20:31:25 -08:00
Alexey Milovidov
7bded0a5e7
Merge pull request #58316 from ClickHouse/reintroduce_is_deleted 
Re-introduce `is_deleted` column for ReplacingMergeTree
 2023-12-31 00:57:19 +01:00
Alexey Milovidov
063463b0a8
Merge pull request #58359 from ClickHouse/all-system-tables-local 
Attach all system tables in `clickhouse-local`
 2023-12-30 17:51:27 +01:00
Alexey Milovidov
c9467d8d47
Update ReplicatedMergeTreeSink.cpp 2023-12-30 15:19:50 +03:00
Alexey Milovidov
2421173574
Merge branch 'master' into insert-quorum-host-name-check 2023-12-30 13:13:24 +01:00
Alexey Milovidov
8fc05e25fe
Merge pull request #58310 from azat/kafka-fix-stat-leak-resubmit 
Create consumers for Kafka tables on fly with TTL (resubmit)
 2023-12-30 13:03:16 +01:00
Alexey Milovidov
e1812f3b58
Merge pull request #58266 from ClickHouse/vdimir/simple_fix_tuple_elimination 
Analyzer: fix tuple comparison when result is always null
 2023-12-30 13:02:38 +01:00
Alexey Milovidov
aa6ecd2d59
Merge pull request #58343 from azat/s3/optional-gcs-compose 
Avoid sending ComposeObject requests after upload to GCS
 2023-12-30 12:40:04 +01:00
Alexey Milovidov
40ca9c202d
Merge pull request #58346 from ClickHouse/check-what-would-be-ifremove-array-joined-columns-from-key-condition 
Check what happen if remove array joined columns from KeyCondition
 2023-12-30 12:38:57 +01:00
Alexey Milovidov
f058394d92
Merge pull request #58351 from ClickHouse/fix_00002 
Keep exception format string in retries ctl
 2023-12-30 12:37:36 +01:00
Alexey Milovidov
39b239683c Attach all system tables in clickhouse-local 2023-12-29 21:25:22 +01:00
Nikolai Kochetov
b95bdef09e Update StorageS3 and StorageS3Cluster 2023-12-29 17:41:11 +00:00
Kruglov Pavel
f57939096c
Merge branch 'master' into ignore-mv-with-dropped-target-table 2023-12-29 17:02:23 +01:00
Nikolai Kochetov
5521e5d9b1 Refactor StorageHDFS and StorageFile virtual columns filtering 2023-12-29 15:58:01 +00:00
robot-ch-test-poll3
07ba672e37
Merge pull request #58142 from canhld94/final_less_compare 
MergeTree FINAL to not compare rows from same non-L0 part
 2023-12-29 16:47:14 +01:00
Azat Khuzhin
a12df35be4 Eliminate possible race between ALTER_METADATA and MERGE_PARTS 
v2: move metadata version check after checking that the part is not covering part
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 16:46:10 +01:00
Azat Khuzhin
c7fa93d704 Add infrastructure for testing replicated MergeTree queue 
- replicated_queue_fail_next_entry - to fail next queue entry
- replicated_queue_unfail_entries - to "unfail" all queue entries (if
  any)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 16:43:01 +01:00
Alexander Tokmakov
1013f6b23f
Merge branch 'master' into reintroduce_is_deleted 2023-12-29 15:46:24 +01:00
Alexander Tokmakov
72a0797b88 keep exception format string in retries ctl 2023-12-29 15:21:46 +01:00
Alexey Milovidov
ea03cc82aa
Merge pull request #58320 from ClickHouse/mv3 
Refreshable materialized views again
 2023-12-29 14:44:50 +01:00
Azat Khuzhin
853fdfe775 Clean cached messages on destroy kafka consumer 
The callchain of the kafka consumer is very tricky, so for the sake of
common sense let's just clean the messages on moving out consumer (and
in dtor, but this is just to keep that two code path in sync).

(Also reported by @filimonov)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 14:30:21 +01:00
Azat Khuzhin
b3d6caf37f Unsubscribe kafka consumer before cleaning it by TTL 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 14:03:53 +01:00
Nikolai Kochetov
0e8232a8c3 Check what happen if remove array joined columns from KeyCondition 2023-12-29 12:24:19 +00:00
Azat Khuzhin
f578541ded Fix destructing kafka consumer via member orders 
We've discussed this with @filimonov and he pointed out that everything
else (except for rdkafka_stat/rdkafka_stat_mutex) is done via members
orders, so let's do it in the same style.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 13:19:11 +01:00
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Duc Canh Le
91a87d6b6c better implementation 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-29 07:27:10 +00:00
Igor Nikonov
208a9193f6 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-28 21:28:36 +00:00
Alexander Tokmakov
852f397a97 fix lost blobs after dropping a replica with broken detached parts 2023-12-28 21:47:19 +01:00
Kruglov Pavel
fbd3f7cd59
Merge pull request #56132 from Avogar/flatten-only-true-nested 
Flatten only true Nested type if flatten_nested=1, not all Array(Tuple)
 2023-12-28 20:58:28 +01:00
Michael Kolupaev
c4f4516a37 Fix WriteBuffer assert if refresh is cancelled at the wrong moment 2023-12-28 18:34:28 +00:00
Nikolai Kochetov
490a8bce9e Remove commented code. 2023-12-28 18:01:08 +00:00
Michael Kolupaev
ea138fe8c9 space 2023-12-28 17:56:06 +00:00