thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-11 17:02:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
132,785 Commits 9,299 Branches 1,782 Tags 2.1 GiB
c291a11e5d
Commit Graph

51975 Commits

Author SHA1 Message Date
Yarik Briukhovetskyi
94533ecb9f
Update DateTimeTransforms.h 2024-01-05 17:59:21 +01:00
Yarik Briukhovetskyi
90a986e262
Update DateTimeTransforms.h 2024-01-05 17:58:22 +01:00
yariks5s
12585ded09 init 2024-01-05 16:51:35 +00:00
Sema Checherinda
c84bad3259
Update src/Common/MatchGenerator.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2024-01-05 17:25:32 +01:00
Sema Checherinda
c01a6ccc8b
Update src/Common/MatchGenerator.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2024-01-05 17:25:19 +01:00
Sema Checherinda
2fce803e6a
Update src/Common/MatchGenerator.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2024-01-05 17:25:14 +01:00
Sema Checherinda
e19ee6ff67
Update src/Common/MatchGenerator.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2024-01-05 17:24:59 +01:00
Sema Checherinda
0d50ad29e7
Update src/Common/MatchGenerator.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2024-01-05 17:24:30 +01:00
Nikolay Degterinsky
56aa835954
Merge pull request #58444 from bharatnc/ncb/support-hints-for-db-engines 
support hints for database engines
 2024-01-05 17:12:44 +01:00
Alexander Tokmakov
f94f33c554
Merge pull request #57755 from azat/rmt/ALTER_METADATA-MERGE_PARTS-race 
[RFC] Eliminate possible race between ALTER_METADATA and MERGE_PARTS
 2024-01-05 17:11:57 +01:00
Sergei Trifonov
8d6b012223
Merge pull request #57877 from azat/merges-mutations-throttling 
Add ability to throttle merges/mutations
 2024-01-05 17:08:50 +01:00
Yakov Olkhovskiy
84dd4a74bc fix exception leaking from destructor 2024-01-05 15:45:36 +00:00
Nikolai Kochetov
c6858f8666 Fixing tidy 2024-01-05 15:22:41 +00:00
Dani Pozo
a012377102 Hide credentials in system.backup_log base_backup_name column 2024-01-05 15:39:36 +01:00
Alexander Tokmakov
97481bf0a3
Merge pull request #58520 from tomahawk28/master 
Enable ordinary databases while restoration
 2024-01-05 15:31:25 +01:00
Alexey Milovidov
8b9613d5a2 Merge branch 'master' of github.com:ClickHouse/ClickHouse into pretty-type-names-default 2024-01-05 14:53:41 +01:00
Sema Checherinda
3dff106772 fix clang tidy 2024-01-05 13:42:30 +00:00
Sema Checherinda
c59a72b987 fix special build, add randomization to stateless tests 2024-01-05 13:42:30 +00:00
Sema Checherinda
8ab4aa3423 enable randomization setting 2024-01-05 13:42:30 +00:00
Sema Checherinda
680ce20509 build fix 2024-01-05 13:42:30 +00:00
Sema Checherinda
f7d033cac2 speed up generation, less allocations 2024-01-05 13:42:30 +00:00
Sema Checherinda
a409b6c420 speed up generation 2024-01-05 13:42:30 +00:00
Sema Checherinda
53f36f8b21 object storage key template 2024-01-05 13:42:30 +00:00
Robert Schulze
f6ecf2b617
Add decode function for sqids 2024-01-05 13:33:40 +00:00
Raúl Marín
c0a44ffce4
Revert "Merging #53757" 2024-01-05 13:46:18 +01:00
Vitaly Baranov
7c867a09af
Merge pull request #57263 from Enmk/patch-4 
Fixed potential exception due to stale profile UUID
 2024-01-05 13:43:54 +01:00
Robert Schulze
8169b3e17f
Merge remote-tracking branch 'rschu1ze/master' into punycode-revert-revert 2024-01-05 12:19:46 +00:00
Robert Schulze
2cc2a9089c
Incorporate review feedback 2024-01-05 12:18:21 +00:00
Nikolai Kochetov
8bcfa888a5 Fixing index hint 2024-01-05 11:50:09 +00:00
Nikolai Kochetov
eff6232418 Merge branch 'master' into try-to-remove-pk-analysis-on-ast 2024-01-05 10:54:46 +00:00
Nikolai Kochetov
15b66031bc Update prepared sets. 2024-01-05 10:53:07 +00:00
Nikolai Kochetov
7cfb2e1b44
Merge branch 'master' into analyzer-execution-names 2024-01-05 11:47:30 +01:00
Nikolai Kochetov
84b566d1a8
Merge pull request #58317 from ClickHouse/analyzer-using-func-arg 
Analyzer: support functional arguments in USING clause
 2024-01-05 10:43:42 +01:00
Nikolai Kochetov
80c1e59d3f Merge branch 'master' into analyzer-execution-names 2024-01-05 09:31:25 +00:00
sunny19930321
8c8f7c83ff hive threadpool read orc failed 2024-01-05 15:58:17 +08:00
Duc Canh Le
c3586b2e35 fix docs 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-05 07:23:22 +00:00
Duc Canh Le
934e006213 add document 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-05 07:13:46 +00:00
Duc Canh Le
29c4d96b48 fix build and add a tests 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-05 06:54:56 +00:00
sunny19930321
f9b57bfb2c fix: hive threadpool read orc failed 2024-01-05 14:36:28 +08:00
Alexey Milovidov
f89bd40575 Merge branch 'master' into pretty-type-names-default 2024-01-05 00:03:13 +01:00
Robert Schulze
2166df0640
Merge pull request #57882 from lingtaolf/optimization/BF_support_rg 
Utilize `ngrambf` and `tokenbf` indexes by function `match()`
 2024-01-05 00:00:50 +01:00
Robert Schulze
2928d8742e
Merge pull request #58517 from rschu1ze/minor-sqids-fixup 
Minor fixups for `sqid()`
 2024-01-04 23:12:14 +01:00
Vitaly Baranov
2a385bc573 Fix currentProfiles() 2024-01-04 21:06:45 +01:00
Nikolai Kochetov
4ab6bcbd25
Merge pull request #58353 from ClickHouse/hdfs-virtuals 
Refactor StorageHDFS and StorageFile virtual columns filtering
 2024-01-04 20:11:41 +01:00
Raúl Marín
0522d859c2
Merge pull request #58334 from Algunenano/minmax_non_numeric 
Speedup MIN/MAX for non numeric types
 2024-01-04 19:48:18 +01:00
Nikolai Kochetov
85c1bb80fd Merge branch 'master' into hdfs-virtuals 2024-01-04 18:00:10 +00:00
Jihyuk Bok
491df7bf6e enable ordinary databases while restoration 2024-01-04 18:46:55 +01:00
Nikolai Kochetov
85ea5c1f82 Remove more code 2024-01-04 17:46:06 +00:00
Nikolai Kochetov
085484c5f1
Merge pull request #58505 from kitaisreal/merge-tree-prefetched-read-pool-disable-for-limit-only-queries 
MergeTreePrefetchedReadPool disable for LIMIT only queries
 2024-01-04 18:39:25 +01:00
Robert Schulze
f276a7f9d4
Merge remote-tracking branch 'ClickHouse/master' into optimization/BF_support_rg 2024-01-04 17:15:17 +00:00
Alexander Tokmakov
9f5015737b fix a stupid case of intersecting parts 2024-01-04 17:45:42 +01:00
Robert Schulze
76b7cddb18
Update docs 2024-01-04 16:29:43 +00:00
Robert Schulze
98d602c3d5
Reserve enough space in result column upfront 2024-01-04 16:21:08 +00:00
Robert Schulze
03e344c36a
Fix preprocessor guard 2024-01-04 16:15:51 +00:00
Robert Schulze
296e1ac8aa
FunctionSqid.cpp --> sqid.cpp 2024-01-04 16:11:39 +00:00
Dmitry Novik
39b15f9130
Add a comment 2024-01-04 16:33:52 +01:00
Nikolai Kochetov
d7883eeaf4 Fix some tests. 2024-01-04 15:15:28 +00:00
Alexander Tokmakov
d2752f4b66
Merge pull request #57603 from evillique/allow-not-resolving-hostnames 
Allow avoiding resolving hostnames in DDLWorker
 2024-01-04 15:46:28 +01:00
Nikolai Kochetov
494a32f4e4 Review fixes 2024-01-04 14:41:04 +00:00
Kruglov Pavel
7e6e835e2e
Merge pull request #57520 from Avogar/ignore-mv-with-dropped-target-table 
Ignore MVs with dropped target table during pushing to views
 2024-01-04 15:33:27 +01:00
Maksim Kita
d9f68f4a2c Fixed tests 2024-01-04 17:16:47 +03:00
Blargian
459946035c #58363 🚧 modified Pretty in FormatSettings.h to have PrettyColor which can be 0,1 or auto. modified output_format_pretty_color in FormatFactory.cpp to make use of this, added the default to Settings.h. Implemented the logic for enabling/disabling based on output_format_pretty_color in PrettyBlockOutputFormat.h 2024-01-04 16:10:36 +02:00
Raúl Marín
5bfddfebb6 Fix instantiation detection 2024-01-04 14:08:58 +00:00
Alexander Tokmakov
972756486a
Merge pull request #58350 from ClickHouse/rdb_dont_wait_inactive 
An option to avoid waiting for inactive Replicated database replicas
 2024-01-04 14:50:11 +01:00
Ilya Yatsishin
2593a566eb
Merge pull request #58453 from canhld94/fix_graphite_segfault 
Fix segfault when graphite table does not have agg function
 2024-01-04 14:44:11 +01:00
Raúl Marín
971d030ec5
Merge pull request #58271 from Algunenano/speedup_numbers 
Speed up iota (filling array with consecutive numbers)
 2024-01-04 14:17:42 +01:00
Alexander Tokmakov
74fb390444 fix build 2024-01-04 13:36:42 +01:00
Maksim Kita
8573c66b09 Fixed code review issues 2024-01-04 15:29:25 +03:00
Alexander Tokmakov
10e3ae9e68 Merge branch 'master' into allow-not-resolving-hostnames 2024-01-04 13:22:15 +01:00
Maksim Kita
b5997e6a96 MergeTreePrefetchedReadPool disable for LIMIT only queries 2024-01-04 15:06:38 +03:00
Raúl Marín
b7f77d98c8 Merge remote-tracking branch 'blessed/master' into minmax_non_numeric 2024-01-04 11:40:49 +00:00
Duc Canh Le
2e14cfb526 add settings for output compression level and window size 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-04 08:16:00 +00:00
Antonio Andelic
13749f550f Lower log levels for some Raft logs 2024-01-04 07:59:44 +00:00
Antonio Andelic
8f26e2af67 Even better Keeper startup 2024-01-04 07:54:14 +00:00
Duc Canh Le
4e35e760e4 Merge branch 'master' into fix_graphite_segfault 
Fix flaky tests
 2024-01-04 01:39:26 +00:00
Alexey Milovidov
43d7e07118 Merge branch 'pretty-type-names-default' of github.com:ClickHouse/ClickHouse into pretty-type-names-default 2024-01-04 00:39:52 +01:00
Alexey Milovidov
0a4a78f0f2 Merge branch 'master' into pretty-type-names-default 2024-01-04 00:31:47 +01:00
Nikita Taranov
66d2db5283
New parallel replicas coordinator implementation (#57968) 2024-01-04 00:27:04 +01:00
Bhavna Jindal
c2ebb9a5d9 fix for const column input 2024-01-03 13:05:52 -08:00
Anton Popov
48a3968d5a fix query info with projection 2024-01-03 20:06:36 +00:00
Nikolai Kochetov
950140cbf5 Fixing build. 2024-01-03 18:43:52 +00:00
Robert Schulze
f652c6050c
Implement idnaEn/Decode() 2024-01-03 18:08:32 +00:00
Nikolai Kochetov
7a271f09ed Check if I can remove KeyCondition analysis on AST. 2024-01-03 17:50:46 +00:00
Nikolai Kochetov
d06de83ac1 Fix KeyCondition for file/url/s3 2024-01-03 17:44:28 +00:00
Bhavna Jindal
62521d30ce fix style-check 2024-01-03 09:34:58 -08:00
Bhavna Jindal
8a7eaed07e Added additional testcases 2024-01-03 09:16:32 -08:00
János Benjamin Antal
81b5d8fddd Delay reading from StorageKafka by creating a query plan step 2024-01-03 17:13:41 +00:00
Anton Popov
cf4604bfb6 fix build and tests 2024-01-03 16:59:13 +00:00
Yakov Olkhovskiy
85f03478ef
Revert "Revert "Use CH Buffer for HTTP out stream, add metrics for interfaces"" 2024-01-03 11:47:15 -05:00
Bharat Nallan Chakravarthy
d63765fb19 Merge upstream/master into ncb/support-hints-for-db-engines 2024-01-03 08:44:11 -08:00
Dmitry Novik
558d103f96 Apply optimization only in case of type match 2024-01-03 16:18:00 +00:00
Alexander Tokmakov
b1688b1953 Merge branch 'master' into rdb_dont_wait_inactive 2024-01-03 17:03:01 +01:00
Anton Popov
ce1b978a2d Merge remote-tracking branch 'upstream/master' into HEAD 2024-01-03 16:01:44 +00:00
Anton Popov
7dd128f90f Revert "remove projection from StorageSnapshot" 
This reverts commit a01acf5d2a.
 2024-01-03 16:00:50 +00:00
Anton Popov
6b6f395cc7
Merge pull request #57631 from Avogar/cache-for-reading-compact-subcolumns 
Read column once while reading more that one subcolumn from it in Compact parts
 2024-01-03 16:31:54 +01:00
Nikolai Kochetov
a3a24bdb40 Merge branch 'master' of github.com:ClickHouse/ClickHouse into hdfs-virtuals 2024-01-03 15:10:57 +00:00
Nikolai Kochetov
7834519212
Merge pull request #58255 from ClickHouse/filter-virtual-columns-storage-merge 
Refactor StorageMerge virtual columns filtering.
 2024-01-03 16:09:49 +01:00
Robert Schulze
8d4b519bb1
Replace std::regex by re2 2024-01-03 15:06:20 +00:00
Val Doroshchuk
5c221d123d MaterializedMySQL: Introduce MySQL Binlog Client 
One binlog connection for many databases.

Suggesting to disable this feature by default for now. It should be explicitly enabled by SETTINGS use_binlog_client=1.
But if you would permanently enable it in MaterializedMySQLSettings, it should keep old behavior and all tests should pass too.

1. Introduced `IBinlog` and its impl to read the binlog events from socket - `BinlogFromSocket`, or file - `BinlogFromFile`. Based on prev impl of `EventBase` and the same old binlog parsers. It fully keeps BC with old version. Fixed `./check-mysql-binlog` to test new impl.
2. Introduced `BinlogEventsDispatcher`, it reads the event from the source `IBinlog` and sends it to currently attached `IBinlog` instances.
3. Introduced `BinlogClient`, which is used to group a list of `BinlogEventsDispatcher` by MySQL binlog connection which is defined by `user:password@host:port`. All dispatchers with the same binlog position should be merged to one.
4. Introduced `BinlogClientFactory`, which is a singleton and it is used to track all binlogs created over the instance.
5. Introduced `use_binlog_client` setting to `MaterializedMySQL`, which forces to reuse a `BinlogClient` if it already exists in `BinlogClientCatalog` or create new one. By default, it is disabled.
6. Introduced `max_bytes_in_binlog_queue` setting to define the limit of bytes in binlog's queue of events. If bytes in the queue increases this limit, `BinlogEventsDispatcher` will stop reading new events from source `IBinlog` until the space for new events will be freed.
7. Introduced `max_milliseconds_to_wait_in_binlog_queue` setting to define max ms to wait when the max bytes exceeded.
7. Introduced `max_milliseconds_to_wait_in_binlog_queue` setting to define max ms to wait when the max bytes exceeded.
8. Introduced `max_bytes_in_binlog_dispatcher_buffer` setting to define max bytes in the binlog dispatcher's buffer before it is flushed to attached binlogs.
9. Introduced `max_flush_milliseconds_in_binlog_dispatcher` setting to define max milliseconds in the binlog dispatcher's buffer to wait before it is flushed to attached binlogs.
10. Introduced `system.mysql_binlogs` system table, which shows a list of active binlogs.
11. Introduced `UnparsedRowsEvent` and `MYSQL_UNPARSED_ROWS_EVENT`, which defines that an event is not parsed and should be explicitly parsed later.
12. Fixed bug when not possible to apply DDL since syntax error or unsupported SQL.

@larspars is the author of following:
`GTIDSets::contains()`
`ReplicationHelper`
`shouldReconnectOnException()`
 2024-01-03 15:26:09 +01:00
Dmitry Novik
5308e24b8c Another fixup + reference update 2024-01-03 14:12:56 +00:00
Dmitry Novik
1c3364046e Fixup 2024-01-03 13:54:17 +00:00
Nikolai Kochetov
9dc1f4d99c
Update StorageMerge.cpp 2024-01-03 14:53:40 +01:00
Kruglov Pavel
570d1c013b
Merge pull request #57053 from Avogar/better-parsing-exceptions 
Better exception messages in input formats
 2024-01-03 14:47:51 +01:00
Kruglov Pavel
c8acc7c2d1
Fix build 2024-01-03 14:44:00 +01:00
Raúl Marín
ff90f64bc1 Merge remote-tracking branch 'blessed/master' into speedup_numbers 2024-01-03 13:33:22 +00:00
Raúl Marín
1c40700ea1 Merge remote-tracking branch 'blessed/master' into minmax_non_numeric 2024-01-03 14:09:28 +01:00
Alexey Milovidov
fe97671d42
Merge pull request #58440 from ClickHouse/kitaisreal-enable-jit-for-aggregation-without-key 
Merging #53757
 2024-01-03 14:05:50 +01:00
Raúl Marín
1f960a32de Fix OSX build 2024-01-03 14:04:30 +01:00
Alexey Milovidov
74c3b0dff1
Update src/DataTypes/DataTypeTuple.cpp 
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
 2024-01-03 15:55:27 +03:00
Duc Canh Le
be825b1290 fix segfault when graphite table does not have agg function 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-03 12:20:08 +00:00
Nikolai Kochetov
aee933a437 Merge branch 'master' into hdfs-virtuals 2024-01-03 12:16:57 +00:00
Robert Schulze
9a49f0cbb3
Factorize common code 2024-01-03 11:26:52 +00:00
Dmitry Novik
76f58fb49e Add a comment 2024-01-03 11:17:24 +00:00
Dmitry Novik
cb4d571a45 Support ALIAS columns in USING clause 2024-01-03 11:14:19 +00:00
Raúl Marín
91fc3b3456 Linter 2024-01-03 10:54:42 +00:00
Robert Schulze
b5d7ea2b02
Introduce OrNull() overloads for punycodeEncode/Decode() 2024-01-03 10:50:00 +00:00
Nikolai Kochetov
d5dcb6661d Review fixes. 2024-01-03 10:12:08 +00:00
Nikolai Kochetov
78776a060c
Update src/Storages/StorageMerge.cpp 
Co-authored-by: Dmitry Novik <n0vik@clickhouse.com>
 2024-01-03 11:06:57 +01:00
Nikolai Kochetov
eeed23b1bc Fix sanitizer assert. 2024-01-03 09:45:25 +00:00
Raúl Marín
d491758939
Revert "Use CH Buffer for HTTP out stream, add metrics for interfaces" 2024-01-03 10:42:15 +01:00
Duc Canh Le
12fda5f309 fix 02771_multidirectory_globs_storage_file 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-03 09:22:29 +00:00
Robert Schulze
45613f1d14
Refactor punycodeEncode/Decode() functions to inerit from IFunction directly 2024-01-03 09:18:25 +00:00
Robert Schulze
2186aa8f21
Revert "Revert "Implement punycode encoding/decoding"" 
This reverts commit 345d29a3c1.
 2024-01-03 08:03:01 +00:00
Duc Canh Le
3502245ecb use fs::absolute instead of fs::canonical 
To allow users to work with symlinks in user_files_path

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-03 04:58:26 +00:00
Alexey Milovidov
7c5e329cb7 Check what will happen if we only indent named tuples 2024-01-02 23:27:25 +01:00
Alexey Milovidov
7392919909 Merge branch 'master' into pretty-type-names-default 2024-01-02 22:57:26 +01:00
Alexey Milovidov
eb930efebb
Merge pull request #57732 from ClickHouse/load-metadata-threads 
Increase `load_metadata_threads` to 16
 2024-01-02 22:26:18 +01:00
Alexey Milovidov
8af3b9db2d Merge branch 'enable-jit-for-aggregation-without-key' of github.com:kitaisreal/ClickHouse into kitaisreal-enable-jit-for-aggregation-without-key 2024-01-02 22:01:59 +01:00
Alexey Milovidov
44432b1ffc Fix build 2024-01-02 21:54:13 +01:00
Alexey Milovidov
deac0b5f3e Merge branch 'master' into insert-quorum-host-name-check 2024-01-02 21:46:12 +01:00
avogar
488e36de46 More fixes 2024-01-02 20:20:39 +00:00
Alexey Milovidov
029b1326e0
Merge pull request #58399 from ClickHouse/minor-fixes-lambda 
Fix a comment; remove unused method; stop using pointers
 2024-01-02 19:21:00 +01:00
Nikolai Kochetov
9c25cb6692 Cleanup 2024-01-02 18:08:04 +00:00
Nikolai Kochetov
4f99a8bc1f Remove more unused code. 2024-01-02 17:54:20 +00:00
Nikolai Kochetov
1b20ce5162 Cleanup 2024-01-02 17:50:06 +00:00
Nikolai Kochetov
c808b03e55 Remove unneeded code 2024-01-02 17:27:33 +00:00
Nikolai Kochetov
8936c8376a Use predicate in getTaskIteratorExtension. 2024-01-02 17:14:16 +00:00
Bhavna Jindal
c4ebc75209 remove stl-cpp submodule 2024-01-02 08:53:35 -08:00
Bhavna Jindal
d4eb8c7352 handled NaN and inf values 2024-01-02 08:52:28 -08:00
Bhavna Jindal
c0cff7b4f3 handled edge cases and addresses review comments 2024-01-02 08:52:28 -08:00
Bhavna Jindal
6d58c99408 fixed the failing fasttest and style-check 2024-01-02 08:52:28 -08:00
Bhavna Jindal
5c34ddab09 implemneted seriesDecomposeSTL method 2024-01-02 08:52:23 -08:00
Bhavna Jindal
a407cd270d Added new library stl-cpp for sesonal decompose of time series data 2024-01-02 08:48:10 -08:00
avogar
c5ba97f0cd Fix tests 2024-01-02 16:38:45 +00:00
Kruglov Pavel
f2dfe8bdda
Fix build 2024-01-02 16:42:17 +01:00
Nikolai Kochetov
3e3fed1cbe Add reading step to URL 2024-01-02 15:18:13 +00:00
Yakov Olkhovskiy
5633fb8145
Merge pull request #56064 from ClickHouse/feature-server-iface-metrics 
Use CH Buffer for HTTP out stream, add metrics for interfaces
 2024-01-02 10:17:52 -05:00
Kruglov Pavel
e69bda7f08
Merge pull request #57438 from Avogar/fix-resetting-buffers 
Fix working with read buffers in StreamingFormatExecutor
 2024-01-02 15:34:35 +01:00
Alexander Tokmakov
d3d5976d3e fix 2024-01-02 15:13:25 +01:00
Nikolai Kochetov
0f76967f97 Add reading step to Azure. 2024-01-02 13:46:08 +00:00
Alexander Tokmakov
d13abac912
Merge pull request #58333 from ClickHouse/fix_no_such_key_detached_part 
Fix lost blobs after dropping a replica with broken detached parts
 2024-01-02 14:21:55 +01:00
Nikolai Kochetov
b7cc6d4615 Fixing tests. 2024-01-02 13:08:04 +00:00
kevinyhzou
733e68cfd5 review fix 2024-01-02 17:47:45 +08:00
Duc Canh Le
d623702378 Merge branch 'master' into final_no_copy 
Resolve conflicts

Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-02 06:07:49 +00:00
Bharat Nallan Chakravarthy
6e2c4f04aa support hints for database engine 2024-01-01 21:31:22 -08:00
Alexey Milovidov
6a4409200e DatabaseReplicated: too much success 2024-01-01 22:01:25 +01:00
robot-ch-test-poll4
657da6b55e
Merge pull request #58398 from ClickHouse/bad-formatting-timediff 
Fix bad formatting of the `timeDiff` compatibility alias
 2024-01-01 21:37:06 +01:00
Alexey Milovidov
7ee8708fa6 Remove useless headers 2024-01-01 19:29:51 +01:00
Alexey Milovidov
e4444e1db4 More minor changes 2024-01-01 19:17:04 +01:00
Alexey Milovidov
4172057a4e Fix a comment; remove unused method; stop using pointers 2024-01-01 18:46:38 +01:00
Alexey Milovidov
974cf132d4 Fix bad formatting of the timeDiff compatibility alias 2024-01-01 17:45:23 +01:00
Bharat Nallan Chakravarthy
de821fca8a remove leftover cruft 2023-12-31 21:31:23 -08:00
Bharat Nallan Chakravarthy
1e0ccf917c make getImpl private 2023-12-31 20:36:18 -08:00
Bharat Nallan Chakravarthy
c342688821 fail even earlier for invalid engine 2023-12-31 20:36:18 -08:00
Bharat Nallan Chakravarthy
f16ae027fc fix validation of database engines 2023-12-31 17:45:03 -08:00
Yakov Olkhovskiy
db97764e98 fix tests, some refactoring 2023-12-31 12:56:37 +00:00
Bharat Nallan Chakravarthy
00b5e28d6a Merge upstream/master into ncb/system-database-engines 2023-12-30 23:10:48 -08:00
Bharat Nallan Chakravarthy
3e8439a78b add to context access and suggest 2023-12-30 20:38:35 -08:00
Bharat Nallan Chakravarthy
3c4ac0ba08 add system.database_engines table 2023-12-30 20:31:25 -08:00
Alexey Milovidov
7bded0a5e7
Merge pull request #58316 from ClickHouse/reintroduce_is_deleted 
Re-introduce `is_deleted` column for ReplacingMergeTree
 2023-12-31 00:57:19 +01:00
Alexey Milovidov
b752a1c62e
Merge pull request #58365 from bharatnc/ncb/refactor-db-factory 
independent registration of database engines
 2023-12-31 00:56:24 +01:00
Alexey Milovidov
2f3e17c737 Remove commented code 2023-12-30 18:39:12 +01:00
Alexey Milovidov
68cc3eaa96 Merge branch 'master' of github.com:ClickHouse/ClickHouse into feature-server-iface-metrics 2023-12-30 18:38:45 +01:00
Alexey Milovidov
a1bd76388e Remove commented code 2023-12-30 18:38:41 +01:00
Bharat Nallan Chakravarthy
6c4a4991e2 few more fixes 2023-12-30 09:26:53 -08:00
Alexey Milovidov
c1c7218d53
Merge pull request #58369 from ClickHouse/remove-function 
Remove function `reverseDNSQuery`
 2023-12-30 17:51:56 +01:00
Alexey Milovidov
063463b0a8
Merge pull request #58359 from ClickHouse/all-system-tables-local 
Attach all system tables in `clickhouse-local`
 2023-12-30 17:51:27 +01:00
Alexey Milovidov
01c7562cc3 Merge with master 2023-12-30 16:01:17 +01:00
Alexey Milovidov
aad63c8273 Merge branch 'master' into feature-server-iface-metrics 2023-12-30 15:44:42 +01:00
Alexey Milovidov
78523639e0 Remove function 2023-12-30 14:52:54 +01:00
Antonio Andelic
d86e59eb08
Merge pull request #58308 from ClickHouse/keeper-preprocess-fix 
Handle another case for preprocessing in Keeper
 2023-12-30 14:13:04 +01:00
Alexey Milovidov
c9467d8d47
Update ReplicatedMergeTreeSink.cpp 2023-12-30 15:19:50 +03:00
Alexey Milovidov
9dfca3ea45
Update ZooKeeperWithFaultInjection.h 2023-12-30 15:16:02 +03:00
Alexey Milovidov
5300e80e7c
Update ZooKeeperWithFaultInjection.h 2023-12-30 15:14:44 +03:00
Alexey Milovidov
2421173574
Merge branch 'master' into insert-quorum-host-name-check 2023-12-30 13:13:24 +01:00
Alexey Milovidov
8fc05e25fe
Merge pull request #58310 from azat/kafka-fix-stat-leak-resubmit 
Create consumers for Kafka tables on fly with TTL (resubmit)
 2023-12-30 13:03:16 +01:00
Alexey Milovidov
e1812f3b58
Merge pull request #58266 from ClickHouse/vdimir/simple_fix_tuple_elimination 
Analyzer: fix tuple comparison when result is always null
 2023-12-30 13:02:38 +01:00
Alexey Milovidov
8365722c95
Merge pull request #58336 from amosbird/macro2constexpr 
Organize symbols from src/* into DB namespace
 2023-12-30 12:40:36 +01:00
Alexey Milovidov
aa6ecd2d59
Merge pull request #58343 from azat/s3/optional-gcs-compose 
Avoid sending ComposeObject requests after upload to GCS
 2023-12-30 12:40:04 +01:00
Alexey Milovidov
40ca9c202d
Merge pull request #58346 from ClickHouse/check-what-would-be-ifremove-array-joined-columns-from-key-condition 
Check what happen if remove array joined columns from KeyCondition
 2023-12-30 12:38:57 +01:00
Alexey Milovidov
f058394d92
Merge pull request #58351 from ClickHouse/fix_00002 
Keep exception format string in retries ctl
 2023-12-30 12:37:36 +01:00
Alexey Milovidov
ebd95586d2
Merge pull request #58358 from ClickHouse/format-constexpr 
Make function `format` return a constant on constant arguments.
 2023-12-30 12:34:56 +01:00
Alexey Milovidov
1d344026be
Merge pull request #58360 from ClickHouse/enum-transform 
Support for Enum in function `transform`
 2023-12-30 12:32:49 +01:00
Bharat Nallan Chakravarthy
906745feaa validation and left over fixes 2023-12-29 20:06:07 -08:00
robot-ch-test-poll3
c0290d1cfd
Merge pull request #58354 from azat/xml-conf-dot 
[RFC] Correctly handle keys with dot in the name in configurations XMLs
 2023-12-30 03:59:16 +01:00
Amos Bird
907e844635
Fix examples 2023-12-30 10:48:19 +08:00
Bharat Nallan Chakravarthy
af534db4d9 style fixes and cleanup 2023-12-29 16:38:17 -08:00
Bharat Nallan Chakravarthy
ab28da85d3 try wire everything up 2023-12-29 16:38:17 -08:00
Bharat Nallan Chakravarthy
8e83d3604d register all remaining databases 2023-12-29 16:38:02 -08:00
Alexey Milovidov
aa462fa7cd
Merge branch 'master' into feature-server-iface-metrics 2023-12-29 22:37:52 +01:00
Alexey Milovidov
e1eca1ee39 Support for Enum in function transform 2023-12-29 21:48:38 +01:00
Alexey Milovidov
7c2d868ef1 Revert "Support for Enum in function transform" 
This reverts commit 06628f76d7.
 2023-12-29 21:48:14 +01:00
Alexey Milovidov
06628f76d7 Support for Enum in function transform 2023-12-29 21:47:53 +01:00
Alexey Milovidov
39b239683c Attach all system tables in clickhouse-local 2023-12-29 21:25:22 +01:00
Alexey Milovidov
9d8c26331e Make format constexpr 2023-12-29 21:09:58 +01:00
Nikolay Degterinsky
47c3696a46
Fix build 2023-12-29 20:41:33 +01:00
kssenii
653d48a552 Flexible disks configuration 2023-12-29 19:57:58 +01:00
avogar
51ca7f6e0a Fix test 2023-12-29 17:58:01 +00:00
Nikolai Kochetov
b95bdef09e Update StorageS3 and StorageS3Cluster 2023-12-29 17:41:11 +00:00
Amos Bird
18395a387c
Fix another unitest 2023-12-30 01:14:21 +08:00
Raúl Marín
5fb7f9f861 Ignore other numeric types for now 2023-12-29 18:09:20 +01:00
Kruglov Pavel
c03e36e012
Merge branch 'master' into better-parsing-exceptions 2023-12-29 18:07:32 +01:00
Alexander Tokmakov
790ededf80
Update BackupCoordinationStageSync.cpp 2023-12-29 17:58:24 +01:00
Azat Khuzhin
9a8b308a3d Correctly handle keys with dot in the name in configurations XMLs 
For this I've added escape of the keys returned by keys() method, and
handle this escaping in get*() methods.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 17:30:23 +01:00
Raúl Marín
22ef5443bb Move findNumeric to .cpp 2023-12-29 17:22:06 +01:00
Raúl Marín
ed6b9703a1 More iota 2023-12-29 17:20:18 +01:00
Raúl Marín
4b7fcfbc75 Use iota in more places 2023-12-29 17:20:18 +01:00
Alexander Tokmakov
302ddeb6d0
Update BackupCoordinationStageSync.cpp 2023-12-29 17:08:13 +01:00
Kruglov Pavel
f57939096c
Merge branch 'master' into ignore-mv-with-dropped-target-table 2023-12-29 17:02:23 +01:00
Nikolai Kochetov
5521e5d9b1 Refactor StorageHDFS and StorageFile virtual columns filtering 2023-12-29 15:58:01 +00:00
robot-ch-test-poll3
07ba672e37
Merge pull request #58142 from canhld94/final_less_compare 
MergeTree FINAL to not compare rows from same non-L0 part
 2023-12-29 16:47:14 +01:00
Azat Khuzhin
a12df35be4 Eliminate possible race between ALTER_METADATA and MERGE_PARTS 
v2: move metadata version check after checking that the part is not covering part
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 16:46:10 +01:00
Azat Khuzhin
c7fa93d704 Add infrastructure for testing replicated MergeTree queue 
- replicated_queue_fail_next_entry - to fail next queue entry
- replicated_queue_unfail_entries - to "unfail" all queue entries (if
  any)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 16:43:01 +01:00
Alexander Tokmakov
1013f6b23f
Merge branch 'master' into reintroduce_is_deleted 2023-12-29 15:46:24 +01:00
Alexander Tokmakov
72a0797b88 keep exception format string in retries ctl 2023-12-29 15:21:46 +01:00
Alexander Tokmakov
bfc10bd234 an option to avoid waiting for inactive Replicated db replicas 2023-12-29 15:19:16 +01:00
Alexey Milovidov
ea03cc82aa
Merge pull request #58320 from ClickHouse/mv3 
Refreshable materialized views again
 2023-12-29 14:44:50 +01:00
Raúl Marín
bda6104f84 Replace std::iota with DB::iota where possible 2023-12-29 14:38:22 +01:00
Azat Khuzhin
853fdfe775 Clean cached messages on destroy kafka consumer 
The callchain of the kafka consumer is very tricky, so for the sake of
common sense let's just clean the messages on moving out consumer (and
in dtor, but this is just to keep that two code path in sync).

(Also reported by @filimonov)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 14:30:21 +01:00
Sergei Trifonov
e2b43c2cfa
Update src/Databases/DatabaseOrdinary.cpp 2023-12-29 14:29:30 +01:00
Sergei Trifonov
4f15031f79
Update src/Common/AsyncLoader.cpp 2023-12-29 14:28:16 +01:00
Sergei Trifonov
d7aca766c2
Update src/Common/AsyncLoader.h 2023-12-29 14:27:01 +01:00
Azat Khuzhin
b3d6caf37f Unsubscribe kafka consumer before cleaning it by TTL 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 14:03:53 +01:00
Raúl Marín
e692b0a5bd Move iota implementation to its own file 2023-12-29 13:47:33 +01:00
Amos Bird
0d801c4f6b
Fix unitest 2023-12-29 20:40:52 +08:00
Nikolai Kochetov
0e8232a8c3 Check what happen if remove array joined columns from KeyCondition 2023-12-29 12:24:19 +00:00
serxa
62139f0073 fix recursion for finish 2023-12-29 12:19:56 +00:00
Azat Khuzhin
f578541ded Fix destructing kafka consumer via member orders 
We've discussed this with @filimonov and he pointed out that everything
else (except for rdkafka_stat/rdkafka_stat_mutex) is done via members
orders, so let's do it in the same style.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 13:19:11 +01:00
Raúl Marín
e2d0ca675f Merge remote-tracking branch 'blessed/master' into speedup_numbers 2023-12-29 13:15:39 +01:00
Alexey Milovidov
09181b6c37
Merge pull request #58335 from lingtaolf/feature/sparkBar_alias_to_sparkbar 
sparkBar as an alias to sparkbar
 2023-12-29 13:02:55 +01:00
Raúl Marín
fd438cc5b6 Merge remote-tracking branch 'blessed/master' into minmax_non_numeric 2023-12-29 12:39:29 +01:00
Azat Khuzhin
8c54380d80 Avoid sending ComposeObject requests after upload to GCS 
This should not be required anymore, but leave it as an option, since
likely this is required for old files.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
f4a7789cd4 Convert various S3::Client settings into separate ClientSettings struct 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Azat Khuzhin
2b2d10689b Make S3::Client::ComposeObject private 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-29 11:53:49 +01:00
Duc Canh Le
91a87d6b6c better implementation 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-29 07:27:10 +00:00
Amos Bird
6b6e40831c
Move symbols from src/* into namespace DB 2023-12-29 14:37:08 +08:00
Bharat Nallan Chakravarthy
1dce048b58 initial refactor of db registration 2023-12-28 18:59:45 -08:00
凌涛
4630398d23 optimize 2023-12-29 10:38:13 +08:00
凌涛
7bd8488db5 Funciont sparkBar alias sparkbar 2023-12-29 10:08:17 +08:00
kevinyhzou
842252bcad review fix 2023-12-29 09:32:31 +08:00
robot-ch-test-poll4
ef5837a008
Merge pull request #58318 from ClickHouse/fix-fuzzer-sparse 
Fixed logical error in CheckSortedTransform
 2023-12-28 23:57:01 +01:00
Raúl Marín
8a90f12dc9 Speedup MIN/MAX for non numeric types 2023-12-28 23:28:26 +01:00
Igor Nikonov
208a9193f6 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-28 21:28:36 +00:00
Igor Nikonov
8538dc041d Fix test 2023-12-28 21:28:00 +00:00
Alexander Tokmakov
852f397a97 fix lost blobs after dropping a replica with broken detached parts 2023-12-28 21:47:19 +01:00
Kruglov Pavel
fbd3f7cd59
Merge pull request #56132 from Avogar/flatten-only-true-nested 
Flatten only true Nested type if flatten_nested=1, not all Array(Tuple)
 2023-12-28 20:58:28 +01:00
avogar
c6085b3eb9 Fix working with read buffers in StreamingFormatExecutor 2023-12-28 18:47:57 +00:00
Michael Kolupaev
c4f4516a37 Fix WriteBuffer assert if refresh is cancelled at the wrong moment 2023-12-28 18:34:28 +00:00
Nikolai Kochetov
490a8bce9e Remove commented code. 2023-12-28 18:01:08 +00:00
Michael Kolupaev
ea138fe8c9 space 2023-12-28 17:56:06 +00:00
Michael Kolupaev
96c68e5aae Remove pausing, enable multithreading, kick off refresh on table creation unless the query says EMPTY 2023-12-28 17:56:06 +00:00
Michael Kolupaev
4d732cdf1e Add to system.process, improve test slightly 2023-12-28 17:56:05 +00:00
Michael Kolupaev
edd120e8be Make it experimental 2023-12-28 17:56:05 +00:00
Michael Kolupaev
f0417d0ec3 Things 2023-12-28 17:56:05 +00:00
Michael Kolupaev
0fc7535eba Fixes 2023-12-28 17:56:05 +00:00
Michael Kolupaev
609b2c216c Fix some of the CI 2023-12-28 17:56:05 +00:00
Michael Kolupaev
8b8ef41407 Documentation 2023-12-28 17:56:05 +00:00
Michael Kolupaev
64e6deb197 Slightly more things 2023-12-28 17:56:05 +00:00
Michael Kolupaev
dda0606f67 Things 2023-12-28 17:56:05 +00:00
Michael Kolupaev
98dbd105ad Overhaul timestamp arithmetic 2023-12-28 17:56:04 +00:00
Michael Kolupaev
a524e8c51e Overhaul dependencies 2023-12-28 17:56:04 +00:00
Michael Kolupaev
bd18522cad Overhaul RefreshTask 2023-12-28 17:56:04 +00:00
Michael Kolupaev
29a8edb40e Simple review comments 2023-12-28 17:56:04 +00:00
koloshmet
49367186e3 fix fix fix 2023-12-28 17:56:04 +00:00
koloshmet
fb420a160b proper tmp table cleanup 2023-12-28 17:56:04 +00:00
koloshmet
0999a6d98e proper tmp table cleanup 2023-12-28 17:56:04 +00:00
koloshmet
238741dafe fixed style 2023-12-28 17:56:04 +00:00
koloshmet
4305457883 fixed tests 2023-12-28 17:56:04 +00:00
koloshmet
c52aa984ee refreshable materialized views 2023-12-28 17:56:04 +00:00
Nikolai Kochetov
4c68716df7 Fix another test. 2023-12-28 17:51:11 +00:00
Nikolai Kochetov
d7a473e386 Fix some test. 2023-12-28 17:34:28 +00:00
Nikita Mikhaylov
e15b1c6e5f Fixed 2023-12-28 17:25:27 +00:00
Alexander Tokmakov
95e4b0002f fix a bug in PartsSplitter 2023-12-28 17:25:36 +01:00
avogar
e66701dd10 Add setting ignore_materialized_views_with_dropped_target_table 2023-12-28 15:00:39 +00:00
Nikolai Kochetov
50e9c9bb4e Fixing tests. 2023-12-28 14:59:33 +00:00
Azat Khuzhin
ecf7188d52 Fix use-after-free in KafkaConsumer due to statistics callback 
CI founds [1]:

    Exception: Sanitizer assert found for instance �=================================================================
    ==1==ERROR: AddressSanitizer: heap-use-after-free on address 0x5250006a4100 at pc 0x55d4ed46d2e2 bp 0x7f7e33b40190 sp 0x7f7e33b3f950
    WRITE of size 5390 at 0x5250006a4100 thread T2 (TCPHandler)
       8 0x55d50eba9497 in DB::KafkaConsumer::setRDKafkaStat(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./src/Storages/Kafka/KafkaConsumer.h:117:22
       12 0x55d51e0eebfe in cppkafka::stats_callback_proxy(rd_kafka_s*, char*, unsigned long, void*) build_docker/./contrib/cppkafka/src/configuration.cpp:92:5
       13 0x55d51e151e3d in rd_kafka_poll_cb build_docker/./contrib/librdkafka/src/rdkafka.c:3790:7
       14 0x55d51e15531b in rd_kafka_consumer_close build_docker/./contrib/librdkafka/src/rdkafka.c:3200:31
       15 0x55d51e0f3241 in cppkafka::Consumer::close() build_docker/./contrib/cppkafka/src/consumer.cpp:293:33
       16 0x55d51e0f3241 in cppkafka::Consumer::~Consumer() build_docker/./contrib/cppkafka/src/consumer.cpp:82:9
       20 0x55d50eb8d12e in DB::KafkaConsumer::~KafkaConsumer() build_docker/./src/Storages/Kafka/KafkaConsumer.cpp:179:1

    0x5250006a4100 is located 0 bytes inside of 8736-byte region [0x5250006a4100,0x5250006a6320)
    freed by thread T2 (TCPHandler) here:
       0 0x55d4ed4a26b2 in operator delete(void*, unsigned long) (/usr/bin/clickhouse+0xa94b6b2) (BuildId: 74ec4a14a5109c41de109e82d56d8d863845144d)
       1 0x55d50eb8ca55 in void std::__1::__libcpp_operator_delete[abi:v15000]<void*, unsigned long>(void*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/new:256:3
       2 0x55d50eb8ca55 in void std::__1::__do_deallocate_handle_size[abi:v15000]<>(void*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/new:282:10
       3 0x55d50eb8ca55 in std::__1::__libcpp_deallocate[abi:v15000](void*, unsigned long, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/new:296:14
       4 0x55d50eb8ca55 in std::__1::allocator<char>::deallocate[abi:v15000](char*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator.h:128:13
       5 0x55d50eb8ca55 in std::__1::allocator_traits<std::__1::allocator<char>>::deallocate[abi:v15000](std::__1::allocator<char>&, char*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:282:13
       6 0x55d50eb8ca55 in std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::~basic_string() build_docker/./contrib/llvm-project/libcxx/include/string:2334:9
       7 0x55d50eb8ca55 in DB::KafkaConsumer::~KafkaConsumer() build_docker/./src/Storages/Kafka/KafkaConsumer.cpp:179:1

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/745d9bb47f3425e28e5660ed7c730038ffece4ee/integration_tests__asan__analyzer__%5B6_6%5D/integration_run_parallel4_0.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-28 15:48:43 +01:00
serxa
54fbb059a0 Merge branch 'master' into dependency-failure 2023-12-28 14:43:17 +00:00
Azat Khuzhin
4a14112af1 Move StorageKafka::createConsumer() into KafkaConsumer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit ebad1bf4f3)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
87f3f6619a Fix data-race between StorageKafka::startup() and cleanConsumers() 
Actually now we can create consumer object in the ctor, no need to do
this in startup(), since consumer now do not connects to kafka.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 03218202d3)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
3c139d7135 Update comment for statistics.interval.ms librdkafka option 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 1f03a21033)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
6f85306510 Use separate thread for kafka consumers cleanup 
Since pool may exceed threads, while we need to run this thread always
to avoid memory leaking.

And this should not be a problem since librdkafka has multiple threads
for each consumer (5!) anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 06a9e9a9ca)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
8ac68b64d7 Allow setThreadName() to truncate thread name instead of throw an error 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit a7453f7f14)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
7d2b82c37c Add ability to configure TTL for kafka consumers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit b19b70b8fc)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
bea1610219 Preserve KafkaConsumer objects 
This will make system.kafka_consumers more useful, since after TTL
consumer object will be removed prio this patch, but after, all
information will be preserved.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 2ff0bfb0a1)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
71fdde76c2 Enable stats for system.kafka_consumers back by default 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit db74549940)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
d66be02dc3 Create consumers for Kafka tables on fly (but keep them for 1min since last used) 
Pool of consumers created a problem for librdkafka internal statistics,
you need to read from the queue always, while in ClickHouse consumers
created regardless are there any readers or not (attached materialized
views or direct SELECTs).

Otherwise, this statistics messages got queued and never released,
which:
- creates live memory leak
- and also makes destroy very slow, due to librdkafka internals (it
  moves entries from this queue into another linked list, but in a
  with sorting, which is incredibly slow for linked lists)

So the idea is simple, let's create a pool of consumers only when they
are required, and destroy them after some timeout (right now it is 60
seconds) if nobody uses them, that way this problem should gone.

This should also reduce number of internal librdkafka threads, when
nobody reads from Kafka tables.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit e7592c140e)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
a6841c8915 Properly set shutdown_called in StorageKafka::shutdown() 
Fixes: https://github.com/ClickHouse/ClickHouse/pull/42777
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 51d4f583e6)
 2023-12-28 15:32:39 +01:00
Azat Khuzhin
3541d9a05f Remove StorageKafka::num_created_consumers (in favor of all_consumers.size()) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit 123d63e824)
 2023-12-28 15:32:39 +01:00
avogar
146de5b220 Ignore MVs with dropped target table during pushing to views 2023-12-28 13:56:57 +00:00
avogar
e1a9baa5b0 Fix 2023-12-28 13:51:37 +00:00
Nikolai Kochetov
737563296b
Merge branch 'master' into filter-virtual-columns-storage-merge 2023-12-28 14:47:41 +01:00
Antonio Andelic
8ace9978b3 Handle another case for preprocessing 2023-12-28 13:20:47 +00:00
Alexander Tokmakov
bdada351c8 Revert "Merge pull request #58274 from ClickHouse/revert-58267" 
This reverts commit 583b9637c2, reversing
changes made to 224e937620.
 2023-12-28 14:07:59 +01:00
Alexander Tokmakov
5fcbf9cfb0 Revert "Merge pull request #58251 from ClickHouse/reintroduce-compatibility-with-a-misfeature" 
This reverts commit a811d5b761, reversing
changes made to 583b9637c2.
 2023-12-28 14:06:56 +01:00
Alexander Tokmakov
38fe70c68a
Revert "Refreshable materialized views (takeover)" 2023-12-28 13:12:20 +01:00
Duc Canh Le
238c5e66d5 use ChunkInfo to carry part level 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-28 11:01:18 +00:00
Raúl Marín
2fdb56bb4b Merge remote-tracking branch 'blessed/master' into speedup_numbers 2023-12-28 11:26:16 +01:00
Alexey Milovidov
4bb8592434 Update autogenerated version to 23.13.1.1 and contributors 2023-12-28 11:22:16 +01:00
凌涛
4bd6cb9cc9 Inverted Index support function match 2023-12-28 17:49:43 +08:00
Alexey Milovidov
a2faa65b08
Merge pull request #56946 from ClickHouse/mv 
Refreshable materialized views (takeover)
 2023-12-28 09:29:44 +01:00
kevinyhzou
41cdedbf26 review 2023-12-28 16:16:54 +08:00
Alexey Milovidov
345d29a3c1
Revert "Implement punycode encoding/decoding" 2023-12-28 09:12:22 +03:00
Duc Canh Le
e622cf9ee9 address review comments 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-12-28 05:01:55 +00:00
Alexey Milovidov
524d53199d
Merge branch 'master' into mv 2023-12-28 04:11:48 +01:00
Alexey Milovidov
a811d5b761
Merge pull request #58251 from ClickHouse/reintroduce-compatibility-with-a-misfeature 
Reintroduce compatibility with `is_deleted` on a syntax level
 2023-12-28 04:11:04 +01:00
Alexey Milovidov
c7efd2afea Revert #58267 2023-12-28 04:09:33 +01:00
Alexey Milovidov
40a5dbdeba
Merge branch 'master' into mv 2023-12-28 03:16:27 +01:00
Alexey Milovidov
c52886eb81
Revert "Create consumers for Kafka tables on fly (but keep them for some period since last used)" 2023-12-28 03:35:57 +03:00
Alexey Milovidov
1d9dbfd18b
Merge pull request #49103 from ClickHouse/check-about-global-sorting 
Fixed a sorting order breakage in TTL GROUP BY
 2023-12-28 01:35:14 +01:00
Alexey Milovidov
30f6d03b37
Merge pull request #57977 from bianpengyuan/add-s3-read-only-setting 
Add a setting to specify s3 disk is read only.
 2023-12-28 01:24:50 +01:00
Alexey Milovidov
e7c987058c
Merge pull request #58250 from Avogar/json-compact-each-row-parallel-parsing 
Fix parallel parsing for JSONCompactEachRow
 2023-12-28 01:23:30 +01:00
robot-ch-test-poll2
04178a9028
Merge pull request #58231 from Algunenano/new_minmax 
Speedup MIN and MAX for native types
 2023-12-28 01:22:38 +01:00
Alexey Milovidov
38d050060e
Merge branch 'master' into merge-tree-final-extract-non-intersecting-parts-ranges 2023-12-28 01:18:18 +01:00
Raúl Marín
87eb18eb74 Speed up numbers table function 2023-12-28 01:02:03 +01:00
Alexey Milovidov
d7a35773c1
Merge pull request #58252 from Algunenano/i51543 
Avoid throwing ABORTED on normal situations
 2023-12-28 00:28:51 +01:00
Alexey Milovidov
8d984df135
Merge pull request #58237 from azat/build/fwd-decl-exception 
Some code refactoring (was an attempt to improve build time, but failed)
 2023-12-28 00:21:09 +01:00
Alexey Milovidov
c024dc9c3d
Merge pull request #58265 from ClickHouse/remove-mayBenefitFromIndexForIn 
Remove mayBenefitFromIndexForIn
 2023-12-28 00:15:04 +01:00
Alexey Milovidov
dcbd3b9c26
Merge pull request #58267 from ClickHouse/fix_is_deleted_compatibility 
Re-introduce `is_deleted` column for ReplacingMergeTree
 2023-12-28 00:13:01 +01:00
Alexey Milovidov
d005193815
Merge pull request #58269 from ClickHouse/tavplubix-patch-9 
Allow a few retries when committing a part during shutdown
 2023-12-28 00:12:02 +01:00
Alexey Milovidov
5406c79c89
Merge pull request #58264 from ClickHouse/fs-cache-better-multi-cache-configuration 
Make fs caches with the same path use the same cache objects
 2023-12-28 00:10:15 +01:00
Michael Kolupaev
4d4d8e0545 space 2023-12-27 20:25:35 +00:00
Michael Kolupaev
b9cbecb0df Remove pausing, enable multithreading, kick off refresh on table creation unless the query says EMPTY 2023-12-27 20:24:56 +00:00
Michael Kolupaev
de8567660c Add to system.process, improve test slightly 2023-12-27 20:24:55 +00:00
Michael Kolupaev
18cc5d1b76 Make it experimental 2023-12-27 20:24:55 +00:00
Michael Kolupaev
538b23d862 Things 2023-12-27 20:24:55 +00:00
Michael Kolupaev
802961f0a2 Fixes 2023-12-27 20:24:55 +00:00
Michael Kolupaev
673743e2ac Fix some of the CI 2023-12-27 20:24:55 +00:00
Michael Kolupaev
7786b12a89 Documentation 2023-12-27 20:24:55 +00:00
Michael Kolupaev
418423a304 Slightly more things 2023-12-27 20:24:55 +00:00
Michael Kolupaev
ef4cc5ec7f Things 2023-12-27 20:24:55 +00:00
Michael Kolupaev
a7c369e14f Overhaul timestamp arithmetic 2023-12-27 20:24:55 +00:00
Michael Kolupaev
01369a0a8a Overhaul dependencies 2023-12-27 20:24:54 +00:00
Michael Kolupaev
01345981e2 Overhaul RefreshTask 2023-12-27 20:24:54 +00:00
Michael Kolupaev
5dc04a13a7 Simple review comments 2023-12-27 20:24:54 +00:00
koloshmet
808cb0fa05 fix fix fix 2023-12-27 20:24:54 +00:00
koloshmet
f1161566b4 proper tmp table cleanup 2023-12-27 20:24:54 +00:00
koloshmet
f14114dafc proper tmp table cleanup 2023-12-27 20:24:54 +00:00
koloshmet
d1932763f3 fixed style 2023-12-27 20:24:54 +00:00
koloshmet
3556c15c51 fixed tests 2023-12-27 20:24:54 +00:00
koloshmet
c762898adb refreshable materialized views 2023-12-27 20:24:54 +00:00