Commit Graph

31068 Commits

Author SHA1 Message Date
Alexey Milovidov
bff96b0af6
Merge branch 'master' into fix-access-rights-for-describe 2022-09-05 04:34:57 +03:00
Alexey Milovidov
79e3723272 Merge branch 'master' into s3-sharding-2 2022-09-05 02:24:35 +02:00
Alexey Milovidov
d20d49f39f
Merge pull request #40958 from arenadata/ADQM-566
stylecheck renewed and version in docker set explicitely
2022-09-05 03:23:38 +03:00
alesapin
45b1f9dc41
Merge branch 'master' into stress_s3 2022-09-04 23:17:10 +02:00
Igor Nikonov
8fece1e2d2
Merge branch 'master' into sort_mode_rename 2022-09-04 21:44:33 +02:00
kssenii
48dc32faf8 Better test 2022-09-04 19:54:34 +02:00
kssenii
e2df4b1da1 Merge remote-tracking branch 'origin/fix-races-in-write-buffer-from-s3' into fix-races-in-write-buffer-from-s3 2022-09-04 19:50:19 +02:00
kssenii
52375d2eb0 Merge master 2022-09-04 19:48:04 +02:00
Vitaly Baranov
cfb250c46b Fix checking access rights for DESCRIBE TABLE url. 2022-09-04 18:58:39 +02:00
Nikita Taranov
16af4aebc8
impl (#40952) 2022-09-04 16:26:31 +02:00
Azat Khuzhin
2e85f9f0ad Remove completely processed WAL files
Previously all WAL files had been stored, though with the time of use
this can take too much space on disk, and also the startup time will be
increased.

But it is pretty easy to prune old WAL files (the one parts from which
had been completely written to disk already).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-04 14:18:14 +02:00
Kseniia Sumarokova
5f757be43d
Merge pull request #40943 from kssenii/fix-deadlock-in-s3-write-buffer
Fix deadlock in WriteBufferFromS3
2022-09-04 13:17:41 +02:00
alesapin
a67703b76b
Merge branch 'master' into stress_s3 2022-09-04 13:00:28 +02:00
Alexey Milovidov
712de53033 Sharding s3 key names 2022-09-04 04:15:45 +02:00
Alexey Milovidov
193cd1b3b2
Merge pull request #39138 from nickitat/control_block_size_in_aggregator
Control block size in aggregator
2022-09-04 04:51:00 +03:00
Alexey Milovidov
dbca269ec7
Merge pull request #40321 from azat/mem/untracked_memory_limit_increase
Remove ThreadStatus::untracked_memory_limit_increase
2022-09-04 04:42:24 +03:00
Alexey Milovidov
660c1439ec Fix build 2022-09-04 03:27:55 +02:00
Alexey Milovidov
13a129bee7 Merge branch 'master' into remove-useless-line 2022-09-04 03:24:40 +02:00
Alexey Milovidov
097221a3af
Merge pull request #40726 from amosbird/projection-fix-1
Disable projection when grouping set is used.
2022-09-04 04:23:30 +03:00
Alexey Milovidov
22f7bfdcb5
Update gtest_sensitive_data_masker.cpp 2022-09-04 03:42:58 +03:00
Alexey Milovidov
c6f7ddc5e6
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-04 03:42:24 +03:00
Azat Khuzhin
9479e2143f Do not try to load empty debug files
This will avoid CANNOT_PARSE_ELF error for builds that has empty debug
file in clickhouse-common-static-dbg package, i.e. debug build.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-02 23:23:05 +02:00
Azat Khuzhin
597197c01d Improve error messages in Elf loader
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-02 23:22:25 +02:00
Azat Khuzhin
45afaa6fb8 Fix loading external symbols
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-02 23:22:25 +02:00
Igor Nikonov
5d7fa55f36
Merge branch 'master' into sort_mode_rename 2022-09-02 23:19:04 +02:00
Yakov Olkhovskiy
fdcced8962
Merge pull request #40850 from canhld94/ch_canh_fix_set
Fix tryGetPreparedSet crash when same set expression built from different column(s)
2022-09-02 14:40:35 -04:00
kssenii
2dcf3fef94 Fix tests 2022-09-02 19:27:52 +02:00
Alexander Tokmakov
2044470895 review fixes 2022-09-02 19:18:44 +02:00
Suzy Wang
83b029e716 re arrange sensitive data masker unit test scenario order 2022-09-02 10:11:57 -07:00
Kseniia Sumarokova
1e81e0ef67
Update WriteBufferFromS3.cpp 2022-09-02 18:54:34 +02:00
Kseniia Sumarokova
7dea71c83f
Merge pull request #40884 from TKaxe/fix_removeUnneededColumn
fix bug when remove unneeded columns in subquery
2022-09-02 18:51:18 +02:00
Kseniia Sumarokova
1370bf0de7
Merge pull request #40820 from kssenii/cache-assert-directory-ownership
Create status file for cache directory
2022-09-02 18:42:11 +02:00
kssenii
239847ad16 Fix races 2022-09-02 18:27:07 +02:00
Anton Popov
dd776eb3d5 fix enumerateStreams 2022-09-02 16:26:38 +00:00
Yakov Olkhovskiy
5727517713 add variadic constructor 2022-09-02 16:22:57 +00:00
Anton Popov
464818c142 try to fix filling of missed Nested columns with multiple levels 2022-09-02 15:05:58 +00:00
Sergei Trifonov
067e712fdf
Merge branch 'master' into s3-detailed-metrics 2022-09-02 16:59:52 +02:00
Sergei Trifonov
3daf1f5c77 fix build 2022-09-02 16:57:51 +02:00
Ilya Golshtein
74561b40ae stylecheck renewed 2022-09-02 17:42:18 +03:00
Lloyd-Pottiger
5db220c7fb Merge branch 'master' of github.com:ClickHouse/ClickHouse into feat/support-read-only-for-embeddedrocksdb 2022-09-02 22:40:28 +08:00
Kruglov Pavel
77071381e4
fix build 2022-09-02 16:37:33 +02:00
kssenii
c9b512e33e Fix 2022-09-02 16:15:54 +02:00
kssenii
0db38fcbc9 Fix 2022-09-02 15:32:46 +02:00
Anton Popov
9dda9658a8 Merge remote-tracking branch 'upstream/master' into HEAD 2022-09-02 12:48:27 +00:00
Vladimir C
963c0111bf
Merge pull request #39418 from vdimir/join_and_sets
Filter joined streams for `full_sorting_join` by each other before sorting
2022-09-02 13:57:06 +02:00
alesapin
3f02995161 Better exceptions handling 2022-09-02 13:54:41 +02:00
Antonio Andelic
b816d46520
Merge branch 'master' into fix-keeper-system-path-check 2022-09-02 13:42:16 +02:00
alesapin
521e2cc22e Merge branch 'master' into stress_s3 2022-09-02 12:57:29 +02:00
alesapin
45fe9c800a
Merge pull request #40898 from georgthegreat/patch-2
Remove remaining usage of // Y_IGNORE magic comment
2022-09-02 12:24:57 +02:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage 2022-09-02 12:13:07 +02:00
Alexander Tokmakov
3582a51a72
Merge pull request #40920 from ClickHouse/fix-typos
Fix typos found by new `codespell`
2022-09-02 13:11:16 +03:00
Lloyd-Pottiger
5a6b2106b5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into feat/support-read-only-for-embeddedrocksdb 2022-09-02 18:10:00 +08:00
Alexander Tokmakov
5b6a8677ef
Merge pull request #40890 from arthurpassos/patch-1
CaresPTRResolver small safety improvement
2022-09-02 12:29:23 +03:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Antonio Andelic
ad2196155c Add test for system node modification 2022-09-02 08:14:06 +00:00
Antonio Andelic
b7eebfc626 Correctly check if the node is using system path 2022-09-02 07:47:12 +00:00
Antonio Andelic
f5fc3db0dd Small fix 2022-09-02 07:42:22 +00:00
Antonio Andelic
1a48beec30 Use server UUID 2022-09-02 07:40:39 +00:00
Antonio Andelic
dbf67f26d8 Add prefix path 2022-09-02 07:24:26 +00:00
Robert Schulze
319d8b00a7
Merge pull request #39010 from FrankChen021/tracing_context_propagation
Improve the opentelemetry tracing context propagation across threads
2022-09-02 07:56:43 +02:00
Robert Schulze
c7c00f9002
Merge pull request #40739 from ClickHouse/clang-tidy-for-headers
Enable clang-tidy for headers
2022-09-02 07:54:50 +02:00
Yakov Olkhovskiy
b6e3680d60 test with TLS 2022-09-02 04:47:04 +00:00
Duc Canh Le
65b64010a0 move out-of-bound check into set::areTypesEqual 2022-09-02 11:56:13 +08:00
Duc Canh Le
763328ef53 Merge branch 'master' into ch_canh_fix_set 2022-09-02 11:45:13 +08:00
Lloyd-Pottiger
59dccd6e49 fix test
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
2022-09-02 11:14:49 +08:00
Sergei Trifonov
29e922f05c support all profile events for disk s3 2022-09-02 02:22:52 +02:00
Yakov Olkhovskiy
e7959d74bc add composable factory and protocol 2022-09-01 22:06:54 +00:00
Kruglov Pavel
5a9d2460dd
Merge pull request #40851 from Avogar/fix-json-object-inference
Fix nested JSON Objects schema inference
2022-09-01 21:07:21 +02:00
Kruglov Pavel
7a4a65bc36
Make better exception message in schema inference 2022-09-01 20:36:08 +02:00
Kruglov Pavel
f53aa86a20
Merge pull request #40485 from arthurpassos/fix-parquet-chunked-array-deserialization
Add support for extended (chunked) arrays for Parquet format
2022-09-01 19:40:40 +02:00
Sergei Trifonov
66b5cf566c fix bugs 2022-09-01 19:02:04 +02:00
Sergei Trifonov
ac9dc53c73 Merge branch 'master' into s3-detailed-metrics 2022-09-01 18:31:02 +02:00
Sergei Trifonov
bcb6475c4a add separate s3 profile events for disk s3 2022-09-01 18:30:55 +02:00
Dmitry Novik
ddadb362cf
Merge pull request #39762 from quickhouse/betterorderbyoptimization
Fixed `Unknown identifier (aggregate-function)` exception which appears when a user tries to calculate WINDOW ORDER BY/PARTITION BY expressions over aggregate functions
2022-09-01 18:08:06 +02:00
Dmitry Novik
2ca619d40f
Merge pull request #40320 from azat/memory-tracker-fix-peak-race
Fix rare race in updating peak of memory tracker
2022-09-01 18:07:33 +02:00
Frank Chen
9d63cbe811 Merge 'origin/master' into tracing_context_propagation to resolve conflicts 2022-09-01 23:18:59 +08:00
Frank Chen
1da35a486c Disable query_span for internal queries 2022-09-01 23:16:05 +08:00
Yuriy Chernyshov
304dc30f0f
Remove remaining usage of // Y_IGNORE magic comment 2022-09-01 18:15:56 +03:00
alesapin
dbe3b777ec Fix build 2022-09-01 17:07:56 +02:00
kssenii
ec55c90f36 Fix 2022-09-01 17:04:34 +02:00
alesapin
89930c315b Merge branch 'master' into stress_s3 2022-09-01 17:03:55 +02:00
kssenii
4fa291cf59 Fix hdfs read buffer too 2022-09-01 16:46:46 +02:00
Robert Schulze
1bbbae8da7
Merge pull request #40785 from ClickHouse/remove-unnecessary-context-access-method
Minor: remove unnecessary ContextAccess::make() method
2022-09-01 16:41:06 +02:00
Antonio Andelic
9231084c57 Style fix 2022-09-01 14:40:50 +00:00
Antonio Andelic
a0c1ce5eb7 Merge branch 'master' into embeddedrocksdb-delete-update-support 2022-09-01 14:40:24 +00:00
kssenii
3134bdeb13 Fix 2022-09-01 16:35:28 +02:00
Alexander Tokmakov
04d431793b
Update src/Common/ZooKeeper/ZooKeeperImpl.cpp
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-09-01 17:12:22 +03:00
Alexander Tokmakov
36454a340d
Update src/Common/ZooKeeper/ZooKeeperArgs.cpp
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-09-01 17:12:16 +03:00
Vladimir C
12e6fc4182
Merge branch 'master' into join_and_sets 2022-09-01 14:56:14 +02:00
Vladimir C
f2cf7d7762
Merge pull request #40334 from ClickHouse/vdimir/analyze-stuck-limit
Limit number of analyze for one query, att. 2
2022-09-01 14:40:20 +02:00
Kseniia Sumarokova
c6c67a248d
Merge pull request #40792 from canhld94/ch_canh_intersect_distinct
Implement intersect + except distinct
2022-09-01 14:35:26 +02:00
Antonio Andelic
62ff61404f Improve remove recursive 2022-09-01 12:16:13 +00:00
Arthur Passos
fb42afbbac
CaresPTRResolver small safety improvement
Previous to #40769, only `hostent::h_aliases` was being accessed. After that PR got merged, `hostent::h_name` started being accessed as well. This PR moves the first `hostent::h_aliases != nullptr` check that could prevent `hostent::h_name` from being accessed. During debugging, I observed that even when there are not aliases, `hostent::h_aliases` is not null. That's why it hasn't caused any problems, but proposing this change to be on the safe side.
2022-09-01 08:59:14 -03:00
Fangyuan Deng
bc7d661668
Merge branch 'master' into UniqSketch 2022-09-01 19:31:53 +08:00
pzhdfy
acec516271 add docs 2022-09-01 19:31:01 +08:00
Nikita Mikhaylov
c7b0b87574
Better exception handling in DiskObjectStorageTransaction (#40813)
Co-authored-by: alesapin <alesapin@clickhouse.com>
2022-09-01 13:29:59 +02:00
alesapin
f2bcd63618
Merge pull request #40844 from ClickHouse/fix_incorrect_remove_in_zero_copy
Fix incorrect directory remove in zero copy replication
2022-09-01 13:21:58 +02:00
alesapin
1ae7e82126
Merge pull request #40819 from vitlibar/fix-locking-when-writing-backup
Fix locking while writing backup in multiple threads
2022-09-01 13:20:21 +02:00
Vitaly Baranov
6dcca686cb
Merge pull request #40827 from vitlibar/fix-incremental-backups-for-log-family
Fix incremental backups for Log family.
2022-09-01 13:13:32 +02:00
kssenii
82879543d3 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-01 13:13:01 +02:00
kssenii
9ae95f8c04 Fix unit test 2022-09-01 13:11:47 +02:00
kssenii
ca4363e2e3 Merge master 2022-09-01 13:09:00 +02:00
Anton Popov
f7bdf07adc
Merge pull request #38715 from CurtizJ/fix-read-in-order-fixed-prefix
Better support of `optimize_read_in_order` in case of fixed prefix of sorting key
2022-09-01 12:59:18 +02:00
Anton Popov
cc49198204
Merge pull request #40754 from CurtizJ/fix-fuzzer-with-object
Avoid logical errors in some common operations with `ColumnObject`
2022-09-01 12:57:32 +02:00
Antonio Andelic
6442824da8 Fix KeeperMap with Replicated database 2022-09-01 10:47:05 +00:00
kssenii
0d1f7afb17 Merge remote-tracking branch 'upstream/master' into fix-removing-directories 2022-09-01 12:28:51 +02:00
Kseniia Sumarokova
849262bd99
Merge pull request #40858 from kssenii/move-cache-files
Move cache files from Common
2022-09-01 12:28:00 +02:00
Sema Checherinda
7b59fdc042
Merge pull request #40779 from CheSema/detached-parts-metric
Metric for the number of detached parts
2022-09-01 12:24:42 +02:00
Duc Canh Le
018f4118bb
Fix view with merge table function (#40734) 2022-09-01 12:20:00 +02:00
luocongkai
56e4179cad fix bug when remove unneeded columns in subquery 2022-09-01 17:10:32 +08:00
Robert Schulze
b3fc6eafe7
Merge pull request #40750 from ClickHouse/last-command-shortcut
Make last command available in clickhouse-client interactive mode
2022-09-01 10:27:31 +02:00
Robert Schulze
de64c6b103
Merge branch 'master' into clang-tidy-for-headers 2022-09-01 10:24:56 +02:00
Robert Schulze
560b6082ee
Merge branch 'master' into remove-unnecessary-context-access-method 2022-09-01 10:10:31 +02:00
Antonio Andelic
c7d1dd942c Address PR comments 2022-09-01 07:56:06 +00:00
Robert Schulze
56eece40ec
Merge pull request #40736 from LevyCory/add-offset-to-formatDateTime
Add timezone offset support  to `formatDateTime`
2022-09-01 09:50:17 +02:00
Antonio Andelic
cecdcb5059
Merge pull request #40627 from ClickHouse/fix-install-snapshot
Keeper fix for install snapshot and preprocessing performance improvements
2022-09-01 09:13:01 +02:00
Robert Coelho
19097c433d
fix compile error 2022-09-01 01:34:41 -04:00
Frank Chen
d9a516288c Resolve comments 2022-09-01 11:56:10 +08:00
Robert Coelho
e0597700eb
Fix uncaught DNS_ERROR on failed connection to replicas
DNS_ERROR would cause the replica to not be marked as unusable, resulting in the replica being repeatedly reattempted on subsequent queries and for connection failover to break.
(This is common in Kubernetes setups where a replica has failed and it's DNS record is returning NXDOMAIN)

On SELECT, would additionally result in an intermittent query error if the failed replica is chosen:
"Code: 198. DB::Exception: Received from localhost:9000. DB::Exception: Not found address of host: chi-clickhouse-main-2-0: While executing Remote. (DNS_ERROR)"
2022-08-31 20:34:14 -04:00
Nikolay Degterinsky
e98ceb2575
Merge pull request #40251 from ClibMouse/Structured-logging-custom-keys
Feature Improvement: Custom Key Names for Structured Logging Support
2022-09-01 00:36:23 +02:00
kssenii
10a7e73d52 Fix unit test 2022-08-31 21:07:39 +02:00
Robert Schulze
912663b719
Revert "Move CatBoost evaluation into clickhouse-library-bridge" 2022-08-31 20:54:43 +02:00
alesapin
251bdab9a0 Better grep 2022-08-31 20:08:47 +02:00
kssenii
a78571de11 Fix 2022-08-31 18:47:47 +02:00
Kruglov Pavel
86516d3bb4
Merge pull request #40740 from amosbird/row-policy-index-fix-1
Use index when row_policy_filter is always false
2022-08-31 18:46:14 +02:00
Kruglov Pavel
b7ea529307
Merge pull request #40746 from Avogar/fix-csv-schema-inference
Fix possible error Attempt to read after eof in CSV schema inference
2022-08-31 18:29:01 +02:00
Kseniia Sumarokova
6e61a4f49f
Update unit test 2022-08-31 18:07:40 +02:00
kssenii
0f747e7987 Move cache files 2022-08-31 17:50:55 +02:00
Antonio Andelic
f043790e6d Add support for update 2022-08-31 14:33:53 +00:00
Robert Schulze
ca01286028
Merge pull request #39629 from ClickHouse/catboost-bridge
Move CatBoost evaluation into clickhouse-library-bridge
2022-08-31 16:16:11 +02:00
avogar
acf87c1d10 Fix nested JSON Objects schema inference 2022-08-31 14:10:29 +00:00
Anton Popov
5a3e24c4e4
Merge pull request #40483 from canhld94/ch_canh_fix_json
Fix columnObject crash with array of variadic dimension elems
2022-08-31 16:07:58 +02:00
Duc Canh Le
55c20ce45b fix tryGetPreparedSet crash when same set expression built from different columns 2022-08-31 21:43:08 +08:00
kssenii
3a183236a2 Fix 2022-08-31 15:39:53 +02:00
alesapin
10b456ce64
Merge branch 'master' into fix-locking-when-writing-backup 2022-08-31 15:12:23 +02:00
Antonio Andelic
0e6b3b870a Add support for delete in RocksDB 2022-08-31 13:08:27 +00:00
alesapin
c427491bcb Better 2022-08-31 15:05:45 +02:00
alesapin
a4f71ba6b8 Fix one more bug 2022-08-31 14:53:33 +02:00
alesapin
2d976753e1 Fix incorrect directory remove in zero copy replication 2022-08-31 13:50:59 +02:00
Vitaly Baranov
007ae0e6cc Fix incremental backups for Log family. 2022-08-31 12:57:28 +02:00
Robert Schulze
cedf75ed5e
Enable clang-tidy for headers
clang-tidy now also checks code in header files. Because the analyzer
finds tons of issues, activate the check only for directory "base/" (see
file ".clang-tidy"). All other directories, in particular "src/" are
left to future work.

While many findings were fixed, some were not (and suppressed instead).
Reasons for this include: a) the file is 1:1 copypaste of a 3rd-party
lib (e.g. pcg_extras.h) and fixing stuff would make upgrades/fixes more
difficult b) a fix would have broken lots of using code
2022-08-31 10:48:15 +00:00
Antonio Andelic
7d5e13c6ba Use async exist for write 2022-08-31 10:40:32 +00:00
Antonio Andelic
9dd1a9859d Merge branch 'master' into keeper-storage 2022-08-31 10:12:11 +00:00
vdimir
6c15db9b11 Setting max_analyze_depth forbid only second analyze 2022-08-31 11:57:41 +02:00
vdimir
473c9106a5 Revert "Create new query context in StorageRabbitMQ/Nat"
This reverts commit 2d6ac7344a25134a625e879da63b9a4aa20d9a69.
2022-08-31 11:57:41 +02:00
vdimir
63f7301ee0 Create new query context in StorageRabbitMQ/Nat 2022-08-31 11:57:41 +02:00
vdimir
60454bd722 Fix error codes in InterpreterSelectQuery.cpp 2022-08-31 11:57:41 +02:00
vdimir
c829a1914d Disable max_analyze_depth for StorageRabbitMQ/NATS 2022-08-31 11:57:41 +02:00
vdimir
f38f39ed6f Use max_analyze_depth 2022-08-31 11:57:41 +02:00
vdimir
928adb2b38 [WIP] Limit number of analyze for one query 2022-08-31 11:57:41 +02:00
Kseniia Sumarokova
164ebbe009
Merge branch 'master' into cache-assert-directory-ownership 2022-08-31 11:56:59 +02:00
Kseniia Sumarokova
9abcfce529
Merge pull request #40759 from kssenii/fix-write-through-cache-logical-error
Fix logical error from write-though cache from stress test
2022-08-31 11:55:18 +02:00
Antonio Andelic
1af9a81297 Sync after reconnect 2022-08-31 08:29:40 +00:00
Antonio Andelic
1ff14aed02 Fix dropping table 2022-08-31 08:23:44 +00:00
Antonio Andelic
5ef4c52290 Add support for keys limit 2022-08-31 08:14:28 +00:00
Robert Schulze
5ff0ecb867
Merge branch 'master' into remove-unnecessary-context-access-method 2022-08-31 10:14:27 +02:00
Antonio Andelic
97fd3d9123
Merge branch 'master' into fix-install-snapshot 2022-08-31 08:42:00 +02:00
Duc Canh Le
aebd1cb7de
Merge branch 'master' into ch_canh_intersect_distinct 2022-08-31 12:08:20 +08:00
Fangyuan Deng
5c2f35c302
Merge branch 'master' into UniqSketch 2022-08-31 10:23:46 +08:00
Mallik Hassan
87a28c25f5 Update src/Loggers/Loggers.cpp
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
2022-08-30 23:20:19 -03:00
root
f02f7ce99c addressed minor change requests 2022-08-30 23:20:19 -03:00
root
834ab5cd2b removed redundant initializer for base class 'OwnPatternFormatter' 2022-08-30 23:20:19 -03:00
root
790a437eb2 Addressed review requests: removed this pointer in functions and config variable in class 2022-08-30 23:20:19 -03:00
root
bb25abd0ae to resolve build error(no member named 'Util' in namespace 'Poco'), added <Poco/Util/AbstractConfiguration.h> in OwnJSONPatternFormatter.h 2022-08-30 23:20:19 -03:00
root
b8501bea06 style check correction 2022-08-30 23:20:19 -03:00
root
d5db88fa27 Structured-logging-custom-keys PR init 2022-08-30 23:20:19 -03:00
Anton Popov
3504781529
Merge branch 'master' into fix-read-in-order-fixed-prefix 2022-08-30 23:32:43 +02:00
Sema Checherinda
412973daee
Update src/Interpreters/AsynchronousMetrics.cpp
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2022-08-30 21:57:39 +02:00
Sema Checherinda
6b23abe2b6
Update src/Interpreters/AsynchronousMetrics.cpp
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2022-08-30 21:57:33 +02:00
Dmitry Novik
003483b616 Add compatibility setting 2022-08-30 20:49:40 +02:00
kssenii
a5f69517ae Allow to load marks in threadpool 2022-08-30 20:20:05 +02:00
Sema Checherinda
9bb4f556c3 work on remarks on the review 2022-08-30 20:18:35 +02:00
kssenii
23db13ca69 Merge master 2022-08-30 19:39:41 +02:00
kssenii
19c4442933 Fix 2022-08-30 19:38:57 +02:00
Robert Schulze
7a4978e600
docs: mention where else shortcuts are used 2022-08-30 17:00:25 +00:00
Robert Schulze
413a5759e1
style: eliminate some newlines
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
2022-08-30 17:00:20 +00:00
Robert Schulze
9baf0d745a
feat: make last command available in clickhouse-client interactive mode 2022-08-30 16:59:14 +00:00
Vitaly Baranov
77d741dc25 Add comments. 2022-08-30 18:58:13 +02:00
Vitaly Baranov
86872b2307 Fix locking while writing backup in multiple threads. 2022-08-30 18:10:54 +02:00
Antonio Andelic
d498213973 Style fixes 2022-08-30 13:50:02 +00:00
Antonio Andelic
bf03dfc661 Fix create 2022-08-30 13:41:13 +00:00
Alexander Tokmakov
6a50c20734
Merge branch 'master' into zookeeper_client_fault_injection 2022-08-30 15:43:30 +03:00
Dmitry Novik
0a8378d9cd
Merge branch 'master' into betterorderbyoptimization 2022-08-30 14:23:22 +02:00
Alexander Tokmakov
6fdfb964d0
Revert "Add Annoy index" 2022-08-30 15:10:10 +03:00
vdimir
0f6f3c73b0
Minor fix 2022-08-30 11:57:28 +00:00
Vladimir C
7266c007b7
Merge pull request #39131 from amosbird/cte-global-with-fix1
Fix unused unknown columns introduced by WITH statement
2022-08-30 13:52:19 +02:00
Anton Popov
ee691961d9 fix build 2022-08-30 11:43:27 +00:00
Alexander Tokmakov
022f440ad0
Merge pull request #40769 from arthurpassos/caresptrresolver-hosts-patch
Fix CaresPTRResolver not reading hosts file
2022-08-30 14:35:10 +03:00
Kseniia Sumarokova
54f6f5bb2c
Merge branch 'master' into fix-write-through-cache-logical-error 2022-08-30 13:30:13 +02:00
Kruglov Pavel
624af5bffd
Update src/Storages/MergeTree/KeyCondition.cpp 2022-08-30 13:18:05 +02:00
Kruglov Pavel
be81d21fdf
Merge pull request #40597 from helifu/master
Fix a crash while the grpc port is conflict
2022-08-30 13:17:25 +02:00
Kseniia Sumarokova
5eb188311a
Merge pull request #40586 from kssenii/fix-race
Fix possible race in fs cache
2022-08-30 12:58:31 +02:00
Alexander Tokmakov
0ec7f068cc
Merge pull request #40677 from ClickHouse/query-insert-oom
Avoid deadlock in case of new query and OOM
2022-08-30 13:54:34 +03:00
Antonio Andelic
53da469147 Merge branch 'master' into keeper-storage 2022-08-30 10:22:04 +00:00
Duc Canh Le
626b3e4147
Merge branch 'master' into ch_canh_intersect_distinct 2022-08-30 18:14:07 +08:00
Duc Canh Le
8590cc46c4 implement intersect + except distinct 2022-08-30 18:09:01 +08:00
Kseniia Sumarokova
487100a4fd
Merge pull request #40764 from kssenii/fix-logical-error-in-cache
Remove incorrect check in fs cache
2022-08-30 11:59:56 +02:00
Kseniia Sumarokova
c88db2ef97
Merge pull request #40751 from kssenii/fix-mysql-timeouts
Fix issue with mysql db / table function timeouts
2022-08-30 11:59:01 +02:00
Antonio Andelic
80365e0448 Simplify drop and create 2022-08-30 09:19:59 +00:00
Antonio Andelic
21cdb9a500 Don't remove from cache if it wasn't added during rollback 2022-08-30 08:14:53 +00:00
Antonio Andelic
2416ddfae7 Add extra message about startup preprocessing 2022-08-30 08:14:31 +00:00
Robert Schulze
c185353bf0
Minor: remove unnecessary ContextAccess::make() method 2022-08-30 08:06:42 +00:00
Robert Schulze
cc4225109f
Merge pull request #37215 from Vector-Similarity-Search-for-ClickHouse/annoy-2
Test failures are unrelated, merging.
2022-08-30 09:25:57 +02:00
Robert Schulze
566daad4a1
Merge pull request #40723 from k-morozov/ref/remove-unnecessary-allocation
refactoring: remove unnecessary allocation
2022-08-30 09:22:21 +02:00
Frank Chen
922a0dee74 Fix style
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
2022-08-30 15:14:12 +08:00
Frank Chen
f17d56b528 Merge branch 'master' into tracing_context_propagation 2022-08-30 14:24:36 +08:00
Frank Chen
83cbdef3c6 Fix context re-initialization for ThreadPool 2022-08-30 12:26:23 +08:00
Cory Levy
cd371110ad Fix timezone offset formatting 2022-08-29 21:01:27 -04:00
Sema Checherinda
e436b4f4cc Metric for the number of detached parts 2022-08-30 02:12:57 +02:00
Robert Schulze
4ed1e1a5be
perf: don't copy data around unnecessarily 2022-08-29 20:27:06 +00:00
Robert Schulze
35a37c91f8
chore: incorporate review feedback 2022-08-29 20:27:06 +00:00
robot-clickhouse
64fa077148
style: fix style 2022-08-29 20:27:06 +00:00
Robert Schulze
4d511332c4
chore: delete obsolete modelEvaluate() function
- superseded by catboostEvaluate() which no longer uses the internal
  repository for external models

- also removed was statement SYSTEM RELOAD MODELS and the monitoring view
  SYSTEM.SYSTEMMODELS
2022-08-29 20:27:06 +00:00
Robert Schulze
6b2b3c1eb3
feat: implement catboost in library-bridge
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes #27870
2022-08-29 20:26:45 +00:00
Dmitry Novik
e25ed9547e
Update src/Interpreters/ProcessList.h 2022-08-29 20:26:37 +02:00
Dmitry Novik
865ee5d0d6 Refactor code 2022-08-29 20:24:35 +02:00
Arthur Passos
961365c7a4 Fix CaresPTRResolver not reading hosts file 2022-08-29 15:11:39 -03:00
Dmitry Novik
1169315580 Add OvercommitTracker blocking 2022-08-29 19:44:05 +02:00
Maksim Kita
88141cae98
Merge pull request #40732 from azat/thread-status-fix-leak
Fix memory leak while pushing to MVs w/o query context (from Kafka/...)
2022-08-29 19:36:25 +02:00
Kseniia Sumarokova
c5c48e44ea
Merge branch 'master' into fix-mysql-timeouts 2022-08-29 19:33:29 +02:00
kssenii
db2bc31e17 Remove incorrect assertion 2022-08-29 19:32:47 +02:00
Konstantin Morozov
d185b7a332 refactoring: public ctors 2022-08-29 20:19:20 +03:00
FArthur-cmd
862b53b06f Merge branch 'annoy-2' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-08-29 16:43:39 +00:00
FArthur-cmd
3305af8db2 fix case when query is already matched 2022-08-29 16:43:24 +00:00
Dmitry Novik
d32492ce8a Fix GROUPING function SQL compatibility 2022-08-29 18:26:56 +02:00
kssenii
545c6c8be4 Fix 2022-08-29 17:50:27 +02:00
Dmitry Novik
cfe509c3de Block overcommit tracker in ProcessList near allocations 2022-08-29 17:49:01 +02:00
Filatenkov Artur
d73f661732
Merge branch 'master' into annoy-2 2022-08-29 17:33:13 +03:00
Anton Popov
506bee1e2f avoid logical errors at some common operation with ColumnObject 2022-08-29 14:31:09 +00:00
kssenii
b1dab84d97 Review fixes 2022-08-29 16:23:14 +02:00
kssenii
0a6c4b9265 Fix 2022-08-29 16:20:53 +02:00
kssenii
877ade9a50 Merge remote-tracking branch 'upstream/master' into fix-race 2022-08-29 16:05:27 +02:00
Antonio Andelic
57471d11db Fix uncommitted auth 2022-08-29 12:55:30 +00:00
Vladimir C
5cbe7e0846
Merge pull request #40548 from ClickHouse/vdimir/warn-suppress-40330
Add config option warning_supress_regexp
2022-08-29 14:02:00 +02:00
Alexander Tokmakov
8bdb589c2b Merge branch 'master' into zookeeper_client_fault_injection 2022-08-29 13:34:57 +02:00
Antonio Andelic
54ce92a5c4 Merge branch 'master' into fix-install-snapshot 2022-08-29 11:27:34 +00:00
vdimir
24f62e8486
Throw an error in CreatingSetsOnTheFlyTransform in case of input for finished 2022-08-29 11:27:08 +00:00
vdimir
b0e2616aa9
Style fixes in CreateSetAndFilterOnTheFlyTransform and related 2022-08-29 11:26:21 +00:00
Alexander Tokmakov
69387acffa
Revert "Support for DWARF-5 in in house DWARF parser" 2022-08-29 14:25:53 +03:00
Alexander Tokmakov
8d90d30d37
Merge pull request #40589 from ClickHouse/remove_wrong_code_from_mutations
Remove wrong code for skipping mutations in MergeTree
2022-08-29 14:18:59 +03:00
avogar
85fa372ad7 Fix possible error Attempt to read after eof in CSV schema inference 2022-08-29 11:18:53 +00:00
Anton Popov
2a3e012931
Merge branch 'master' into fix-read-in-order-fixed-prefix 2022-08-29 13:17:26 +02:00
Alexander Tokmakov
eda0582ec0
Merge pull request #40641 from ClickHouse/fix_startup_of_dropped_replica
Do not try to strartup dropped replica
2022-08-29 14:15:15 +03:00
vdimir
3e06cf3a9d
Upd max_rows_in_set_to_optimize_join description 2022-08-29 10:10:47 +00:00
vdimir
7915b6948f
Fix build after rebase 2022-08-29 09:49:16 +00:00
vdimir
afb6b7d9cf
Test plan and pipeline for filtering step for join 2022-08-29 09:49:15 +00:00
vdimir
afeff512b5
Aux port for ReadHeadBalancedProcessor is empty Block 2022-08-29 09:49:14 +00:00
vdimir
95f87dc34e
fix sanitizer assert in CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:12 +00:00
vdimir
c67ab33d90
small fix CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:11 +00:00
vdimir
51e02d09f6
set preserves_sorting = true for CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:10 +00:00
vdimir
714c53ab24
fix typos 2022-08-29 09:49:09 +00:00
vdimir
8e1632f824
Create sets for joins: better code 2022-08-29 09:49:08 +00:00
vdimir
7228091ff1
rename CreateSetAndFilterOnTheFlyTransform 2022-08-29 09:49:07 +00:00
vdimir
67a9acc8db
rename CreatingSetOnTheFlyStep -> CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:07 +00:00
vdimir
d82a75ae75
cleanup PingPongProcessor 2022-08-29 09:49:06 +00:00
vdimir
e472e13c70
move PingPongProcessor/ReadHeadBalancedProceesor into separate file 2022-08-29 09:49:05 +00:00
vdimir
6ae15db3ad
check join kind for before applying max_rows_in_set_to_optimize_join 2022-08-29 09:49:04 +00:00
vdimir
51a51694d6
Create sets for joins: better code 2022-08-29 09:49:01 +00:00
vdimir
c778bba13f
Create sets for joins: wip 2022-08-29 09:47:00 +00:00
vdimir
31a167848d
Fix set finish condition in CreatingSetsOnTheFlyTransform 2022-08-29 09:46:59 +00:00
vdimir
71708d595f
Create sets for joins: wip 2022-08-29 09:46:59 +00:00
vdimir
8f06430ebd
Create sets for joins: upd 2022-08-29 09:46:58 +00:00
vdimir
3292566603
Format bytes in CreatingSetsOnTheFlyTransform logs 2022-08-29 09:46:57 +00:00
vdimir
031aaf3a45
Add Creating/FilterBySetsOnTheFlyStep for full sorting join 2022-08-29 09:46:57 +00:00
vdimir
c5bc7b0a0c
Resize pipeline after full sort join 2022-08-29 09:46:56 +00:00
Azat Khuzhin
f9812d9917 Fix memory leak while pushing to MVs w/o query context (from Kafka/...)
While pushign to MVs, there is a low-level code that create
ThreadGroupStatus/ThreadStatus, it is required to gather some metrics
for system.query_views_log.

But, one should not use ThreadGroupStatus of the MainThreadStatus, since
this structure can hold some state, that may not be cleaned, plus this
may be racy, instead it is better to create new ThreadGroupStatus and
attach it instead.

Also this place misses detachQuery(), and because of this it leaks
ThreadGroupStatus::finished_threads_counters_memory. But it is only the
problem pushing to MVs is done w/o query context (i.e. from Kafka/...),
since when it has query context detachQuery() will be called eventually.

Before this patch series, when I've tried the reproducer with
500 MVs attached to Kafka engine (that @den-crane suggested), jemalloc
report looks like this:

    $ ../jeprof --text ~/ch/tmp/upstream/clickhouse-binary --base jeprof.44384.0.i0.heap jeprof.44384.167.i167.heap
    Using local file /home/azat/ch/tmp/upstream/clickhouse-binary.
    Using local file jeprof.44384.167.i167.heap.
    Total: 915.6 MB
       910.7  99.5%  99.5%    910.7  99.5% Snapshot (inline)
         9.5   1.0% 100.5%      9.5   1.0% std::__1::__libcpp_operator_new (inline)
         0.5   0.1% 100.6%      0.5   0.1% DB::TasksStatsCounters::create

And with focus to this place:

    $ ../jeprof --focus Snapshot --text ~/ch/tmp/upstream/clickhouse-binary --base jeprof.44384.0.i0.heap jeprof.44384.167.i167.heap
    Using local file /home/azat/ch/tmp/upstream/clickhouse-binary.
    Using local file jeprof.44384.167.i167.heap.
    Total: 915.6 MB
       910.7 100.0% 100.0%    910.7 100.0% Snapshot (inline)
         0.0   0.0% 100.0%    910.7 100.0% DB::QueryPipeline::reset
         0.0   0.0% 100.0%    910.7 100.0% DB::StorageKafka::streamToViews
         0.0   0.0% 100.0%    910.7 100.0% DB::StorageKafka::threadFunc
         0.0   0.0% 100.0%    910.7 100.0% ProfileEvents::Counters::getPartiallyAtomicSnapshot
         0.0   0.0% 100.0%    910.7 100.0% ~ThreadStatus
         0.0   0.0% 100.0%    910.7 100.0% ~ViewRuntimeData
         0.0   0.0% 100.0%    910.7 100.0% ~ViewRuntimeStats (inline)

Actually this report does not looks great (you understand it because I
stripped it), because --text does not that smart, but if you will use
--pdf for the report you will see the stacktrace (will attach pdf to the
pull request).

But after this patch series the process RSS does not goes beyond
~700MiB.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-29 11:36:33 +02:00
Azat Khuzhin
6da5707f8f Fix possible missing detachQuery() in case of exception in readers
This can create leaks, since detachQuery() responsible for cleaning,
i.e. ThreadGroupStatus::finished_threads_counters_memory

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-29 11:30:17 +02:00
Azat Khuzhin
b16891da8d Avoid using of ThreadGroupStatus of the MainThreadStatus
One should not use MainThreadStatus, since ThreadGroupStatus can hold
some states, and it is better not to play with this, since this may
create leaks.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-29 11:30:17 +02:00
Azat Khuzhin
9fff08eac7 WriteBufferFromS3: remove unused ThreadGroupStatus
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-29 11:30:17 +02:00
alesapin
133ca01447 Merge branch 'master' into stress_s3 2022-08-29 11:25:28 +02:00
alesapin
4ed375ca5b
Merge pull request #40720 from ClickHouse/fix_benign_race
Fix benign race in database replicated worker
2022-08-29 11:13:07 +02:00
Azat Khuzhin
269453a646 Avoid leaking of ThreadGroupStatus::finished_threads_counters_memory
Cleanup them in ThreadStatus::detachQuery(), anyway they cannot be
received after by the client.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-29 11:08:50 +02:00
alesapin
7ce0afc0df
Merge pull request #40670 from Avogar/kafka
Add setting to disable limit on kafka_num_consumers
2022-08-29 10:53:35 +02:00
Amos Bird
15a69bce84
Use index when row_policy_filter is always false 2022-08-29 16:44:32 +08:00
Antonio Andelic
b4af21b6fb Small fixes 2022-08-29 07:06:51 +00:00
Amos Bird
abf51bccfd
Disable projection when grouping set is used. 2022-08-29 14:35:48 +08:00
Konstantin Morozov
6636bdec2a refactoring: fix format 2022-08-29 07:44:40 +03:00
Cory Levy
d0ace9a8fb Add the %z format descriptor to formatDateTime 2022-08-28 22:28:11 -04:00
Alexey Milovidov
71f6c52c2d
Merge pull request #40727 from amosbird/column-transformer-fix1
Correct format of APPLY transformer param
2022-08-29 03:48:04 +03:00
Alexey Milovidov
307e8f2da9
Merge pull request #40729 from ClickHouse/remove-useless-method
Remove useless method
2022-08-29 03:46:34 +03:00
Alexey Milovidov
ae97e880d7
Merge pull request #40710 from azat/DWARF-5
Support for DWARF-5 in in house DWARF parser
2022-08-29 00:13:33 +03:00
Alexey Milovidov
27782ceef8 Remove useless method 2022-08-28 22:33:42 +02:00
Konstantin Morozov
38a85ade27 refactoring: small up 2022-08-28 22:33:12 +03:00
Alexey Milovidov
f462160bd1 Merge remote-tracking branch 'origin/master' into remove-useless-line 2022-08-28 21:00:29 +02:00
Konstantin Morozov
b4de735a90 refactoring: fix space 2022-08-28 20:55:08 +03:00
Konstantin Morozov
08e9e799d0 refactoring: fix 2022-08-28 20:53:00 +03:00
Konstantin Morozov
75bd61fad5 Merge remote-tracking branch 'origin/ref/remove-unnecessary-allocation' into ref/remove-unnecessary-allocation
# Conflicts:
#	src/Access/ContextAccess.cpp
#	src/Access/ContextAccess.h
2022-08-28 20:50:29 +03:00
Konstantin Morozov
7cd2821aed refactoring: template make helper 2022-08-28 20:42:55 +03:00
Amos Bird
d1fbe51b81
Correct format of APPLY transformer param 2022-08-29 01:21:12 +08:00
Konstantin Morozov
4050ab819e refactoring: remove unnecessary allocation 2022-08-28 17:23:17 +03:00
Konstantin Morozov
d4d80dd3c1 refactoring: remove unnecessary allocation 2022-08-28 17:09:13 +03:00
Alexey Milovidov
f518844f06
Merge pull request #40709 from ClickHouse/all_new_table_functions_must_be_documented
All new table functions must be documented
2022-08-28 17:05:20 +03:00
Alexey Milovidov
00074a5e14
Merge pull request #40711 from ClickHouse/all_new_functions_must_be_documented
All new functions must be documented
2022-08-28 17:05:05 +03:00
Azat Khuzhin
e03870bc8b Common/Dwarf: fix clang-tidy 2022-08-28 15:51:46 +02:00
alesapin
4f9faab924 Fix benign race in database replicated worker 2022-08-28 15:07:29 +02:00
Kseniia Sumarokova
a0bc5b6ea4
Merge pull request #40705 from azat/stress/max_read_buffer_size
Fix LOGICAL_ERROR with max_read_buffer_size=0 during reading marks
2022-08-28 12:41:16 +02:00
Robert Schulze
df934d8762
Merge pull request #40217 from zvonand/zvonand-minmax
Fix conversion Date32 / DateTime64 / Date to narrow types
2022-08-28 09:42:39 +02:00
Alexey Milovidov
d3a7945b1b
Merge pull request #40646 from ClickHouse/force-documentation-3
Every profile event must have documentation
2022-08-28 04:10:11 +03:00
Alexey Milovidov
b42d26acfe Remove one line from XML, because we do not care 2022-08-28 02:44:02 +02:00
Alexey Milovidov
5b57c91ad6 Rename table function MeiliSearch to meilisearch 2022-08-28 02:08:51 +02:00