thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
87,779 Commits 9,326 Branches 1,784 Tags 2.1 GiB
cf48390cb9
Commit Graph

7810 Commits

Author SHA1 Message Date
kssenii
c4832f7633 Fix 2022-03-28 09:28:02 +02:00
Mikhail Artemenko
1886d9c2ea refactor parseTypeOfField & fix tests 2022-03-27 23:00:28 +03:00
kssenii
c6706b8bfa Better 2022-03-25 19:30:13 +01:00
kssenii
141c290344 Fix build 2022-03-25 18:31:15 +01:00
taiyang-li
b79cec6806 Merge branch 'use_minmax_index' of https://github.com/bigo-sg/ClickHouse into use_minmax_index 2022-03-25 23:33:49 +08:00
taiyang-li
eee8949150 fix code 2022-03-25 23:33:46 +08:00
Alexander Tokmakov
9ec0951de5 update comment for mismatching checksums 2022-03-25 15:54:05 +01:00
taiyang-li
4aaa361f2e Merge remote-tracking branch 'ck/master' into use_minmax_index 2022-03-25 22:48:03 +08:00
taiyang-li
fbadd8fc0e only deduce absolute path of hdfs config when it's relative path do not exists 2022-03-25 22:28:47 +08:00
taiyang-li
c013cd0b52 Merge branch 'master' into hdfs_config_prefix 2022-03-25 22:13:45 +08:00
taiyang-li
3ca487030d Merge branch 'master' into improve_access_type 2022-03-25 22:11:08 +08:00
avogar
ecc57b436c Check remote_url_allow_hosts before schema inference in URL engine 2022-03-25 13:30:45 +00:00
Antonio Andelic
394682b983 Merge branch 'master' into parallel-s3-downloading 2022-03-25 13:00:44 +00:00
kssenii
6769b84638 Merge master 2022-03-24 20:47:43 +01:00
kssenii
d9d9f3bc6d Fix 2022-03-24 19:10:53 +01:00
kssenii
e91e30aaa4 Fix 2022-03-24 17:10:27 +01:00
Sergei Trifonov
a90e83665d
Merge pull request #35343 from ClickHouse/parallel-multipart-upload-for-s3storage 
allow parallelization of multipart S3 storage upload; fixes #34350
 2022-03-24 15:58:29 +01:00
kssenii
d6ab6de30a Minor changes 2022-03-24 15:32:08 +01:00
taiyang-li
c0eb9f73a9 remote std::cout 2022-03-24 21:17:43 +08:00
taiyang-li
6eccd09e2b deduce hdfs config path 2022-03-24 18:04:29 +08:00
Antonio Andelic
62a495a5fc Small refactor 2022-03-24 09:45:30 +00:00
Nikolai Kochetov
283e20a9a5
Merge pull request #35395 from amosbird/distributedmultiplejoin 
Validate some thoughts over making sets
 2022-03-24 10:30:26 +01:00
Antonio Andelic
c52cee2219 Merge branch 'master' into parallel-s3-downloading 2022-03-24 08:11:52 +00:00
Maksim Kita
1df1721648
Merge pull request #35546 from kitaisreal/refactor-external-models-loader 
ExternalModelsLoader refactoring
 2022-03-24 01:24:45 +01:00
kssenii
d4161b5925 Add optin read_from_cache_if_exists_otherwise_bypass_cache (for merges) 2022-03-23 20:24:00 +01:00
Maksim Kita
3df6a86387 ExternalModelsLoader refactoring 2022-03-23 20:05:17 +01:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
Yakov Olkhovskiy
1631656803
Merge pull request #34353 from zvonand/issue_33147 
Extended usage of Milliseconds, Microseconds, Nanoseconds
 2022-03-23 13:33:07 -04:00
kssenii
e39aba37a2 Minor changes 2022-03-23 18:11:52 +01:00
Alexander Tokmakov
9aed0507b7 Merge branch 'master' into mvcc_prototype 2022-03-23 18:07:22 +01:00
Sergei Trifonov
bf9c6be6c6 move threadPoolCallbackRunner from Common into Interpreters folder to fix split build issue 2022-03-23 17:57:19 +01:00
Antonio Andelic
cbd20af706 Merge branch 'master' into parallel-s3-downloading 2022-03-23 16:35:05 +00:00
kssenii
c2c7f365ce Merge master 2022-03-23 16:58:35 +01:00
kssenii
8c1be8950e Improve test 2022-03-23 16:55:18 +01:00
kssenii
34c0690a69 Better 2022-03-23 15:43:14 +01:00
Antonio Andelic
1a5f5c32bb Remove commented out code 2022-03-23 14:36:25 +00:00
李扬
9cc528b01f
Update HiveFile.h 2022-03-23 21:57:58 +08:00
xiedeyantu
9441111833
fix INSERT INTO table FROM INFILE does not display progress bar (#35429) 
* fix INSERT INTO table FROM INFILE does not display progress bar

* Update StorageFile.cpp

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-03-23 14:24:59 +01:00
kssenii
f3e1ca44a9 Add system.remote_data_paths table 2022-03-23 13:01:18 +01:00
Antonio Andelic
93ad209910 Attach memory tracker 2022-03-23 11:52:31 +00:00
Alexander Tokmakov
bcec53aadc Merge branch 'master' into mvcc_prototype 2022-03-23 11:28:43 +01:00
Antonio Andelic
131b3a091c Refactor StorageS3 2022-03-23 08:57:01 +00:00
Antonio Andelic
bb6b9f3799 Merge branch 'parallel-downloading-url-engine' into parallel-s3-downloading 2022-03-23 07:46:14 +00:00
Antonio Andelic
9829f2d051 Merge branch 'master' into parallel-downloading-url-engine 2022-03-23 07:41:14 +00:00
taiyang-li
ae3d55c6a2 merge master and fix conflict 2022-03-23 14:31:12 +08:00
taiyang-li
6fe4c3d17e some minor changes 2022-03-23 12:13:42 +08:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00
taiyang-li
68d5b538aa fix build error 2022-03-23 11:15:42 +08:00
Mikhail f. Shiryaev
e6f5a3f98b
Apply black formatter to all *.py files in the repo 2022-03-22 17:39:58 +01:00
Anton Popov
93dc5c0b81
Merge pull request #35488 from CurtizJ/add-subcolumns-sizes 
Add sizes of subcolumns to `system.parts_columns` table
 2022-03-22 16:53:23 +01:00
Anton Popov
f693eba568 fix tests with approx rows 2022-03-22 14:30:40 +00:00
Antonio Andelic
6785ad165a Fix issue for mutliple download threads 2022-03-22 14:06:30 +00:00
Antonio Andelic
df99a9cc3c Merge branch 'parallel-downloading-url-engine' into parallel-s3-downloading 2022-03-22 12:49:35 +00:00
Antonio Andelic
5da3058368 Merge branch 'master' into parallel-downloading-url-engine 2022-03-22 12:20:36 +00:00
Antonio Andelic
246c66bf3e
Merge pull request #35393 from bigo-sg/hive_column_pruning_bug 
Fix erasing nonexistent column
 2022-03-22 08:59:33 +01:00
Anton Popov
0c210a831c add sizes of subcolumns to system.parts_columns table 2022-03-21 21:12:26 +00:00
kssenii
a9ba14e386 Fix checks 2022-03-21 21:53:46 +01:00
mergify[bot]
9a505476a5
Merge branch 'master' into mvcc_prototype 2022-03-21 20:33:07 +00:00
tavplubix
2fdc937ae1
Merge pull request #35480 from ClickHouse/fix_34586 
Fix show create for information_schema
 2022-03-21 22:20:34 +03:00
Alexander Tokmakov
3c762f566d Merge branch 'master' into mvcc_prototype 2022-03-21 20:16:29 +01:00
mergify[bot]
9b9959aae0
Merge branch 'master' into read_in_order_max_rows_to_read 2022-03-21 17:33:37 +00:00
Vitaly Baranov
72a2f31984
Merge pull request #33985 from vitlibar/improve-backup-and-restore 
Improve backup and restore (experimental)
 2022-03-21 17:59:51 +01:00
tavplubix
9226ea7b70
Merge pull request #35467 from ClickHouse/resubmit_21474 
Resubmit #21474
 2022-03-21 19:15:28 +03:00
Nikolai Kochetov
5e239762c6
Merge pull request #35334 from amosbird/fixpartitionpruneroverflow 
Fix incorrect partition pruning when constant in predicate has no supertype of related key columns
 2022-03-21 17:05:19 +01:00
Alexander Tokmakov
dba82a7732 fix show create for information_schema 2022-03-21 17:01:39 +01:00
Antonio Andelic
04e6e6ab5e Add ParallelReadBuffer for S3 2022-03-21 14:52:26 +00:00
kssenii
afd0c64a1a Better 2022-03-21 15:04:16 +01:00
mergify[bot]
e11ef05c2b
Merge branch 'master' into issue_33147 2022-03-21 13:40:17 +00:00
lgbo-ustc
967d5a8055 Merge remote-tracking branch 'ck/master' into hive_column_pruning_bug 2022-03-21 19:52:06 +08:00
kssenii
eabbce69a7 Add system drop remote filesystem cache query, allow to write cache on insert 2022-03-21 12:32:44 +01:00
Alexander Tokmakov
7cba97aeab Merge branch 'master' into resubmit_21474 2022-03-21 12:09:00 +01:00
Vitaly Baranov
7a46e7a0d1
Merge branch 'master' into improve-backup-and-restore 2022-03-21 11:49:26 +01:00
Vitaly Baranov
51fc556fcd Implement BACKUP and RESTORE for MatViews. 2022-03-21 11:45:21 +01:00
mergify[bot]
e5a5ab2a40
Merge branch 'master' into distributedmultiplejoin 2022-03-21 10:00:51 +00:00
Antonio Andelic
b8c43ff2f2 Merge branch 'master' into parallel-downloading-url-engine 2022-03-21 09:11:01 +00:00
kssenii
59b7394caf WriteSettings 2022-03-21 09:52:48 +01:00
kssenii
1d80262a2a Add write settings, file cache settings 2022-03-21 09:50:50 +01:00
taiyang-li
49b6f3dfc5 merge master and fix conflict 2022-03-21 15:05:43 +08:00
taiyang-li
bf05b94940 fix build 2022-03-21 15:03:28 +08:00
taiyang-li
d2bea0a72c Merge branch 'master' into improve_access_type 2022-03-21 14:25:19 +08:00
Alexey Milovidov
0ce4696c49
Revert "[WIP] New row policies" 2022-03-21 08:41:33 +03:00
taiyang-li
7d50bd1eb3 add access type hive 2022-03-21 11:19:45 +08:00
Amos Bird
243de091bb
Validate some thoughts over making sets 2022-03-21 10:58:44 +08:00
lgbo-ustc
f7aa40af5b update codes 2022-03-21 09:25:20 +08:00
Alexander Tokmakov
9e05b12d2c Merge branch 'master' into mvcc_prototype 2022-03-20 22:42:26 +01:00
Vitaly Baranov
ce25afb2e9 Storages and databases are hollow by default now. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
c884cf901f Use BackupEntryFromAppendOnlyFile for Log engine, 
and use a BackupEntriesBatch for Memort engine to improve performance.
A lot of minor corrections.
 2022-03-20 20:02:15 +01:00
Vitaly Baranov
e72a343994 Implement BACKUP & RESTORE for the Memory table engine. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7f89b98308 Rework BackupSettings and RestoreSettings a little, pass StorageRestoreSettings to storages. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
258a472001 Shorter names: rename IRestoreFromBackupTask -> IRestoreTask. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
5a883c6b5a Fixed checking access rights while producing backup, add IStorage::hasHollowBackup(). 2022-03-20 20:01:54 +01:00
Vitaly Baranov
7a63feb3f7 Make restore tasks explicit. 2022-03-20 20:01:31 +01:00
Vitaly Baranov
9fb50422ce
Merge pull request #35345 from vitlibar/new-row-policies 
[WIP] New row policies
 2022-03-20 20:00:25 +01:00
Alexey Milovidov
9b05801e0f
Merge pull request #35284 from CurtizJ/fix-mutations-sparse-columns 
Fix mutations in tables with enabled sparse columns
 2022-03-20 21:55:59 +03:00
mergify[bot]
7ac606fa65
Merge branch 'master' into parallel-multipart-upload-for-s3storage 2022-03-20 18:25:28 +00:00
alesapin
1bd5736e34 Fix build one more time 2022-03-19 20:00:08 +01:00
alesapin
9e24677a30 Fix build 2022-03-19 18:09:01 +01:00
alesapin
97e84e6dc2 fix build 2022-03-19 17:47:35 +01:00
alesapin
f2c5e2d3a0 Don't spam logs in zero copy replication 2022-03-19 17:31:33 +01:00
Kseniia Sumarokova
a918e68d9a
Merge pull request #35355 from kssenii/add-hosts-filter 
Add more checks with remoteHostsFilter
 2022-03-19 09:41:23 +01:00
Alexander Tokmakov
418d52044a try fix merge assignment 2022-03-18 20:31:44 +01:00
Alexander Tokmakov
c28008a2df more comments, minor fixes 2022-03-18 12:01:26 +01:00
Amos Bird
4d85c2fc8b
Use castColumnAccurateOrNull 2022-03-18 17:47:04 +08:00
Amos Bird
10996b1434
Fix mixed constant type during partition pruning 2022-03-18 17:47:03 +08:00
lgbo-ustc
e78cfe3b26 update codes 2022-03-18 15:07:52 +08:00
Alexander Tokmakov
07d952b728 use snapshots for semistructured data, durability fixes 2022-03-17 18:26:18 +01:00
Antonio Andelic
b789a41662 Fix ASAN problems 2022-03-17 15:18:20 +00:00
Anton Popov
11ab24acb1
Merge pull request #35356 from CurtizJ/fix-clang-tidy 
Fix clang-tidy
 2022-03-17 16:17:23 +01:00
Vitaly Baranov
470582e262 Change type of RowPolicyKind: bool -> enum. 2022-03-17 15:28:05 +01:00
Alexander Tokmakov
d04dc03fa4 Merge branch 'master' into mvcc_prototype 2022-03-17 15:24:32 +01:00
Nikolai Kochetov
ee9c2ec735
Merge pull request #34780 from azat/mt-delayed-part-flush 
Do not delay final part writing by default (fixes possible Memory limit exceeded during INSERT)
 2022-03-17 12:30:51 +01:00
Anton Popov
fe2dc85788 fix clang-tidy 2022-03-17 11:16:32 +00:00
kssenii
a2cd165d38 Add remote host filter 2022-03-17 11:48:42 +01:00
alesapin
92ab33f560
Merge pull request #35348 from ClickHouse/fix_yet_another_zero_copy_bug 
Fix race in zero copy replication
 2022-03-17 11:34:38 +01:00
alesapin
bb251938dc
Merge pull request #35344 from ClickHouse/changelog-22.3 
Changelog 22.3
 2022-03-17 11:25:36 +01:00
Antonio Andelic
103a3fa140 Merge branch 'master' into parallel-downloading-url-engine 2022-03-17 10:11:58 +00:00
Antonio Andelic
44de5a581c Fix split build 2022-03-17 10:09:16 +00:00
Antonio Andelic
cc623bbf5e Fix redirect_url tests 2022-03-17 08:03:22 +00:00
Alexey Milovidov
68ef49ea51 Fix something stupid 2022-03-17 05:57:13 +01:00
alesapin
03ff6d52f6 Fix race in zero copy replication 2022-03-16 21:51:33 +01:00
Anton Popov
de2cc23e15 fix race 2022-03-16 20:16:59 +00:00
Alexander Tokmakov
4b3e13a4fe Merge branch 'master' into mvcc_prototype 2022-03-16 21:06:19 +01:00
Alexander Tokmakov
c2ac8d4a5c review fixes 2022-03-16 21:05:34 +01:00
Sergei Trifonov
bbe657ba24 allow parallelization of multipart S3 storage upload fixes #34350 2022-03-16 19:41:41 +01:00
Antonio Andelic
0979155f2f Address PR comments 2022-03-16 15:45:17 +00:00
Anton Popov
0ba78c3c3a Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-16 15:28:09 +00:00
Antonio Andelic
84f86bf3da Limit number of bytes read per worker 2022-03-16 12:34:14 +00:00
Antonio Andelic
2291b7c1e1 Merge branch 'master' into parallel-downloading-url-engine 2022-03-16 09:59:52 +00:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts 
Reload remote_url_allow_hosts after config update.
 2022-03-15 22:07:16 +01:00
Alexander Tokmakov
1f571b7734 Merge branch 'master' into mvcc_prototype 2022-03-15 14:45:06 +01:00
Nikolai Kochetov
97aa6c82ce Reload remote_url_allow_hosts after config update. 2022-03-15 13:00:31 +00:00
Anton Popov
b5a04ff938
Merge pull request #35207 from bigo-sg/add_distributed_depth_in_querylog 
Add column  distributed_depth for system.query_log and system.processes
 2022-03-15 13:38:21 +01:00
Kseniia Sumarokova
427ae808ba
Merge pull request #35191 from DevTeamBK/RemoteHostFilter_improvement 
Added RemoteHostFilter check for MYSQL and postgresSQL
 2022-03-15 09:58:08 +01:00
Antonio Andelic
067b79b00b Merge branch 'master' into parallel-downloading-url-engine 2022-03-15 07:55:41 +00:00
lgbo-ustc
abfaa82bca fixed hive query bugs 2022-03-15 12:01:34 +08:00
Saad Ur Rahman
679abaafdc
[Storages] bugfix 
Extracting Version Id for S3::URI.
 2022-03-14 22:25:40 -04:00
Saad Ur Rahman
fb450b7068
[Storages] bugfix 
Corrected order of parameters by swapping bucket and versionId.
 2022-03-14 22:20:02 -04:00
Saad Ur Rahman
171260f40b
[Storages] Added Version ID parameter for S3. 2022-03-14 22:03:58 -04:00
Anton Popov
ccbddd53a3 fix mutations in tables with enabled sparse columns 2022-03-15 01:48:21 +00:00
alesapin
fbb1ebd9b8
Merge pull request #35274 from CurtizJ/fix-check-table-sparse-columns 
Fix check table in case when there exist sparse columns
 2022-03-14 21:56:04 +01:00
Alexander Tokmakov
9702b5177d Merge branch 'master' into mvcc_prototype 2022-03-14 21:45:38 +01:00
Alexander Tokmakov
278d779a01 log cleanup, more comments 2022-03-14 21:43:34 +01:00
Maksim Kita
2fdcf53a76 Fix clang-tidy warnings in Server, Storages folders 2022-03-14 18:17:35 +00:00
Anton Popov
063917786e minor fixes 2022-03-14 17:29:18 +00:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Anton Popov
428bbd6377 fix check table in case when there exist sparse columns 2022-03-14 15:22:23 +00:00
Heena Bansal
c774458f96
Update src/Storages/StoragePostgreSQL.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-14 10:38:12 -04:00
Heena Bansal
01c8b2f71e
Update src/Storages/StorageMySQL.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-14 10:37:59 -04:00
Antonio Andelic
d3353f3f0c Merge branch 'master' into parallel-downloading-url-engine 2022-03-14 12:47:23 +00:00
Antonio Andelic
556fe2bcc5 Don't send RANGE with HEAD call 2022-03-14 12:46:22 +00:00
Antonio Andelic
ed3d71d83f
Merge pull request #35072 from azat/buffer-memory-tracker-leak 
Avoid MEMORY_LIMIT_EXCEEDED during INSERT into Buffer with AggregateFunction
 2022-03-14 12:31:04 +01:00
Maksim Kita
ad6b3693e1
Merge pull request #35123 from zhanghuajieHIT/fix_build_fail_with_gcc 
fix build fail with gcc
 2022-03-14 10:36:15 +01:00
Antonio Andelic
9dda2863d3 Split download threads when multiple URLs are used 2022-03-14 09:27:09 +00:00
HeenaBansal2009
09be30ac26 Added suggestion from review comments 2022-03-11 07:33:23 -08:00
zhanghuajie
53a8987b3b fix build fail with gcc --fix warnings without disabling some parameters 2022-03-11 21:59:19 +08:00
Antonio Andelic
f5d3a8a31d Polishing 2022-03-11 13:38:19 +00:00
taiyang-li
1f9e050152 add column distributed_depth for system.query_log and system.processes 2022-03-11 17:57:34 +08:00
Antonio Andelic
28e9508c4e Improve shared pool and add settings 2022-03-11 08:33:34 +00:00
Antonio Andelic
cce318273b Use a shared IO thread pool 2022-03-11 08:33:34 +00:00
Antonio Andelic
df0f5e20d0 Refactor 2022-03-11 08:33:34 +00:00
Antonio Andelic
e051587fc2 Format code 2022-03-11 08:33:34 +00:00
Antonio Andelic
29c32ed831 Refactor code 2022-03-11 08:33:34 +00:00
Antonio Andelic
0a1a3a230e Add support for parallel http range requests 2022-03-11 08:33:34 +00:00
Antonio Andelic
58557b9bec Test check for ranges support 2022-03-11 08:33:34 +00:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs 
Local cache for remote filesystem
 2022-03-11 07:23:10 +01:00
Alexander Tokmakov
7f47f20aba add kill transaction query 2022-03-10 22:29:58 +01:00
Anton Popov
37efe2ddb5
Apply suggestions from code review 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-10 22:24:19 +01:00
alesapin
c0d8ccc91b
Merge pull request #35178 from Varinara/master 
Added disk_name to system.part_log
 2022-03-10 22:22:37 +01:00
HeenaBansal2009
d981463b05 Added RemoteHostFilter check for MYSQL and postgresSQL 2022-03-10 08:58:48 -08:00
Maksim Kita
493169910b
Merge pull request #35174 from zhangyifan27/fix_typo 
fix typos
 2022-03-10 17:10:44 +01:00
Kseniia Sumarokova
3fc399b6e9
Merge pull request #35158 from kssenii/fix-materialized-postgresql 
Fix materialised postrgesql adding new table after manually removing it
 2022-03-10 17:02:32 +01:00
Kseniia Sumarokova
e30b0c5d57
Merge pull request #35162 from kssenii/fix-materialized-postgresql-table-override 
Fix materialised postgres `table overrides` for partition by, etc
 2022-03-10 17:01:24 +01:00
Varinara
f5523f7ff0 added disk_name to system.part_log 2022-03-10 18:44:19 +03:00
Rajkumar
3d3b6d1956 clang-tidy report issues with Medium priority 2022-03-10 07:23:49 -08:00
mergify[bot]
df01290e73
Merge branch 'master' into fix_typo 2022-03-10 13:35:04 +00:00
Alexander Tokmakov
061fa6a6f2 Merge branch 'master' into mvcc_prototype 2022-03-10 13:13:04 +01:00
kssenii
1dc3f36a11 Better 2022-03-10 12:19:20 +01:00
kssenii
787a0805a5 Merge master 2022-03-10 11:42:19 +01:00
Kseniia Sumarokova
e6ee891c9c
Merge pull request #34957 from bigo-sg/hive_random_access_file_cache 
Optimization for first time to read a random access readbuffer in hive
 2022-03-10 11:36:22 +01:00
zhangyifan27
e6fa9f699a fix typo 2022-03-10 18:29:42 +08:00
kssenii
3cd1da1e11 Fix 2022-03-10 11:11:59 +01:00
Alexander Tokmakov
0906b59fba fixes 2022-03-09 21:38:18 +01:00
kssenii
af9d8d278e Fix 2022-03-09 19:25:43 +01:00
Vladimir C
ce266b5a3e
Merge pull request #35146 from amosbird/fixpartitionprunerin 2022-03-09 13:23:45 +01:00
Nikolai Kochetov
6bfee7aca2
Merge pull request #35132 from azat/parallel_distributed_insert_select-view 
Support view() for parallel_distributed_insert_select
 2022-03-09 09:10:34 +01:00
Azat Khuzhin
e2960e1a52 Avoid MEMORY_LIMIT_EXCEEDED during INSERT into Buffer with AggregateFunction 
In case of Buffer table has columns of AggregateFunction type,
aggregate states for such columns will be allocated from the query
context but those states can be destroyed from the server context (in
case of background flush), and thus memory will be leaked from the query
since aggregate states can be shared, and eventually this will lead to
MEMORY_LIMIT_EXCEEDED error.

To avoid this, prohibit sharing the aggregate states.

But note, that this problem only about memory accounting, not memory
usage itself.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-09 10:57:49 +03:00
Amos Bird
a19224bc9b
Fix partition pruner: non-monotonic function IN 2022-03-09 15:48:42 +08:00
Azat Khuzhin
3a5a39a9df Do not delay final part writing by default 
For async s3 writes final part flushing was defered until all the INSERT
block was processed, however in case of too many partitions/columns you
may exceed max_memory_usage limit (since each stream has overhead).

Introduce max_insert_delayed_streams_for_parallel_writes (with default
to 1000 for S3, 0 otherwise), to avoid this.

This should "Memory limit exceeded" errors in performance tests.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:17:36 +03:00
Alexander Tokmakov
d2f838bd91 fix another race condition 2022-03-08 20:11:47 +01:00
Azat Khuzhin
4843e210c3 Support view() for parallel_distributed_insert_select 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:05:57 +03:00
Azat Khuzhin
ced34dea84 Take flush_time into account for scheduling background flush of the Buffer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 21:58:10 +03:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0 Fix split build 2022-03-08 18:05:55 +01:00
Azat Khuzhin
c4b6342853
Improvements for parallel_distributed_insert_select (and related) (#34728) 
* Add a warning if parallel_distributed_insert_select was ignored

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Respect max_distributed_depth for parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Print warning for non applied parallel_distributed_insert_select only for initial query

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Remove Cluster::getHashOfAddresses()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Forbid parallel_distributed_insert_select for remote()/cluster() with different addresses

Before it uses empty cluster name (getClusterName()) which is not
correct, compare all addresses instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix max_distributed_depth check

max_distributed_depth=1 must mean not more then one distributed query,
not two, since max_distributed_depth=0 means no limit, and
distribute_depth is 0 for the first query.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix INSERT INTO remote()/cluster() with parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add a test for parallel_distributed_insert_select with cluster()/remote()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Return <remote> instead of empty cluster name in Distributed engine

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Make user with sharding_key and w/o in remote()/cluster() identical

Before with sharding_key the user was "default", while w/o it it was
empty.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 15:24:39 +01:00
Antonio Andelic
bc5d7aea57
Merge pull request #34876 from azat/long-INSERT-fix 
Fix possible "Part directory doesn't exist" during INSERT
 2022-03-08 12:44:53 +01:00
Kseniia Sumarokova
1eb2bae792
Merge pull request #34954 from bigo-sg/hive_read_columns_pruning 
read columns pruning for hive
 2022-03-08 10:17:24 +01:00
lgbo-ustc
256e92ffee Merge remote-tracking branch 'ck/master' into hive_random_access_file_cache 2022-03-08 14:14:40 +08:00
Azat Khuzhin
caffc144b5 Fix possible "Part directory doesn't exist" during INSERT 
In #33291 final part commit had been defered, and now it can take
significantly more time, that may lead to "Part directory doesn't exist"
error during INSERT:

    2022.02.21 18:18:06.979881 [ 11329 ] {insert} <Debug> executeQuery: (from 127.1:24572, user: default) INSERT INTO db.table (...) VALUES
    2022.02.21 20:58:03.933593 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18044_18044_0 to 20220214_270654_270654_0.
    2022.02.21 21:16:50.961917 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18197_18197_0 to 20220214_270689_270689_0.
    ...
    2022.02.22 21:16:57.632221 [ 64878 ] {} <Warning> db.table: Removing temporary directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/
    ...
    2022.02.23 12:23:56.277480 [ 11329 ] {insert} <Trace> db.table: Renaming temporary part tmp_insert_20220214_18232_18232_0 to 20220214_273459_273459_0.
    2022.02.23 12:23:56.299218 [ 11329 ] {insert} <Error> executeQuery: Code: 107. DB::Exception: Part directory /clickhouse/data/db/table/tmp_insert_20220214_18232_18232_0/ doesn't exist. Most likely it is a logical error. (FILE_DOESNT_EXIST) (version 22.2.1.1) (from 127.1:24572) (in query: INSERT INTO db.table (...) VALUES), Stack trace (when copying this message, always include the lines below):

Follow-up for: #28760
Refs: #33291

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 07:44:11 +03:00
lgbo-ustc
a8cfc2458a update codes 2022-03-08 11:55:15 +08:00
taiyang-li
b4174b0bef merge master and fix conflicts 2022-03-08 11:39:25 +08:00
Maksim Kita
2f9361008b
Merge pull request #35089 from 1lann/1lann/fix-update_lag-typo 
Fix typo of update_lag
 2022-03-07 23:12:35 +01:00
Alexander Tokmakov
8acfb8d27f Merge branch 'master' into mvcc_prototype 2022-03-07 17:40:15 +01:00
Alexander Tokmakov
ea2f65fef6 fix tests with DiskS3 2022-03-07 17:35:47 +01:00
Kseniia Sumarokova
5511f2f6e6
Merge pull request #34940 from bigo-sg/hive_client_connection_pool 
Use connection pool in HiveMetastoreClient
 2022-03-07 17:14:56 +01:00
Kseniia Sumarokova
28b9ec01c0
Merge pull request #34945 from bigo-sg/hive_bug_fixed 
unexpected result when use `in` in hive query
 2022-03-07 17:13:11 +01:00
Anton Popov
0bc57da238 Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-07 14:46:08 +00:00
alesapin
d90bee1df9
Merge pull request #35080 from azat/mutate-exceptions 
Do not hide exceptions during mutations
 2022-03-07 12:22:49 +01:00
Vladimir C
678f05ca1e
Merge pull request #34912 from kssenii/fix-filelog-metadata-path 2022-03-07 11:45:29 +01:00
1lann
5423c5a45c Fix typo of update_lag 
In external dictionary providers, the allowed keys for configuration seemed to have a typo
of "update_lag" as "update_tag", preventing the use of "update_lag". This change fixes that.
 2022-03-07 18:31:20 +08:00
lgbo-ustc
8ae5296ee8 fixed compile errors 2022-03-07 17:26:48 +08:00
lgbo-ustc
eab925554d fixed code styles 2022-03-07 14:01:03 +08:00
lgbo-ustc
0c83b96d8c fixed code style 2022-03-07 13:13:33 +08:00
lgbo-ustc
cfeedd2cb5 fixed code style 2022-03-07 12:28:31 +08:00
lgbo-ustc
4507cc58aa update codes 2022-03-07 12:05:07 +08:00
lgbo-ustc
c37eedd887 update codes 2022-03-07 10:30:54 +08:00
lgbo-ustc
75a50a30c4 update codes 2022-03-07 09:43:53 +08:00
lgbo-ustc
d907b70cc4 update codes: get actual read block 2022-03-07 09:26:05 +08:00
lgbo-ustc
f4d8fb46c5 update codes 2022-03-07 09:26:05 +08:00
lgbo-ustc
62c1bd5ae9 hive read columns pruning 2022-03-07 09:26:05 +08:00
Azat Khuzhin
bc224dee36 Do not hide exceptions during mutations 
system.mutations includes only the message, but not stacktrace, and it
is not always obvious to understand the culprit w/o stacktrace.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-06 13:39:49 +03:00
Kseniia Sumarokova
3ec6cd3128
Update StorageFileLog.cpp 2022-03-06 11:03:22 +01:00
alexey-milovidov
f9b7df6ba1
Merge pull request #35050 from CurtizJ/fix-async-inserts-system-table 
Fix reading from `system.asynchronous_inserts` table
 2022-03-06 02:25:53 +03:00
Maksim Kita
7ae1f0fa3b
Merge pull request #34911 from larspars/master 
Allow LowCardinality strings for ngrambf_v1/tokenbf_v1 indexes. Fixes #21865
 2022-03-04 19:17:48 +01:00
Azat Khuzhin
2ef9d32448 Revert "Remove VERSION_DATE from system.build_options" 
As requested by @kitaisreal

This reverts commit 4a404532fb.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:32:54 +03:00
Azat Khuzhin
4a404532fb Remove VERSION_DATE from system.build_options 
It was set only bu utils/release/release_lib.sh, and seems that this
script is not used anymore, at least that part of it.

Also note, that GIT_DATE is the same, and it is date time, not only
date.

Plus VERSION_DATE is not installed for releases anyway.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:37 +03:00
Azat Khuzhin
b0f964a14f Remove LIBRARY_ARCHITECTURE from system.build_options 
CMAKE_LIBRARY_ARCHITECTURE and it is useless, since it is reported only
if the compiler reports subdir arch triplet [1]

  [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1531678

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:32 +03:00
Azat Khuzhin
494fe91f86 Fix LINK_FLAGS in system.build_options 
Fixes: 79f6f5a202
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:32 +03:00
Azat Khuzhin
c426eef07d Fix generating USE_* for system.build_options 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-04 15:31:32 +03:00
Anton Popov
c836a57000 fix reading from system.asynchronous_inserts table 2022-03-04 11:46:15 +00:00
Anton Popov
df3b07fe7c Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-03 22:25:28 +00:00
Anton Popov
aea7bfb59a
Merge pull request #34992 from azat/fix-asynchronous_inserts-race 
Fix race between INSERT async_insert=1 and system.asynchronous_inserts
 2022-03-03 20:55:19 +03:00
kssenii
d19f199e93 Revert 2022-03-03 15:25:27 +01:00
Kseniia Sumarokova
b11b34dc8c
Merge pull request #34849 from kssenii/fix-too-many-columns 
Fix reading too many columns for s3 and url storages
 2022-03-03 13:57:22 +01:00
Kseniia Sumarokova
ad09554c4c
Merge pull request #34996 from kssenii/fix-filelog-assertion 
Fix possible segfault in filelog storage
 2022-03-03 13:52:54 +01:00
Frank Chen
b4829465d9
Improve the opentelemetry span logs for INSERT on distributed table (#34480) 2022-03-03 12:53:29 +01:00
mergify[bot]
e169813004
Merge branch 'master' into fix-too-many-columns 2022-03-02 18:46:35 +00:00
Maksim Kita
b1a956c5f1 clang-tidy check performance-move-const-arg fix 2022-03-02 18:15:27 +00:00
Maksim Kita
1f5837359e clang-tidy check performance-noexcept-move-constructor fix 2022-03-02 18:15:27 +00:00
mreddy017
f893002b69 Fix vulnerable code related to std::move and noexcept 
This commit fixes the vulnerable code related to std::move and noexcept identified by clangtidy tool.
 2022-03-02 18:15:27 +00:00
kssenii
d5952109fb Merge master 2022-03-02 18:15:25 +01:00
mergify[bot]
add225c83e
Merge branch 'master' into fix-filelog-assertion 2022-03-02 17:06:08 +00:00
Amos Bird
d4cdf04683
Add missing locks (#34025) 2022-03-02 16:23:29 +01:00
kssenii
5e84c75942 Fix 2022-03-02 15:31:34 +01:00
kssenii
ef344a581b Fix bug in FileLog storage 2022-03-02 15:28:17 +01:00
alesapin
b1f5805647
Merge pull request #34609 from ClickHouse/unrestricted-zk-reads 
allow unrestricted reads from zookeeper
 2022-03-02 14:53:12 +01:00
Azat Khuzhin
57f636a1e8 Fix race between INSERT async_insert=1 and system.asynchronous_inserts 
CI report [1]:

    [c190f600f8c6] 2022.03.02 01:07:34.553012 [ 23552 ] {76b6113b-1479-46c9-90ab-e78a3c9f3dbb}  executeQuery: Code: 60. DB::Exception: Both table name and UUID are empty. (UNKNOWN_TABLE) (version 22.3.1.1) (from [::1]:42040) (comment: '02015_async_inserts_stress_long.sh') (in query: SELECT * FROM system.asynchronous_inserts FORMAT Null), Stack trace (when copying this message, always include the lines below):

    0. ClickHouse/contrib/libcxx/include/exception:133: Poco::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int) @ 0xf50e04c in /fasttest-workspace/build/programs/clickhouse
    1. ClickHouse/src/Common/Exception.cpp:58: DB::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int, bool) @ 0x663ebfa in /fasttest-workspace/build/programs/clickhouse
    2. DB::StorageID::assertNotEmpty() const @ 0xbc08591 in /fasttest-workspace/build/programs/clickhouse
    3. ClickHouse/contrib/libcxx/include/string:1444: DB::StorageID::getDatabaseName() const @ 0xe50d2b6 in /fasttest-workspace/build/programs/clickhouse
    4. ClickHouse/contrib/libcxx/include/string:1957: DB::StorageSystemAsynchronousInserts::fillData(std::__1::vector::mutable_ptr, std::__1::allocator::mutable_ptr > >&, std::__1::shared_ptr, DB::SelectQueryInfo const&) const @ 0xdac636c in /fasttest-workspace/build/programs/clickhouse

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/34973/e6fc6a22d5c018961c18247242dd3a40b8c54ff2/fast_test__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-02 15:28:06 +03:00
alesapin
9249c5d50e Use tryget instead of get 2022-03-02 13:09:12 +03:00
Maksim Kita
53116faeeb
Update MergeTreeIndexFullText.cpp 2022-03-02 11:08:35 +01:00
Kseniia Sumarokova
a9ab149b31
Merge pull request #34859 from Vxider/windowview-multi-column-groupby 
Fix bugs for multiple columns group by in WindowView
 2022-03-02 10:09:47 +01:00
Filatenkov Artur
f48f35cad0
Merge pull request #34975 from Vector-Similarity-Search-for-ClickHouse/fix-typo 
Fix typo
 2022-03-02 09:59:06 +03:00
Anton Popov
d7cd9aa69b fix reading of missed subcolumns 2022-03-02 03:31:40 +03:00
NikitaEvs
06f47673f4 Fix typo 2022-03-01 21:42:27 +00:00
alesapin
e2989c2b85 Fix storage system zookeeper 2022-03-01 21:23:26 +01:00
alesapin
ec3e4251e1 Fix style 2022-03-01 20:34:25 +01:00
kssenii
a594f388a4 Merge master 2022-03-01 19:43:45 +01:00
kssenii
aa1c71a877 Merge master 2022-03-01 19:25:04 +01:00
kssenii
755e63ed03 Keep compatibility 2022-03-01 19:21:59 +01:00
Anton Popov
c1fdcf7a64 Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-01 20:21:39 +03:00
Anton Popov
04a3a10148 minor fixes 2022-03-01 20:20:53 +03:00
alesapin
cba5fe44a9 Merge branch 'master' into unrestricted-zk-reads 2022-03-01 18:09:21 +01:00
Anton Popov
2758db5341 add more comments 2022-03-01 19:32:55 +03:00
taiyang-li
74bc819f12 fix conflict with master and update contrib/arrow version 2022-03-01 20:14:23 +08:00
kssenii
092ec45b47 Merge master 2022-03-01 12:06:56 +01:00
lgbo-ustc
ca470e1b94 lazy initialization about getting hive metadata in HiveStorage 2022-03-01 19:04:44 +08:00
Kseniia Sumarokova
781621eefe
Merge pull request #34946 from bigo-sg/hive_table_function 
Add hive table function
 2022-03-01 11:28:36 +01:00
alesapin
4b61e4795c
Merge pull request #34949 from nikitamikhaylov/system_log_tables_and_settings 
Recreate system.{*}_log table on settings changes
 2022-03-01 11:15:19 +01:00
lgbo-ustc
5ed41bda9b fixed code style 2022-03-01 17:20:32 +08:00
lgbo-ustc
5ae99df87c fxied code style 2022-03-01 15:35:57 +08:00
lgbo-ustc
6e568c1530 update codes 2022-03-01 15:24:40 +08:00
lgbo-ustc
91a45d799e optimization for first time to read a random access readbuffer 2022-03-01 15:22:07 +08:00
Mikhail Artemenko
96cfc7f07a add table function 2022-02-28 22:43:43 +03:00
Nikita Mikhaylov
d6036f6da3 Better 
(cherry picked from commit 4ae445c9e227581ea9f1cbe9aa9d1ba82e1236c9)
 2022-02-28 15:27:52 +00:00
Kruglov Pavel
011813957d
Merge pull request #34938 from azat/create-as-ignore-ttl 
Ignore per-column TTL in CREATE TABLE AS if new table engine does not support it
 2022-02-28 16:58:15 +03:00
kssenii
9b64a8fe39 Fix odbc bridge 2022-02-28 14:29:05 +01:00
lgbo-ustc
99cd25d70e add new table function: hive() 2022-02-28 20:51:33 +08:00
mergify[bot]
b4574a777f
Merge branch 'master' into mvcc_prototype 2022-02-28 11:24:39 +00:00
lgbo-ustc
6473767c99 fixed code style 2022-02-28 17:10:56 +08:00
lgbo-ustc
5885cfd869 fixed bug : unexpected result when using in clause for filtering partitions 2022-02-28 16:47:50 +08:00
Hongbin
c9bc442114
fix comments 2022-02-28 16:44:35 +08:00
Azat Khuzhin
644f9168fa Ignore per-column TTL in CREATE TABLE AS if new table engine does not support it 
Follow-up for: #6968
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-28 10:29:26 +03:00
lgbo-ustc
c5e02be44e fixed code-style 2022-02-28 15:22:54 +08:00
lgbo-ustc
2176d74cd1 Use connection pool in HiveMetastoreClient 
1. remove lock for hive metastore client access
2. auo reconnect when connection is broken
 2022-02-28 15:11:38 +08:00
Hongbin
99bd56e2de
Fix some code comments style 2022-02-28 08:15:37 +08:00
Mikhail Artemenko
54ea1c172a add named_collections 2022-02-26 21:58:37 +03:00
mergify[bot]
8d84d22618
Merge branch 'master' into windowview-multi-column-groupby 2022-02-26 00:50:49 +00:00
kssenii
2ba9010a34 Fix 2022-02-25 17:53:19 +01:00
kssenii
6c8401bfbd Fix 2022-02-25 16:35:37 +01:00
Lars Eidnes
2629614dfe Allow LowCardinality strings for ngrambf_v1/tokenbf_v1 indexes. Fixes #21865 2022-02-25 15:36:36 +01:00
Anton Popov
fcdebea925 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-25 13:41:30 +03:00
kssenii
cfad79bf74 Remove redundant 2022-02-25 09:37:48 +01:00
Alexander Tokmakov
11ae0d144b fix 2022-02-25 00:51:21 +03:00
Sergei Trifonov
2d25c79e37 analyze select queries from system.zookeeper table with LIKE pattern and fetch nodes using prefix recursively 2022-02-24 13:40:47 +03:00
Vxider
06469eb793 remove blank row 2022-02-24 14:16:24 +08:00
Vxider
43475f79bf windowview_multi_column_groupby 2022-02-24 14:06:37 +08:00
Alexander Tokmakov
711aad6953 fix 2022-02-24 01:31:21 +03:00
Alexander Tokmakov
aa6b9a2abc Merge branch 'master' into mvcc_prototype 2022-02-23 23:22:03 +03:00
kssenii
003b807b00 Fix 2022-02-23 20:33:05 +01:00
tavplubix
43626b3ffd
Update src/Storages/FileLog/StorageFileLog.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-02-23 21:07:37 +03:00
Alexander Tokmakov
5a26f856d9 remove trash that shouldn't have been merged 2022-02-22 23:41:33 +03:00
Dmitry Novik
2fd4baaa64
Merge pull request #34387 from nvartolomei/nv/move-part-settings-cleanup 
Remove useless setting experimental_query_deduplication_send_all_part_uuids
 2022-02-22 06:11:00 -08:00
Sergei Trifonov
7fe3bef866 add test for unrestricted zk reads 2022-02-22 16:51:30 +03:00
kssenii
c637385dd0 Merge master 2022-02-22 13:17:51 +01:00
Kseniia Sumarokova
eeea322556
Merge pull request #34629 from amosbird/remotefsimprove 
Some refactoring and improvement over async and remote buffer related stuff
 2022-02-22 11:36:40 +01:00
Dmitry Novik
1df43a7f57
Merge pull request #34385 from nvartolomei/nv/move-part-count 
Disable optimize_trivial_count when deduplication for part movement feature is enabled
 2022-02-21 08:53:09 -08:00
Anton Popov
065305ab65
Merge pull request #34764 from ucasfl/hints-index 
Add name hints for data skipping indices
 2022-02-21 16:50:59 +03:00
Mikhail f. Shiryaev
5ac8cdbc69
Merge pull request #34786 from ClickHouse/make_drop_column_metadata_only 
Make drop of alias column metadata only
 2022-02-21 14:11:55 +01:00
mergify[bot]
314ab73b11
Merge branch 'master' into nv/move-part-settings-cleanup 2022-02-21 10:18:44 +00:00
Dmitry Novik
4428e7aa1b
Merge branch 'master' into nv/move-part-count 2022-02-21 02:14:23 -08:00
alesapin
d7cae5ffb4 Fix build 2022-02-21 11:54:52 +03:00
alesapin
852757219f Make drop of alias column metadata only 2022-02-21 11:46:16 +03:00
Vitaly Baranov
aee67a6693
Merge pull request #31484 from eungenue/Implement-SSL-X509-certificate-authentication 
Implement ssl x509 certificate authentication
 2022-02-21 11:30:52 +03:00
Vitaly Baranov
0d377de5f0 Support syntax CREATE USER IDENTIFIED WITH ssl_certificate CN ... 2022-02-21 07:01:00 +03:00
Vitaly Baranov
7b97c986cb
Revert "Allow restrictive row policies without permissive" 2022-02-21 06:54:28 +03:00
feng lv
07280e0ab1 Add name hints for data skipping indices 
fix test
 2022-02-20 11:48:22 +00:00
Vitaly Baranov
874b2c8dcb
Merge pull request #34596 from vitlibar/allow-restrictive-without-permissive 
Allow restrictive row policies without permissive
 2022-02-19 21:45:28 +07:00
Azat Khuzhin
fef5f146e7 Fix ENOENT with fsync_part_directory and Vertical merge 
fsync of the temporary part directory is superfluous anyway, and besides
that directory is not exists at that time, that will lead to ENOENT
error:

    2022.02.18 17:02:51.634565 [ 35639 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/system/text_log/tmp_merge_202202_1864_3192_14/, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

    0. DB::Exception::Exception() @ 0xb26ecfa in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    1. DB::throwFromErrnoWithPath() @ 0xb2700ea in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    2. DB::LocalDirectorySyncGuard::LocalDirectorySyncGuard() @ 0x14905531 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    3. DB::DiskLocal::getDirectorySyncGuard() const @ 0x148af3e3 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    4. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x157bef13 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug

Note, that IMergeTreeDataPart::renameTo() anyway will have fsync for the
directory.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-19 07:50:59 +03:00
Nikolai Kochetov
e4d5db6161
Merge pull request #34717 from azat/merge-mutate-memory-tracker 
Fix possible memory_tracker use-after-free (for async s3 writes) for merges/mutations
 2022-02-18 19:28:43 +01:00
Vladimir C
9b7d011ee7
Merge pull request #34529 from vdimir/join-nullable-on-pipeline 
Apply join_use_nulls on types before join
 2022-02-18 18:34:44 +01:00
Azat Khuzhin
65e9b4879d Fix possible memory_tracker use-after-free for merges/mutations 
There are two possible cases for execution merges/mutations:
1) from background thread
2) from OPTIMIZE TABLE query

1) is pretty simple, it's memory tracking structure is as follow:

    current_thread::memory_tracker = level=Thread / description="(for thread)" ==
      background_thread_memory_tracker = level=Thread / description="(for thread)"
    current_thread::memory_tracker.parent = level=Global / description="(total)"

  So as you can see it is pretty simple and MemoryTrackerThreadSwitcher
  does not do anything icky for this case.

2) is complex, it's memory tracking structure is as follow:

    current_thread::memory_tracker = level=Thread / description="(for thread)"
    current_thread::memory_tracker.parent = level=Process / description="(for query)" ==
      background_thread_memory_tracker = level=Process / description="(for query)"

  Before this patch to track memory (and related things, like sampling,
  profiling and so on) for OPTIMIZE TABLE query dirty hacks was done to
  do this, since current_thread memory_tracker was of Thread scope, that
  does not have any limits.

  And so if will change parent for it to Merge/Mutate memory tracker
  (which also does not have some of settings) it will not be correctly
  tracked.

  To address this Merge/Mutate was set as parent not to the
  current_thread memory_tracker but to it's parent, since it's scope is
  Process with all settings.

  But that parent's memory_tracker is the memory_tracker of the
  thread_group, and so if you will have nested ThreadPool inside
  merge/mutate (this is the case for s3 async writes, which has been
  added in #33291) you may get use-after-free of memory_tracker.

  Consider the following example:

    MemoryTrackerThreadSwitcher()
      thread_group.memory_tracker.parent = merge_list_entry->memory_tracker
      (see also background_thread_memory_tracker above)

    CurrentThread::attachTo()
      current_thread.memory_tracker.parent = thread_group.memory_tracker

    CurrentThread::detachQuery()
      current_thread.memory_tracker.parent = thread_group.memory_tracker.parent
      # and this is equal to merge_list_entry->memory_tracker

    ~MemoryTrackerThreadSwitcher()
      thread_group.memory_tracker = thread_group.memory_tracker.parent

  So after the following we will get incorrect memory_tracker (from the
  mege_list_entry) when the next job in that ThreadPool will not have
  thread_group, since in this case it will not try to update the
  current_thread.memory_tracker.parent and use-after-free will happens.

So to address the (2) issue, settings from the parent memory_tracker
should be copied to the merge_list_entry->memory_tracker, to avoid
playing with parent memory tracker.

Note, that settings from the query (OPTIMIZE TABLE) is not available at
that time, so it cannot be used (instead of parent's memory tracker
settings).

v2: remove memory_tracker.setOrRaiseHardLimit() from settings

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-18 16:23:54 +03:00
Amos Bird
f459e8fc95
Less getMark calls 2022-02-18 19:55:19 +08:00
Alexander Tokmakov
f4a46a13fb fixes 2022-02-18 00:26:37 +03:00
Anton Popov
0a7895ebb9 add comments and small refactoring 2022-02-17 22:00:25 +03:00
zvonand
90c857c5e3 merge 2022-02-17 18:23:37 +03:00
tavplubix
0f5ee19d0b
Merge pull request #34633 from zhangjmruc/master 
For ReplatedMergeTree, early break for multiple leaders case when log has been updated by the other leader
 2022-02-17 14:01:50 +03:00
Alexander Tokmakov
dae044f86b Merge branch 'master' into mvcc_prototype 2022-02-17 13:49:37 +03:00
Kruglov Pavel
6dcb766879
Merge pull request #34465 from Avogar/fix-url-globs 
Improve schema inference with globs in FIle/S3/HDFS/URL engines
 2022-02-17 13:33:27 +03:00
Vitaly Baranov
2de6e8e575 Change type of RowPolicyKind: bool -> enum. 2022-02-17 14:18:10 +07:00
Amos Bird
d3bd8b5f93
Cosmetic fix 2022-02-17 14:31:22 +08:00
Amos Bird
ba19c7cf44
Slightly better interface of compressed buffer 2022-02-17 14:31:22 +08:00
Jianmei Zhang
ef0c3b99ff Merge remote-tracking branch 'upstream/master' 2022-02-17 14:02:27 +08:00
Sergei Trifonov
b6bb479c48 add setting to enable unrestricted reads from zookeeper 2022-02-16 23:03:44 +03:00
Sergei Trifonov
f342c497ef fix style 2022-02-16 20:22:03 +03:00
Azat Khuzhin
774744a86d Fix allow_experimental_projection_optimization with enable_global_with_statement 
allow_experimental_projection_optimization requires one more
InterpreterSelectQuery, which with enable_global_with_statement will
apply ApplyWithAliasVisitor if the query is not subquery.

But this should not be done for queries from
MergeTreeData::getQueryProcessingStage()/getQueryProcessingStageWithAggregateProjections()
since this will duplicate WITH statements over and over.

This will also fix scalar.xml perf tests, that leads to the following
error now:

    scalar.query0.prewarm0: DB::Exception: Stack size too large.

And since it has very long query in the log, this leads to the following
perf test error:

    _csv.Error: field larger than field limit (131072)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-16 19:14:47 +03:00
kssenii
47f94120da Merge master 2022-02-16 14:43:28 +01:00
kssenii
3bd3e51aa0 Fix tests 2022-02-16 14:08:41 +01:00
Mikhail f. Shiryaev
4f84406136
Merge pull request #34641 from ClickHouse/version-and-release 
refactor version_helper, create release script
 2022-02-16 14:00:55 +01:00
Maksim Kita
d6e88f56cd
Merge pull request #34623 from CurtizJ/minor-subcolumns-fix 
Fix quadratic complexity while adding subcolumns
 2022-02-16 12:38:00 +01:00
Mikhail f. Shiryaev
c5db40f679
Deprecate sh script for StorageSystemContributors, update generated file 2022-02-16 12:16:43 +01:00
Nikolai Kochetov
f9d2dae88e
Merge pull request #34424 from yakov-olkhovskiy/ephemeral-column 
Ephemeral column issue #9436
 2022-02-16 12:04:57 +01:00
Kruglov Pavel
dd863ca2a0
Merge branch 'master' into fix-url-globs 2022-02-16 12:45:31 +03:00
Jianmei Zhang
25c761b3b6 Early break for multiple leaders case when log updated by other leader 2022-02-16 16:06:41 +08:00
taiyang-li
a4baec6d26 fix building 2022-02-16 15:12:43 +08:00
taiyang-li
afcb295273 fix compile error 2022-02-16 14:51:56 +08:00
taiyang-li
f19f0d847f fix code style 2022-02-16 12:23:06 +08:00
taiyang-li
dec083ab44 add minmax index for hivengine; remove libhdfspp dependency for hive engine 2022-02-16 12:19:16 +08:00
Anton Popov
e4fddaa03a fix quadratic complexity while adding subcolumns 2022-02-16 02:42:50 +03:00
Anton Popov
a661eaf39f better performance of getting storage snapshot 2022-02-16 02:17:22 +03:00
alesapin
bc2d0ee7c7
Merge pull request #34215 from ClickHouse/revert-34211-revert-34153-add_func_tests_over_s3 
Add func tests run with s3 and fix several bugs
 2022-02-15 19:07:11 +03:00
Alexander Tokmakov
e37ef4560c fix 2022-02-15 18:00:45 +03:00
Sergei Trifonov
a507f83d8d allow unrestricted reads from zookeeper 2022-02-15 17:12:37 +03:00
Nikolai Kochetov
ab288642f6 Merge branch 'master' into ephemeral-column 2022-02-15 10:03:34 +00:00
Nikolai Kochetov
d6cbac1ed3
Merge pull request #34577 from ClickHouse/alwasy-remove-unused-actions-for-add-missing-defaults 
Always remove unused actions from addMissingDefaults
 2022-02-15 11:01:29 +01:00
alesapin
447cd56cb9 Fix comments 2022-02-15 12:11:50 +03:00
李扬
f52b67b939
Merge branch 'master' into rocksdb_metacache 2022-02-15 02:16:29 -06:00
Alexander Tokmakov
1e4e569151 Merge branch 'master' into mvcc_prototype 2022-02-15 02:26:47 +03:00
Alexander Tokmakov
ae5aa8c12d write part version before other files 2022-02-15 02:24:51 +03:00
Alexander Tokmakov
cbd3b45646 add EXPLAIN CURRENT TRANSACTION 2022-02-14 22:47:17 +03:00
alesapin
e15396d90c Fix race condition: 2022-02-14 22:19:49 +03:00
Nikolai Kochetov
b3ea360cd2 Fix a little bit more 2022-02-14 19:05:30 +00:00
Kseniia Sumarokova
382b8e0388
Merge pull request #34432 from ClickHouse/static-files-disk-uploader-create-symlinks 
`static-files-disk-uploader`: add a mode to create symlinks
 2022-02-14 18:10:53 +01:00
vdimir
99ca89c0ca
Fix StorageJoin and Asof or join_use_nulls in pipeline 2022-02-14 14:14:27 +00:00
alesapin
bb69455395
Merge pull request #34504 from CurtizJ/ttl-move-if-exists 
Support `TTL TO [DISK|VOLUME] [IF EXISTS]`
 2022-02-14 14:56:18 +03:00
alesapin
b75d551281 Fix clang tidy and add check for master 2022-02-14 14:37:41 +03:00
alesapin
b2886a429b Fix lock during fetch 2022-02-14 12:20:27 +03:00
alesapin
beb4400978 Fix 'same local part' check 2022-02-13 23:08:29 +03:00
alesapin
89373155fc Merge branch 'master' into revert-34211-revert-34153-add_func_tests_over_s3 2022-02-13 21:07:54 +03:00
Yakov Olkhovskiy
579fe6c97a major rework, transform added to the insert pipe 2022-02-13 17:42:59 +00:00
zvonand
dcc0f53f8e updated Interval definitions 2022-02-13 17:54:03 +03:00
Maksim Kita
cb09aadc72
Merge pull request #34524 from FrankChen021/urlengine 
Fix compression support in URL engine
 2022-02-13 14:39:36 +01:00
Maksim Kita
ae1da31d19
Merge pull request #34558 from ucasfl/storage-log 
try fix data race in StorageLog
 2022-02-13 14:33:36 +01:00
mergify[bot]
8a47ed2b3f
Merge branch 'master' into static-files-disk-uploader-create-symlinks 2022-02-13 06:12:39 +00:00
alexey-milovidov
1774836d4f
Merge pull request #34555 from kitaisreal/table-functions-insert-partition-by-refactoring 
TableFunctionFile added performance test
 2022-02-13 09:08:51 +03:00
feng lv
4ccd1c3278 try fix data race in StorageLog 2022-02-13 04:52:16 +00:00
Maksim Kita
e2c8ba9ab2 Added performance test 2022-02-12 16:05:35 +00:00
李扬
daa27d0bda
Merge branch 'master' into rocksdb_metacache 2022-02-12 07:50:12 -06:00
alexey-milovidov
747b6b2058
Merge pull request #34528 from CurtizJ/key-condition-float 
Fix comparison between integers and floats in index analysis
 2022-02-12 10:19:09 +03:00
alexey-milovidov
ea71dc9d11
Merge pull request #34510 from kitaisreal/table-functions-insert-partition-by-refactoring 
Improve performance of insert into table functions URL, S3, File, HDFS
 2022-02-12 10:14:00 +03:00
alexey-milovidov
9bb2eba281
Merge pull request #34429 from ClickHouse/filimonov-SystemAsynchronousInserts 
asynchronous_inserts engine AsynchronousInserts -> SystemAsynchronousInserts
 2022-02-12 10:08:04 +03:00
Frank Chen
7b7bc8f6c3 Address review comment 2022-02-12 10:46:01 +08:00
Dmitry Novik
50131fd732 Fix cancelation for S3 and HDFS 2022-02-11 18:14:55 +00:00
Anton Popov
6a8e35930f fix comparison with integers and floats in index analysis 2022-02-11 18:20:37 +03:00
tavplubix
6d1fae834a
Merge pull request #34187 from qoega/default-table-engine 
Default table engine
 2022-02-11 18:14:29 +03:00
Anton Popov
2fcd69baf7 fix comparison with integers and floats in index analysis 2022-02-11 17:15:27 +03:00
Alexander Tokmakov
07e66e690d Merge branch 'master' into mvcc_prototype 2022-02-11 15:53:32 +03:00
mergify[bot]
f099048dd1
Merge branch 'master' into urlengine 2022-02-11 11:03:21 +00:00
Frank Chen
c8cddd7077 Fix compression for URL engine 2022-02-11 17:45:49 +08:00
Anton Popov
f012871a7c better caching of common types of object columns 2022-02-11 01:20:30 +03:00
Maksim Kita
13cbf79ecb Improve performance of insert into table functions URL, S3, File, HDFS 2022-02-10 20:06:23 +00:00
alesapin
705529ca03 Followup 2022-02-10 22:50:15 +03:00
alesapin
ef61c9b47c fix 2022-02-10 22:49:33 +03:00
alesapin
3af06b23f8 POC 2022-02-10 22:45:52 +03:00
alesapin
088f8cacf3 Small improvements 2022-02-10 19:50:21 +03:00
Anton Popov
70986a70a1 support TTL TO [DISK|VOLUME] [IF EXISTS] 2022-02-10 19:26:23 +03:00
Kruglov Pavel
bf8c028533
Merge pull request #34479 from Avogar/fix-filesystem-error 
Fix possible error 'file_size: Operation not supported'
 2022-02-10 14:17:05 +03:00
alesapin
f764da35ca Also zero copy mutations 2022-02-10 14:15:08 +03:00
Kruglov Pavel
a4f5610764
Merge pull request #34476 from CurtizJ/avoid-settings-copy 
Avoid unnecessary copying of `Settings`
 2022-02-10 14:13:46 +03:00
alesapin
70221b272b Better solution 2022-02-10 12:57:11 +03:00
Anton Popov
298838f891 avoid unnecessary copying of Settings 2022-02-10 12:13:51 +03:00
avogar
bfa96463ca Fix possible error 'file_size: Operation not supported' 2022-02-10 09:23:27 +03:00
Anton Popov
dcd7312d75 cache common type on objects in MergeTree 2022-02-09 23:47:53 +03:00