thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-28 02:21:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
104,963 Commits 9,188 Branches 1,782 Tags 2.2 GiB
6c4f53bdca
Commit Graph

5523 Commits

Author SHA1 Message Date
vdimir
08b8d335f0
add integration/test_temporary_data_in_cache 2022-12-01 11:49:25 +00:00
vdimir
7a814d2854
wip: temporary files evict fs cache 2022-12-01 11:49:24 +00:00
Elena Torró
5630412944
Merge branch 'master' into fix_reset_setting_on_cluster 2022-11-30 21:47:43 +01:00
Elena
0a35a072d9 Add RESET_SETTING to check if is a settings alter 2022-11-30 21:44:19 +01:00
Robert Schulze
6f564c59bd
Merge pull request #42641 from Enmk/fix_profile_names_mismatch 
Fixed exception when user tries to log in
 2022-11-30 13:01:57 +01:00
Anton Popov
df6d2f2163 Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-29 23:21:23 +00:00
Alexey Milovidov
72bf1115e9
Merge branch 'master' into password-reset-message 2022-11-29 23:51:46 +01:00
Vasily Nemkov
3d0c07ac5b Updated tests to match improved behaviour of currentProfiles() 2022-11-30 02:32:27 +04:00
Anton Popov
2a1fd48e91 fix tests 2022-11-29 17:33:35 +00:00
robot-clickhouse
ad66e01ace Automatic style fix 2022-11-29 14:48:05 +00:00
Alexander Tokmakov
12a7d64ef5
Update test.py 2022-11-29 17:33:02 +03:00
Antonio Andelic
db46842c8a Merge branch 'master' into more-aggresive-batching-keeper 2022-11-29 12:58:33 +00:00
alesapin
a800a80ab6
Merge pull request #43770 from ClickHouse/fix_flaky_store_cleanup2 
Fix flaky `test_store_cleanup`
 2022-11-29 13:11:57 +01:00
Antonio Andelic
e39f97a255 Properly close connections and add retries 2022-11-29 09:13:23 +00:00
Anton Popov
486da48ae7 fix tests 2022-11-28 21:15:41 +00:00
Alexander Tokmakov
40d28e3489 fix flaky test 2022-11-28 18:31:22 +01:00
Alexander Tokmakov
06e6844ec0
Update test.py 2022-11-28 18:22:03 +03:00
Antonio Andelic
af7f0257c3
Merge pull request #43658 from ClickHouse/try-fix-flaky-keeper-map 
Fix flaky `KeeperMap` integration tests
 2022-11-28 12:04:04 +01:00
Nikita Taranov
8ed5cfc265
Memory bound merging for distributed aggregation in order (#40879) 
* impl

* fix style

* make executeQueryWithParallelReplicas similar to executeQuery

* impl for parallel replicas

* cleaner code for remote sorting properties

* update test

* fix

* handle when nodes of old versions participate

* small fixes

* temporary enable for testing

* fix after merge

* Revert "temporary enable for testing"

This reverts commit cce7f8884c.

* review fixes

* add bc test

* Update src/Core/Settings.h
 2022-11-28 00:41:31 +01:00
Antonio Andelic
cade21b76e Fix 4LW test 2022-11-26 19:24:10 +00:00
Antonio Andelic
65dba31c3a
Merge branch 'master' into try-fix-flaky-keeper-map 2022-11-26 19:00:30 +01:00
Sema Checherinda
15a6ce254b
Merge pull request #41145 from CheSema/lock-free-drop-partition 
Delete parts by replacing them with empty parts
 2022-11-25 12:54:30 +01:00
Antonio Andelic
f3b01d455b Use dummy pool 2022-11-25 11:12:01 +00:00
Anton Popov
c65be55da1 Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-24 14:30:23 +00:00
Alexander Tokmakov
52d33fff00
Merge pull request #43366 from zzsmdfj/issue_15357_MaterializeMySQL_support_drop_mulit_table 
MaterializeMySQL_support_drop_mulit_table
 2022-11-24 14:54:24 +03:00
Sema Checherinda
6fd7dcf26f
Merge branch 'master' into lock-free-drop-partition 2022-11-24 12:34:19 +01:00
alesapin
f64f0461fe
Merge pull request #43288 from ClickHouse/Enable_compress_marks_and_primary_key 
Enable compress marks and primary key for CI
 2022-11-24 12:24:04 +01:00
Sema Checherinda
4f592bc208 fix isolation breach, droping outdated part should't be ignored 2022-11-23 15:16:09 +00:00
Sema Checherinda
f0294cd6ed fix test_merge_tree_s3_restore 2022-11-23 15:16:09 +00:00
Sema Checherinda
597421014e fix style check 2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a fix several critical races, fix tests 2022-11-23 15:16:08 +00:00
Sema Checherinda
49539e6862 fix test_partition 2022-11-23 15:16:08 +00:00
Sema Checherinda
62d3e9da4d fix test_merge_tree_hdfs 2022-11-23 15:16:08 +00:00
Sema Checherinda
d43d7ed6ea fix test_merge_tree_empty_parts and test_ttl_replicated 2022-11-23 15:16:08 +00:00
Sema Checherinda
dd93d331c1 fix 01168_mutations_isolation 2022-11-23 15:16:08 +00:00
Sema Checherinda
9edad4a078 make test_detached_parts_metrics wait delete inactive and empty parts 2022-11-23 15:16:08 +00:00
Sema Checherinda
44a8f2f460 work with test_merge_tree_s3 and test_merge_tree_s3_restore 2022-11-23 15:16:08 +00:00
Sema Checherinda
cddf44034d address test_transactions 2022-11-23 15:16:08 +00:00
Sema Checherinda
dc19141d84 fix test_detached_parts_metrics 2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070 address the review comments 2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9 Delete parts over replacing them with empty parts 2022-11-23 15:16:08 +00:00
Alexander Tokmakov
f44eb4e021
Update test.py 2022-11-23 14:10:49 +03:00
kssenii
6044a9257c Merge remote-tracking branch 'upstream/master' into named-collections-sql-commands 2022-11-23 12:00:55 +01:00
Alexey Milovidov
bedfe68503
Merge branch 'master' into nv/fix-mutate-fetch 2022-11-22 20:06:56 +01:00
Dmitry Novik
e67bda65a0
Merge pull request #43453 from ClickHouse/fix-hard-limit-in-test 
Do not reuse jemalloc memory in test_global_overcommit
 2022-11-22 13:24:34 +01:00
Alexander Tokmakov
cab33c02af
Merge branch 'master' into revert_SingleValueDataString 2022-11-22 14:27:58 +03:00
Antonio Andelic
843401c3a3
Merge pull request #43323 from ClickHouse/fix-describe-delta-lake-hudi 
Fix `DESCRIBE` for `deltaLake` and `hudi` table functions
 2022-11-22 09:09:39 +01:00
kssenii
a279bac767 Merge remote-tracking branch 'upstream/master' into named-collections-sql-commands 2022-11-21 21:49:34 +01:00
Kruglov Pavel
581e57be9f
Merge pull request #43236 from xiedeyantu/fix_hdfscluster_connection_fail 
fix skip_unavailable_shards does not work using hdfsCluster table function
 2022-11-21 19:22:16 +01:00
Igor Nikonov
47ce029cbf
Merge pull request #43326 from ClickHouse/igor/test_read_only_fast_fix 
Fix: make test_read_only_table more stable
 2022-11-21 18:25:14 +01:00
Dmitry Novik
b17dc24a94 Do not reuse jemalloc memory in test 2022-11-21 16:10:47 +00:00
Raúl Marín
35e9e169a1
Merge branch 'master' into revert_SingleValueDataString 2022-11-21 15:11:13 +01:00
Kseniia Sumarokova
a59dac6337
Merge pull request #43143 from azat/disks/s3-check-fix 
Add server UUID for disks access checks (read/read-by-offset/write/delete) to avoid possible races
 2022-11-21 15:03:46 +01:00
alesapin
b78b602ea1
Merge branch 'master' into Enable_compress_marks_and_primary_key 2022-11-21 12:05:04 +01:00
Antonio Andelic
3cb202a63b
Merge pull request #43026 from JackyWoo/keeper_manually_assign_leader 
Keeper support manually assigning leader
 2022-11-21 09:30:00 +01:00
Antonio Andelic
9a8d82f0fc Merge branch 'master' into fix-describe-delta-lake-hudi 2022-11-21 08:23:33 +00:00
zzsmdfj
a0f391745d to 15357_MaterializeMySQL_support_drop_mulit_table-fix style-black 2022-11-21 11:48:57 +08:00
Azat Khuzhin
3e42ffd372 tests: fix hdfs disks (chicken and an egg problem with current cluster.py) 
Right now cluster.py first creates HDFS and then creates clickhouse in
one API call, so you cannot interract and add missing mkdirs for the
clickhouse, fix this by using root dir where it is possible.
 2022-11-20 16:28:35 +01:00
Azat Khuzhin
0678fba3d1 tests: fix test_disk_types (by adding mkdir for HDFS) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-20 16:28:35 +01:00
Kseniia Sumarokova
f0dbfbb0f4
Merge pull request #42800 from azat/disks/web-fix 
Do not suppress exceptions in web disk (and fix retries for requests from web disk)
 2022-11-20 16:07:45 +01:00
Alexander Tokmakov
9464c32bb8 Merge branch 'master' into revert_SingleValueDataString 2022-11-19 18:00:04 +01:00
Azat Khuzhin
c029549859 Allow to drop tables from s3_plain disk (so as from web disk) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-19 10:10:27 +01:00
zzsmdfj
e177927efb to 15357_MaterializeMySQL_support_drop_mulit_table-fix code-style 2022-11-19 12:05:27 +08:00
alesapin
c7d648174b
Merge pull request #43328 from ClickHouse/better-format-version-check 
Check content of `format_version` file in `MergeTreeData`
 2022-11-18 22:34:37 +01:00
Vitaly Baranov
a348332eab
Merge pull request #43227 from vitlibar/improve-masking-sensitive-info 
Improve masking sensitive info
 2022-11-18 15:37:50 +01:00
zzsmdfj
9239f50a87
Merge branch 'master' into issue_15357_MaterializeMySQL_support_drop_mulit_table 2022-11-18 21:33:31 +08:00
zzsmdfj
6a998917b0 to MaterializeMySQL_support_drop_mulit_table 2022-11-18 21:29:13 +08:00
alesapin
06894534e8 Followup fix 2022-11-17 18:52:18 +01:00
robot-clickhouse
8c55679b01 Automatic style fix 2022-11-17 17:13:23 +00:00
alesapin
cecf032b9e Disable compressed marks for some tests 2022-11-17 18:06:59 +01:00
kssenii
69d71a8e5f Mask collection values in logs 2022-11-17 16:01:42 +01:00
robot-clickhouse
926b35b7c4 Automatic style fix 2022-11-17 14:46:25 +00:00
kssenii
5d9b34b59d More tests 2022-11-17 15:34:35 +01:00
robot-clickhouse
13e051a5de Automatic style fix 2022-11-17 13:46:21 +00:00
Antonio Andelic
d41af8b741 Check and compare content of format_version file 2022-11-17 13:41:36 +00:00
Igor Nikonov
d2d6e75d97 Fix: make test_read_only_table more stable 
+ add retries to INSERT queries after keeper node restart
 2022-11-17 13:26:32 +00:00
Nikita Taranov
7beb58b0cf
Optimize merge of uniqExact without_key (#43072) 
* impl for uniqExact

* rm unused (read|write)Text methods

* fix style

* small fixes

* impl for variadic uniqExact

* refactor

* fix style

* more agressive inlining

* disable if max_threads=1

* small improvements

* review fixes

* Revert "rm unused (read|write)Text methods"

This reverts commit a7e7480584.

* encapsulate is_able_to_parallelize_merge in Data

* encapsulate is_exact & argument_is_tuple in Data
 2022-11-17 13:19:02 +01:00
robot-clickhouse
1ad7362db7 Automatic style fix 2022-11-17 11:54:13 +00:00
Antonio Andelic
d5848d53cc Fix DESCRIBE for deltaLake and hudi table functions 2022-11-17 11:46:17 +00:00
kssenii
1964291fba Add tests 2022-11-17 12:35:43 +01:00
JackyWoo
5a09de279c fix test 2022-11-17 10:35:01 +08:00
chen
cb83337586
Update test.py 2022-11-17 06:41:43 +08:00
xiedeyantu
fc1d06f096 fix 2022-11-16 23:43:23 +08:00
Raúl Marín
45a611bc1d Fix style with black 2022-11-16 13:28:04 +01:00
Raúl Marín
4e7ccd1c06 Merge remote-tracking branch 'blessed/master' into revert_SingleValueDataString 2022-11-16 12:59:47 +01:00
xiedeyantu
7dc941cacd fix 2022-11-16 19:47:57 +08:00
Alexander Tokmakov
7e6fe94a8e
Merge pull request #43266 from ClickHouse/global-overcommit-test 
Make test_global_overcommit_tracker non-parallel
 2022-11-16 14:28:56 +03:00
alesapin
32358d2180
Merge pull request #43256 from ClickHouse/vdimir/msg_in_wait_zookeeper_to_start 
Better message in wait_zookeeper_to_start
 2022-11-16 11:58:53 +01:00
JackyWoo
01b63fd98a fix style 2022-11-16 12:51:51 +08:00
Anton Popov
f94182384a Merge remote-tracking branch 'upstream/master' into HEAD 2022-11-16 03:30:29 +00:00
xiedeyantu
c23cd091a3 fix 2022-11-16 09:59:44 +08:00
JackyWoo
f8d71f60a2
Merge branch 'master' into keeper_manually_assign_leader 2022-11-16 09:34:40 +08:00
xiedeyantu
8144516e53 fix 2022-11-16 09:33:46 +08:00
JackyWoo
ace40a2538 fix tests 2022-11-16 09:33:46 +08:00
xiedeyantu
3d218795be fix 2022-11-16 09:18:45 +08:00
Vitaly Baranov
2e242fa991 Add tests. 2022-11-16 01:35:11 +01:00
Dmitry Novik
fcdc9dfaac Make test_global_overcommit_tracker non-parallel 2022-11-15 20:52:18 +00:00
Vitaly Baranov
22abe9a4fb
Merge pull request #42950 from azat/disks/s3_plain/wide-part-fix 
Fixes for s3_plain disk that will allow to attach Wide parts
 2022-11-15 17:56:16 +01:00
robot-clickhouse
aaeeeaf1fc Automatic style fix 2022-11-15 16:40:01 +00:00
vdimir
03968eb694
Better message in wait_zookeeper_to_start 2022-11-15 16:17:12 +00:00
Antonio Andelic
6beab87610
Merge pull request #43080 from ucasfl/hudi-udtf 
Add table function hudi and deltaLake
 2022-11-15 15:42:25 +01:00
Kseniia Sumarokova
9fc96ab7fd
Merge branch 'master' into disks/web-fix 2022-11-15 12:16:32 +01:00
xiedeyantu
09c749fbd0 update 2022-11-15 13:30:29 +08:00
xiedeyantu
ec6698395e fix skip_unavailable_shards does not work using hdfsCluster table function 2022-11-15 13:25:15 +08:00
Kruglov Pavel
386b69acae
Merge pull request #43131 from xiedeyantu/fix_s3cluster_connection_fail 
fix skip_unavailable_shards does not work using s3Cluster table function
 2022-11-14 20:48:00 +01:00
Raúl Marín
790cfe2b7e Allow backward incompatible change for toDate32 since it was declared as such in 22.8 2022-11-14 18:10:36 +01:00
Kruglov Pavel
eacd7e1876
Merge pull request #42931 from zzsmdfj/MaterializedMySQL_improvement_bug_fix 
Fix issue 42856 and MaterializedMySQL improvement
 2022-11-14 16:54:12 +01:00
flynn
5bd8e2338b
Merge branch 'master' into hudi-udtf 2022-11-14 22:28:04 +08:00
Raúl Marín
58da0398a0 Merge remote-tracking branch 'blessed/master' into revert_SingleValueDataString 2022-11-14 12:46:00 +01:00
Antonio Andelic
41c0100dd7
Merge pull request #43115 from ClickHouse/fix-4lw-command-test 
Fix `test_keeper_four_word_command::test_cmd_stat`
 2022-11-14 08:33:11 +01:00
Azat Khuzhin
f2260ab724 Add a test for unavalable web server and web disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-12 13:25:10 +01:00
Azat Khuzhin
de5d777ca2 tests/integration: add ability not to retry server start 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-12 13:20:51 +01:00
Igor Nikonov
ee5756b5b4
Merge pull request #43122 from ClickHouse/revert-43116-revert-42607-igor/insert_zk_retries_retry 
Revert "Revert " Keeper retries during insert (clean)""
 2022-11-11 18:22:38 +01:00
Raúl Marín
5a3f324311 Try adding compatibility on write instead of on read 2022-11-11 18:12:34 +01:00
Dmitry Novik
31794284cb
Merge pull request #43105 from ClickHouse/overcommit-test 
Rewrite tests for memory overcommit
 2022-11-11 17:34:53 +01:00
xiedeyantu
5504f3af9b fix skip_unavailable_shards does not work using s3Cluster table function 2022-11-12 00:03:36 +08:00
Kruglov Pavel
4ecbcbf5e1
Merge branch 'master' into MaterializedMySQL_improvement_bug_fix 2022-11-11 12:54:02 +01:00
zzsmdfj
a9a208c3cd to MaterializedMySQL_improvement_bug_fix-fix code style 2022-11-11 11:46:22 +08:00
zzsmdfj
227dcd68ba Merge branch 'MaterializedMySQL_improvement_bug_fix' of github.com:zzsmdfj/ClickHouse into MaterializedMySQL_improvement_bug_fix 2022-11-11 11:25:03 +08:00
zzsmdfj
0e614aec12 to MaterializedMySQL_improvement_bug_fix-fix code sytle 2022-11-11 11:24:29 +08:00
zzsmdfj
b01eea5c10
Merge branch 'master' into MaterializedMySQL_improvement_bug_fix 2022-11-11 10:45:36 +08:00
zzsmdfj
d4e270b522 to MaterializedMySQL_improvement_bug_fix-fix codestyle 2022-11-11 10:44:16 +08:00
Stig Bakken
9020631826 Ignore SAVEPOINT queries in MaterializedMySQL 2022-11-10 23:27:21 +01:00
Azat Khuzhin
afc6c8bb73 tests: add ATTACH FROM s3_plain for wide part 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-10 17:42:32 +01:00
robot-clickhouse
fe283af82b Automatic style fix 2022-11-10 16:08:05 +00:00
Dmitry Novik
4e15d648e7 Add test_global_overcommit_tracker/__init__.py 2022-11-10 16:02:45 +00:00
Dmitry Novik
b1e942e14c Do not run global test with sanitizers 2022-11-10 16:00:10 +00:00
Igor Nikonov
f8c907c463
Revert "Revert " Keeper retries during insert (clean)"" 2022-11-10 13:14:04 +01:00
Alexander Tokmakov
fb8c527a2f
Revert " Keeper retries during insert (clean)" 2022-11-10 14:12:24 +03:00
Antonio Andelic
c50e795788 Fix expected ZXID 2022-11-10 11:04:51 +00:00
JackyWoo
8e881e6009
Merge branch 'master' into keeper_manually_assign_leader 2022-11-10 14:38:04 +08:00
flynn
da77199bf5 fix style 
fix style
 2022-11-10 03:35:38 +00:00
flynn
6cbfab4e95 fix test 2022-11-10 03:10:25 +00:00
robot-clickhouse
eb612a699d Automatic style fix 2022-11-10 01:08:12 +00:00
Dmitry Novik
981ece08ed Rewrite tests for memory overcommit 2022-11-10 00:55:39 +00:00
flynn
0ffdf3e0c7 fix style 2022-11-09 12:42:13 +00:00
flynn
c226a3cc9a fix style 2022-11-09 12:32:12 +00:00
JackyWoo
6a2957c269 fix test 2022-11-09 19:15:56 +08:00
flynn
592f6b31d9 fix and add test 2022-11-09 10:04:53 +00:00
Igor Nikonov
345304abe0 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-11-08 20:26:50 +00:00
Mikhail f. Shiryaev
cca2620c44
Merge pull request #42891 from ClickHouse/sync-integration-logs 
Wait for all files are in sync before archiving them
 2022-11-08 13:30:32 +01:00
JackyWoo
8f57ecf595 fix style 2022-11-08 19:56:20 +08:00
JackyWoo
a9d59a7e3f Simplify keeper_utils.is_leader method 2022-11-08 19:43:36 +08:00
Kseniia Sumarokova
d2b777c059
Merge pull request #41054 from rubin-do/hudi 
Add Hudi and DeltaLake table engines
 2022-11-08 12:03:06 +01:00
JackyWoo
9060141f16 revert last change 2022-11-08 18:44:43 +08:00
JackyWoo
eeedd7d9ba Keeper support manually assigning leader 2022-11-08 12:01:53 +08:00
Kseniia Sumarokova
4fc7926c16
Merge branch 'master' into system_table_add_column 2022-11-07 23:36:57 +01:00
alesapin
e6c3ea7ab8
Merge pull request #42890 from AVMusorin/s3_plain_system_name 
changed type name for s3 plain storage
 2022-11-07 22:46:25 +01:00
alesapin
257e766e0e Trying to fix two flaky tests 2022-11-07 20:49:41 +01:00
kssenii
819be6bb32 Merge remote-tracking branch 'upstream/master' into hudi 2022-11-07 11:30:53 +01:00
Antonio Andelic
9ac829d4c4
Merge pull request #41766 from JackyWoo/manual_snapshot_for_keeper 
Manually snapshot creation for keeper
 2022-11-07 09:18:58 +01:00
Antonio Andelic
e08f94d0f8
Merge pull request #42659 from ClickHouse/optimize-entire-partition 
Support for optimizing old parts for entire partition only
 2022-11-07 09:06:11 +01:00
Igor Nikonov
503649a638
Merge branch 'master' into igor/insert_zk_retries_retry 2022-11-06 22:55:13 +01:00
Kseniia Sumarokova
cd5c6acbd2
Merge pull request #42960 from kssenii/fix-pg-db-datetime-table-attach 
Fix postgres db engine attaching table with datetime
 2022-11-06 22:39:15 +01:00
robot-clickhouse
f44e4fe7c6 Automatic style fix 2022-11-05 22:36:32 +00:00
Igor Nikonov
e94b9cd003 Fixes 
+ enable retries in integration test
+ typo in fault injection traces
 2022-11-05 22:28:32 +00:00
JackyWoo
9404e089fc
Merge branch 'master' into manual_snapshot_for_keeper 2022-11-05 17:37:36 +08:00
Igor Nikonov
859022660e Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-11-04 23:31:51 +00:00
Aleksandr Musorin
5cb69d8a22 changed type name for S3_Plain storage 
renamed a disk for S3PlainObjectStorage in system.disks table from s3 to s3_plain
 2022-11-04 17:35:51 +01:00
Vitaly Baranov
52b1f4aed9
Merge pull request #42484 from vitlibar/mask-sensitive-info-in-logs 
Mask some information in logs
 2022-11-04 14:09:38 +01:00
Kruglov Pavel
a7a5a70189
Merge pull request #42814 from kgoralski/failed_metrics_for_async_inserts 
Add `FailedAsyncInsertQuery` event metric for async inserts
 2022-11-04 13:41:39 +01:00
alesapin
82729a88c8
Merge pull request #42833 from vitlibar/backup-s3-increase-upload-part-size-exponentially 
Increase the size of upload part exponentially for backup to S3.
 2022-11-04 13:41:10 +01:00
kssenii
c9b4bc66b5 Fiux 2022-11-04 12:19:30 +01:00
Krzysztof Góralski
f526ddba68 Update tests/integration/test_failed_async_inserts/test.py 
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
 2022-11-03 22:26:24 +01:00
Krzysztof Góralski
14e922aca1 style check fixes test.py 2022-11-03 22:26:24 +01:00
Krzysztof Góralski
69f56d8d68 style check fixes 2022-11-03 22:26:24 +01:00
Krzysztof Góralski
be53a3596c Review: Failed queries metrics for Async Inserts, added integration test instead of stateless one 2022-11-03 22:26:24 +01:00
Daniil Rubin
99d57325bb Merge branch 'master' into hudi 2022-11-03 19:29:23 +00:00
Kruglov Pavel
b124875257
Merge branch 'master' into improve-streaming-engines 2022-11-03 13:22:06 +01:00
zzsmdfj
eeb9c57c59 to fix issue 42856 and MaterializedMySQL improvement 2022-11-03 20:21:02 +08:00
Antonio Andelic
66ffdda973
Merge branch 'master' into optimize-entire-partition 2022-11-03 09:52:13 +01:00
Antonio Andelic
25b19e6ba0
Merge branch 'master' into manual_snapshot_for_keeper 2022-11-03 08:55:01 +01:00
Igor Nikonov
9a315b7899 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-11-02 15:32:54 +00:00
Vitaly Baranov
e0133688bc
Merge branch 'master' into mask-sensitive-info-in-logs 2022-11-02 16:26:13 +01:00
avogar
9f39a6a049 Fix possible heap-use-after-free 2022-11-02 14:17:48 +00:00
Mikhail f. Shiryaev
550c776ffe
Wait for all files are in sync before archiving them 2022-11-02 13:16:27 +01:00
Vitaly Baranov
b9f2f17331 Add test and logging. 2022-11-01 12:23:20 +01:00
Vitaly Baranov
73f5664109 Fix tests. 2022-11-01 11:10:58 +01:00
Igor Nikonov
9cc64a0fcc Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-31 18:21:21 +00:00
Vitaly Baranov
d1c69a0753 Avoid changing query if it doesn't contain password. 2022-10-31 17:04:44 +01:00
Azat Khuzhin
8345c5f881 tests: cover ATTACH of BACKUP from s3_plain disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Vitaly Baranov
901ae126e1 Improve test test_on_cluster. 2022-10-31 10:50:33 +01:00
Vitaly Baranov
aa5cfd6980 Improve tests. 2022-10-31 10:50:33 +01:00
Vitaly Baranov
854e1e918f Use assert_cast() instead of as() in some places. 2022-10-31 10:50:33 +01:00
Vitaly Baranov
43efbad642 Wipe passwords from distributed queries too. 2022-10-31 10:50:33 +01:00
Vitaly Baranov
dcf8724adf Move prepareQueryForLogging() to a separate header. 2022-10-31 10:50:33 +01:00
Vitaly Baranov
5d2a222fe4 Mask sensitive information in logs. 2022-10-31 10:50:33 +01:00
avogar
8e13d1f1ec Improve and refactor Kafka/StorageMQ/NATS and data formats 2022-10-28 16:41:10 +00:00
Azat Khuzhin
7a5432feaa Fix possible SIGSEGV for web disks when file does not exists 
It can be triggered in multiple ways, either when file does not exists
and you are trying to create MergeTree table from web (that has special
code for UUID handling) or by simply OPTIMIZE TABLE FINAL for MergeTree
table that is located on web disk, in both cases you will get the
following:

<details>

<summary>stacktrace</summary>

    2022.10.28 14:08:40.631226 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> ContextAccess (default): Access granted: OPTIMIZE ON default.data_from_web
    2022.10.28 14:08:40.632017 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Debug> default.data_from_web (a3e65e1f-5fd4-47ed-9dbd-307f2586b52d) (MergerMutator): Selected 1 parts from all_1_1_0 to all_1_1_0
    2022.10.28 14:08:40.632496 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> default.data_from_web (a3e65e1f-5fd4-47ed-9dbd-307f2586b52d): Trying to reserve 1.00 MiB using storage policy from min volume index 0
    2022.10.28 14:08:40.632752 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> DiskObjectStorage(DiskWebServer): Reserved 1.00 MiB on remote disk `web_disk`, having unreserved 16.00 EiB.
    2022.10.28 14:08:40.634155 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Debug> MergeTask::PrepareStage: Merging 1 parts: from all_1_1_0 to all_1_1_0 into Compact
    2022.10.28 14:08:40.634498 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> WebObjectStorage: Loading metadata for directory: http://127.0.0.1:8080/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1
    2022.10.28 14:08:40.635025 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> DiskWeb: Adding directory: store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/
    2022.10.28 14:08:40.635355 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> ReadWriteBufferFromHTTP: Sending request to http://127.0.0.1:8080/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index
    2022.10.28 14:08:40.639618 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> DiskWeb: Cannot load disk metadata. Error: Code: 86. DB::Exception: Received error from remote server /store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index. HTTP status code: 404 Not Found, body: <!doctype html><html><head><title>404 Not Found</title><style>
    body { background-color: #fcfcfc; color: #333333; margin: 0; padding:0; }
    h1 { font-size: 1.5em; font-weight: normal; background-color: #9999cc; min-height:2em; line-height:2em; border-bottom: 1px inset black; margin: 0; }
    h1, p { padding-left: 10px; }
    code.url { background-color: #eeeeee; font-family:monospace; padding:0 2px;}
    </style>
    </head><body><h1>Not Found</h1><p>The requested resource <code class="url">/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index</code> was not found on this server.</p></body></html>: while loading disk metadata. (RECEIVED_ERROR_FROM_REMOTE_IO_SERVER) (version 22.11.1.1)
    2022.10.28 14:08:40.640527 [ 5488 ] {} <Trace> BaseDaemon: Received signal 11
    2022.10.28 14:08:40.641529 [ 9027 ] {} <Fatal> BaseDaemon: ########################################
    2022.10.28 14:08:40.642759 [ 9027 ] {} <Fatal> BaseDaemon: (version 22.11.1.1, build id: 12145DA78CE5E9EBB10A034177FAE5967EF81A4A) (from thread 6043) (query_id: a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1) (query: optimize table data_from_web final) Received signal Segmentation fault (11)
    2022.10.28 14:08:40.643260 [ 9027 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Unknown si_code.
    2022.10.28 14:08:40.643769 [ 9027 ] {} <Fatal> BaseDaemon: Stack trace: 0x7ffff416c0f2 0x7ffff7cd1ca8 0x7ffff679ae5e 0x7fffd52e7906 0x7fffd50c65aa 0x7fffca7a0d42 0x7fffcaee79ec 0x7fffcaf242f8 0x7fffcaf242b5 0x7fffcaf2427d 0x7fffcaf24255 0x7fffcaf2421d 0x7ffff65c3686 0x7ffff65c2295 0x7fffcaeee2a9 0x7fffcaef2c43 0x7fffcaee3c0e 0x7fffcc4a7851 0x7fffcc4a768f 0x7fffcc4abb2d 0x7fffcfdce828 0x7fffd03e3eaa 0x7fffd03dfe3b 0x7fffc8ec42d4 0x7fffc8ed51d2 0x7ffff4bdd839 0x7ffff4bde0a8 0x7ffff48ab261 0x7ffff48a769a 0x7ffff48a6335 0x7ffff409f8fd 0x7ffff4121a60
    2022.10.28 14:08:40.644411 [ 9027 ] {} <Fatal> BaseDaemon: 4. ? @ 0x7ffff416c0f2 in ?
    2022.10.28 14:08:40.676390 [ 9027 ] {} <Fatal> BaseDaemon: 5. /src/ch/clickhouse/src/Common/StringUtils/StringUtils.cpp:9: detail::startsWith(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, char const*, unsigned long) @ 0x1ca8 in /src/ch/clickhouse/.cmake/src/Common/StringUtils/libstring_utilsd.so
    2022.10.28 14:08:40.730727 [ 9027 ] {} <Fatal> BaseDaemon: 6. /src/ch/clickhouse/src/Common/StringUtils/StringUtils.h:19: startsWith(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) @ 0x59ae5e in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    2022.10.28 14:08:40.923955 [ 9027 ] {} <Fatal> BaseDaemon: 7. /src/ch/clickhouse/src/Disks/ObjectStorages/Web/MetadataStorageFromStaticFilesWebServer.cpp:58: DB::MetadataStorageFromStaticFilesWebServer::exists(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const @ 0x6e7906 in /src/ch/clickhouse/.cmake/src/libdbmsd.so
    2022.10.28 14:08:41.291996 [ 9027 ] {} <Fatal> BaseDaemon: 8. /src/ch/clickhouse/src/Disks/ObjectStorages/DiskObjectStorage.cpp:181: DB::DiskObjectStorage::exists(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const @ 0x4c65aa in /src/ch/clickhouse/.cmake/src/libdbmsd.so
    2022.10.28 14:08:41.704697 [ 9027 ] {} <Fatal> BaseDaemon: 9. /src/ch/clickhouse/src/Storages/MergeTree/DataPartStorageOnDisk.cpp:74: DB::DataPartStorageOnDisk::exists() const @ 0xda0d42 in /src/ch/clickhouse/.cmake/src/libclickhouse_storages_mergetreed.so
    2022.10.28 14:08:43.032459 [ 9027 ] {} <Fatal> BaseDaemon: 10. /src/ch/clickhouse/src/Storages/MergeTree/MergeTask.cpp:147: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x14e79ec in /src/ch/clickhouse/.cmake/src/libclickhouse_storages_mergetreed.so
    ...
    Segmentation fault (core dumped)

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-28 15:15:49 +02:00
Antonio Andelic
5fae8b0cc9 Merge branch 'master' into optimize-entire-partition 2022-10-28 11:44:10 +00:00
Igor Nikonov
3682913172 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-27 22:01:53 +00:00
Alexander Tokmakov
86b81a2b6f
Merge pull request #42234 from arthurpassos/fix_ares_fd_process_race_condition 
Fix c-ares crash
 2022-10-27 17:17:22 +03:00
Igor Nikonov
649e8d302c Sync with retiring branch igor/insert_zk_retries 2022-10-26 17:41:44 +00:00
Igor Nikonov
8435cba352 Merge remote-tracking branch 'origin/master' into igor/insert_zk_retries_retry 2022-10-26 17:35:47 +00:00
Antonio Andelic
35a9672704 Merge branch 'master' into optimize-entire-partition 2022-10-26 11:09:25 +00:00
Antonio Andelic
97aaebfa18 Address PR comments 2022-10-26 10:06:56 +00:00
Antonio Andelic
611c2e2bd7 Support for optimizing old parts for entire partition only 2022-10-25 13:40:18 +00:00
Nikolai Kochetov
5c4444237e
Merge pull request #42617 from ClickHouse/revert-revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3 
Revert revert 41268 disable s3 parallel write for part moves to disk s3
 2022-10-25 14:29:27 +02:00
Nikita Taranov
49f6692a2e
Adapt internal data structures to 512-bit era (#42564) 
* impl

* update tests

* fix tests
 2022-10-25 13:56:28 +02:00
Raúl Marín
0016bc2b73
Improve the time to recover keeper connections (#42541) 
Co-authored-by: Nikita Mikhaylov <nikitamikhaylov@clickhouse.com>
Co-authored-by: robot-clickhouse <robot-clickhouse@clickhouse.com>
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2022-10-25 13:26:29 +02:00
JackyWoo
9a36a509fe fix test 2022-10-25 17:15:49 +08:00
Alexey Milovidov
9ee7131f67
Merge pull request #42423 from ClickHouse/optimize-partitions-in-background 
Optimize partitions in background
 2022-10-24 19:41:50 +02:00
Nikolai Kochetov
beb40e4579 Split tests. 2022-10-24 14:22:06 +00:00
Vitaly Baranov
cd158b021f
Merge pull request #42562 from vitlibar/subqueries-in-row-policy-filters 
Fix using subqueries in row policy filters
 2022-10-24 13:38:15 +02:00
Igor Nikonov
54013009d3 Code migration from #41734 to align diff and git blame. 
Initial implementation was different and it changed the entire
ReplicatedMergeTreeSink::commitPart() which change history provided by git blame.
Then RetriesControl.retryLoop() was introduced later which significantly reduces
the diff since it's like while() used before.
So, check outing the current version will keep more original history in
git blame, which is useful here
 2022-10-24 08:20:11 +00:00
JackyWoo
42e391a019 fix test 2022-10-22 22:47:03 +08:00
JackyWoo
39c88c74e8 check whether last manually created snapshot is done 2022-10-22 22:31:17 +08:00
JackyWoo
bf291790b7
Merge branch 'master' into manual_snapshot_for_keeper 2022-10-22 16:57:50 +08:00
Azat Khuzhin
dbb4f8216f Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Functions/GregorianDate.h
 2022-10-22 08:20:05 +02:00
Nikolay Degterinsky
22f87e2bf9
Merge pull request #42549 from evillique/disks-better-logs 
Better logs in clickhouse-disks
 2022-10-22 04:31:44 +02:00
Azat Khuzhin
4eeb7337ed Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Common/intExp.h
- src/Functions/DateTimeTransforms.h
- src/Functions/FunctionsConversion.h
 2022-10-21 22:51:56 +02:00
Alexander Tokmakov
7d11054134
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-21 18:27:22 +03:00
Vitaly Baranov
2033ea1216 Support subqueries in row policy filters. 2022-10-21 15:42:29 +02:00
Alexander Tokmakov
5ffd026261
Merge pull request #42529 from azat/tests/integration/fix-sanitizer-reports 
Fix sanitizer reports in integration tests
 2022-10-21 16:01:43 +03:00
Alexander Tokmakov
1343f62e8f
Update test.py 2022-10-21 15:59:27 +03:00
Azat Khuzhin
0365bc56aa Add backward compatiblity check for some functions 
* State of aggregate functions

This test goes though all aggregate functions that:
- has only one argument
- support string as an argument

And do a simple check by creating the aggregate state with one string.

Yes this is not covers everything (does not cover functions with
different number of arguments, types, different states in case of
multiple values - uniqCombined, but as for uniqCombined it will be
checked via uniqHLL12), but at least something.

And about type, String had been selected, since it more likely that
there will be used some hash function internally.

* Function(String)

Check every function which has only one argument, and it's type is
String.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:43 +02:00
Antonio Andelic
3091d609fa
Merge branch 'master' into optimize-partitions-in-background 2022-10-21 11:01:41 +02:00
Nikolay Degterinsky
d7d0794d2f Better logs in clickhouse-disks 2022-10-20 23:17:29 +00:00
Nikolai Kochetov
57ffcb7ed0 Revert "Merge pull request #41681 from ClickHouse/revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3" 
This reverts commit 7956c2becf, reversing
changes made to 57be648984.
 2022-10-20 19:51:27 +02:00
Arthur Passos
50f9acb221
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-20 13:35:58 -03:00
Arthur Passos
2fe59e7e8f lower thread and iterations number 2022-10-20 13:30:20 -03:00
Alexander Tokmakov
905a4ded42
Merge pull request #42535 from ClickHouse/tavplubix-patch-7 
Disable `test_storage_nats`, because it's permanently broken
 2022-10-20 16:58:46 +03:00
Alexander Tokmakov
5b03bbce2e
Merge pull request #41981 from ClickHouse/fixes_for_replicated_merge_tree2 
Some fixes for ReplicatedMergeTree
 2022-10-20 16:57:51 +03:00
Alexander Tokmakov
89ff3d4731
Update test.py 2022-10-20 16:54:52 +03:00
Azat Khuzhin
c399b021cf Fix sanitizer reports in integration tests 
Before they was ignored because first there was a check for a sign of
sanitizer (==================), but it was done by
clickhouse-server.log, while sanitizer write to stderr.log.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-20 15:45:40 +02:00
Antonio Andelic
9d7e44409a
Merge pull request #41342 from ClickHouse/keeper-upload-snapshot-to-s3 
Upload Keeper snapshots to S3
 2022-10-20 09:14:59 +02:00
Antonio Andelic
77eb353839
Merge pull request #42243 from ClickHouse/use-multiread-more 
Use MultiRead where possible
 2022-10-20 09:04:54 +02:00
Alexander Tokmakov
f958c155f3
Merge branch 'master' into fixes_for_replicated_merge_tree2 2022-10-19 17:25:24 +03:00
Antonio Andelic
05e6b94b54 Polishing 2022-10-19 12:59:28 +00:00
Antonio Andelic
e5408aac99 Simplify logic 2022-10-19 12:34:20 +00:00
Arthur Passos
7f0a2d32c8 fix test errors 2022-10-18 20:15:58 -03:00
Vitaly Baranov
1365105bc4 Implement backup to S3 2022-10-19 00:04:41 +02:00
Alexander Tokmakov
ea77ff9022 Merge branch 'master' into fixes_for_replicated_merge_tree2 2022-10-18 17:25:25 +02:00
Antonio Andelic
e828dce331 Disable auto optimize by default 2022-10-18 12:11:23 +00:00
Antonio Andelic
bd2522fdd0 Merge branch 'master' into optimize-partitions-in-background 2022-10-18 09:13:59 +00:00
Antonio Andelic
ee2f5977d5
Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-18 10:02:50 +02:00
chen
163b3c9f52
fix test 
fix test
 2022-10-18 14:53:21 +08:00
xiedeyantu
10acf43db4 fix test 2022-10-18 09:20:59 +08:00
Vitaly Baranov
eb637a6f81
Merge pull request #42232 from azat/backups/s3 
Support BACKUP to S3 with as-is path/data structure
 2022-10-18 02:37:07 +02:00
Yakov Olkhovskiy
3ce1fa09d6
Merge pull request #41198 from ClickHouse/composable-protocol 
Composable protocol
 2022-10-17 11:23:12 -04:00
Antonio Andelic
7c7395575f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-17 11:02:36 +00:00
Antonio Andelic
878dee40b1 Use ZooKeeper in a single test 2022-10-17 07:36:49 +00:00
xiedeyantu
2a356715da fix test 2022-10-16 16:46:00 +08:00
Arthur Passos
f414edebae fix black style 2022-10-14 15:54:48 -03:00
Arthur Passos
299d848473 style fix? 2022-10-14 15:23:10 -03:00
Arthur Passos
78202f9660
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-14 12:56:39 -03:00
Arthur Passos
6d22bb78e0 custom py script in integ tests 2022-10-14 12:39:08 -03:00
Anton Popov
95fdb0a877 remove outdated parts asynchronously 2022-10-14 14:52:57 +00:00
Azat Khuzhin
8830f0608d Support BACKUP to S3 with as-is path/data structure 
Right now backup to S3 does not make a lot of sense, since:
- it has random names, and to decoding them
- requires metadata from local disk (/var/lib/disks/DISK/BACKUP_NAME)
- or send_metadata (but it is also tricky even with it)

So this patch adds simpler interface for S3, it is only suitable for
BACKUP/RESTORE, so don't try to use it for MergeTree engine.

It is done by adding separate disk - `s3_plain` for this, that:
- does not support any extended features, like renames/hardlinks/attrs/...
  (so basically everything that MergeTree requires)
- only write/read/unlink/list files

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-14 12:59:25 +02:00
nvartolomei
ea47ab1400
assert unused value in test_replicated_merge_tree_compatibility 2022-10-12 18:12:25 +01:00