thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
102,529 Commits 9,326 Branches 1,783 Tags 2.1 GiB
616efdd336
Commit Graph

5314 Commits

Author SHA1 Message Date
Alexey Milovidov
9ee7131f67
Merge pull request #42423 from ClickHouse/optimize-partitions-in-background 
Optimize partitions in background
 2022-10-24 19:41:50 +02:00
Nikolai Kochetov
beb40e4579 Split tests. 2022-10-24 14:22:06 +00:00
Vitaly Baranov
cd158b021f
Merge pull request #42562 from vitlibar/subqueries-in-row-policy-filters 
Fix using subqueries in row policy filters
 2022-10-24 13:38:15 +02:00
Igor Nikonov
54013009d3 Code migration from #41734 to align diff and git blame. 
Initial implementation was different and it changed the entire
ReplicatedMergeTreeSink::commitPart() which change history provided by git blame.
Then RetriesControl.retryLoop() was introduced later which significantly reduces
the diff since it's like while() used before.
So, check outing the current version will keep more original history in
git blame, which is useful here
 2022-10-24 08:20:11 +00:00
JackyWoo
42e391a019 fix test 2022-10-22 22:47:03 +08:00
JackyWoo
39c88c74e8 check whether last manually created snapshot is done 2022-10-22 22:31:17 +08:00
JackyWoo
bf291790b7
Merge branch 'master' into manual_snapshot_for_keeper 2022-10-22 16:57:50 +08:00
Azat Khuzhin
dbb4f8216f Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Functions/GregorianDate.h
 2022-10-22 08:20:05 +02:00
Nikolay Degterinsky
22f87e2bf9
Merge pull request #42549 from evillique/disks-better-logs 
Better logs in clickhouse-disks
 2022-10-22 04:31:44 +02:00
Azat Khuzhin
4eeb7337ed Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Common/intExp.h
- src/Functions/DateTimeTransforms.h
- src/Functions/FunctionsConversion.h
 2022-10-21 22:51:56 +02:00
Alexander Tokmakov
7d11054134
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-21 18:27:22 +03:00
Vitaly Baranov
2033ea1216 Support subqueries in row policy filters. 2022-10-21 15:42:29 +02:00
Alexander Tokmakov
5ffd026261
Merge pull request #42529 from azat/tests/integration/fix-sanitizer-reports 
Fix sanitizer reports in integration tests
 2022-10-21 16:01:43 +03:00
Alexander Tokmakov
1343f62e8f
Update test.py 2022-10-21 15:59:27 +03:00
Azat Khuzhin
0365bc56aa Add backward compatiblity check for some functions 
* State of aggregate functions

This test goes though all aggregate functions that:
- has only one argument
- support string as an argument

And do a simple check by creating the aggregate state with one string.

Yes this is not covers everything (does not cover functions with
different number of arguments, types, different states in case of
multiple values - uniqCombined, but as for uniqCombined it will be
checked via uniqHLL12), but at least something.

And about type, String had been selected, since it more likely that
there will be used some hash function internally.

* Function(String)

Check every function which has only one argument, and it's type is
String.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:43 +02:00
Antonio Andelic
3091d609fa
Merge branch 'master' into optimize-partitions-in-background 2022-10-21 11:01:41 +02:00
Nikolay Degterinsky
d7d0794d2f Better logs in clickhouse-disks 2022-10-20 23:17:29 +00:00
Nikolai Kochetov
57ffcb7ed0 Revert "Merge pull request #41681 from ClickHouse/revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3" 
This reverts commit 7956c2becf, reversing
changes made to 57be648984.
 2022-10-20 19:51:27 +02:00
Arthur Passos
50f9acb221
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-20 13:35:58 -03:00
Arthur Passos
2fe59e7e8f lower thread and iterations number 2022-10-20 13:30:20 -03:00
Alexander Tokmakov
905a4ded42
Merge pull request #42535 from ClickHouse/tavplubix-patch-7 
Disable `test_storage_nats`, because it's permanently broken
 2022-10-20 16:58:46 +03:00
Alexander Tokmakov
5b03bbce2e
Merge pull request #41981 from ClickHouse/fixes_for_replicated_merge_tree2 
Some fixes for ReplicatedMergeTree
 2022-10-20 16:57:51 +03:00
Alexander Tokmakov
89ff3d4731
Update test.py 2022-10-20 16:54:52 +03:00
Azat Khuzhin
c399b021cf Fix sanitizer reports in integration tests 
Before they was ignored because first there was a check for a sign of
sanitizer (==================), but it was done by
clickhouse-server.log, while sanitizer write to stderr.log.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-20 15:45:40 +02:00
Antonio Andelic
9d7e44409a
Merge pull request #41342 from ClickHouse/keeper-upload-snapshot-to-s3 
Upload Keeper snapshots to S3
 2022-10-20 09:14:59 +02:00
Antonio Andelic
77eb353839
Merge pull request #42243 from ClickHouse/use-multiread-more 
Use MultiRead where possible
 2022-10-20 09:04:54 +02:00
Alexander Tokmakov
f958c155f3
Merge branch 'master' into fixes_for_replicated_merge_tree2 2022-10-19 17:25:24 +03:00
Antonio Andelic
05e6b94b54 Polishing 2022-10-19 12:59:28 +00:00
Antonio Andelic
e5408aac99 Simplify logic 2022-10-19 12:34:20 +00:00
Arthur Passos
7f0a2d32c8 fix test errors 2022-10-18 20:15:58 -03:00
Vitaly Baranov
1365105bc4 Implement backup to S3 2022-10-19 00:04:41 +02:00
Alexander Tokmakov
ea77ff9022 Merge branch 'master' into fixes_for_replicated_merge_tree2 2022-10-18 17:25:25 +02:00
Antonio Andelic
e828dce331 Disable auto optimize by default 2022-10-18 12:11:23 +00:00
Antonio Andelic
bd2522fdd0 Merge branch 'master' into optimize-partitions-in-background 2022-10-18 09:13:59 +00:00
Antonio Andelic
ee2f5977d5
Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-18 10:02:50 +02:00
chen
163b3c9f52
fix test 
fix test
 2022-10-18 14:53:21 +08:00
xiedeyantu
10acf43db4 fix test 2022-10-18 09:20:59 +08:00
Vitaly Baranov
eb637a6f81
Merge pull request #42232 from azat/backups/s3 
Support BACKUP to S3 with as-is path/data structure
 2022-10-18 02:37:07 +02:00
Yakov Olkhovskiy
3ce1fa09d6
Merge pull request #41198 from ClickHouse/composable-protocol 
Composable protocol
 2022-10-17 11:23:12 -04:00
Antonio Andelic
7c7395575f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-17 11:02:36 +00:00
Antonio Andelic
878dee40b1 Use ZooKeeper in a single test 2022-10-17 07:36:49 +00:00
xiedeyantu
2a356715da fix test 2022-10-16 16:46:00 +08:00
Arthur Passos
f414edebae fix black style 2022-10-14 15:54:48 -03:00
Arthur Passos
299d848473 style fix? 2022-10-14 15:23:10 -03:00
Arthur Passos
78202f9660
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-14 12:56:39 -03:00
Arthur Passos
6d22bb78e0 custom py script in integ tests 2022-10-14 12:39:08 -03:00
Azat Khuzhin
8830f0608d Support BACKUP to S3 with as-is path/data structure 
Right now backup to S3 does not make a lot of sense, since:
- it has random names, and to decoding them
- requires metadata from local disk (/var/lib/disks/DISK/BACKUP_NAME)
- or send_metadata (but it is also tricky even with it)

So this patch adds simpler interface for S3, it is only suitable for
BACKUP/RESTORE, so don't try to use it for MergeTree engine.

It is done by adding separate disk - `s3_plain` for this, that:
- does not support any extended features, like renames/hardlinks/attrs/...
  (so basically everything that MergeTree requires)
- only write/read/unlink/list files

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-14 12:59:25 +02:00
nvartolomei
ea47ab1400
assert unused value in test_replicated_merge_tree_compatibility 2022-10-12 18:12:25 +01:00
Yakov Olkhovskiy
6c8e52eb67
Merge branch 'master' into composable-protocol 2022-10-08 20:33:32 -04:00
robot-clickhouse
3bed015a1f Automatic style fix 2022-10-08 00:09:24 +00:00
Yakov Olkhovskiy
6bb62d4d03 add test for PROXYv1 2022-10-08 00:01:58 +00:00
Vitaly Baranov
69ebf12dab
Merge pull request #42146 from azat/backups/metadata-overflow-fix 
Fix reusing of files > 4GB from base backup
 2022-10-08 00:22:28 +02:00
Alexander Tokmakov
b8d9066004 Revert "Resurrect parallel distributed insert select with s3Cluster (#41535)" 
This reverts commit 860e34e760.
 2022-10-07 15:53:30 +02:00
Azat Khuzhin
94566abda9 Fix reusing of files > 4GB from base backup 
Previosly u64 numbers was truncated to u32 numbers during writing to the
mdatadata xml file, and further incremental backup cannot reuse them,
since the file in base backup is smaller.

P.S. There can be other places, I thought about enabling
-Wshorten-64-to-32, but there are lots of warnings right now.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-07 14:56:09 +02:00
Yakov Olkhovskiy
2229087896
Merge branch 'master' into composable-protocol 2022-10-07 02:23:26 -04:00
Yakov Olkhovskiy
997fa5e295 review suggestions 2022-10-07 01:16:22 +00:00
Nikita Mikhaylov
860e34e760
Resurrect parallel distributed insert select with s3Cluster (#41535) 2022-10-06 13:47:32 +02:00
Nikita Mikhaylov
766107df0a
Fixed the restriction on maximum size of replicated fetches (#42090) 2022-10-06 13:18:46 +02:00
Kseniia Sumarokova
316824c02a
Merge pull request #42085 from elenatorro/use_send_metadata_setting_on_storage_config 
Use send_metadata instead of send_object_metadata
 2022-10-06 12:33:25 +02:00
Kseniia Sumarokova
3cff776109
Merge pull request #41254 from filimonov/system_disks_unreserved_space 
Add unreserved_space column to system.disks
 2022-10-06 12:28:23 +02:00
Elena
3914b8afda Remove 'send_metadata' setting 2022-10-05 14:54:08 +02:00
Elena
5bbd2d1729 Use send_metadata instead of send_object_metadata 2022-10-05 13:43:59 +02:00
Kseniia Sumarokova
6775c2c0a8
Update test.py 2022-10-04 16:52:53 +02:00
Kseniia Sumarokova
0b0c7e359f
Update test.py 2022-10-04 16:04:57 +02:00
Kseniia Sumarokova
39d5dfb3f6
Fix integration test 2022-10-04 16:03:11 +02:00
Kseniia Sumarokova
1ffa3291d6
Update storage_conf.xml 2022-10-04 11:18:02 +02:00
kssenii
2ab884359e More efficient WriteBufferFromAzureBlobStorage 2022-10-03 22:15:04 +02:00
kssenii
c6d7cd5820 Fix write into azure blob storage 2022-10-03 18:54:07 +02:00
Kseniia Sumarokova
0bfbb5c38a
Merge branch 'master' into system_disks_unreserved_space 2022-10-03 14:25:47 +02:00
Alexander Tokmakov
a2c29076d6 avoid detached broken part duplicates 2022-10-03 12:27:17 +02:00
Alexander Tokmakov
7b5b7e4963
Merge pull request #41945 from ClickHouse/enable_more_s3_tests 
Enable some disabled S3 tests
 2022-10-03 13:25:50 +03:00
Alexander Tokmakov
035be1feae fix 2022-10-02 14:58:33 +02:00
robot-clickhouse
4cd2dc8d0d Automatic style fix 2022-10-02 12:57:04 +00:00
Alexander Tokmakov
4ce8f47400 fix 2022-10-02 14:50:24 +02:00
Vitaly Baranov
baf48ec400
Merge pull request #41562 from vitlibar/refactor-wiping-info-from-logs 
Refactor wiping sensitive information from logs.
 2022-10-02 12:58:58 +02:00
Alexander Tokmakov
e40917a9c5 fix 2022-10-02 00:44:46 +02:00
Kseniia Sumarokova
fe6de361c6
Merge pull request #41933 from ClickHouse/kssenii-patch-5 
Fix broken test_disks_app_func
 2022-10-02 00:17:28 +02:00
Vitaly Baranov
a5b2a49ab1 Refactor wiping sensitive information from logs. 2022-10-01 17:17:45 +02:00
Alexey Milovidov
645dd997d3
Update test.py 2022-10-01 18:16:25 +03:00
robot-clickhouse
26ad8b2f13 Automatic style fix 2022-10-01 11:55:58 +00:00
Kseniia Sumarokova
3afec69021
Update test.py 2022-10-01 13:49:50 +02:00
kssenii
fb3bcd2fe9 Fix use_environment_credentials 2022-09-30 12:25:39 +02:00
Alexander Tokmakov
bd305960e7
Merge pull request #41898 from ClickHouse/fix-tests-sync-replica 
Another sync replicas in test_recovery_replica
 2022-09-29 19:36:20 +03:00
alesapin
0097f15ee7
Merge pull request #41832 from ClickHouse/make_copy_instead_of_hardlink 
Use copy for some operations instead of hardlinks
 2022-09-29 17:12:08 +02:00
Alexander Tokmakov
4e422b8046
Merge pull request #41741 from ClickHouse/fix_intersecting_parts 
Fix intersecting parts
 2022-09-29 18:00:22 +03:00
robot-clickhouse
acdc3504aa Automatic style fix 2022-09-29 11:21:41 +00:00
Kseniia Sumarokova
6270f513fc
Update test.py 2022-09-29 13:03:03 +02:00
Yakov Olkhovskiy
028af59702
Update tests/integration/test_recovery_replica/test.py 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-09-28 12:33:38 -04:00
alesapin
905d4585ac
Merge branch 'master' into make_copy_instead_of_hardlink 2022-09-28 16:47:45 +02:00
Yakov Olkhovskiy
10f6ba90e0
add sync_replicas 2022-09-28 10:00:59 -04:00
alesapin
4b4c63749a Correct test version 2022-09-28 14:38:11 +02:00
Daniil Rubin
95b5d2d29d
Merge branch 'master' into hudi 2022-09-28 14:24:55 +03:00
alesapin
2eef384807 Add comments and test 2022-09-28 13:09:48 +02:00
Alexey Milovidov
e5ffd74a56
Merge branch 'master' into trying_to_fix_compressed_marks 2022-09-28 02:30:10 +03:00
Alexander Tokmakov
0915bfdb17 fix 2022-09-27 17:42:36 +02:00
Alexander Tokmakov
b4a10c330c fix 2022-09-27 17:34:24 +02:00
Antonio Andelic
036d1c8cbc
Merge pull request #40991 from ClickHouse/start-embedded-keeper-async 
Always start embedded Keeper in async mode
 2022-09-27 14:44:19 +02:00
robot-clickhouse
08af2a32bf Automatic style fix 2022-09-27 09:53:26 +00:00
alesapin
4d2a5d752d Add failing test 2022-09-27 11:36:20 +02:00
Antonio Andelic
655bde8e3f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-27 07:28:26 +00:00
JackyWoo
d1c85f68e8 manually snapshot creation for keeper 2022-09-26 18:29:15 +08:00
robot-clickhouse
7a6386c732 Automatic style fix 2022-09-25 16:13:17 +00:00
Yakov Olkhovskiy
1407f6bcda add test 2022-09-25 15:46:12 +00:00
kssenii
3725397040 Try fix azure tests 2022-09-23 15:24:10 +02:00
Alexander Tokmakov
3eebceaa38
Update test.py 2022-09-23 14:01:00 +03:00
Kseniia Sumarokova
ea43cb5648
Merge pull request #41261 from kssenii/s3-header-auth 
Support s3 authorisation headers from ast arguments
 2022-09-23 12:48:08 +02:00
Kseniia Sumarokova
a0c5da3459
Merge pull request #41652 from kssenii/fix-disk-web 
Fixes for disk web
 2022-09-23 12:43:50 +02:00
kssenii
0e8343e6f3 Revert "Better config for cache" 
This reverts commit 0ca6aadbdb.
 2022-09-22 17:57:44 +02:00
kssenii
109d2ad022 Better 2022-09-22 16:22:05 +02:00
Antonio Andelic
44a3d6babe Address PR comments 2022-09-22 13:03:27 +00:00
Kseniia Sumarokova
79304422b3
Update test.py 2022-09-22 13:11:35 +02:00
kssenii
cb442a3ad7 Fixes for disk web 2022-09-21 21:26:55 +02:00
kssenii
92c2219e82 Fix test 2022-09-21 20:51:09 +02:00
Antonio Andelic
6798b500e9 Wait on startup for Keeper 2022-09-21 15:12:16 +00:00
Antonio Andelic
aa700836b8 Merge branch 'master' into start-embedded-keeper-async 2022-09-21 13:46:54 +00:00
Kseniia Sumarokova
1e7308832c
Merge branch 'master' into system_disks_unreserved_space 2022-09-21 13:04:39 +02:00
kssenii
f917b268b3 Merge remote-tracking branch 'upstream/master' into s3-header-auth 2022-09-21 12:56:25 +02:00
Antonio Andelic
fc6c1faa57 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-21 06:57:58 +00:00
kssenii
b8079e4577 Merge remote-tracking branch 'upstream/master' into rename-some-cache-commands 2022-09-20 13:02:18 +02:00
alesapin
8d88b68a85
Merge pull request #41415 from ClickHouse/revert_prefixes_s3 
Revert prefixes s3 + add retries on write requests unexpected errors
 2022-09-20 12:55:04 +02:00
Mikhail Filimonov
fb3ebbb2d2
Apply style fix for tests/integration/test_disk_types/test.py 2022-09-20 12:12:01 +02:00
Vitaly Baranov
f58f8104b5
Merge pull request #41474 from vitlibar/mask-some-info-in-logs 
Mask some information in logs.
 2022-09-20 12:11:02 +02:00
Alexey Milovidov
03bbf0676f
Merge branch 'master' into revert_prefixes_s3 2022-09-20 06:14:07 +03:00
Sergei Trifonov
cf2db48c29
Merge pull request #40631 from ClickHouse/readonly-settings-allow 
Allow to modify constrained settings in readonly mode
 2022-09-20 02:18:14 +02:00
Sergei Trifonov
358c5a0139
Merge pull request #41532 from ClickHouse/s3-api-calls-events-for-disk-s3 
Detailed S3 profile events `DiskS3*` per S3 API call split for S3 ObjectStorage
 2022-09-20 02:10:50 +02:00
Vitaly Baranov
b1199878ad
Merge pull request #39977 from vitlibar/improve-recovery-of-replicated-access-storage 
Improve recovery of ReplicatedAccessStorage after errors.
 2022-09-20 00:36:30 +02:00
Sergei Trifonov
7902922ae4
Merge branch 'master' into readonly-settings-allow 2022-09-19 21:00:12 +02:00
serxa
7a1d4a404a add DiskS3* profile events per S3 API calls 2022-09-19 18:10:47 +00:00
serxa
f8aa738511 more conventional profile events names 2022-09-19 17:23:22 +00:00
Vitaly Baranov
052ecaa211
Merge branch 'master' into improve-recovery-of-replicated-access-storage 2022-09-19 15:26:21 +02:00
Vitaly Baranov
f75dd93965 Mask some information in logs. 2022-09-19 15:00:06 +02:00
Sergei Trifonov
ff2e1ff8d8
Merge branch 'master' into support_s3_api_event_metrics 2022-09-19 14:35:45 +02:00
Sergei Trifonov
26da32f53f
Update tests/integration/test_profile_events_s3/test.py 2022-09-19 14:35:23 +02:00
Alexander Tokmakov
c6efb1773e
Merge branch 'master' into fix_drop_partition_on_cluster 2022-09-19 15:18:05 +03:00
kssenii
e3cd3686af Rename 2022-09-19 14:02:51 +02:00
Sergei Trifonov
f7a1ed3141
Merge branch 'master' into readonly-settings-allow 2022-09-19 12:52:36 +02:00
Azat Khuzhin
00d4ac542d tests: fix test_hive_query flakiness 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-19 07:54:49 +02:00
alesapin
1d0dbfb112
Merge pull request #41444 from ClickHouse/one_wrong_config_assert 
One more validation for broken config in keeper
 2022-09-18 15:56:48 +02:00
Alexey Milovidov
6b752bff3a
Merge branch 'master' into revert_prefixes_s3 2022-09-18 07:10:57 +03:00
alesapin
b6d61318d0 One more validation for broken config in keeper 2022-09-18 00:04:16 +02:00
robot-clickhouse
838bc678cc Automatic style fix 2022-09-17 18:38:12 +00:00
Alexey Milovidov
c480531423 Remove trash error message from integration tests 2022-09-17 20:27:19 +02:00
Alexey Milovidov
ecfd8cb5a6 Remove trash error message from integration tests 2022-09-17 20:24:56 +02:00
Vitaly Baranov
406c76ee4e Fix test. 2022-09-16 19:02:06 +02:00
Vitaly Baranov
1206e77883 Add test checking that ReplicatedAccessStorage can continue after reloading ZooKeeper. 2022-09-16 19:02:06 +02:00
alesapin
456da30c9f Revert "Merge pull request #41249 from ClickHouse/revert-40968-s3-sharding-2" 
This reverts commit ac60300997, reversing
changes made to a9c272283a.
 2022-09-16 14:06:26 +02:00
Robert Schulze
b32b02d844
Merge pull request #40897 from ClickHouse/catboost-bridge-resurrected 
Move CatBoost evaluation into clickhouse-library-bridge
 2022-09-16 13:12:09 +02:00
alesapin
4e780cf99f
Merge pull request #41378 from ClickHouse/rename_test 
Check test flakiness
 2022-09-16 11:51:04 +02:00
Antonio Andelic
296fe01fb8 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-16 08:00:15 +00:00
Alexey Milovidov
da01982652
Merge pull request #41046 from azat/build/llvm-15 
Switch to llvm/clang 15
 2022-09-16 07:31:06 +03:00
Alexander Tokmakov
03c193ccca fix ON CLUSTER with Replicated database cluster 2022-09-15 21:15:57 +02:00
alesapin
1dd50e3ce8 Rename test 2022-09-15 17:07:42 +02:00
Antonio Andelic
c7775b7f16 Add tests for S3 snapshot upload 2022-09-15 13:37:17 +00:00
serxa
a960fdea89 Merge branch 'readonly-settings-allow' of github.com:ClickHouse/ClickHouse into readonly-settings-allow 2022-09-15 11:38:06 +00:00
serxa
4971c1f04d review fixes 2022-09-15 11:34:14 +00:00
kssenii
559c696230 Fix 2022-09-14 20:35:28 +02:00
mateng0915
372a0b7794 added events into s3 integration test 2022-09-14 19:19:47 +08:00
alesapin
51a302a70f
Merge branch 'master' into revert-40968-s3-sharding-2 2022-09-14 11:53:14 +02:00
Mikhail Filimonov
a136465eb4
Fix test integration/test_disk_types/test.py 2022-09-14 11:08:37 +02:00
Antonio Andelic
36f46891dd
Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown 
Close sessions on Keeper shutdown
 2022-09-13 19:09:34 +02:00
kssenii
420ac4eb43 s3 header auth in ast 2022-09-13 15:13:28 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage 
KeeperMap storage engine
 2022-09-13 14:15:09 +02:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze 
Fix SYSTEM UNFREEZE for ordinary database
 2022-09-13 14:55:11 +03:00
alesapin
9af591a328
Revert "Sharding s3 key names (2)" 2022-09-13 12:29:02 +02:00
Antonio Andelic
45cde90219 Fix test 2022-09-13 09:51:31 +00:00
Sergei Trifonov
9cd78585c3 fix tests 2022-09-13 00:12:40 +02:00
Robert Schulze
fac1be9700
chore: restore SYSTEM RELOAD MODEL(S) and moniting view SYSTEM.MODELS 
- This commit restores statements "SYSTEM RELOAD MODEL(S)" which provide
  a mechanism to update a model explicitly. It also saves potentially
  unnecessary reloads of a model from disk after it's initial load.

  To keep the complexity low, the semantics of "SYSTEM RELOAD MODEL(S)
  was changed from eager to lazy. This means that both statements
  previously immedately reloaded the specified/all models, whereas now
  the statements only trigger an unload and the first call to
  catboostEvaluate() does the actual load.

- Monitoring view SYSTEM.MODELS is also restored but with some obsolete
  fields removed. The view was not documented in the past and for now it
  remains undocumented. The commit is thus not considered a breach of
  ClickHouse's public interface.
 2022-09-12 19:33:02 +00:00
Azat Khuzhin
b698a4ff65 Apply changes to http handlers on fly without server restart 
This has been implemented by simply restarting http servers in case of
http_handlers directive in configuration xml had been changed.

But, for this I have to change the handlers interface to accept
configuration separatelly, since the configuration that contains in the
server is the configuration with which server had been started.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Co-authored-by: Antonio Andelic
 2022-09-12 17:34:51 +02:00
Antonio Andelic
deee0d639f Address PR comments 2022-09-12 14:43:52 +00:00
Sergei Trifonov
fe88a7991d
Merge branch 'master' into readonly-settings-allow 2022-09-12 16:20:40 +02:00
Antonio Andelic
1cd43d8e5c Merge branch 'master' into keeper-storage 2022-09-12 13:42:51 +00:00
Antonio Andelic
e985ee3354 Close sessions on Keeper shutdown 2022-09-12 12:22:48 +00:00
kssenii
02c3d8b0a0 Fix 2022-09-11 16:42:56 +02:00
Alexey Milovidov
5802c2fdd2
Merge pull request #40713 from ClickHouse/remove-useless-line 
Remove one line from XML, because we do not care
 2022-09-11 09:53:30 +03:00
Alexey Milovidov
a5388affb8
Merge pull request #41023 from ClickHouse/remove-trash-3 
Remove trash from config
 2022-09-11 09:52:45 +03:00
Alexey Milovidov
91338ea771
Merge branch 'master' into remove-useless-line 2022-09-11 02:03:39 +03:00
Azat Khuzhin
46eafb4732 tests: skip test_send_crash_reports under ASan 
Since it also does not fits into timeouts [1]:

    2022-09-08 21:16:20 [ 377 ] DEBUG : Command:['docker', 'exec', '-u', 'root', 'roottestsendcrashreports_node_1', 'bash', '-c', 'pkill -SEGV clickhouse'] (cluster.py:95, run_and_check)
    ...
    2022-09-08 21:16:22 [ 377 ] DEBUG : run container_id:roottestsendcrashreports_node_1 detach:False nothrow:False cmd: ['cat', '/result.txt'] (cluster.py:1744, exec_in_container)
    ...
    2022-09-08 21:16:36 [ 377 ] DEBUG : Stdout:INITIAL_STATE (cluster.py:103, run_and_check)

And server logs:

    2022.09.08 21:16:21.112076 [ 228 ] {} <Fatal> BaseDaemon: ########################################
    2022.09.08 21:16:21.112170 [ 228 ] {} <Fatal> BaseDaemon: (version 22.9.1.1, build id: 0F7336E0A4D64134C51C8365DADCB78A9B39AA3B) (from thread 1) (no query) Received signal Segmentation fault (11)
    2022.09.08 21:16:21.112244 [ 228 ] {} <Fatal> BaseDaemon: Address: 0xde Access: read. Unknown si_code.
    2022.09.08 21:16:21.112321 [ 228 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fbe21d09376 0x40a4f71a 0xe293a4b 0xdc5c51a 0x38dee227 0xdc326a0 0x38e319d9 0xdc2b4e2 0xdc25fdb 0x7fbe21b2c083 0xdb636ae
    2022.09.08 21:16:21.112419 [ 228 ] {} <Fatal> BaseDaemon: 3. pthread_cond_wait @ 0x7fbe21d09376 in ?
    2022.09.08 21:16:21.122914 [ 228 ] {} <Fatal> BaseDaemon: 4. ./build_docker/../contrib/libcxx/src/condition_variable.cpp:0: std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) @ 0x40a4f71a in /usr/bin/clickhouse
    2022.09.08 21:16:21.233016 [ 228 ] {} <Fatal> BaseDaemon: 5.1. inlined from ./build_docker/../contrib/libcxx/include/atomic:952: unsigned long std::__1::__cxx_atomic_load<unsigned long>(std::__1::__cxx_atomic_base_impl<unsigned long> const*, std::__1::memory_order)
    2022.09.08 21:16:21.233135 [ 228 ] {} <Fatal> BaseDaemon: 5.2. inlined from ../contrib/libcxx/include/atomic:1582: std::__1::__atomic_base<unsigned long, false>::load(std::__1::memory_order) const
    2022.09.08 21:16:21.233183 [ 228 ] {} <Fatal> BaseDaemon: 5.3. inlined from ../contrib/libcxx/include/atomic:1586: std::__1::__atomic_base<unsigned long, false>::operator unsigned long() const
    2022.09.08 21:16:21.233234 [ 228 ] {} <Fatal> BaseDaemon: 5.4. inlined from ../src/Daemon/BaseDaemon.cpp:967: operator()
    2022.09.08 21:16:21.233303 [ 228 ] {} <Fatal> BaseDaemon: 5.5. inlined from ../contrib/libcxx/include/__mutex_base:402: void std::__1::condition_variable::wait<BaseDaemon::waitForTerminationRequest()::$_0>(std::__1::unique_lock<std::__1::mutex>&, BaseDaemon::waitForTerminationRequest()::$_0)
    2022.09.08 21:16:21.233334 [ 228 ] {} <Fatal> BaseDaemon: 5. ../src/Daemon/BaseDaemon.cpp:967: BaseDaemon::waitForTerminationRequest() @ 0xe293a4b in /usr/bin/clickhouse
    2022.09.08 21:16:21.350675 [ 228 ] {} <Fatal> BaseDaemon: 6. ./build_docker/../programs/server/Server.cpp:0: DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xdc5c51a in /usr/bin/clickhouse
    2022.09.08 21:16:21.394092 [ 228 ] {} <Fatal> BaseDaemon: 7. ./build_docker/../contrib/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x38dee227 in /usr/bin/clickhouse
    2022.09.08 21:16:21.654195 [ 228 ] {} <Fatal> BaseDaemon: 8. ./build_docker/../programs/server/Server.cpp:466: DB::Server::run() @ 0xdc326a0 in /usr/bin/clickhouse
    2022.09.08 21:16:21.666991 [ 228 ] {} <Fatal> BaseDaemon: 9. ./build_docker/../contrib/poco/Util/src/ServerApplication.cpp:0: Poco::Util::ServerApplication::run(int, char**) @ 0x38e319d9 in /usr/bin/clickhouse
    2022.09.08 21:16:21.916078 [ 228 ] {} <Fatal> BaseDaemon: 10. ./build_docker/../programs/server/Server.cpp:0: mainEntryClickHouseServer(int, char**) @ 0xdc2b4e2 in /usr/bin/clickhouse
    2022.09.08 21:16:21.929922 [ 228 ] {} <Fatal> BaseDaemon: 11. ./build_docker/../programs/main.cpp:0: main @ 0xdc25fdb in /usr/bin/clickhouse
    2022.09.08 21:16:21.929981 [ 228 ] {} <Fatal> BaseDaemon: 12. __libc_start_main @ 0x7fbe21b2c083 in ?
    2022.09.08 21:16:30.357032 [ 228 ] {} <Fatal> BaseDaemon: 13. _start @ 0xdb636ae in /usr/bin/clickhouse
    2022.09.08 21:16:31.383233 [ 228 ] {} <Fatal> BaseDaemon: Integrity check of the executable skipped because the reference checksum could not be read. (calculated checksum: 6200AC7C1270DC293DF3302E1C64399B)
    ...
    2022.09.08 21:16:40.564453 [ 228 ] {} <Information> SentryWriter: Sending crash report

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/a0b85eaca8d4003c9fbc4571b30830d30f1984e9/integration_tests__asan__[3/3].html

Though another option is to increase waiting time.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
robot-clickhouse
5bc2845e49 Automatic style fix 2022-09-10 03:17:34 +00:00
Alexey Milovidov
f2227e125f Fix test 2022-09-10 05:11:09 +02:00
Alexey Milovidov
61278c81e4 Merge branch 'master' into remove-useless-line 2022-09-10 05:07:10 +02:00
robot-clickhouse
79ccce23e9 Automatic style fix 2022-09-09 20:09:54 +00:00
Igor Nikonov
dbfb448290 Fix: integration test, standalone keeper mode 
There is logic regarding which keeper binary use to start keeper cluster in an integration test
There 2 options:
(1) standalone keeper binary (expected binary name clickhouse-keeper)
(2) clickhouse binary with keeper inside

Fixed:
- option (1) didn't work since docker_compose_keeper.yaml didn't create
target clickhouse-keeper at all
- if clickhouse-keeper existed, option (1) was taken but
  clickhouse-keeper could be just a link to clickhouse binary (the link
  is created always during build if cmake option BUILD_STANDALONE_KEEPER is OFF)
 2022-09-09 14:51:34 +00:00
robot-clickhouse
aeac3916ef Automatic style fix 2022-09-09 12:27:13 +00:00
Antonio Andelic
32483aeec8 Merge branch 'master' into keeper-storage 2022-09-09 12:24:01 +00:00
Sergei Trifonov
76eb001e78 add test for changeable_in_readonly + fix SQL in docs 2022-09-09 14:01:29 +02:00
Alexander Tokmakov
48927ba0ac
Merge branch 'master' into no-hardlinks-while-making-backup-of-mergetree-in-atomic-db 2022-09-09 14:24:44 +03:00
Vitaly Baranov
10629a66e5 Fix black. 2022-09-08 20:58:51 +02:00
Vitaly Baranov
e48cd2b6f6 Add more test cases. 2022-09-08 18:14:25 +02:00
robot-clickhouse
807b09f6b0 Automatic style fix 2022-09-08 15:52:00 +00:00
Sergei Trifonov
f92b6a6176
Merge branch 'master' into readonly-settings-allow 2022-09-08 17:44:32 +02:00
Sergei Trifonov
62541ab764 fix more tests and clang tidy build 2022-09-08 17:43:09 +02:00
Antonio Andelic
b99996961d Address PR comments 2022-09-08 12:45:36 +00:00
Vitaly Baranov
122009a2bd Use table lock if database is ordinary and zero-copy-replication is enabled. 2022-09-08 13:54:59 +02:00
Robert Schulze
c16707ff00
chore: delete obsolete modelEvaluate() function + SYSTEM.MODELS view 
- The deleted function modelEvaluate() was superseded by
  catboostEvaluate().

- Also delete the external model repository, as modelEvaluate() was it's
  last user. Additionally remove the system view SYSTEM.MODELS for
  inspecting the repository.

- SYSTEM RELOAD MODELS is also obsolete. HOWEVER, it was retained and
  made a no-op instead of deleted.

  Why?
  The reason is that RBAC in distributed setups works by storing
  privileges (granted and revoked) as plain SQL statements in Keeper.
  Nodes read these statements at startup and parse them. If a privilege
  for SYSTEM RELOAD MODELS exists but parser doesn't recognize it
  nodes would fail to come up.

  Considered but rejected alternatives:
  - Ignore SYSTEM RELOAD MODELS during parsing RBAC privileges and
    return an error for regular SYSTEM RELOAD MODELS SQL. Special-case
    of no-op behavior, too brittle.
  - Remove SYSTEM RELOAD MODELS manually from Keeper via command-line
    manipulation of Keeper nodes or via SQL by dropping the privileges.
    Needs user intervention during upgrade.
 2022-09-08 09:10:11 +00:00
Robert Schulze
60f9f6855d
feat: implement catboost in library-bridge 
This commit moves the catboost model evaluation out of the server
process into the library-bridge binary. This serves two goals: On the
one hand, crashes / memory corruptions of the catboost library no longer
affect the server. On the other hand, we can forbid loading dynamic
libraries in the server (catboost was the last consumer of this
functionality), thus improving security.

SQL syntax:

  SELECT
    catboostEvaluate('/path/to/model.bin', FEAT_1, ..., FEAT_N) > 0 AS prediction,
    ACTION AS target
  FROM amazon_train
  LIMIT 10

Required configuration:

  <catboost_lib_path>/path/to/libcatboostmodel.so</catboost_lib_path>

*** Implementation Details ***

The internal protocol between the server and the library-bridge is
simple:

- HTTP GET on path "/extdict_ping":
  A ping, used during the handshake to check if the library-bridge runs.

- HTTP POST on path "extdict_request"
  (1) Send a "catboost_GetTreeCount" request from the server to the
      bridge, containing a library path (e.g /home/user/libcatboost.so) and
      a model path (e.g. /home/user/model.bin). Rirst, this unloads the
      catboost library handler associated to the model path (if it was
      loaded), then loads the catboost library handler associated to the
      model path, then executes GetTreeCount() on the library handler and
      finally sends the result back to the server. Step (1) is called once
      by the server from FunctionCatBoostEvaluate::getReturnTypeImpl(). The
      library path handler is unloaded in the beginning because it contains
      state which may no longer be valid if the user runs
      catboost("/path/to/model.bin", ...) more than once and if "model.bin"
      was updated in between.
  (2) Send "catboost_Evaluate" from the server to the bridge, containing
      the model path and the features to run the interference on. Step (2)
      is called multiple times (once per chunk) by the server from function
      FunctionCatBoostEvaluate::executeImpl(). The library handler for the
      given model path is expected to be already loaded by Step (1).

Fixes #27870
 2022-09-08 09:01:32 +00:00
Vitaly Baranov
31ed722572
Merge pull request #41044 from vitlibar/more-conventional-conversion-yaml-to-xmk 
More conventional conversion yaml to xml
 2022-09-07 13:46:32 +02:00
Vitaly Baranov
9c847ceec9 No hardlinks while making backup of MergeTree in atomic database. 2022-09-07 11:44:50 +02:00
Daniil Rubin
f75376b7f8 Apply black formatter to tests 2022-09-07 11:05:40 +03:00
Daniil Rubin
4cc397da84 Add tests for deltalake storage 2022-09-07 11:05:40 +03:00
Daniil Rubin
0e9c3f299c Fix bug in test 2022-09-07 11:05:40 +03:00
Daniil Rubin
7d8cc20c46 Add select test 2022-09-07 11:05:40 +03:00
Daniil Rubin
6b95faf2b3 Add createquery test 2022-09-07 11:05:40 +03:00
Daniil Rubin
3e87fd0ee4 Add test template 2022-09-07 11:05:40 +03:00
Vitaly Baranov
63e992d52d Edit test configs. 2022-09-06 17:09:26 +02:00
Sergei Trifonov
6217559175
Merge branch 'master' into readonly-settings-allow 2022-09-06 16:10:48 +02:00
Antonio Andelic
5eb034a920 Merge branch 'master' into start-embedded-keeper-async 2022-09-06 12:14:47 +00:00
Antonio Andelic
4a68bfef39 Fix tests with async Keeper start 2022-09-06 12:12:17 +00:00
Sergei Trifonov
f77809ddbc
Merge pull request #40900 from ClickHouse/s3-detailed-metrics 
S3 detailed metrics
 2022-09-06 13:20:57 +02:00
Anton Ivashkin
e91d22a5b7 Merge branch 'fix-ordinary-system-unfreeze' of github.com:PolyProgrammist/ClickHouse into fix-ordinary-system-unfreeze 2022-09-06 12:18:05 +03:00
Anton Ivashkin
5e7cf1f238 Fix test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] 2022-09-06 12:14:40 +03:00
Alexey Milovidov
59bd94dcf8
Merge pull request #40975 from vitlibar/fix-access-rights-for-describe 
Fix access rights for DESCRIBE TABLE
 2022-09-06 02:38:34 +03:00
Alexey Milovidov
4c30dbc905
Merge pull request #40968 from ClickHouse/s3-sharding-2 
Sharding s3 key names (2)
 2022-09-05 22:47:25 +03:00
ianton-ru
39e1fc7a0f
Merge branch 'master' into fix-ordinary-system-unfreeze 2022-09-05 17:10:59 +03:00
Sergei Trifonov
1c247308f1
Merge branch 'master' into s3-detailed-metrics 2022-09-05 15:55:41 +02:00
Sergei Trifonov
225773634c fix integration test for profile events s3 2022-09-05 15:54:28 +02:00
Kseniia Sumarokova
cff1dd6a8e
Merge branch 'master' into move-strange-test 2022-09-05 15:19:57 +02:00
Antonio Andelic
e07539d1b9 Merge branch 'master' into keeper-storage 2022-09-05 09:54:13 +00:00
Antonio Andelic
3a0581e990
Merge pull request #40543 from Lloyd-Pottiger/feat/support-read-only-for-embeddedrocksdb 
Add read-only support for EmbeddedRocksDB
 2022-09-05 09:31:07 +02:00
Alexey Milovidov
cda8052df0 Fix trash 2022-09-05 03:46:10 +02:00
Alexey Milovidov
e70d526896 Fix trash 2022-09-05 03:35:43 +02:00
Alexey Milovidov
bff96b0af6
Merge branch 'master' into fix-access-rights-for-describe 2022-09-05 04:34:57 +03:00
Alexey Milovidov
cc50aa39a2 Merge branch 's3-sharding-2' of github.com:ClickHouse/ClickHouse into s3-sharding-2 2022-09-05 02:25:01 +02:00
Alexey Milovidov
79e3723272 Merge branch 'master' into s3-sharding-2 2022-09-05 02:24:35 +02:00
robot-clickhouse
89d40b6f70 Automatic style fix 2022-09-05 00:23:36 +00:00
Alexey Milovidov
a0b5567d3e Move strange test 2022-09-05 02:16:48 +02:00
Alexey Milovidov
007680d93f Preparation 2022-09-05 02:13:51 +02:00
Vitaly Baranov
6c164905ca Add test. 2022-09-04 18:57:51 +02:00
Alexey Milovidov
c4adc9ed8f Remove trash 2022-09-04 04:28:08 +02:00
Alexey Milovidov
13a129bee7 Merge branch 'master' into remove-useless-line 2022-09-04 03:24:40 +02:00
Lloyd-Pottiger
ec68ed8a93 fix test 
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
 2022-09-03 10:44:07 +08:00
Sergei Trifonov
005b2588d8
Merge branch 'master' into readonly-settings-allow 2022-09-02 16:37:43 +02:00
Lloyd-Pottiger
b76af9284e style check 
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
 2022-09-02 19:29:57 +08:00
Lloyd-Pottiger
f499381f57 move functional test to integration test 
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
 2022-09-02 18:45:37 +08:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage 2022-09-02 12:13:07 +02:00
Lloyd-Pottiger
5a6b2106b5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into feat/support-read-only-for-embeddedrocksdb 2022-09-02 18:10:00 +08:00
robot-clickhouse
46f74fcbaf Automatic style fix 2022-09-02 07:49:09 +00:00
Antonio Andelic
b02bf49939 Use path prefix 2022-09-02 07:34:26 +00:00
Lloyd-Pottiger
59dccd6e49 fix test 
Signed-off-by: Lloyd-Pottiger <yan1579196623@gamil.com>
 2022-09-02 11:14:49 +08:00
Sergei Trifonov
7fec55eea4 work in progress 2022-09-02 04:12:05 +02:00
Frank Chen
9d63cbe811 Merge 'origin/master' into tracing_context_propagation to resolve conflicts 2022-09-01 23:18:59 +08:00
Vitaly Baranov
6dcca686cb
Merge pull request #40827 from vitlibar/fix-incremental-backups-for-log-family 
Fix incremental backups for Log family.
 2022-09-01 13:13:32 +02:00
Sema Checherinda
7b59fdc042
Merge pull request #40779 from CheSema/detached-parts-metric 
Metric for the number of detached parts
 2022-09-01 12:24:42 +02:00
Vitaly Baranov
007ae0e6cc Fix incremental backups for Log family. 2022-08-31 12:57:28 +02:00
Antonio Andelic
9dd1a9859d Merge branch 'master' into keeper-storage 2022-08-31 10:12:11 +00:00
root
ea517de61a style check correction 2022-08-30 23:20:19 -03:00
root
f02f7ce99c addressed minor change requests 2022-08-30 23:20:19 -03:00
root
84a617675d style check correction in test.py 2022-08-30 23:20:19 -03:00
root
3ced3c9933 style check correction in test.py 2022-08-30 23:20:19 -03:00
root
1d29a494db Modified test code to cover various scenarios for custom keys 2022-08-30 23:20:19 -03:00
root
d5db88fa27 Structured-logging-custom-keys PR init 2022-08-30 23:20:19 -03:00