thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-20 06:32:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
100,392 Commits 9,144 Branches 1,778 Tags 2 GiB
774a86021f
Commit Graph

5105 Commits

Author SHA1 Message Date
Azat Khuzhin
8345c5f881 tests: cover ATTACH of BACKUP from s3_plain disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
7a5432feaa Fix possible SIGSEGV for web disks when file does not exists 
It can be triggered in multiple ways, either when file does not exists
and you are trying to create MergeTree table from web (that has special
code for UUID handling) or by simply OPTIMIZE TABLE FINAL for MergeTree
table that is located on web disk, in both cases you will get the
following:

<details>

<summary>stacktrace</summary>

    2022.10.28 14:08:40.631226 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> ContextAccess (default): Access granted: OPTIMIZE ON default.data_from_web
    2022.10.28 14:08:40.632017 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Debug> default.data_from_web (a3e65e1f-5fd4-47ed-9dbd-307f2586b52d) (MergerMutator): Selected 1 parts from all_1_1_0 to all_1_1_0
    2022.10.28 14:08:40.632496 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> default.data_from_web (a3e65e1f-5fd4-47ed-9dbd-307f2586b52d): Trying to reserve 1.00 MiB using storage policy from min volume index 0
    2022.10.28 14:08:40.632752 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> DiskObjectStorage(DiskWebServer): Reserved 1.00 MiB on remote disk `web_disk`, having unreserved 16.00 EiB.
    2022.10.28 14:08:40.634155 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Debug> MergeTask::PrepareStage: Merging 1 parts: from all_1_1_0 to all_1_1_0 into Compact
    2022.10.28 14:08:40.634498 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> WebObjectStorage: Loading metadata for directory: http://127.0.0.1:8080/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1
    2022.10.28 14:08:40.635025 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> DiskWeb: Adding directory: store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/
    2022.10.28 14:08:40.635355 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> ReadWriteBufferFromHTTP: Sending request to http://127.0.0.1:8080/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index
    2022.10.28 14:08:40.639618 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> DiskWeb: Cannot load disk metadata. Error: Code: 86. DB::Exception: Received error from remote server /store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index. HTTP status code: 404 Not Found, body: <!doctype html><html><head><title>404 Not Found</title><style>
    body { background-color: #fcfcfc; color: #333333; margin: 0; padding:0; }
    h1 { font-size: 1.5em; font-weight: normal; background-color: #9999cc; min-height:2em; line-height:2em; border-bottom: 1px inset black; margin: 0; }
    h1, p { padding-left: 10px; }
    code.url { background-color: #eeeeee; font-family:monospace; padding:0 2px;}
    </style>
    </head><body><h1>Not Found</h1><p>The requested resource <code class="url">/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index</code> was not found on this server.</p></body></html>: while loading disk metadata. (RECEIVED_ERROR_FROM_REMOTE_IO_SERVER) (version 22.11.1.1)
    2022.10.28 14:08:40.640527 [ 5488 ] {} <Trace> BaseDaemon: Received signal 11
    2022.10.28 14:08:40.641529 [ 9027 ] {} <Fatal> BaseDaemon: ########################################
    2022.10.28 14:08:40.642759 [ 9027 ] {} <Fatal> BaseDaemon: (version 22.11.1.1, build id: 12145DA78CE5E9EBB10A034177FAE5967EF81A4A) (from thread 6043) (query_id: a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1) (query: optimize table data_from_web final) Received signal Segmentation fault (11)
    2022.10.28 14:08:40.643260 [ 9027 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Unknown si_code.
    2022.10.28 14:08:40.643769 [ 9027 ] {} <Fatal> BaseDaemon: Stack trace: 0x7ffff416c0f2 0x7ffff7cd1ca8 0x7ffff679ae5e 0x7fffd52e7906 0x7fffd50c65aa 0x7fffca7a0d42 0x7fffcaee79ec 0x7fffcaf242f8 0x7fffcaf242b5 0x7fffcaf2427d 0x7fffcaf24255 0x7fffcaf2421d 0x7ffff65c3686 0x7ffff65c2295 0x7fffcaeee2a9 0x7fffcaef2c43 0x7fffcaee3c0e 0x7fffcc4a7851 0x7fffcc4a768f 0x7fffcc4abb2d 0x7fffcfdce828 0x7fffd03e3eaa 0x7fffd03dfe3b 0x7fffc8ec42d4 0x7fffc8ed51d2 0x7ffff4bdd839 0x7ffff4bde0a8 0x7ffff48ab261 0x7ffff48a769a 0x7ffff48a6335 0x7ffff409f8fd 0x7ffff4121a60
    2022.10.28 14:08:40.644411 [ 9027 ] {} <Fatal> BaseDaemon: 4. ? @ 0x7ffff416c0f2 in ?
    2022.10.28 14:08:40.676390 [ 9027 ] {} <Fatal> BaseDaemon: 5. /src/ch/clickhouse/src/Common/StringUtils/StringUtils.cpp:9: detail::startsWith(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, char const*, unsigned long) @ 0x1ca8 in /src/ch/clickhouse/.cmake/src/Common/StringUtils/libstring_utilsd.so
    2022.10.28 14:08:40.730727 [ 9027 ] {} <Fatal> BaseDaemon: 6. /src/ch/clickhouse/src/Common/StringUtils/StringUtils.h:19: startsWith(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) @ 0x59ae5e in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    2022.10.28 14:08:40.923955 [ 9027 ] {} <Fatal> BaseDaemon: 7. /src/ch/clickhouse/src/Disks/ObjectStorages/Web/MetadataStorageFromStaticFilesWebServer.cpp:58: DB::MetadataStorageFromStaticFilesWebServer::exists(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const @ 0x6e7906 in /src/ch/clickhouse/.cmake/src/libdbmsd.so
    2022.10.28 14:08:41.291996 [ 9027 ] {} <Fatal> BaseDaemon: 8. /src/ch/clickhouse/src/Disks/ObjectStorages/DiskObjectStorage.cpp:181: DB::DiskObjectStorage::exists(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const @ 0x4c65aa in /src/ch/clickhouse/.cmake/src/libdbmsd.so
    2022.10.28 14:08:41.704697 [ 9027 ] {} <Fatal> BaseDaemon: 9. /src/ch/clickhouse/src/Storages/MergeTree/DataPartStorageOnDisk.cpp:74: DB::DataPartStorageOnDisk::exists() const @ 0xda0d42 in /src/ch/clickhouse/.cmake/src/libclickhouse_storages_mergetreed.so
    2022.10.28 14:08:43.032459 [ 9027 ] {} <Fatal> BaseDaemon: 10. /src/ch/clickhouse/src/Storages/MergeTree/MergeTask.cpp:147: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x14e79ec in /src/ch/clickhouse/.cmake/src/libclickhouse_storages_mergetreed.so
    ...
    Segmentation fault (core dumped)

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-28 15:15:49 +02:00
Alexander Tokmakov
86b81a2b6f
Merge pull request #42234 from arthurpassos/fix_ares_fd_process_race_condition 
Fix c-ares crash
 2022-10-27 17:17:22 +03:00
Nikolai Kochetov
5c4444237e
Merge pull request #42617 from ClickHouse/revert-revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3 
Revert revert 41268 disable s3 parallel write for part moves to disk s3
 2022-10-25 14:29:27 +02:00
Nikita Taranov
49f6692a2e
Adapt internal data structures to 512-bit era (#42564) 
* impl

* update tests

* fix tests
 2022-10-25 13:56:28 +02:00
Raúl Marín
0016bc2b73
Improve the time to recover keeper connections (#42541) 
Co-authored-by: Nikita Mikhaylov <nikitamikhaylov@clickhouse.com>
Co-authored-by: robot-clickhouse <robot-clickhouse@clickhouse.com>
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2022-10-25 13:26:29 +02:00
Alexey Milovidov
9ee7131f67
Merge pull request #42423 from ClickHouse/optimize-partitions-in-background 
Optimize partitions in background
 2022-10-24 19:41:50 +02:00
Nikolai Kochetov
beb40e4579 Split tests. 2022-10-24 14:22:06 +00:00
Vitaly Baranov
cd158b021f
Merge pull request #42562 from vitlibar/subqueries-in-row-policy-filters 
Fix using subqueries in row policy filters
 2022-10-24 13:38:15 +02:00
Azat Khuzhin
dbb4f8216f Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Functions/GregorianDate.h
 2022-10-22 08:20:05 +02:00
Nikolay Degterinsky
22f87e2bf9
Merge pull request #42549 from evillique/disks-better-logs 
Better logs in clickhouse-disks
 2022-10-22 04:31:44 +02:00
Azat Khuzhin
4eeb7337ed Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32 
Conflicts:
- src/Common/intExp.h
- src/Functions/DateTimeTransforms.h
- src/Functions/FunctionsConversion.h
 2022-10-21 22:51:56 +02:00
Alexander Tokmakov
7d11054134
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-21 18:27:22 +03:00
Vitaly Baranov
2033ea1216 Support subqueries in row policy filters. 2022-10-21 15:42:29 +02:00
Alexander Tokmakov
5ffd026261
Merge pull request #42529 from azat/tests/integration/fix-sanitizer-reports 
Fix sanitizer reports in integration tests
 2022-10-21 16:01:43 +03:00
Alexander Tokmakov
1343f62e8f
Update test.py 2022-10-21 15:59:27 +03:00
Azat Khuzhin
0365bc56aa Add backward compatiblity check for some functions 
* State of aggregate functions

This test goes though all aggregate functions that:
- has only one argument
- support string as an argument

And do a simple check by creating the aggregate state with one string.

Yes this is not covers everything (does not cover functions with
different number of arguments, types, different states in case of
multiple values - uniqCombined, but as for uniqCombined it will be
checked via uniqHLL12), but at least something.

And about type, String had been selected, since it more likely that
there will be used some hash function internally.

* Function(String)

Check every function which has only one argument, and it's type is
String.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:43 +02:00
Antonio Andelic
3091d609fa
Merge branch 'master' into optimize-partitions-in-background 2022-10-21 11:01:41 +02:00
Nikolay Degterinsky
d7d0794d2f Better logs in clickhouse-disks 2022-10-20 23:17:29 +00:00
Nikolai Kochetov
57ffcb7ed0 Revert "Merge pull request #41681 from ClickHouse/revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3" 
This reverts commit 7956c2becf, reversing
changes made to 57be648984.
 2022-10-20 19:51:27 +02:00
Arthur Passos
50f9acb221
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-20 13:35:58 -03:00
Arthur Passos
2fe59e7e8f lower thread and iterations number 2022-10-20 13:30:20 -03:00
Alexander Tokmakov
905a4ded42
Merge pull request #42535 from ClickHouse/tavplubix-patch-7 
Disable `test_storage_nats`, because it's permanently broken
 2022-10-20 16:58:46 +03:00
Alexander Tokmakov
5b03bbce2e
Merge pull request #41981 from ClickHouse/fixes_for_replicated_merge_tree2 
Some fixes for ReplicatedMergeTree
 2022-10-20 16:57:51 +03:00
Alexander Tokmakov
89ff3d4731
Update test.py 2022-10-20 16:54:52 +03:00
Azat Khuzhin
c399b021cf Fix sanitizer reports in integration tests 
Before they was ignored because first there was a check for a sign of
sanitizer (==================), but it was done by
clickhouse-server.log, while sanitizer write to stderr.log.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-20 15:45:40 +02:00
Antonio Andelic
9d7e44409a
Merge pull request #41342 from ClickHouse/keeper-upload-snapshot-to-s3 
Upload Keeper snapshots to S3
 2022-10-20 09:14:59 +02:00
Antonio Andelic
77eb353839
Merge pull request #42243 from ClickHouse/use-multiread-more 
Use MultiRead where possible
 2022-10-20 09:04:54 +02:00
Alexander Tokmakov
f958c155f3
Merge branch 'master' into fixes_for_replicated_merge_tree2 2022-10-19 17:25:24 +03:00
Antonio Andelic
05e6b94b54 Polishing 2022-10-19 12:59:28 +00:00
Antonio Andelic
e5408aac99 Simplify logic 2022-10-19 12:34:20 +00:00
Arthur Passos
7f0a2d32c8 fix test errors 2022-10-18 20:15:58 -03:00
Vitaly Baranov
1365105bc4 Implement backup to S3 2022-10-19 00:04:41 +02:00
Alexander Tokmakov
ea77ff9022 Merge branch 'master' into fixes_for_replicated_merge_tree2 2022-10-18 17:25:25 +02:00
Antonio Andelic
e828dce331 Disable auto optimize by default 2022-10-18 12:11:23 +00:00
Antonio Andelic
bd2522fdd0 Merge branch 'master' into optimize-partitions-in-background 2022-10-18 09:13:59 +00:00
Antonio Andelic
ee2f5977d5
Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-18 10:02:50 +02:00
Vitaly Baranov
eb637a6f81
Merge pull request #42232 from azat/backups/s3 
Support BACKUP to S3 with as-is path/data structure
 2022-10-18 02:37:07 +02:00
Yakov Olkhovskiy
3ce1fa09d6
Merge pull request #41198 from ClickHouse/composable-protocol 
Composable protocol
 2022-10-17 11:23:12 -04:00
Antonio Andelic
7c7395575f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-17 11:02:36 +00:00
Antonio Andelic
878dee40b1 Use ZooKeeper in a single test 2022-10-17 07:36:49 +00:00
Arthur Passos
f414edebae fix black style 2022-10-14 15:54:48 -03:00
Arthur Passos
299d848473 style fix? 2022-10-14 15:23:10 -03:00
Arthur Passos
78202f9660
Merge branch 'master' into fix_ares_fd_process_race_condition 2022-10-14 12:56:39 -03:00
Arthur Passos
6d22bb78e0 custom py script in integ tests 2022-10-14 12:39:08 -03:00
Azat Khuzhin
8830f0608d Support BACKUP to S3 with as-is path/data structure 
Right now backup to S3 does not make a lot of sense, since:
- it has random names, and to decoding them
- requires metadata from local disk (/var/lib/disks/DISK/BACKUP_NAME)
- or send_metadata (but it is also tricky even with it)

So this patch adds simpler interface for S3, it is only suitable for
BACKUP/RESTORE, so don't try to use it for MergeTree engine.

It is done by adding separate disk - `s3_plain` for this, that:
- does not support any extended features, like renames/hardlinks/attrs/...
  (so basically everything that MergeTree requires)
- only write/read/unlink/list files

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-14 12:59:25 +02:00
nvartolomei
ea47ab1400
assert unused value in test_replicated_merge_tree_compatibility 2022-10-12 18:12:25 +01:00
Yakov Olkhovskiy
6c8e52eb67
Merge branch 'master' into composable-protocol 2022-10-08 20:33:32 -04:00
robot-clickhouse
3bed015a1f Automatic style fix 2022-10-08 00:09:24 +00:00
Yakov Olkhovskiy
6bb62d4d03 add test for PROXYv1 2022-10-08 00:01:58 +00:00