thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-19 14:11:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
100,228 Commits 9,138 Branches 1,778 Tags 2 GiB
bceca73f6f
Commit Graph

100228 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Azat Khuzhin
bceca73f6f Throw NOT_IMPLEMENTED form ReadOnlyMetadataStorage::getLastChanged() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
3b7abbbff4 Rename s/listPrefix/findAllFiles, s/listPrefixInPath/getDirectoryContents/ 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
8345c5f881 tests: cover ATTACH of BACKUP from s3_plain disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
fa7535c90d Slightly optimize MetadataStorageFromPlainObjectStorage::isDirectory() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
82ea67eb51 Slightly better MetadataStorageFromPlainObjectStorage::isFile() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
95fb2ad3cf Implement ATTACH of MergeTree table for s3_plain disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
4e42521f44 Introduce ReadOnlyMetadataStorage 
And use it for:
- MetadataStorageFromPlainObjectStorage
- MetadataStorageFromStaticFilesWebServer

This will allow to reduce ~100-200 lines of duplicated code, and plus
make the code less error prone.

Note, for now I tried to make this without behaviour changes.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
f8ba24f040 Implement S3ObjectStorage::listPrefixInPath() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:26 +01:00
Azat Khuzhin
18e4fdf40f Introduce IObjectStorage::listPrefixInPath() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:25 +01:00
Azat Khuzhin
ee18140c48 Remove listPrefix() implementation for disks that does not support send_metadata 
The reason for removing is it because not compatible with restoring
(with send_metadata set) anyway:
- HDFS - is not compatible with send_metadata, and besides it's
  implementaion is not correct, since it is simply `ls -l`, while the
  following is required: `find . -maxdepth 1 -type f`
- Web - is not compatible with send_metadata anyway
- Local - is not compatible with send_metadata anyway

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:25 +01:00
Azat Khuzhin
88db8ae7fa Add a comment for IObjectStorage::listPrefix() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:25 +01:00
Azat Khuzhin
94d9600fb8 Implement MetadataStorageFromPlainObjectStorage::getLastChanged() (as for web) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:25 +01:00
Azat Khuzhin
b34ffda272 Implement MetadataStorageFromPlainObjectStorage::getLastModified() (used by MergeTree) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:25 +01:00
Azat Khuzhin
f6d0c03ee5 Fix path to files in MetadataStorageFromPlainObjectStorage 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-31 12:02:25 +01:00
Nikolai Kochetov
742d9af397
Merge pull request #42582 from azat/memory-tracker/fixes 
Fix frequent memory drift message and clarify things in comments
 2022-10-31 11:43:20 +01:00
Nikolay Degterinsky
05e178b6d3
Merge pull request #42767 from azat/disks/web-sigsegv-fix 
Fix possible SIGSEGV for web disks when file does not exists
 2022-10-30 20:27:21 +01:00
Robert Schulze
088bc8b8b5
Merge pull request #42737 from arenadata/ADQM-605 
Add support of Date32 arguments for formatDateTime and FROM_UNIXTIME functions
 2022-10-30 20:07:46 +01:00
Roman Vasin
24a4f4afe7
Merge branch 'master' into ADQM-605 2022-10-30 18:03:07 +03:00
Alexey Milovidov
88f3db9115
Merge pull request #42718 from ClickHouse/move-sonar-to-nightly 
Move SonarCloud Job to nightly
 2022-10-29 18:07:50 +02:00
Alexey Milovidov
7569d866f8
Merge pull request #42764 from ClickHouse/auto/v22.3.14.23-lts 
Update version_date.tsv and changelogs after v22.3.14.23-lts
 2022-10-29 17:42:33 +02:00
Alexey Milovidov
3c69f33b7d
Merge branch 'master' into auto/v22.3.14.23-lts 2022-10-29 17:42:26 +02:00
Alexey Milovidov
cd9756bbaa
Merge pull request #42759 from ClickHouse/auto/v22.3.14.18-lts 
Update version_date.tsv and changelogs after v22.3.14.18-lts
 2022-10-29 17:40:22 +02:00
Nikolay Degterinsky
6d1b547586
Merge pull request #42724 from evillique/fix-parser-bug 
Fix a bug in ParserFunction
 2022-10-29 13:59:20 +02:00
Alexey Milovidov
2a7f5823a6
Merge pull request #42783 from yukotakagi/patch-7 
Update README.md
 2022-10-29 11:46:09 +02:00
Ilya Yatsishin
4bd30c4fa6
Merge pull request #42781 from den-crane/patch-53 2022-10-29 09:35:29 +02:00
Yuko Takagi
004ca99712
Update README.md 2022-10-28 13:58:14 -06:00
Denny Crane
7fa71d8c04
Update column.md 2022-10-28 16:04:00 -03:00
Denny Crane
e903efda0a
Update column.md 2022-10-28 16:00:37 -03:00
Julio Jimenez
53acc23856
sc2086 
Signed-off-by: Julio Jimenez <julio@clickhouse.com>
 2022-10-28 14:17:29 -04:00
Julio Jimenez
51f2d6046c
sc2086 
Signed-off-by: Julio Jimenez <julio@clickhouse.com>
 2022-10-28 14:10:23 -04:00
Julio Jimenez
dc10c7a446
no newline at end of file. 
Signed-off-by: Julio Jimenez <julio@clickhouse.com>
 2022-10-28 13:57:47 -04:00
Julio Jimenez
10bee23c58
no newline at end of file. 
Signed-off-by: Julio Jimenez <julio@clickhouse.com>
 2022-10-28 13:56:12 -04:00
Julio Jimenez
144648a633
disable sc2086 
Signed-off-by: Julio Jimenez <julio@clickhouse.com>
 2022-10-28 13:47:37 -04:00
Julio Jimenez
3cb479f7de Merge branch 'move-sonar-to-nightly' of https://github.com/ClickHouse/ClickHouse into move-sonar-to-nightly 2022-10-28 13:33:35 -04:00
Julio Jimenez
0872d5c440
style 
Signed-off-by: Julio Jimenez <julio@clickhouse.com>
 2022-10-28 13:33:12 -04:00
Julio Jimenez
36d48865a9
Merge branch 'master' into move-sonar-to-nightly 2022-10-28 13:32:51 -04:00
Maksim Kita
d7eee86827
Merge pull request #42773 from UnamedRus/patch-2 
Fix Polygon dict xml config
 2022-10-28 18:15:01 +03:00
UnamedRus
d4c6268806
Fix Polygon dict xml config 
Missing  <attribute> tag
 2022-10-28 18:47:32 +04:00
Nikolay Degterinsky
4d4a79091f
Merge pull request #42770 from besteffects/patch-2 
Update database.md
 2022-10-28 16:25:18 +02:00
Alexey Milovidov
f64e632707
Merge pull request #42469 from amosbird/issue_42456 
Fix misbehavior of key analysis
 2022-10-28 16:18:00 +02:00
Roman Vasin
4a4fb20731 Add Date32 to 01411_from_unixtime and improve 00718_format_datetime 2022-10-28 14:08:28 +00:00
Vitalii S
665fcf55aa
Update database.md 
Minor text correction
 2022-10-28 10:08:04 -04:00
Azat Khuzhin
7a5432feaa Fix possible SIGSEGV for web disks when file does not exists 
It can be triggered in multiple ways, either when file does not exists
and you are trying to create MergeTree table from web (that has special
code for UUID handling) or by simply OPTIMIZE TABLE FINAL for MergeTree
table that is located on web disk, in both cases you will get the
following:

<details>

<summary>stacktrace</summary>

    2022.10.28 14:08:40.631226 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> ContextAccess (default): Access granted: OPTIMIZE ON default.data_from_web
    2022.10.28 14:08:40.632017 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Debug> default.data_from_web (a3e65e1f-5fd4-47ed-9dbd-307f2586b52d) (MergerMutator): Selected 1 parts from all_1_1_0 to all_1_1_0
    2022.10.28 14:08:40.632496 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> default.data_from_web (a3e65e1f-5fd4-47ed-9dbd-307f2586b52d): Trying to reserve 1.00 MiB using storage policy from min volume index 0
    2022.10.28 14:08:40.632752 [ 6043 ] {6165bf5f-e76b-4bca-941c-7c7ff5e3b46b} <Trace> DiskObjectStorage(DiskWebServer): Reserved 1.00 MiB on remote disk `web_disk`, having unreserved 16.00 EiB.
    2022.10.28 14:08:40.634155 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Debug> MergeTask::PrepareStage: Merging 1 parts: from all_1_1_0 to all_1_1_0 into Compact
    2022.10.28 14:08:40.634498 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> WebObjectStorage: Loading metadata for directory: http://127.0.0.1:8080/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1
    2022.10.28 14:08:40.635025 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> DiskWeb: Adding directory: store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/
    2022.10.28 14:08:40.635355 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> ReadWriteBufferFromHTTP: Sending request to http://127.0.0.1:8080/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index
    2022.10.28 14:08:40.639618 [ 6043 ] {a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1} <Trace> DiskWeb: Cannot load disk metadata. Error: Code: 86. DB::Exception: Received error from remote server /store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index. HTTP status code: 404 Not Found, body: <!doctype html><html><head><title>404 Not Found</title><style>
    body { background-color: #fcfcfc; color: #333333; margin: 0; padding:0; }
    h1 { font-size: 1.5em; font-weight: normal; background-color: #9999cc; min-height:2em; line-height:2em; border-bottom: 1px inset black; margin: 0; }
    h1, p { padding-left: 10px; }
    code.url { background-color: #eeeeee; font-family:monospace; padding:0 2px;}
    </style>
    </head><body><h1>Not Found</h1><p>The requested resource <code class="url">/store/a3e/a3e65e1f-5fd4-47ed-9dbd-307f2586b52d/tmp_merge_all_1_1_1/.index</code> was not found on this server.</p></body></html>: while loading disk metadata. (RECEIVED_ERROR_FROM_REMOTE_IO_SERVER) (version 22.11.1.1)
    2022.10.28 14:08:40.640527 [ 5488 ] {} <Trace> BaseDaemon: Received signal 11
    2022.10.28 14:08:40.641529 [ 9027 ] {} <Fatal> BaseDaemon: ########################################
    2022.10.28 14:08:40.642759 [ 9027 ] {} <Fatal> BaseDaemon: (version 22.11.1.1, build id: 12145DA78CE5E9EBB10A034177FAE5967EF81A4A) (from thread 6043) (query_id: a3e65e1f-5fd4-47ed-9dbd-307f2586b52d::all_1_1_1) (query: optimize table data_from_web final) Received signal Segmentation fault (11)
    2022.10.28 14:08:40.643260 [ 9027 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Unknown si_code.
    2022.10.28 14:08:40.643769 [ 9027 ] {} <Fatal> BaseDaemon: Stack trace: 0x7ffff416c0f2 0x7ffff7cd1ca8 0x7ffff679ae5e 0x7fffd52e7906 0x7fffd50c65aa 0x7fffca7a0d42 0x7fffcaee79ec 0x7fffcaf242f8 0x7fffcaf242b5 0x7fffcaf2427d 0x7fffcaf24255 0x7fffcaf2421d 0x7ffff65c3686 0x7ffff65c2295 0x7fffcaeee2a9 0x7fffcaef2c43 0x7fffcaee3c0e 0x7fffcc4a7851 0x7fffcc4a768f 0x7fffcc4abb2d 0x7fffcfdce828 0x7fffd03e3eaa 0x7fffd03dfe3b 0x7fffc8ec42d4 0x7fffc8ed51d2 0x7ffff4bdd839 0x7ffff4bde0a8 0x7ffff48ab261 0x7ffff48a769a 0x7ffff48a6335 0x7ffff409f8fd 0x7ffff4121a60
    2022.10.28 14:08:40.644411 [ 9027 ] {} <Fatal> BaseDaemon: 4. ? @ 0x7ffff416c0f2 in ?
    2022.10.28 14:08:40.676390 [ 9027 ] {} <Fatal> BaseDaemon: 5. /src/ch/clickhouse/src/Common/StringUtils/StringUtils.cpp:9: detail::startsWith(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, char const*, unsigned long) @ 0x1ca8 in /src/ch/clickhouse/.cmake/src/Common/StringUtils/libstring_utilsd.so
    2022.10.28 14:08:40.730727 [ 9027 ] {} <Fatal> BaseDaemon: 6. /src/ch/clickhouse/src/Common/StringUtils/StringUtils.h:19: startsWith(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) @ 0x59ae5e in /src/ch/clickhouse/.cmake/src/libclickhouse_common_iod.so
    2022.10.28 14:08:40.923955 [ 9027 ] {} <Fatal> BaseDaemon: 7. /src/ch/clickhouse/src/Disks/ObjectStorages/Web/MetadataStorageFromStaticFilesWebServer.cpp:58: DB::MetadataStorageFromStaticFilesWebServer::exists(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const @ 0x6e7906 in /src/ch/clickhouse/.cmake/src/libdbmsd.so
    2022.10.28 14:08:41.291996 [ 9027 ] {} <Fatal> BaseDaemon: 8. /src/ch/clickhouse/src/Disks/ObjectStorages/DiskObjectStorage.cpp:181: DB::DiskObjectStorage::exists(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const @ 0x4c65aa in /src/ch/clickhouse/.cmake/src/libdbmsd.so
    2022.10.28 14:08:41.704697 [ 9027 ] {} <Fatal> BaseDaemon: 9. /src/ch/clickhouse/src/Storages/MergeTree/DataPartStorageOnDisk.cpp:74: DB::DataPartStorageOnDisk::exists() const @ 0xda0d42 in /src/ch/clickhouse/.cmake/src/libclickhouse_storages_mergetreed.so
    2022.10.28 14:08:43.032459 [ 9027 ] {} <Fatal> BaseDaemon: 10. /src/ch/clickhouse/src/Storages/MergeTree/MergeTask.cpp:147: DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x14e79ec in /src/ch/clickhouse/.cmake/src/libclickhouse_storages_mergetreed.so
    ...
    Segmentation fault (core dumped)

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-28 15:15:49 +02:00
Roman Vasin
5755728b33 Add Date32 and DateTime64 into exception messages 2022-10-28 13:11:09 +00:00
Roman Vasin
fc16752844 Update docs for FROM_UNIXTIME 2022-10-28 12:56:52 +00:00
Alexander Tokmakov
11f1f5ae4d
Merge pull request #42747 from jrdi/fix-implicit-cast-CaresPTRResolver 
Minor fix implicit cast CaresPTRResolver
 2022-10-28 15:50:11 +03:00
Roman Vasin
15f3f56812 Change 1 argument of function to first argument of function etc 2022-10-28 12:44:47 +00:00
robot-clickhouse
4f62bd6ba0 Update version_date.tsv and changelogs after v22.3.14.23-lts 2022-10-28 11:43:06 +00:00
robot-clickhouse
51e59405bc Update version_date.tsv and changelogs after v22.3.14.18-lts 2022-10-28 09:41:29 +00:00
Maksim Kita
835daee0f3
Merge pull request #42728 from ucasfl/fix 
Fix truncate table does not hold lock correctly
 2022-10-28 11:44:24 +03:00