thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
79,667 Commits 9,326 Branches 1,784 Tags 2.1 GiB
e6fd4bfb50
Commit Graph

657 Commits

Author SHA1 Message Date
zhanglistar
96b8df5899 1. chagne all fsync to fdatasync 2021-11-10 17:40:16 +08:00
kssenii
618c332f9c Rename read_threadpool to threadpool 2021-11-10 07:46:18 +00:00
alexey-milovidov
7006147268
Merge pull request #31159 from snar/freebsd-disk-space 
correct disk space calculations
 2021-11-10 02:48:29 +03:00
Kruglov Pavel
17085183bf
Merge pull request #31042 from Avogar/hdfs-bad-url 
Add additional hdfs url check
 2021-11-09 16:23:59 +03:00
kssenii
36f213e51c Fix build 2021-11-09 07:11:44 +00:00
Kseniia Sumarokova
c458f73138
Update AsynchronousReadIndirectBufferFromRemoteFS.cpp 2021-11-08 22:12:04 +03:00
kssenii
83d64f1c2d Better 2021-11-08 13:32:32 +03:00
kssenii
b4bebc5977 Add comments 2021-11-08 12:01:24 +03:00
Alexandre Snarskii
248709ea16 correct disk space calculations 2021-11-07 19:51:48 +03:00
kssenii
9189966106 Fix 2021-11-05 20:01:21 +03:00
kssenii
4bc2a29345 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-11-04 14:41:24 +00:00
kssenii
b95f0738fb Fix 2021-11-04 14:39:53 +00:00
kssenii
ec11179f91 Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers 2021-11-03 14:33:31 +03:00
kssenii
45ea820297 Reduce memory usage for some formats 2021-11-03 14:30:03 +03:00
avogar
3839d521a0 Add additional hdfs url check 2021-11-03 14:01:50 +03:00
Vladimir C
77edd081a5
Fix whitespace in src/Disks/S3/DiskS3.cpp 2021-11-01 13:00:21 +03:00
Vladimir C
2e58a3bf62
Merge branch 'master' into MDB-15168 2021-11-01 11:59:13 +03:00
kssenii
2940d9fd19 Squashed commit of the following: 
commit a1acc7ed3485bd158cf42f26b2d2a1bca84b7269
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Sun Oct 31 14:47:25 2021 +0000

    Fix

commit 12a27d445b
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Fri Oct 29 09:24:53 2021 +0000

    Adjutments after merge with master

commit f17e321a73
Merge: 55f1ba857d 11b70a285c
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Fri Oct 29 08:03:25 2021 +0000

    Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http

commit 55f1ba857d
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Fri Oct 29 07:38:42 2021 +0000

    Small fixes

commit aeba8104d1
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Thu Oct 28 10:28:05 2021 +0000

    Some more fixes

commit 82f3754b8a
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Wed Oct 27 20:20:15 2021 +0000

    Slightly better

commit 2647b88a66
Merge: 7024f51b55 2bb586bed3
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Wed Oct 27 19:38:19 2021 +0000

    Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http

commit 7024f51b55
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Wed Oct 27 18:30:25 2021 +0000

    Review fixes and add test

commit 7de5fca075
Merge: 1ecb6ad05d 855b10261c
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Sun Oct 24 17:04:24 2021 +0000

    Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http

commit 1ecb6ad05d
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Sat Oct 23 18:53:59 2021 +0000

    Better

commit 710ee96878
Merge: 28d2485716 158b4c26b7
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Mon Oct 18 08:31:25 2021 +0000

    Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http

commit 28d2485716
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Fri Oct 15 17:27:26 2021 +0000

    Some fixes

commit 7c20ca07b7
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Fri Oct 15 12:59:24 2021 +0000

    Fix style check

commit ccb02cac56
Merge: 9e4fe0f3c8 d1138a8a25
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Fri Oct 15 12:36:30 2021 +0000

    Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http

commit 9e4fe0f3c8
Merge: 9814cb1b45 daed77038b
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Tue Oct 12 09:58:15 2021 +0000

    Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http

commit 9814cb1b45
Merge: 1c8b1b1133 969999ff10
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Tue Oct 12 09:49:08 2021 +0000

    Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http

commit 1c8b1b1133
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Tue Oct 12 09:33:07 2021 +0000

    Better

commit e7dbfff5d2
Merge: 3329b668d6 198adc7ecd
Author: kssenii <sumarokovakseniia@mail.ru>
Date:   Tue Oct 12 06:19:15 2021 +0000

    Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http
 2021-10-31 17:53:08 +03:00
kssenii
12a27d445b Adjutments after merge with master 2021-10-29 09:29:35 +00:00
kssenii
f17e321a73 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-29 08:03:25 +00:00
kssenii
55f1ba857d Small fixes 2021-10-29 07:38:42 +00:00
kssenii
4f3433b4e1 Some clean up before merging 2021-10-29 01:04:52 +03:00
kssenii
7e2ea97e3c Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-28 18:46:21 +03:00
Anton Ivashkin
6cda420b56 Cleanup after fail in localBackup 2021-10-28 17:50:59 +03:00
kssenii
aeba8104d1 Some more fixes 2021-10-28 10:28:05 +00:00
kssenii
3d01fbf603 Review fixes 2021-10-28 11:32:37 +03:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
kssenii
9461e225bb Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-27 22:47:47 +03:00
kssenii
2647b88a66 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-27 19:38:19 +00:00
kssenii
7024f51b55 Review fixes and add test 2021-10-27 19:37:36 +00:00
Azat Khuzhin
9936ee5181 Update gtest_disk_encrypted for new readFile() interface 2021-10-27 09:37:26 +03:00
Azat Khuzhin
7ce01b328e Update gtest_disk_encrypted for new createReadBufferFromFileBase() interface 2021-10-27 09:37:26 +03:00
kssenii
52296e8b04 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-25 00:22:32 +03:00
kssenii
658b33717f A little nicer 2021-10-24 20:43:34 +03:00
kssenii
7de5fca075 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into retriable-http 2021-10-24 17:04:24 +00:00
Azat Khuzhin
34d8f78c33 Explictly pass size in DiskCacheWrapper::writeFile() 
v0: Add explicit default value for DiskLocal::readFile()
v2: Explictly pass size in DiskCacheWrapper::writeFile()
    (since v0 not google-default-arguments compatible)
 2021-10-24 18:58:22 +03:00
Azat Khuzhin
4599e9191b Fix explicit 0 size for readFile() 2021-10-24 18:58:22 +03:00
Azat Khuzhin
2b96433daf Replace estimated_size with optional<size> for IDisk::readFile() 
v2: rebase to fix conflicts in DiskEncrypted.cpp
 2021-10-24 18:58:11 +03:00
Vitaly Baranov
c0b01e7b0b
Merge pull request #30494 from vitlibar/fix-reading-from-empty-file-on-encrypted-disk 
Fix reading from empty file on encrypted disk
 2021-10-24 16:24:47 +03:00
kssenii
1ecb6ad05d Better 2021-10-23 19:00:14 +00:00
Vitaly Baranov
26cb62de32 Add more tests for the "encrypted" disk type. 2021-10-23 20:58:37 +03:00
kssenii
7f2c5dcaae Fix clang-tidy 2021-10-22 01:08:58 +03:00
kssenii
56f0560c46 Asserts and read till end option 2021-10-21 20:54:03 +03:00
Vitaly Baranov
172a6f2e0d Fix reading from empty file on encrypted disk. 2021-10-21 19:31:29 +03:00
kssenii
120b01e089 Better 2021-10-21 18:22:17 +03:00
kssenii
e4e157688d Review fixes 2021-10-21 01:57:43 +03:00
kssenii
31cd71c849 Clean up 2021-10-20 09:37:18 +03:00
kssenii
143e912e4e Fix split build 2021-10-19 22:45:46 +03:00
kssenii
49106f407f Fix some checks 2021-10-19 09:34:27 +03:00
kssenii
d5d4817350 Add disk proxies back, add setReadUntilPosition to ReadBuffer 2021-10-18 18:35:11 +03:00
kssenii
28d2485716 Some fixes 2021-10-15 17:27:26 +00:00
kssenii
7c20ca07b7 Fix style check 2021-10-15 12:59:24 +00:00
kssenii
c1c574e9ca Better 2021-10-15 13:14:57 +03:00
kssenii
3995506d37 Adjust range reader for remote fs reads 2021-10-15 11:36:26 +03:00
kssenii
1c8b1b1133 Better 2021-10-12 09:33:07 +00:00
kssenii
31d98b0992 Remove redundant 2021-10-11 08:17:22 +03:00
kssenii
448d63a0fc Range read with upper bound offset 2021-10-11 01:53:51 +03:00
kssenii
1f8b449bc5 Lazy seek avoiding 2021-10-11 00:51:43 +03:00
kssenii
e3d2942aa6 Correct merge 2021-10-10 20:24:36 +03:00
kssenii
fde6f0507d Merge remote-tracking branch 'origin/retriable-http' into disk-async-read 2021-10-10 20:23:44 +03:00
kssenii
3329b668d6 Fix style check 2021-10-10 15:51:37 +00:00
kssenii
40434519e8 Fix disk web 2021-10-08 16:02:11 +03:00
kssenii
f3f820c6e0 HTTP read buffer retries 2021-10-08 15:41:55 +03:00
kssenii
d6e1d0503d Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-10-07 16:21:00 +03:00
Filatenkov Artur
dc657a2c5e
Add error for multiple keys without current_key_id (#29546) 
* add error for multiple keys without current_key_id

* add exception if one key with non zero id is used without current_key_id

* refactoring
 2021-10-05 14:37:54 +03:00
Azat Khuzhin
a7793101bc Fix data-race in WriteIndirectBuffer (used in StripeLog) 
CI founds [1]:

WARNING: ThreadSanitizer: data race (pid=497)
  Write of size 8 at 0x7b30000260b8 by thread T381 (mutexes: write M546759850860896464):
    ...
    3 DB::DiskMemory::writeFile() obj-x86_64-linux-gnu/../src/Disks/DiskMemory.cpp:339:15 (clickhouse+0x14a1849f)
    4 DB::FileChecker::save() const obj-x86_64-linux-gnu/../src/Common/FileChecker.cpp:111:50 (clickhouse+0x15bbe19f)
    5 DB::TinyLogSink::TinyLogSink() obj-x86_64-linux-gnu/../src/Storages/StorageTinyLog.cpp:214:34 (clickhouse+0x15bca215)

  Previous read of size 8 at 0x7b30000260b8 by thread T460:
    ...
    2 DB::WriteIndirectBuffer::finalize() obj-x86_64-linux-gnu/../src/Disks/DiskMemory.cpp:106:33 (clickhouse+0x14a1e044)
    9 DB::StripeLogSink::onFinish() obj-x86_64-linux-gnu/../src/Storages/StorageStripeLog.cpp:227:30 (clickhouse+0x15bd241a)

  [1]: https://clickhouse-test-reports.s3.yandex.net/29680/d0fc26f91a0141b56a0550741219c3dc43630e03/stress_test_(thread)/stderr.log
 2021-10-04 01:58:35 +03:00
Mike Kot
38232c286d fixing paths 2021-10-02 21:45:05 +02:00
Mike Kot
65e6e211b4 Merge remote-tracking branch 'upstream/master' into improvement/fn-traits 2021-10-02 21:37:50 +02:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
kssenii
7ff092712b Better 2021-10-01 18:13:16 +03:00
Mike Kot
38b02b121a Merge remote-tracking branch 'upstream/master' into improvement/fn-traits 2021-10-01 16:55:28 +02:00
kssenii
55972db866 Small fix and more comments 2021-10-01 11:38:58 +03:00
kssenii
e4bc81ac78 Better 2021-09-30 21:45:57 +03:00
kssenii
cc9ce13bac Finally all fixed 2021-09-30 15:35:59 +03:00
kssenii
85aae03a26 Fix 2021-09-29 22:54:49 +03:00
Mike Kot
7670ea50c1 Replacing std::function to Fn in some places 
to avoid dynamic allocating while keeping desired function signature
clear.
Simplifying SimpleCache (CachedFn) using C++20
 2021-09-29 13:09:39 +02:00
kssenii
20192352ad Better 2021-09-29 11:03:36 +03:00
kssenii
ebca30694d Fix 2021-09-27 18:25:30 +03:00
kssenii
efddc03246 Async read for disk web 2021-09-26 22:49:28 +03:00
alexey-milovidov
85fc160eb4
Merge pull request #29346 from ClickHouse/remove-ya-make 
Remove ya.make
 2021-09-25 19:43:40 +03:00
Alexey Milovidov
cd7f9d981c Remove ya.make 2021-09-25 04:22:54 +03:00
alexey-milovidov
62d606bb76
Update DiskWebServer.h 2021-09-25 02:58:12 +03:00
Alexey Milovidov
4a44bcaced Fix trivial mistake in DiskWebServer 2021-09-25 01:33:40 +03:00
kssenii
6219d541a5 Finally fixed 2021-09-24 15:29:04 +03:00
kssenii
c1ea44b9c2 Fixes 2021-09-24 13:38:08 +03:00
kssenii
1b9565dfa8 Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read 2021-09-22 22:08:44 +03:00
kssenii
e301457e91 Fix prefetch mistake 2021-09-21 15:40:48 +03:00
kssenii
975033e6e3 Fix 2021-09-21 09:44:54 +00:00
kssenii
40ee75fe97 Common thread pool for disks 2021-09-21 10:00:56 +03:00
kssenii
cacc81ca61 Fix 2021-09-21 02:31:02 +03:00
kssenii
0104cb1c8e Better 2021-09-21 01:51:17 +03:00
kssenii
f312760dca Better 2021-09-20 19:57:15 +03:00
kssenii
11ae94b8d3 Async read for remote fs 2021-09-20 16:16:46 +03:00
Kseniia Sumarokova
2a8f355326
Merge pull request #29179 from kssenii/better-retries 
Better retries for disk web
 2021-09-20 08:25:49 +03:00
kssenii
12b90da5de Better retries 2021-09-19 20:50:50 +00:00
Nikita Mikhaylov
c52b8ec083
Introduced MergeTask and MutateTask (#25165) 
Introduced MergeTask and MutateTask
 2021-09-17 00:19:58 +03:00
Kseniia Sumarokova
ef6ba90be1
Update DiskWebServer.h 2021-09-13 20:00:18 +03:00
kssenii
9ec6a9b4c9 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-web-disk-2 2021-09-13 08:32:34 +03:00
kssenii
147e413346 Add retries with backoff, better exceptions 2021-09-13 00:04:05 +03:00
Mike Kot
c4385a2c40 Simplifying exception messages 2021-09-11 00:40:18 +02:00
kssenii
939032b0eb Slightly better 2021-09-10 13:41:41 +03:00
kssenii
c23e17d369 Fix 2021-09-09 15:25:44 +03:00
kssenii
72505db3bd Fix checks 2021-09-09 10:10:34 +03:00
kssenii
1101ee7808 Update example 2021-09-08 22:43:46 +03:00
kssenii
7345ad8e09 Add no throw 2021-09-08 22:40:26 +03:00
kssenii
7141b5d041 Get rid of regexp 2021-09-08 20:22:24 +03:00
Mike Kot
8e9aacadd1 Initial: replacing hardcoded toString for enums with magic_enum 2021-09-06 16:24:03 +02:00
kssenii
97e0a0310a Slightly better 2021-09-04 12:02:07 +03:00
kssenii
32686e11cb Fix style checl 2021-09-04 11:30:30 +03:00
kssenii
6ee94c923f Fix 2021-09-04 00:51:11 +03:00
Kseniia Sumarokova
63a0eb3922
Update DiskWebServer.cpp 2021-08-31 21:51:00 +03:00
kssenii
248fe1fd09 Fix disk 2021-08-31 16:54:43 +03:00
Nikita Mikhaylov
00394014ff
Merge pull request #28382 from DimasKovas/fix_arcadia_build_31_08_2021 
Try to fix arcadia build (generate ya.make)
 2021-08-31 13:49:32 +03:00
Dmitrii Kovalkov
b1eb48804b Run generate-ya-make.sh 2021-08-31 09:05:33 +03:00
Alexey Milovidov
ca6f6306f3 Merge branch 'master' into async-reads 2021-08-31 02:52:58 +03:00
Kseniia Sumarokova
00bea71610
Merge pull request #25251 from kssenii/disk-over-web-server 
ClickHouse over web server with static files
 2021-08-30 16:18:59 +03:00
Dmitrii Kovalkov
9871ad70ff Exclude fuzzers 2021-08-30 11:12:25 +03:00
Dmitrii Kovalkov
8acf0fa16a run generate-ya-make 2021-08-30 10:57:40 +03:00
kssenii
3e7f022d66 Extend usage comment explanation a little 2021-08-28 20:51:20 +00:00
kssenii
f27f519aa2 Fix build and add example 2021-08-28 20:35:51 +00:00
Alexey Milovidov
da4865a5d8 Remove excessive logging from DiskS3 2021-08-25 03:22:26 +03:00
Alexey Milovidov
183891eb0a Add IDisk::isRemote method 2021-08-25 02:27:20 +03:00
Alexey Milovidov
fd27a2faef Remove imperfect code 2021-08-25 02:05:55 +03:00
Alexey Milovidov
7c1d0a3baf Progress on development 2021-08-25 01:24:47 +03:00
Alexey Milovidov
8f57216180 Progress on development 2021-08-25 00:45:58 +03:00
Alexey Milovidov
4d9ad3725d Merge branch 'master' into async-reads 2021-08-25 00:45:40 +03:00
Kseniia Sumarokova
d78e551de9
Update src/Disks/DiskWebServer.cpp 
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
 2021-08-24 12:24:54 +03:00
kssenii
b536653851 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-24 07:29:50 +00:00
kssenii
5307d31924 Throw for alter and silence for drop 2021-08-23 23:48:53 +00:00
Vitaly Baranov
63e8bc1f20 Added new commands BACKUP and RESTORE. 2021-08-23 23:41:45 +03:00
Anton Ivashkin
0e0e07ef46 Allow Multi Part Upload for small objects 2021-08-20 19:13:48 +03:00
Anton Ivashkin
51b543fb5c Call AbortMultipartUpload when Multipart Upload Copy failed 2021-08-20 18:57:03 +03:00
vdimir
e77ff2740a
Add common/unit.h 2021-08-20 13:57:05 +03:00
Anton Ivashkin
b3737afe04 Use Multipart copy upload for large S3 objects 2021-08-19 13:09:58 +03:00
kssenii
709bc1a013 Continued 2021-08-16 14:50:26 +00:00
kssenii
9a02c6ad05 Catch any error if it happens on disk load when server is starting up 2021-08-16 13:18:33 +00:00
Alexey Milovidov
d184b79bba Progress on async reads. 2021-08-16 03:00:32 +03:00
kssenii
073d7fdd5e Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-06 19:42:29 +00:00
Vladimir C
9de16c5017
Merge pull request #27216 from ianton-ru/MDB-13990 2021-08-06 18:12:00 +03:00
Anton Ivashkin
0e040ca40c Add 10 seconds cache for S3 proxy resolver 2021-08-05 18:52:04 +03:00
Maksim Kita
4dc4854fad
Merge pull request #27168 from excitoon-favorites/safers3readbuffer 
Safer `ReadBufferFromS3` for merges and backports
 2021-08-04 15:01:21 +03:00
Vladimir Chebotarev
cc32a61488 Fix. 2021-08-04 10:40:02 +03:00
kssenii
087ddeb0dc Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into taiyang-li-disk-reload 2021-08-02 15:16:09 +00:00
kssenii
b25dfdd6af Use method which was added for s3 instead 2021-08-02 08:15:21 +00:00
Kseniia Sumarokova
322870acd2
Update IDisk.h 2021-08-02 09:36:44 +03:00
Kseniia Sumarokova
9007ddfe22
Update DiskLocal.h 2021-08-02 09:34:39 +03:00
Kseniia Sumarokova
583c2d67cb
Update DiskSelector.cpp 2021-08-02 09:32:23 +03:00
Vitaly Baranov
a05184d83a Fix synchronization while updating from the config of an encrypted disk. 2021-08-01 22:55:36 +03:00
kssenii
f1e96dbc55 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into taiyang-li-disk-reload 2021-07-29 07:53:16 +00:00
Vitaly Baranov
51f882d8cc Support multiple keys for encrypted disk. Display error message if the key is probably wrong. 2021-07-24 23:45:12 +03:00