thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-19 12:52:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
108,360 Commits 9,333 Branches 1,784 Tags 2.2 GiB
60d517c49c
Commit Graph

2411 Commits

Author SHA1 Message Date
avogar
e39e61fc71 Fix heap-use-after-free in PeekableReadBuffer 2022-11-01 12:58:20 +00:00
avogar
8e13d1f1ec Improve and refactor Kafka/StorageMQ/NATS and data formats 2022-10-28 16:41:10 +00:00
Márcio Martins
aaf71d157d Use return type template parameter to select throwing behavior 2022-10-27 10:26:55 +02:00
Márcio Martins
1ca230ff6d Allow coercing in JSONExtract* functions 2022-10-27 10:26:55 +02:00
Márcio Martins
c3bcb1899d Add convenience ctor to ReadBufferFromMemory 2022-10-27 10:26:55 +02:00
ltrk2
2c902bbc65
Implement support for different UUID binary formats (#42108) 
* Implement support for different UUID binary formats

* Declare error codes that the implementation uses

* Make single-argument constructor explicit

* Modernize parts of the solution

* Make improvements based on review comments

* Declare an error code being used
 2022-10-25 21:57:50 +02:00
Nikolai Kochetov
5c4444237e
Merge pull request #42617 from ClickHouse/revert-revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3 
Revert revert 41268 disable s3 parallel write for part moves to disk s3
 2022-10-25 14:29:27 +02:00
Nikita Taranov
49f6692a2e
Adapt internal data structures to 512-bit era (#42564) 
* impl

* update tests

* fix tests
 2022-10-25 13:56:28 +02:00
Azat Khuzhin
905a95e166 Review fixes 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 22:40:13 +02:00
Azat Khuzhin
784f7d988c Fix -Wshorten-64-to-32 in examples 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:43 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Nikolai Kochetov
57ffcb7ed0 Revert "Merge pull request #41681 from ClickHouse/revert-41268-disable-s3-parallel-write-for-part-moves-to-disk-s3" 
This reverts commit 7956c2becf, reversing
changes made to 57be648984.
 2022-10-20 19:51:27 +02:00
Antonio Andelic
9d7e44409a
Merge pull request #41342 from ClickHouse/keeper-upload-snapshot-to-s3 
Upload Keeper snapshots to S3
 2022-10-20 09:14:59 +02:00
Nikolay Degterinsky
486780ba80
Merge pull request #42044 from zvonand/zvonand-csv-dates 
Fix Date from CSV parsing
 2022-10-19 13:10:12 +02:00
zvonand
82c7228e53 fix 2022-10-18 17:03:31 +03:00
Antonio Andelic
7c7395575f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-10-17 11:02:36 +00:00
Sergei Trifonov
f94029731c
Update src/IO/Resource/PriorityPolicy.h 
Co-authored-by: Dmitry Novik <n0vik@clickhouse.com>
 2022-10-14 18:59:16 +02:00
kssenii
0a702d9520 Fix 2022-10-10 17:23:09 +02:00
Alexander Tokmakov
4175f8cde6 abort instead of __builtin_unreachable in debug builds 2022-10-07 21:49:08 +02:00
Nikolay Degterinsky
16a8145deb
Merge pull request #42046 from evillique/fix-bz2-decoding 
Fix bzip2 decoding issue
 2022-10-06 16:02:09 +02:00
Robert Schulze
78be400ac0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup 2022-10-06 12:22:43 +00:00
Nikolay Degterinsky
54ec6a5047
Merge pull request #42071 from evillique/new-http-compression-methods 
Add new HTTP compression methods
 2022-10-06 14:21:39 +02:00
Nikolay Degterinsky
aa332d8f5d Fix build 2022-10-05 08:11:04 +00:00
Nikolay Degterinsky
2a4f2bff9b Better order of compression methods 2022-10-05 00:48:01 +00:00
Nikolay Degterinsky
f7e0e497da Add methods to the tests 2022-10-04 19:04:03 +00:00
flynn
e30297e599 fix and update test 2022-10-04 13:36:28 +00:00
flynn
86cb9f7034 parseDataTimeBestEffort support comma between date and time 2022-10-04 09:52:20 +00:00
Nikolay Degterinsky
99dc396c77 Add new HTTP compression methods 2022-10-04 03:13:46 +00:00
Nikolay Degterinsky
b73b5375a7 Fix bz2 decoding 2022-10-04 00:10:04 +00:00
serxa
bc367e61b8 fix tidy build 2022-10-03 12:51:06 +00:00
Sergei Trifonov
8edac7eba1
Merge branch 'master' into io-scheduler 2022-09-28 22:17:17 +02:00
serxa
268bf40b6e fix special build checks 2022-09-28 20:16:33 +00:00
Robert Schulze
6d70b4a1f6
Generate config_version.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
serxa
703df74756 fix typos 2022-09-27 15:20:43 +00:00
serxa
b057d07977 IO scheduling subsystem 2022-09-27 13:26:41 +00:00
Kseniia Sumarokova
caf6a99f41
Merge pull request #41011 from kssenii/fix-thread-status 
Fix incorrect attach query in threadpool readers, get rid of static threadpools for reads/writes, make threadpool size for reads/writes configurable
 2022-09-27 11:35:07 +02:00
Antonio Andelic
655bde8e3f Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-27 07:28:26 +00:00
Nikita Mikhaylov
e897816572
Evict stale DNS entries from cache in case of network errors (#41707) 2022-09-25 14:06:13 +02:00
kssenii
0a801dad2a Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-23 19:39:07 +02:00
kssenii
30726721ad Fix threadpool reader (for local fs) 2022-09-23 19:35:16 +02:00
Nikolay Degterinsky
9efa4bf766
Merge pull request #41587 from evillique/better-s3-logs 
Better S3 logs
 2022-09-23 15:32:27 +02:00
kssenii
ab702e43fc Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-23 15:21:33 +02:00
Kseniia Sumarokova
ea43cb5648
Merge pull request #41261 from kssenii/s3-header-auth 
Support s3 authorisation headers from ast arguments
 2022-09-23 12:48:08 +02:00
Nikolay Degterinsky
6edf8ec3f2 Better S3 logs 2022-09-22 22:34:13 +00:00
Alexander Tokmakov
021b882455 Revert "Disable parallel s3 multipart upload for part moves. (#41268)" 
This reverts commit 0414d95878.
 2022-09-22 13:59:25 +03:00
alesapin
57be648984
Merge pull request #41649 from ClickHouse/supress_one_more_key_doesnt_exists 
Release AWS SDK log level + replace one exception
 2022-09-22 12:38:03 +02:00
Alexey Milovidov
45afacdae4
Merge pull request #41186 from ClickHouse/fix-three-fourth-of-trash 
Fix more than half of the trash
 2022-09-22 07:28:26 +03:00
alesapin
499f678112 Release AWS SDK log level + replace one exception 2022-09-21 19:56:42 +02:00
Nikolai Kochetov
0414d95878
Disable parallel s3 multipart upload for part moves. (#41268) 
* Disable parallel s3 multipart upload for part moves.

* Add setting s3_allow_parallel_part_upload
 2022-09-21 19:10:32 +02:00
Kruglov Pavel
22e11aef2d
Merge pull request #40910 from Avogar/new-json-formats 
Add new JSON formats, add improvements and refactoring
 2022-09-21 14:19:08 +02:00
kssenii
f917b268b3 Merge remote-tracking branch 'upstream/master' into s3-header-auth 2022-09-21 12:56:25 +02:00
Antonio Andelic
fc6c1faa57 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-21 06:57:58 +00:00
Alexey Milovidov
45bd3cfc30 Merge branch 'master' into fix-three-fourth-of-trash 2022-09-20 21:27:41 +02:00
avogar
868ce8bc16 Fix comments, make better naming, add docs, add setting output_format_json_quote_64bit_floats 2022-09-20 13:49:17 +00:00
Alexey Milovidov
03bbf0676f
Merge branch 'master' into revert_prefixes_s3 2022-09-20 06:14:07 +03:00
serxa
2ef696ffe1 fix issues 2022-09-19 18:40:32 +00:00
serxa
7a1d4a404a add DiskS3* profile events per S3 API calls 2022-09-19 18:10:47 +00:00
serxa
f8aa738511 more conventional profile events names 2022-09-19 17:23:22 +00:00
Sergei Trifonov
ff2e1ff8d8
Merge branch 'master' into support_s3_api_event_metrics 2022-09-19 14:35:45 +02:00
Alexey Milovidov
730655d4fd Fix 8/9 of trash 2022-09-19 08:53:20 +02:00
Alexey Milovidov
6b752bff3a
Merge branch 'master' into revert_prefixes_s3 2022-09-18 07:10:57 +03:00
Alexey Milovidov
f4a48cd4d6 Remove cruft 2022-09-17 22:16:31 +02:00
alesapin
33216dedf9
Merge pull request #41323 from ClickHouse/remove_bad_exception 
Fix parts removal in case of connection loss
 2022-09-17 19:14:53 +02:00
alesapin
84828874e0 Change log level 2022-09-17 19:06:54 +02:00
alesapin
d9c3549cc8 Revert s3 prefixes + add retries on write requests unexpected errors 2022-09-16 14:43:43 +02:00
Antonio Andelic
296fe01fb8 Merge branch 'master' into keeper-upload-snapshot-to-s3 2022-09-16 08:00:15 +00:00
Alexey Milovidov
da01982652
Merge pull request #41046 from azat/build/llvm-15 
Switch to llvm/clang 15
 2022-09-16 07:31:06 +03:00
Antonio Andelic
ad7864cb5f enable S3 if client is built 2022-09-15 14:06:22 +00:00
Antonio Andelic
32a297339b Fix build 2022-09-15 13:36:16 +00:00
Antonio Andelic
9fdd2f2e61 Basic implementation for S3 snapshot upload 2022-09-15 09:11:55 +00:00
alesapin
ad279940d6 Trying print less scary messages in case No such key for outdated parts 2022-09-14 17:25:58 +02:00
mateng0915
f7f976e94e remove extra space 2022-09-14 19:19:47 +08:00
mateng0915
f38e262706 fixed the unnecessory changes 2022-09-14 19:19:47 +08:00
mateng0915
5badb1b186 resolve the review comments 2022-09-14 19:19:40 +08:00
teng.ma
a85c917488 fix master conflict 2022-09-14 19:18:31 +08:00
teng.ma
ebea9f9359 Feature: Support Event Tracing when calling Aws S3 API 
Description:
============
In  Computing && Storage Architecture,  using S3 as remote / Shared storage, the Method to access S3 is using AWS S3 API
There is a gap between ClickHouse DB with Ozone Operation
In ClickHouse, operation is SQL and background task
In S3 , the operation should be AWS S3 API
And one sql maybe can mapped to multiple  API

Solution:
=========
Added Calling API as event into system.events table
 2022-09-14 19:18:31 +08:00
kssenii
52ef3758c4 Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-13 16:34:31 +02:00
kssenii
420ac4eb43 s3 header auth in ast 2022-09-13 15:13:28 +02:00
Kseniia Sumarokova
b2c9c04c7b
Merge pull request #40821 from kssenii/improve-marks-cache-loading 
Allow to load marks in threadpool in advance
 2022-09-13 12:31:58 +02:00
Kruglov Pavel
7d7606a1e3
Merge pull request #41162 from Avogar/fix-http-redirect 
Support relative path in Location header after http redirect
 2022-09-13 12:10:44 +02:00
kssenii
b7d751b782 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-11 13:23:30 +02:00
Azat Khuzhin
4e3135383b Fix clang-tidy warnings (from clang-15) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
avogar
d8cbd51333 Support relative path in Location header after http redirect 2022-09-09 19:26:14 +00:00
alesapin
5e00124f8a More logging for S3 2022-09-08 16:18:21 +02:00
alesapin
6da37b92e1 Improve logging one time 2022-09-08 13:11:53 +02:00
kssenii
3dc19b4fe2 Fix 2022-09-08 12:23:59 +02:00
kssenii
e51313b6b3 Get rid of static threadpools 2022-09-07 17:48:11 +02:00
alesapin
43493389b9 Merge branch 'master' into one_more_logging 2022-09-06 19:40:24 +02:00
alesapin
09e97a6381 Fix style 2022-09-06 18:38:34 +02:00
alesapin
ceed9f418b Return better errors handling 2022-09-06 18:22:44 +02:00
alesapin
422b1658eb Review fix 2022-09-06 14:42:48 +02:00
alesapin
6ea7f1e011 Better exception handling for ReadBufferFromS3 2022-09-06 13:59:55 +02:00
Sergei Trifonov
f77809ddbc
Merge pull request #40900 from ClickHouse/s3-detailed-metrics 
S3 detailed metrics
 2022-09-06 13:20:57 +02:00
alesapin
cbcf267f55 One more line of logging in write buffer from S3 2022-09-05 22:54:38 +02:00
Kruglov Pavel
cadae5a2f6
Merge pull request #40896 from kssenii/fix-memory-usage-for-remote 
Fix extra memory allocation for remote read buffers
 2022-09-05 20:38:57 +02:00
Sergei Trifonov
1c247308f1
Merge branch 'master' into s3-detailed-metrics 2022-09-05 15:55:41 +02:00
kssenii
e2df4b1da1 Merge remote-tracking branch 'origin/fix-races-in-write-buffer-from-s3' into fix-races-in-write-buffer-from-s3 2022-09-04 19:50:19 +02:00
kssenii
52375d2eb0 Merge master 2022-09-04 19:48:04 +02:00
Kseniia Sumarokova
5f757be43d
Merge pull request #40943 from kssenii/fix-deadlock-in-s3-write-buffer 
Fix deadlock in WriteBufferFromS3
 2022-09-04 13:17:41 +02:00
Kseniia Sumarokova
1e81e0ef67
Update WriteBufferFromS3.cpp 2022-09-02 18:54:34 +02:00
kssenii
239847ad16 Fix races 2022-09-02 18:27:07 +02:00
Sergei Trifonov
067e712fdf
Merge branch 'master' into s3-detailed-metrics 2022-09-02 16:59:52 +02:00
Sergei Trifonov
3daf1f5c77 fix build 2022-09-02 16:57:51 +02:00
kssenii
c9b512e33e Fix 2022-09-02 16:15:54 +02:00
kssenii
0db38fcbc9 Fix 2022-09-02 15:32:46 +02:00
alesapin
45fe9c800a
Merge pull request #40898 from georgthegreat/patch-2 
Remove remaining usage of // Y_IGNORE magic comment
 2022-09-02 12:24:57 +02:00
Robert Schulze
c7c00f9002
Merge pull request #40739 from ClickHouse/clang-tidy-for-headers 
Enable clang-tidy for headers
 2022-09-02 07:54:50 +02:00
Sergei Trifonov
29e922f05c support all profile events for disk s3 2022-09-02 02:22:52 +02:00
Sergei Trifonov
66b5cf566c fix bugs 2022-09-01 19:02:04 +02:00
Sergei Trifonov
ac9dc53c73 Merge branch 'master' into s3-detailed-metrics 2022-09-01 18:31:02 +02:00
Sergei Trifonov
bcb6475c4a add separate s3 profile events for disk s3 2022-09-01 18:30:55 +02:00
Yuriy Chernyshov
304dc30f0f
Remove remaining usage of // Y_IGNORE magic comment 2022-09-01 18:15:56 +03:00
kssenii
3134bdeb13 Fix 2022-09-01 16:35:28 +02:00
Nikita Mikhaylov
c7b0b87574
Better exception handling in DiskObjectStorageTransaction (#40813) 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-09-01 13:29:59 +02:00
Vitaly Baranov
6dcca686cb
Merge pull request #40827 from vitlibar/fix-incremental-backups-for-log-family 
Fix incremental backups for Log family.
 2022-09-01 13:13:32 +02:00
kssenii
82879543d3 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-01 13:13:01 +02:00
Kseniia Sumarokova
849262bd99
Merge pull request #40858 from kssenii/move-cache-files 
Move cache files from Common
 2022-09-01 12:28:00 +02:00
Robert Schulze
de64c6b103
Merge branch 'master' into clang-tidy-for-headers 2022-09-01 10:24:56 +02:00
Kruglov Pavel
b7ea529307
Merge pull request #40746 from Avogar/fix-csv-schema-inference 
Fix possible error Attempt to read after eof in CSV schema inference
 2022-08-31 18:29:01 +02:00
kssenii
0f747e7987 Move cache files 2022-08-31 17:50:55 +02:00
Vitaly Baranov
007ae0e6cc Fix incremental backups for Log family. 2022-08-31 12:57:28 +02:00
Robert Schulze
cedf75ed5e
Enable clang-tidy for headers 
clang-tidy now also checks code in header files. Because the analyzer
finds tons of issues, activate the check only for directory "base/" (see
file ".clang-tidy"). All other directories, in particular "src/" are
left to future work.

While many findings were fixed, some were not (and suppressed instead).
Reasons for this include: a) the file is 1:1 copypaste of a 3rd-party
lib (e.g. pcg_extras.h) and fixing stuff would make upgrades/fixes more
difficult b) a fix would have broken lots of using code
 2022-08-31 10:48:15 +00:00
kssenii
a5f69517ae Allow to load marks in threadpool 2022-08-30 20:20:05 +02:00
avogar
85fa372ad7 Fix possible error Attempt to read after eof in CSV schema inference 2022-08-29 11:18:53 +00:00
Azat Khuzhin
9fff08eac7 WriteBufferFromS3: remove unused ThreadGroupStatus 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-29 11:30:17 +02:00
Alexey Milovidov
6e564b18bf
Merge pull request #40600 from FrankChen021/check_url_arg 
Validate the CompressionMethod parameter of URL table engine
 2022-08-27 19:29:55 +03:00
alesapin
7d1e3ecfa3
Merge pull request #40506 from ClickHouse/fix_bad_exception 
Fix lost errors in AWS CPP SDK
 2022-08-25 13:07:35 +02:00
Kruglov Pavel
66e677b307
Merge pull request #40546 from Avogar/fix-inferring-datetimes 
Fix possible error 'Decimal math overflow' while parsing DateTime64
 2022-08-25 11:18:30 +02:00
Frank Chen
4c1fe0d623 validate parameter 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 14:49:17 +08:00
Alexey Milovidov
aaafb797c0
Merge pull request #40541 from ClickHouse/paranoid_check_for_s3_uploads 
Add paranoid option to check blobs to S3 after upload
 2022-08-24 23:51:22 +02:00
Kseniia Sumarokova
4e7578cf3f
Merge pull request #40565 from FrankChen021/url_engine 
Fix segment fault when writing data to URL table engine
 2022-08-24 20:36:45 +02:00
Alexander Gololobov
6c256cbe4a
Merge pull request #40474 from ClickHouse/allow-to-parse-date-as-datetime 
Allow parsing Date as DateTime and DateTime64
 2022-08-24 17:23:44 +02:00
alesapin
f2be18b3a6 Remove trailing 2022-08-24 16:57:59 +02:00
alesapin
f5f0934c13
Update PocoHTTPClient.h 2022-08-24 16:15:43 +02:00
alesapin
e3672d1d40 Fix memsan 2022-08-24 15:25:10 +02:00
alesapin
f0b733313f Merge branch 'master' into fix_bad_exception 2022-08-24 14:44:40 +02:00
alesapin
cac3005f4d Merge branch 'master' into paranoid_check_for_s3_uploads 2022-08-24 14:43:11 +02:00
Kruglov Pavel
109f384e5d
Fix typo 2022-08-24 13:40:19 +02:00
Frank Chen
b432b0d100
Merge branch 'master' into url_engine 2022-08-24 17:59:21 +08:00
Frank Chen
335293549b Fix segment fault when sending data to remote HTTP server via URL table engine 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-24 17:38:16 +08:00
avogar
de489ba1f7 Fix possible error 'Decimal math overflow' while parsing DateTime64 2022-08-23 14:42:23 +00:00
alesapin
07286dd042 Add paranoid option to check blobs to S3 after upload 2022-08-23 15:12:09 +02:00
Kseniia Sumarokova
d814a56277
Merge pull request #40420 from kssenii/fix-cache-bug 
Fix incorrect fallback to skip cache in case of very high concurrency level
 2022-08-23 14:18:38 +02:00
alesapin
a646db4665 Better 2022-08-23 13:56:41 +02:00
alesapin
eef80b0d06 Fix message 2022-08-23 13:23:53 +02:00
alesapin
d97f3b012a Better 2022-08-23 13:20:05 +02:00
alesapin
802d56df49 Merge branch 'master' into fix_bad_exception 2022-08-23 12:22:05 +02:00
Kruglov Pavel
54ed9029b8
Merge pull request #40482 from ClickHouse/Avogar-patch-1 
Fix bug in HadoopSnappyReadBuffer
 2022-08-23 11:47:59 +02:00
alesapin
61d014f6ec
Merge branch 'master' into fix_bad_exception 2022-08-23 00:35:29 +02:00
alesapin
780f39b108 Fix lost errors in AWS CPP SDK 2022-08-23 00:34:39 +02:00
Kruglov Pavel
3666b25e6b
Fix 2022-08-22 18:48:32 +02:00
Kruglov Pavel
2636c032c4
Use loop instead of recursion 2022-08-22 18:48:02 +02:00
Kseniia Sumarokova
f85e5f8808
Merge branch 'master' into fix-cache-bug 2022-08-22 16:54:07 +02:00
Kruglov Pavel
ae81341ab5
Fix hadoop unit test 2022-08-22 12:42:18 +02:00
Kruglov Pavel
019686f6e1
Fix bug in HadoopSnappyReadBuffer 2022-08-22 12:40:58 +02:00
Alexey Milovidov
75c0ed528d Fix test 2022-08-22 01:21:32 +02:00
Alexey Milovidov
2faa23112c Fix test 2022-08-22 00:39:06 +02:00
kssenii
26384b2543 Fix 2022-08-22 00:02:05 +02:00
Alexey Milovidov
bddf1cc312 Allow to parse Date as DateTime and DateTime64 2022-08-21 23:12:41 +02:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
Sema Checherinda
f8480c26e7 do not trigger allocator to return null under sanitizers 2022-08-19 11:22:34 +02:00
Sema Checherinda
128e1fec3d Memory don't do alignment by itself, Allocator does 2022-08-18 16:01:30 +02:00
Sema Checherinda
b101ebdf32 check integer overflow at Memory class 2022-08-18 14:30:52 +02:00
Sema Checherinda
89bada927c fix heap buffer overflow by limiting http chunk size 2022-08-18 14:30:52 +02:00
Nikolai Kochetov
5a85531ef7
Merge pull request #38286 from Avogar/schema-inference-cache 
Add schema inference cache for s3/hdfs/file/url
 2022-08-18 13:07:50 +02:00
kssenii
d63d34f7bb Merge master 2022-08-17 15:49:32 +02:00
avogar
8dd54c043d Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache 2022-08-17 11:47:40 +00:00
kssenii
eb26b219b9 Merge master 2022-08-16 00:56:27 +02:00
Maksim Kita
58c9270a19
Merge branch 'master' into write-buffer-from-s3-potential-deadlock-fix 2022-08-15 19:20:55 +02:00
Maksim Kita
ee5e61c9a5 Fixed tests 2022-08-15 11:09:39 +02:00
Alexey Milovidov
914f4b8155
Merge pull request #40179 from HarryLeeIBM/hlee-s390x-file-enc 
Fix Endian issue in FileEncryption for s390x
 2022-08-13 22:48:48 +03:00
HarryLeeIBM
296835e2d3 Fix Endian issue in FileEncrption for s390x 2022-08-12 18:55:40 -07:00
Alexey Milovidov
6b8f4f1d59 Merge branch 'master' into server-side-time 2022-08-12 21:36:40 +02:00
Alexander Tokmakov
a9c6a883dc
Merge pull request #40158 from ClickHouse/make_tests_less_flaky 
Set sync_request_timeout to 10 to avoid reconnections in tests
 2022-08-12 22:08:53 +03:00
Kseniia Sumarokova
a6cfc7bc3b
Merge pull request #34651 from alexX512/master 
New caching strategies
 2022-08-12 17:23:37 +02:00
Alexey Milovidov
8c3d1f2b5b
Merge branch 'master' into server-side-time 2022-08-12 17:40:02 +03:00
Alexander Tokmakov
fe7e8dbc3c fix my favorite ctor 2022-08-12 16:28:21 +02:00
Alexander Tokmakov
0395dca7be set sync_request_timeout to 10 to avoid reconnections in tests 2022-08-12 15:30:46 +02:00
kssenii
768a207176 Merge master 2022-08-11 22:25:53 +02:00
Maksim Kita
6a7dbc556c WriteBufferFromS3 potential deadlock fix 2022-08-11 18:22:01 +02:00
Kseniia Sumarokova
be69169f97
Merge pull request #38048 from KinderRiven/decoupling_cache_function_and_algorithm 
Decoupling local cache function and cache algorithm
 2022-08-11 16:49:56 +02:00
Alexey Milovidov
0b24dbaec8 Merge 2022-08-10 19:41:44 +02:00
KinderRiven
9d83b93e88 fix rebase 2022-08-11 00:11:40 +08:00
Alexey Milovidov
c283351a5f Fix error 2022-08-10 09:31:24 +02:00
Alexey Milovidov
5bc85a6178 Do not write elapsed_ns in HTTP headers 2022-08-10 09:09:04 +02:00
Tanya Bragin
eb61db3b67 initial changes to close #37492 2022-08-08 19:45:00 -07:00
alexX512
c9c26d4898 Fix review 2022-08-08 21:34:24 +00:00
Alexey Milovidov
db85ebfaa7 Add server-side time to Progress 2022-08-08 06:55:41 +02:00
alexX512
62f84b88eb Review fixes 2022-08-07 20:02:21 +00:00
alexX512
6bf29cb610 Change class LRUCache to class CachBase. Check running CacheBase with default pcahce policy SLRU 2022-08-07 19:59:30 +00:00
alexX512
1133e42367 Add CacheBase clas instead of LRUCache and SLRUCache fo simpler configuration oache policiesf 2022-08-07 19:16:40 +00:00
kssenii
7a9b0bc47f Merge master 2022-08-05 01:48:52 +02:00
Kseniia Sumarokova
bd5610b732
Merge branch 'master' into update-azure-lib 2022-08-03 23:00:57 +02:00
Alexey Milovidov
316528817b
Merge pull request #39179 from azat/fsync-profile-events 
Add profile events for fsync
 2022-08-02 05:44:35 +03:00
Azat Khuzhin
3e627e2861 Add profile events for fsync 
The following new provile events had been added:

- FileSync - Number of times the F_FULLFSYNC/fsync/fdatasync function was called for files.
- DirectorySync - Number of times the F_FULLFSYNC/fsync/fdatasync function was called for directories.
- FileSyncElapsedMicroseconds - Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for files.
- DirectorySyncElapsedMicroseconds - Total time spent waiting for F_FULLFSYNC/fsync/fdatasync syscall for directories.

v2: rewrite test to sh with retries
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-31 23:19:30 +03:00
kssenii
e5f4a619ed Merge master 2022-07-31 20:24:40 +03:00
kssenii
687f0fabfc Move some dependencies 2022-07-31 20:17:22 +03:00
Robert Schulze
a7734672b9
Use std::popcount, ::countl_zero, ::countr_zero functions 
- Introduced with the C++20 <bit> header

- The problem with __builtin_c(l|t)z() is that 0 as input has an
  undefined result (*) and the code did not always check. The std::
  versions do not have this issue.

- In some cases, we continue to use buildin_c(l|t)z(), (e.g. in
  src/Common/BitHelpers.h) because the std:: versions only accept
  unsigned inputs (and they also check that) and the casting would be
  ugly.

(*) https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
 2022-07-31 15:16:51 +00:00
Alexey Milovidov
22bc89690d
Merge pull request #39222 from azat/fix-http-session 
Do not report "Failed communicating with" on and on for parts exchange
 2022-07-31 02:24:05 +03:00
Vitaly Baranov
2f3b27d1ac Fix seeking while reading from encrypted disk. 2022-07-29 12:45:29 +02:00
Vitaly Baranov
3b6bacbf0b Add tests. 2022-07-28 22:38:50 +02:00
Vladimir C
ee515b8862
Merge pull request #39656 from HarryLeeIBM/hlee-s390x-bit-helpers 
Fix Endian issue in BitHelpers for s390x
 2022-07-28 13:31:14 +02:00
Vladimir C
a8be51f4df
Merge pull request #39567 from Algunenano/progress_result_rows_and_bytes 
Add result_rows and result_bytes to progress reports
 2022-07-27 16:43:55 +02:00
HarryLeeIBM
3639ecfc56 Fix Endian issue in BitHelpers 2022-07-27 06:56:04 -07:00
alesapin
2dfb547847
Merge pull request #39054 from SmitaRKulkarni/add-option-for-outfile-to-print-stdout 
Add an option to INTO OUTFILE to also print details to standard output.
 2022-07-27 13:51:38 +02:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache 2022-07-27 11:35:36 +02:00
Kseniia Sumarokova
7d448fc177
Merge pull request #39244 from kssenii/http-buffer-small-fixes 
Some small fixes for http read buffer, allow to retry partial content in case of 200ok
 2022-07-26 14:47:11 +03:00
Raúl Marín
6e3a4b0a3d Add result_rows and result_bytes to progress reports and summary 2022-07-26 10:40:16 +02:00
Kseniia Sumarokova
9abbb35cda
Fix integration test 2022-07-25 23:40:23 +03:00
Alexey Milovidov
388d06fda1
Merge pull request #39535 from ClickHouse/stringref 
Less usage of StringRef
 2022-07-25 04:06:11 +03:00
Robert Schulze
4333750985
Less usage of StringRef 
... replaced by std::string_view, see #39262
 2022-07-24 18:33:52 +00:00
Alexey Milovidov
a897c263f0
Merge branch 'master' into http-buffer-small-fixes 2022-07-24 01:46:10 +03:00
mergify[bot]
0933dd7904
Merge branch 'master' into add-option-for-outfile-to-print-stdout 2022-07-23 19:50:10 +00:00
Kseniia Sumarokova
77ddb8eda0
Merge pull request #39506 from CurtizJ/fix-direct-io 
Fix reading from async read buffer with direct io
 2022-07-23 06:08:48 +02:00
Anton Popov
c1a5514e46 fix reading from async read buffer with direct io 2022-07-22 16:58:30 +00:00
Alexander Tokmakov
bed2206ae9
Merge pull request #39460 from ClickHouse/remove_some_dead_and_commented_code 
Remove some dead and commented code
 2022-07-22 13:24:34 +03:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache 2022-07-21 18:59:14 +02:00
Robert Schulze
ea0a3bf600
Merge branch 'master' into stringref-to-string_view 2022-07-21 18:33:06 +02:00
Alexander Tokmakov
a8da5d96fc remove some dead and commented code 2022-07-21 15:05:48 +02:00
Nikolai Kochetov
91043351aa Fixing build. 2022-07-20 20:30:16 +00:00
Smita Kulkarni
605fc5f121 Addressed review comments 
- Added finalizeImpl() override in ForkWriteBuffer to call finalize() of all the buffers.
- Removed clearing buffer in ForkWriteBuffer destructor.
 2022-07-20 21:02:54 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache 2022-07-20 13:30:42 +02:00
Robert Schulze
df53a556eb
Fix test failures 2022-07-20 07:59:02 +00:00
kssenii
21ebf8874e Merge remote-tracking branch 'upstream/master' into make-cache-composable 2022-07-19 13:36:42 +02:00
Robert Schulze
81ef1099cc
Even less usage of StringRef 
--> see #39300
 2022-07-19 07:01:06 +00:00
Smita Kulkarni
8b523bec16 Addressed review comments 
- Updated destructor of ForkWriteBuffer to clear buffer
 2022-07-18 22:12:26 +02:00
kssenii
0aed5a602a Merge master 2022-07-18 16:16:28 +02:00
kssenii
274457e5cf Allow cache over azure blob storage, add tests 2022-07-18 16:15:04 +02:00
Smita Kulkarni
228b9e7ec4 Addressed review comments 
- Updated exception message in ForkWriteBuffer
- Added test case to tests/queries/0_stateless/02346_into_outfile_and_stdout.sh for calling nextImpl more than once
 2022-07-17 20:09:18 +02:00
Robert Schulze
13482af4ee
First try at reducing the use of StringRef 
- to be replaced by std::string_view
- suggested in #39262
 2022-07-17 17:26:02 +00:00
kssenii
3ccbf5ced0 Review fixes 2022-07-17 15:13:28 +02:00
kssenii
8ad26b3111 Merge master 2022-07-16 22:50:03 +02:00
Smita Kulkarni
6318927fba addressed review comment: ForkWriteBuffer for loop 2022-07-15 18:37:50 +02:00
Smita Kulkarni
d7fc7a4290 Addressed review comments 
- Removed first_buffer, finalizeImpl and logic to different buffer size from ForkWriteBuffer
 2022-07-15 16:08:53 +02:00
kssenii
a06fe4e21b Fix 2022-07-15 13:53:20 +02:00
Robert Schulze
deda29b46b
Pass const StringRef by value, not by reference 
See #39224
 2022-07-15 11:34:56 +00:00
Kruglov Pavel
a944a92d4c
Merge pull request #39224 from Avogar/string-view-by-value 
Pass const std::string_view by value, not by reference
 2022-07-15 13:05:56 +02:00
Smita Kulkarni
5a8d6ba0fc clang tidy feedback addressed: updated src/IO/ForkWriteBuffer.cpp to use const reference 2022-07-15 09:06:25 +02:00
Smita Kulkarni
1dc2187f9c Addressed review comments 
Implementation:
- Added a new buffer ForkWriteBuffer takes a vector of WriteBuffer and writes data to all of them. It uses the buffer of the first element as its buffer and copies data from first buffer to all the other buffers
Testing:
- Updated tests/queries/0_stateless/02346_into_outfile_and_stdout.sh
Documentation:
- Updated the english documentation for SELECT.. INTO OUTFILE with AND STDOUT.
 2022-07-14 22:47:38 +02:00
kssenii
03897589b9 Fix 2022-07-14 21:31:17 +02:00
avogar
9291d33080 Pass const std::string_view & by value, not by reference 2022-07-14 16:11:57 +00:00
Sergei Trifonov
b2317ff710 renaming to be consistent 2022-07-14 17:33:22 +02:00
Azat Khuzhin
a068c397df Do not report "Failed communicating with" on and on for parts exchange 
For parts exchange right now an error message in stored in the
HTTPSession, via HTTPSession::attachSessionData(), and for each request
this HTTPSession::sessionData() is checked and if not empty printed into
the log.

However it should be reported only once, otherwise you will get this
message on and on, even for different tables. Here is an example of such
messages:

    2022.07.13 07:56:09.342997 [ 683 ] {} <Error> test_juq8qk.rmt2 (ede90518-4710-48bb-ab43-caf0b1b157f4): auto DB::StorageReplicatedMergeTree::processQueueEntry(ReplicatedMergeTreeQueue::SelectedEntryPtr)::(anonymous class)::operator()(DB::StorageReplicatedMergeTree::LogEntryPtr &) const: Code: 86. DB::Exception: Received error from remote server /?endpoint=DataPartsExchange%3A%2Ftest%2F01165%2Ftest_juq8qk%2Frmt%2Freplicas%2F1&part=206--1406300905-20220713-1657673769_0_0_0&client_protocol_version=7&compress=false&remote_fs_metadata=s3. HTTP status code: 500 Internal Server Error, body: Code: 236. DB::Exception: Transferring part to replica was cancelled. (ABORTED) (version 22.7.1.1781 (official build)). (RECEIVED_ERROR_FROM_REMOTE_IO_SERVER), Stack trace (when copying this message, always include the lines below):
    # this is the time when this message is written ^
    2022.07.13 07:56:10.528554 [ 814 ] {} <Information> DatabaseCatalog: Removing metadata /var/lib/clickhouse/metadata_dropped/test_juq8qk.rmt2.ede90518-4710-48bb-ab43-caf0b1b157f4.sql of dropped table test_juq8qk.rmt2 (ede90518-4710-48bb-ab43-caf0b1b157f4)
    # now this table had been removed ^
    2022.07.13 07:56:27.442003 [ 683 ] {} <Debug> test_1orbeb.mutations_and_quorum2 (bc811afd-1f57-4f9c-a04d-9e1e1b60e891): Fetching part 201901_0_0_0 from /clickhouse/tables/test_1orbeb/test_01090/mutations_and_quorum/replicas/1
    # here fetch part is scheduled for another table ^
    2022.07.13 07:56:27.442213 [ 683 ] {} <Trace> HTTPCommon: Failed communicating with 250c6af4615d with error 'Received error from remote server /?endpoint=DataPartsExchange%3A%2Ftest%2F01165%2Ftest_juq8qk%2Frmt%2Freplicas%2F1&part=255--1372061156-20220713-1657673769_0_0_0&client_protocol_version=7&compress=false&remote_fs_metadata=s3. HTTP status code: 500 Internal Server Error, body: Code: 236. DB::Exception: Transferring part to replica was cancelled. (ABORTED) (version 22.7.1.1781 (official build))' will try to reconnect session
    # however it still reports an error for the already removed table test_juq8qk.rmt
    2022.07.13 07:56:27.442246 [ 683 ] {} <Trace> ReadWriteBufferFromHTTP: Sending request to http://250c6af4615d:9009/?endpoint=DataPartsExchange%3A%2Fclickhouse%2Ftables%2Ftest_1orbeb%
    # but this is just a noisy message and it still doing the job correctly ^

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-14 15:17:15 +03:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache 2022-07-14 12:29:54 +02:00
Sergei Trifonov
af4a5a5f3a fix conflicts 2022-07-13 19:44:29 +02:00
Sergei Trifonov
f854507729 merge master and resolve conflicts 2022-07-13 19:07:28 +02:00
Sergei Trifonov
60c6a5b23c fix s3 read/write throttling instant 2022-07-13 18:36:08 +02:00
kssenii
0d2d3ac2b4 Merge master 2022-07-13 13:26:15 +02:00
kssenii
6f29eada80 Merge master 2022-07-12 19:28:18 +02:00
Robert Schulze
1a7727a254
Prefix overridden add_executable() command with "clickhouse_" 
A simple HelloWorld program with zero includes except iostream triggers
a build of ca. 2000 source files. The reason is that ClickHouse's
top-level CMakeLists.txt overrides "add_executable()" to link all
binaries against "clickhouse_new_delete". This links against
"clickhouse_common_io", which in turn has lots of 3rd party library
dependencies ... Without linking "clickhouse_new_delete", the number of
compiled files for "HelloWorld" goes down to ca. 70.

As an example, the self-extracting-executable needs none of its current
dependencies but other programs may also benefit.

In order to restore access to the original "add_executable()", the
overriding version is now prefixed. There is precedence for a
"clickhouse_" prefix (as opposed to "ch_"), for example
"clickhouse_split_debug_symbols". In general prefixing makes sense also
because overriding CMake commands relies on undocumented behavior and is
considered not-so-great practice (*).

(*) https://crascit.com/2018/09/14/do-not-redefine-cmake-commands/
 2022-07-11 19:36:18 +02:00
Sergei Trifonov
6ed777d2a8 add azure support 2022-07-11 19:19:57 +02:00
Sergei Trifonov
43779ec280 add max_remote_{read,write}_network_bandwidth_for_server settings 2022-07-11 14:59:39 +02:00
kssenii
bf1c99daab Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
Smita Kulkarni
c7967fb721 Add an option to INTO OUTFILE to also print details to standard output. 
Implementation:
- Added a bool to ASTQueryWithOutput & patched the usage in ClientBase.
- Added a new buffer TeeWriteBuffer which extends from WriteBufferFromFile (used to write data to the file) and has WriteBufferFromFileDescriptor (used to write data to stdout). The WriteBufferFromFileDescriptor uses the same buffer as TeeWriteBuffer.
- Added a new bool select_into_outfile_and_stdout in ClientBase to enable/disable progress rendering.

Testing:
 - Added a test tests/queries/0_stateless/02346_into_outfile_and_stdout.sh

Documentation:
- Updated the english documentation for the new option in SELECT.
 2022-07-09 10:12:46 +02:00
Nikolay Degterinsky
0c783800c8 Better 2022-07-07 01:47:33 +00:00
kssenii
e55169ebad Fix 2022-07-05 20:15:35 +02:00
Kseniia Sumarokova
b3fd8e3bd6
Merge pull request #38802 from kssenii/fix-s3-seekable-with-parallel 
Fix s3 seekable reads with parallel read buffer
 2022-07-05 11:23:16 +02:00
kssenii
b52084265c Merge master 2022-07-04 21:37:43 +02:00
kssenii
f5bf744028 Fix 2022-07-04 16:05:08 +02:00
kssenii
a772a09149 Fix 2022-07-04 00:14:45 +02:00
Robert Schulze
bb441faa42
Merge pull request #38491 from ClickHouse/clang-tidy-in-branches 
Fix some clang-tidy warnings in headers
 2022-06-29 10:22:04 +02:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to 
Replace where possible by std::lock_guard which is more light-weight.
 2022-06-28 19:19:06 +00:00
Robert Schulze
3e528c0385
Fix duplicate include 2022-06-28 08:13:51 +00:00
Robert Schulze
5e28580f11
Fix some clang-tidy warnings in headers 2022-06-27 23:49:07 +02:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache 2022-06-24 16:10:25 +02:00
Alexander Tokmakov
cfe0e434e5
Revert "Add support for io_uring read method" 2022-06-24 12:19:41 +03:00
Alexey Milovidov
812ab9bd6b
Merge pull request #36103 from sauliusvl/uring 
Add support for io_uring read method
 2022-06-24 00:34:29 +03:00
Alexander Tokmakov
74f38710a8 Merge branch 'master' into cleanup_garbage_in_store_dir 2022-06-23 21:43:28 +02:00
kssenii
f7b329ee57 Merge master 2022-06-23 14:56:48 +02:00
alesapin
0f8eed98c4
Merge branch 'master' into better-support-gcp 2022-06-22 12:46:01 +02:00
kssenii
4178abc0a5 Merge master 2022-06-21 22:38:51 +02:00
avogar
c14364e3d9 Check last modification time for URL function too 2022-06-21 17:18:14 +00:00
avogar
d37ad2e6de Implement cache for schema inference for file/s3/hdfs/url 2022-06-21 13:02:48 +00:00
Alexander Tokmakov
ba0fcec993 add background cleanup of store/ subdirs 2022-06-21 12:35:47 +02:00
Saulius Valatka
0dc7854af1 add initial io_uring support 2022-06-21 09:59:46 +03:00
Kseniia Sumarokova
2d039b739c
Update ReadBufferFromFileDescriptor.cpp 2022-06-20 19:13:20 +02:00
kssenii
cea81822bd Fix 2022-06-20 17:19:36 +02:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
kssenii
90be49faa4 Merge master 2022-06-20 14:26:56 +02:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6 
Backup Improvements 6
 2022-06-20 14:13:30 +02:00
Alexey Milovidov
7700c26076
Merge pull request #38196 from ClickHouse/revert-38194-revert-37015-zstd_window_log_max 
Revert "Revert "Add a setting to use more memory for zstd decompression""
 2022-06-19 19:58:03 +03:00
Alexey Milovidov
04b7a34870 Maybe fix error 2022-06-19 05:21:59 +02:00
Alexey Milovidov
73709b0488
Revert "Revert "Add a setting to use more memory for zstd decompression"" 2022-06-18 15:55:35 +03:00
alesapin
16e8b85fbf
Revert "Add a setting to use more memory for zstd decompression" 2022-06-18 14:08:14 +02:00
Alexey Milovidov
e20259e9ca
Merge pull request #37015 from wuxiaobai24/zstd_window_log_max 
Add a setting to use more memory for zstd decompression
 2022-06-18 04:19:27 +03:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache 
Extract some diff from pr #36171
 2022-06-17 12:07:31 +02:00
mergify[bot]
296905771f
Merge branch 'master' into better-support-gcp 2022-06-17 01:19:11 +00:00
kssenii
797482f132 Better 2022-06-16 16:40:29 +02:00
mergify[bot]
7f24574609
Merge branch 'master' into better-support-gcp 2022-06-15 23:18:20 +00:00
Vitaly Baranov
e891eba80e Finalize write buffers used in backups. 2022-06-15 20:26:27 +02:00
Daniel Kutenin
a769dea8ef
Merge branch 'ClickHouse:master' into master 2022-06-15 19:12:10 +01:00
Danila Kutenin
048f56bf4d Fix some tests and comments 2022-06-15 14:40:21 +00:00
kssenii
500f49972b Extract diff from PR 36171 2022-06-15 16:40:18 +02:00
Danila Kutenin
08e3f77a9c Optimize most important parts with NEON SIMD 
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.

That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
 2022-06-15 13:19:29 +00:00
mergify[bot]
12bd2c0e3f
Merge branch 'master' into dt64-neg-subseconds 2022-06-15 02:24:17 +00:00
Anton Popov
a59be0fd5d better support of GCP storage 2022-06-15 00:23:45 +00:00
kssenii
4813b90281 Merge master 2022-06-13 21:32:00 +02:00
kssenii
82629cc2be Continue 2022-06-13 16:32:07 +02:00