thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-18 12:22:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
100,834 Commits 9,330 Branches 1,783 Tags 2.2 GiB
761274d857
Commit Graph

1527 Commits

Author SHA1 Message Date
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
Kseniia Sumarokova
6037da1ca8
Update CachedOnDiskReadBufferFromFile.cpp 2022-09-28 14:04:39 +02:00
Alexander Tokmakov
287fe40173 Merge branch 'master' into fix_intersecting_parts 2022-09-27 16:55:39 +02:00
Kseniia Sumarokova
caf6a99f41
Merge pull request #41011 from kssenii/fix-thread-status 
Fix incorrect attach query in threadpool readers, get rid of static threadpools for reads/writes, make threadpool size for reads/writes configurable
 2022-09-27 11:35:07 +02:00
Kseniia Sumarokova
e4f9d311c3
Merge pull request #41731 from kssenii/fix-azure-tests 
Try fix azure tests
 2022-09-27 11:01:19 +02:00
Alexander Tokmakov
9501f88f8c
Merge branch 'master' into fix_intersecting_parts 2022-09-26 19:53:59 +03:00
alesapin
31f6636a47 Fix endless remove 2022-09-26 16:33:25 +02:00
kssenii
921776625e Fix integration tests 2022-09-26 16:20:00 +02:00
kssenii
a02354458a Review fixes 2022-09-26 12:27:29 +02:00
Kseniia Sumarokova
c53f463d2d
Update S3ObjectStorage.cpp 2022-09-25 15:15:59 +02:00
Kseniia Sumarokova
50f7ce6107
Merge branch 'master' into fix-thread-status 2022-09-24 17:22:45 +02:00
Kseniia Sumarokova
21e09f3e1f
Merge pull request #41733 from kssenii/cache-logging-level-reduce 
Change logging levels in cache
 2022-09-24 17:20:46 +02:00
alesapin
34d9794ab7
Merge pull request #41653 from ClickHouse/investigating_more_bugs 
Add very explicit logging on disk choice for fetch
 2022-09-24 16:15:02 +02:00
Kseniia Sumarokova
307314e1bd
Update CachedOnDiskReadBufferFromFile.cpp 2022-09-24 13:42:58 +02:00
kssenii
0a801dad2a Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-23 19:39:07 +02:00
kssenii
30726721ad Fix threadpool reader (for local fs) 2022-09-23 19:35:16 +02:00
Kseniia Sumarokova
81aa9b9199
Update WriteBufferFromAzureBlobStorage.cpp 2022-09-23 15:34:39 +02:00
kssenii
c122e4dd1f Refactor log levels 2022-09-23 15:32:05 +02:00
kssenii
3725397040 Try fix azure tests 2022-09-23 15:24:10 +02:00
kssenii
ab702e43fc Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-23 15:21:33 +02:00
alesapin
148f018eae Fix style: 2022-09-23 14:32:19 +02:00
alesapin
dff61267b4 More explicit logs 2022-09-23 14:31:05 +02:00
Kseniia Sumarokova
a0c5da3459
Merge pull request #41652 from kssenii/fix-disk-web 
Fixes for disk web
 2022-09-23 12:43:50 +02:00
kssenii
0e8343e6f3 Revert "Better config for cache" 
This reverts commit 0ca6aadbdb.
 2022-09-22 17:57:44 +02:00
kssenii
7b8c63b4e4 Merge remote-tracking branch 'origin/fix-disk-web' into fix-disk-web 2022-09-22 16:24:13 +02:00
kssenii
109d2ad022 Better 2022-09-22 16:22:05 +02:00
alesapin
ca410b7a0b
Merge pull request #41147 from ClickHouse/enable_projections_and_zero_copy 
Enable projections and zero copy replication
 2022-09-22 13:05:09 +02:00
Alexander Tokmakov
021b882455 Revert "Disable parallel s3 multipart upload for part moves. (#41268)" 
This reverts commit 0414d95878.
 2022-09-22 13:59:25 +03:00
Kseniia Sumarokova
956e0e8634
Update WebObjectStorage.cpp 2022-09-21 21:51:32 +02:00
kssenii
cb442a3ad7 Fixes for disk web 2022-09-21 21:26:55 +02:00
Nikolai Kochetov
0414d95878
Disable parallel s3 multipart upload for part moves. (#41268) 
* Disable parallel s3 multipart upload for part moves.

* Add setting s3_allow_parallel_part_upload
 2022-09-21 19:10:32 +02:00
alesapin
3e74b783c2
Merge pull request #41530 from ClickHouse/fix_parts_removal_on_drop 
Slightly better parts removal on drop + add unrelated test
 2022-09-21 12:42:41 +02:00
alesapin
ffa9ab1d1c
Merge branch 'master' into enable_projections_and_zero_copy 2022-09-20 14:58:03 +02:00
alesapin
8d88b68a85
Merge pull request #41415 from ClickHouse/revert_prefixes_s3 
Revert prefixes s3 + add retries on write requests unexpected errors
 2022-09-20 12:55:04 +02:00
alesapin
66dc1e5597 Merge branch 'master' into fix_parts_removal_on_drop 2022-09-20 12:35:03 +02:00
alesapin
e9d31977f3
Merge branch 'master' into enable_projections_and_zero_copy 2022-09-20 12:25:10 +02:00
Kseniia Sumarokova
3c7bbad0b0
Merge pull request #41437 from kssenii/fix-logical-error-write-through-cache 
Fix assertion in write-through cache
 2022-09-20 11:51:19 +02:00
Alexey Milovidov
03bbf0676f
Merge branch 'master' into revert_prefixes_s3 2022-09-20 06:14:07 +03:00
Alexey Milovidov
37a4be249b
Merge pull request #41483 from amosbird/jbod-fix1 
Fix nullptr dereference in DB::VolumeJBOD::reserve
 2022-09-20 06:08:14 +03:00
alesapin
ee24b21aaf Fix revert order 2022-09-19 21:19:36 +02:00
serxa
2ef696ffe1 fix issues 2022-09-19 18:40:32 +00:00
serxa
7a1d4a404a add DiskS3* profile events per S3 API calls 2022-09-19 18:10:47 +00:00
serxa
f8aa738511 more conventional profile events names 2022-09-19 17:23:22 +00:00
Sergei Trifonov
ff2e1ff8d8
Merge branch 'master' into support_s3_api_event_metrics 2022-09-19 14:35:45 +02:00
Kseniia Sumarokova
631aad1671
Merge branch 'master' into fix-logical-error-write-through-cache 2022-09-19 12:29:55 +02:00
Amos Bird
d2601b8192
Add a comment 2022-09-19 14:41:34 +08:00
Amos Bird
09b75901f0
Fix nullptr deref 2022-09-19 14:07:31 +08:00
Alexey Milovidov
6a09340f11
Merge pull request #41459 from azat/io/threadpoolreader-elapsed 
Account any IO that is done from ThreadPoolReader
 2022-09-19 09:04:20 +03:00
alesapin
a5a9d89fb9
Merge branch 'master' into enable_projections_and_zero_copy 2022-09-18 17:20:08 +02:00
alesapin
e4c120742e Add missed throw expression 2022-09-18 16:56:19 +02:00
Kseniia Sumarokova
d6748b182f
Update CachedOnDiskWriteBufferFromFile.cpp 2022-09-18 14:48:24 +02:00
Azat Khuzhin
64b44be513 Account any IO that is done from ThreadPoolReader 
In case of some issues with the disk it is possible that
preadv2(RWF_NOWAIT)==EAGAIN can take a lot of time, so let's measure
even this case.

I've verified with a simple (almost) loop of preadv2(RWF_NOWAIT), and
got this:

    EAGAIN: 8565670 (out of 17148317)
    EAGAIN: 8565670 (out of 17148317)
    EAGAIN: 8565670 (out of 17148317)
    too long at offset=8691694, ret=18446744073709551615 (14.5016)
    too long at offset=8694766, ret=18446744073709551615 (14.4338)
    too long at offset=8694766, ret=18446744073709551615 (16.2144)

So here you can see that preadv2(RWF_NOWAIT) took 14-16 seconds in case
of failure.

Complete code can be found here [1].

  [1]: https://gist.github.com/azat/f4a6b0750316c42bec021064e0a7627c

I came up with this patch because I saw some stranginess in
ProfileEvents, that does not show all the time that was spend in IO when
pread_threadpool method was used.
Also there was some stranginess with trace_log too.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-18 10:05:21 +02:00
Alexey Milovidov
6b752bff3a
Merge branch 'master' into revert_prefixes_s3 2022-09-18 07:10:57 +03:00
Alexey Milovidov
f4a48cd4d6 Remove cruft 2022-09-17 22:16:31 +02:00
kssenii
51745dcb43 Fix assertion in write-through cache 2022-09-17 20:20:32 +02:00
alesapin
d9c3549cc8 Revert s3 prefixes + add retries on write requests unexpected errors 2022-09-16 14:43:43 +02:00
alesapin
456da30c9f Revert "Merge pull request #41249 from ClickHouse/revert-40968-s3-sharding-2" 
This reverts commit ac60300997, reversing
changes made to a9c272283a.
 2022-09-16 14:06:26 +02:00
alesapin
b4d2d217f4
Merge branch 'master' into enable_projections_and_zero_copy 2022-09-16 13:53:17 +02:00
Kseniia Sumarokova
9e37a6356f
Merge pull request #41297 from kssenii/fix-logical-error 
Fix incorrect logical error in disk object storage
 2022-09-15 12:00:53 +02:00
Kseniia Sumarokova
69f38a6228
Merge pull request #41330 from kssenii/fix-composable-cache-setting-incompatibility 
Fix incompatibility of cache after switching setting `do_no_evict_...` from 1 to 0, 0 to 1
 2022-09-15 12:00:22 +02:00
Kseniia Sumarokova
780c0e9b2f
Update CachedOnDiskReadBufferFromFile.cpp 2022-09-15 00:18:23 +02:00
Kseniia Sumarokova
0b1c2c62fd
Update CachedOnDiskReadBufferFromFile.cpp 2022-09-14 20:59:45 +02:00
kssenii
559c696230 Fix 2022-09-14 20:35:28 +02:00
alesapin
9d3ab9a7f7 Merge branch 'master' into enable_projections_and_zero_copy 2022-09-14 15:16:16 +02:00
mateng0915
31a0900756 deleted local file 2022-09-14 19:19:47 +08:00
mateng0915
5badb1b186 resolve the review comments 2022-09-14 19:19:40 +08:00
teng.ma
ebea9f9359 Feature: Support Event Tracing when calling Aws S3 API 
Description:
============
In  Computing && Storage Architecture,  using S3 as remote / Shared storage, the Method to access S3 is using AWS S3 API
There is a gap between ClickHouse DB with Ozone Operation
In ClickHouse, operation is SQL and background task
In S3 , the operation should be AWS S3 API
And one sql maybe can mapped to multiple  API

Solution:
=========
Added Calling API as event into system.events table
 2022-09-14 19:18:31 +08:00
Kseniia Sumarokova
cc639f2e2d
Update CachedOnDiskReadBufferFromFile.cpp 2022-09-14 13:13:02 +02:00
Kseniia Sumarokova
52db0e5c40
Update DiskObjectStorageMetadata.cpp 2022-09-14 11:52:23 +02:00
kssenii
ea31302071 Fix 2022-09-14 11:30:06 +02:00
alesapin
2cecf9e55f Fix bug 2022-09-13 18:30:55 +02:00
kssenii
52ef3758c4 Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-13 16:34:31 +02:00
alesapin
9af591a328
Revert "Sharding s3 key names (2)" 2022-09-13 12:29:02 +02:00
Kseniia Sumarokova
3dd28bb79a
Merge pull request #41042 from kssenii/refactor-cache-for-async-download 
Cache refactoring for async download
 2022-09-13 12:26:35 +02:00
kssenii
a07f3d4bea Fix 2022-09-12 18:45:48 +02:00
kssenii
413fbb6507 Self-review fixes 2022-09-12 15:46:11 +02:00
kssenii
5cba9b789c Merge remote-tracking branch 'upstream/master' into refactor-cache-for-async-download 2022-09-09 14:14:04 +02:00
kssenii
c2c35278ba Refactor cache for async download 2022-09-09 13:12:47 +02:00
alesapin
5e00124f8a More logging for S3 2022-09-08 16:18:21 +02:00
kssenii
3dc19b4fe2 Fix 2022-09-08 12:23:59 +02:00
Kseniia Sumarokova
a270eeef91
Merge pull request #41008 from kssenii/refactor-merge-tree-read 
Small refactoring around merge tree readers (get rid of data part ptr)
 2022-09-07 18:27:33 +02:00
kssenii
e06ed0962e Fix style check and typos check 2022-09-07 17:56:20 +02:00
kssenii
e51313b6b3 Get rid of static threadpools 2022-09-07 17:48:11 +02:00
alesapin
43493389b9 Merge branch 'master' into one_more_logging 2022-09-06 19:40:24 +02:00
Kseniia Sumarokova
3558361a05
Merge branch 'master' into refactor-merge-tree-read 2022-09-06 16:00:43 +02:00
alesapin
422b1658eb Review fix 2022-09-06 14:42:48 +02:00
alesapin
6ea7f1e011 Better exception handling for ReadBufferFromS3 2022-09-06 13:59:55 +02:00
Sergei Trifonov
f77809ddbc
Merge pull request #40900 from ClickHouse/s3-detailed-metrics 
S3 detailed metrics
 2022-09-06 13:20:57 +02:00
Alexey Milovidov
4c30dbc905
Merge pull request #40968 from ClickHouse/s3-sharding-2 
Sharding s3 key names (2)
 2022-09-05 22:47:25 +03:00
Kruglov Pavel
cadae5a2f6
Merge pull request #40896 from kssenii/fix-memory-usage-for-remote 
Fix extra memory allocation for remote read buffers
 2022-09-05 20:38:57 +02:00
kssenii
8f064a1720 Fix 2022-09-05 20:28:06 +02:00
kssenii
83514fa2ef Refactor 2022-09-05 20:08:22 +02:00
Sergei Trifonov
1c247308f1
Merge branch 'master' into s3-detailed-metrics 2022-09-05 15:55:41 +02:00
Alexey Milovidov
712de53033 Sharding s3 key names 2022-09-04 04:15:45 +02:00
Sergei Trifonov
067e712fdf
Merge branch 'master' into s3-detailed-metrics 2022-09-02 16:59:52 +02:00
alesapin
521e2cc22e Merge branch 'master' into stress_s3 2022-09-02 12:57:29 +02:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Sergei Trifonov
ac9dc53c73 Merge branch 'master' into s3-detailed-metrics 2022-09-01 18:31:02 +02:00
Sergei Trifonov
bcb6475c4a add separate s3 profile events for disk s3 2022-09-01 18:30:55 +02:00
alesapin
dbe3b777ec Fix build 2022-09-01 17:07:56 +02:00