Raúl Marín
97d6fc3071
Merge remote-tracking branch 'blessed/master' into perf_experiment
2022-11-17 11:48:46 +01:00
Alexander Tokmakov
9011a18234
Revert "S3 request per second rate throttling"
2022-11-16 22:33:48 +03:00
Kseniia Sumarokova
59cf5def67
Merge branch 'master' into disk-s3-throttler
2022-11-15 12:13:37 +01:00
Alexey Milovidov
127631ee47
Merge branch 'master' into perf_experiment
2022-11-12 18:58:25 +01:00
zzsmdfj
3835373644
to add_oss_function_and_StorageOSS
2022-11-11 16:40:10 +08:00
Sergei Trifonov
8eedd1e046
Merge branch 'master' into disk-s3-throttler
2022-11-08 15:00:56 +01:00
serxa
6d5d9ff421
rename ReadWriteSettings -> RequestSettings
2022-11-08 13:48:23 +00:00
serxa
2daec0b45e
S3 request per second rate throttling + refactoring
2022-11-07 18:05:40 +00:00
Kruglov Pavel
b124875257
Merge branch 'master' into improve-streaming-engines
2022-11-03 13:22:06 +01:00
Nikolay Degterinsky
30ad1a6826
Merge branch 'master' into perf_experiment
2022-11-03 02:18:21 +03:00
Vitaly Baranov
e0133688bc
Merge branch 'master' into mask-sensitive-info-in-logs
2022-11-02 16:26:13 +01:00
Kruglov Pavel
21d50f76ea
Merge pull request #41979 from Avogar/s3-cluster-schema-inference
...
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-11-01 14:00:21 +01:00
Vitaly Baranov
5d2a222fe4
Mask sensitive information in logs.
2022-10-31 10:50:33 +01:00
avogar
8e13d1f1ec
Improve and refactor Kafka/StorageMQ/NATS and data formats
2022-10-28 16:41:10 +00:00
Raúl Marín
891484b462
Merge remote-tracking branch 'blessed/master' into perf_experiment
2022-10-27 13:17:07 +02:00
Raúl Marín
6e0a9452e7
Merge remote-tracking branch 'blessed/master' into perf_experiment
2022-10-25 15:25:06 +02:00
SmitaRKulkarni
96c8260230
Merge branch 'master' into 36316_Support_glob_for_recursive_directory_traversal
2022-10-24 18:34:19 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Raúl Marín
e60415d07d
Make clang-tidy happy
2022-10-18 11:40:12 +02:00
Smita Kulkarni
91433e5b9c
Added ** glob support for recursive directory traversal to filesystem and S3.
...
Implementation:
* Updated parseGlob to not add ‘/‘ restriction when ** is used.
* Updated S3 & filesystem to fetch files and not use regex match if glob is **.
Testing:
* Added a test for filesystem tests/queries/0_stateless/02459_glob_for_recursive_directory_traversal.sh
2022-10-17 09:04:25 +02:00
Kruglov Pavel
3d9f46a1e7
Merge branch 'master' into s3-cluster-schema-inference
2022-10-14 22:07:54 +02:00
avogar
c74b5c8126
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-09-30 16:59:17 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
kssenii
0a801dad2a
Merge remote-tracking branch 'upstream/master' into fix-thread-status
2022-09-23 19:39:07 +02:00
kssenii
ab702e43fc
Merge remote-tracking branch 'upstream/master' into fix-thread-status
2022-09-23 15:21:33 +02:00
kssenii
e34101456a
Fix
2022-09-21 17:11:37 +02:00
kssenii
f917b268b3
Merge remote-tracking branch 'upstream/master' into s3-header-auth
2022-09-21 12:56:25 +02:00
serxa
2ef696ffe1
fix issues
2022-09-19 18:40:32 +00:00
serxa
f8aa738511
more conventional profile events names
2022-09-19 17:23:22 +00:00
mateng0915
f7f976e94e
remove extra space
2022-09-14 19:19:47 +08:00
mateng0915
5badb1b186
resolve the review comments
2022-09-14 19:19:40 +08:00
kssenii
52ef3758c4
Merge remote-tracking branch 'upstream/master' into fix-thread-status
2022-09-13 16:34:31 +02:00
kssenii
420ac4eb43
s3 header auth in ast
2022-09-13 15:13:28 +02:00
kssenii
e51313b6b3
Get rid of static threadpools
2022-09-07 17:48:11 +02:00
Sergei Trifonov
bcb6475c4a
add separate s3 profile events for disk s3
2022-09-01 18:30:55 +02:00
avogar
5ab87f1da4
Small refactoring
2022-08-19 16:42:23 +00:00
avogar
8dd54c043d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache
2022-08-17 11:47:40 +00:00
avogar
c4ff3ffeea
Rename settings
2022-08-15 12:45:18 +00:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache
2022-07-21 18:59:14 +02:00
avogar
6b541aa98f
Fix WriteBuffer finalize when cancel insert into function
2022-07-21 12:18:37 +00:00
Kruglov Pavel
92995a832b
Revert "Fix WriteBuffer finalize in destructor when cacnel query"
2022-07-21 01:45:16 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache
2022-07-20 13:30:42 +02:00
avogar
5c16d6b553
Fix WriteBuffer finalize in destructor when cacnel query
2022-07-19 19:21:30 +00:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache
2022-07-14 12:29:54 +02:00
Sergei Trifonov
15ab3bc99f
use context->getWriteSettings()
2022-07-13 19:48:57 +02:00
Sergei Trifonov
43779ec280
add max_remote_{read,write}_network_bandwidth_for_server settings
2022-07-11 14:59:39 +02:00
avogar
ee54c4f9b7
Add some fixes and add settings in docs
2022-06-30 12:41:56 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache
2022-06-24 16:10:25 +02:00
avogar
59c1c472cb
Better exception messages on wrong table engines/functions argument types
2022-06-23 20:04:06 +00:00
Kruglov Pavel
e5a7f53775
Fix misleading error message while s3 schema inference
2022-06-22 12:36:09 +02:00
avogar
c14364e3d9
Check last modification time for URL function too
2022-06-21 17:18:14 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Alexey Milovidov
73709b0488
Revert "Revert "Add a setting to use more memory for zstd decompression""
2022-06-18 15:55:35 +03:00
alesapin
16e8b85fbf
Revert "Add a setting to use more memory for zstd decompression"
2022-06-18 14:08:14 +02:00
Alexey Milovidov
e20259e9ca
Merge pull request #37015 from wuxiaobai24/zstd_window_log_max
...
Add a setting to use more memory for zstd decompression
2022-06-18 04:19:27 +03:00
Nikolai Kochetov
8991f39412
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-06-02 17:00:08 +00:00
Azat Khuzhin
545a56ce45
Fix sinks with onException() handler
...
It is possible to call onException() even after onFinish(), in case of
onFinish() throws, and in this case onException() should be no-op for
such sinks.
Also there can be caveats with PartitionedSync.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-01 21:50:30 +03:00
Azat Khuzhin
02af58f41d
Fix possible "Cannot write to finalized buffer"
...
It is still possible to get this error since onException does not
finalize format correctly.
Here is an example of such error, that was found by CI [1]:
<details>
[ 2686 ] {fa01bf02-73f6-4f7f-b14f-e725de6d7f9b} <Fatal> : Logical error: 'Cannot write to finalized buffer'.
[ 34577 ] {} <Fatal> BaseDaemon: ########################################
[ 34577 ] {} <Fatal> BaseDaemon: (version 22.6.1.1, build id: AB8040A6769E01A0) (from thread 2686) (query_id: fa01bf02-73f6-4f7f-b14f-e725de6d7f9b) (query: insert into test_02302 select number from numbers(10) settings s3_truncate_on_insert=1;) Received signal Aborted (6)
[ 34577 ] {} <Fatal> BaseDaemon:
[ 34577 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fcbaa5a703b 0x7fcbaa586859 0xfad9bab 0xfad9e05 0xfaf6a3b 0x24a48c7f 0x258fb9b9 0x258f2004 0x258b88f4 0x258b863b 0x2581773d 0x258177ce 0x24bb5e98 0xfad01d6 0xfad0105 0x2419b11d 0xfad01d6 0xfad0105 0x2215afbb 0x2215aa48 0xfad01d6 0xfad0105 0xfcc265d 0x225cc546 0x249a1c40 0x249bc1b6 0x2685902c 0x26859505 0x269d7767 0x269d504c 0x7fcbaa75e609 0x7fcbaa683163
[ 34577 ] {} <Fatal> BaseDaemon: 3. raise @ 0x7fcbaa5a703b in ?
[ 34577 ] {} <Fatal> BaseDaemon: 4. abort @ 0x7fcbaa586859 in ?
[ 34577 ] {} <Fatal> BaseDaemon: 5. ./build_docker/../src/Common/Exception.cpp:47: DB::abortOnFailedAssertion(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfad9bab in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 6. ./build_docker/../src/Common/Exception.cpp:70: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xfad9e05 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 7. ./build_docker/../src/IO/WriteBuffer.h:0: DB::WriteBuffer::write(char const*, unsigned long) @ 0xfaf6a3b in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 8. ./build_docker/../src/Processors/Formats/Impl/ArrowBufferedStreams.cpp:47: DB::ArrowBufferedOutputStream::Write(void const*, long) @ 0x24a48c7f in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 9. long parquet::ThriftSerializer::Serialize<parquet::format::FileMetaData>(parquet::format::FileMetaData const*, arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) @ 0x258fb9b9 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 10. parquet::FileMetaData::FileMetaDataImpl::WriteTo(arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) const @ 0x258f2004 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 11. parquet::WriteFileMetaData(parquet::FileMetaData const&, arrow::io::OutputStream*) @ 0x258b88f4 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 12. parquet::ParquetFileWriter::~ParquetFileWriter() @ 0x258b863b in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 13. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x2581773d in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 14. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x258177ce in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 15. ./build_docker/../src/Processors/Formats/Impl/ParquetBlockOutputFormat.h:27: DB::ParquetBlockOutputFormat::~ParquetBlockOutputFormat() @ 0x24bb5e98 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 16. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 17. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:312: std::__1::unique_ptr<DB::WriteBuffer, std::__1::default_delete<DB::WriteBuffer> >::reset(DB::WriteBuffer*)
[ 34577 ] {} <Fatal> BaseDaemon: 18.2. inlined from ../contrib/libcxx/include/__memory/unique_ptr.h:269: ~unique_ptr
[ 34577 ] {} <Fatal> BaseDaemon: 18. ../src/Storages/StorageS3.cpp:566: DB::StorageS3Sink::~StorageS3Sink() @ 0x2419b11d in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 19. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 20. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 21. ./build_docker/../contrib/abseil-cpp/absl/container/internal/raw_hash_set.h:1662: absl::lts_20211102::container_internal::raw_hash_set<absl::lts_20211102::container_internal::FlatHashMapPolicy<StringRef, std::__1::shared_ptr<DB::SinkToStorage> >, absl::lts_20211102::hash_internal::Hash<StringRef>, std::__1::equal_to<StringRef>, std::__1::allocator<std::__1::pair<StringRef const, std::__1::shared_ptr<DB::SinkToStorage> > > >::destroy_slots() @ 0x2215afbb in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./build_docker/../contrib/libcxx/include/string:1445: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long() const
[ 34577 ] {} <Fatal> BaseDaemon: 22.2. inlined from ../contrib/libcxx/include/string:2231: ~basic_string
[ 34577 ] {} <Fatal> BaseDaemon: 22. ../src/Storages/PartitionedSink.h:14: DB::PartitionedSink::~PartitionedSink() @ 0x2215aa48 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 23. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 24. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 25. ./build_docker/../contrib/libcxx/include/vector:802: std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >::__base_destruct_at_end(std::__1::shared_ptr<DB::IProcessor>*) @ 0xfcc265d in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 26.1. inlined from ./build_docker/../contrib/libcxx/include/vector:402: ~vector
[ 34577 ] {} <Fatal> BaseDaemon: 26.2. inlined from ../src/QueryPipeline/QueryPipeline.cpp:29: ~QueryPipeline
[ 34577 ] {} <Fatal> BaseDaemon: 26. ../src/QueryPipeline/QueryPipeline.cpp:535: DB::QueryPipeline::reset() @ 0x225cc546 in /usr/bin/clickhouse
[ 614 ] {} <Fatal> Application: Child process was terminated by signal 6.
</details>
[1]: https://s3.amazonaws.com/clickhouse-test-reports/37542/8a224239c1d922158b4dc9f5d6609dca836dfd06/stress_test__undefined__actions_.html
Follow-up for: #36979
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-01 21:50:30 +03:00
Nikolai Kochetov
86fbb74703
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-31 18:07:47 +00:00
Kruglov Pavel
0615866aea
Merge pull request #37450 from Avogar/check-format-on-storage-creation
...
Check format name on storage creation
2022-05-30 14:23:20 +02:00
Nikolai Kochetov
5b4658aa5e
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-30 09:47:35 +00:00
alesapin
c7b16065e1
Merge with master
2022-05-25 21:47:05 +02:00
alesapin
6f5c86e55e
Merge branch 'master' into i_object_storage
2022-05-25 20:49:01 +02:00
Nikolai Kochetov
1b85f2c1d6
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-25 16:27:40 +02:00
Kseniia Sumarokova
b50d4549c9
Merge pull request #37356 from amosbird/partition-prune-for-s3
...
"Partition pruning" for s3
2022-05-25 11:03:07 +02:00
avogar
f782fa31c6
Merge branch 'master' of github.com:ClickHouse/ClickHouse into check-format-on-storage-creation
2022-05-25 08:42:54 +00:00
Nikolai Kochetov
3d84aae0ab
Better.
2022-05-24 20:06:08 +00:00
Amos Bird
093d315756
partition pruning for s3
2022-05-24 18:57:55 +08:00
mergify[bot]
51ff49a0ee
Merge branch 'master' into i_object_storage
2022-05-23 20:29:49 +00:00
avogar
37b66c8a9e
Check format name on storage creation
2022-05-23 12:48:48 +00:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names
2022-05-23 12:14:20 +02:00
Nikolai Kochetov
56feef01e7
Move some resources
2022-05-20 19:49:31 +00:00
avogar
2d4b4b9008
Fix inserting defaults for missing values in columnar formats
2022-05-16 14:19:44 +00:00
alesapin
4f1d4dde22
Disable s3 logging by default
2022-05-16 00:16:00 +02:00
avogar
68bb07d166
Better naming
2022-05-13 18:39:19 +00:00
avogar
b17fec659a
Improve performance and memory usage for select of subset of columns for some formats
2022-05-13 13:51:28 +00:00
wuxiaobai24
f3e573bfde
Merge remote-tracking branch 'upstream/master' into zstd_window_log_max
2022-05-13 20:42:54 +08:00
wuxiaobai24
ab5636a46a
fix
2022-05-13 17:52:39 +08:00
alesapin
5dcd25be23
Initial implementation
2022-05-12 00:04:54 +02:00
wuxiaobai24
58755cb156
add zstd_window_log_max setting
2022-05-08 14:02:40 +08:00
avogar
02bd5f6542
Finalize write buffers in case of exception
2022-05-06 17:30:18 +00:00
Kruglov Pavel
c6e26f6128
Merge pull request #36434 from Avogar/fix-s3-cluster
...
Fix bug in s3Cluster schema inference
2022-05-05 13:08:49 +02:00
Kruglov Pavel
97a1a2d665
Fix style
2022-05-04 17:16:02 +02:00
Kruglov Pavel
050eed68e2
Fix bug after resolving conflicts
2022-05-03 14:47:24 +02:00
Kruglov Pavel
21bc5624cc
Fix
2022-05-02 14:16:29 +02:00
Kruglov Pavel
c3570b9f7e
Merge branch 'master' into fix-s3-cluster
2022-05-02 14:15:21 +02:00
Kruglov Pavel
b8ee22eb57
Update StorageS3.cpp
2022-05-02 14:09:54 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying
...
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
previously allowed.
Hence, this change
- removes shared_ptr_helper and as a result all inherited create() methods,
- instead, Storage objects are now created using make_shared<>() by the
caller (for that to work, many constructors had to be made public), and
- all Storage classes were marked as noncopyable using boost::noncopyable.
In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
2022-05-02 08:46:52 +02:00
Sergei Trifonov
c1a0165f7d
merge from master and fix conflicts
2022-04-26 12:39:16 +02:00
Kruglov Pavel
34c342fdd3
Merge pull request #36205 from Avogar/improve-globs
...
Some refactoring around schema inference with globs
2022-04-25 13:14:46 +02:00
Amos Bird
e73d7fea31
Rename narrowBlockInputStream to narrowPipe
2022-04-24 18:33:48 +08:00
Kruglov Pavel
7fb0f2284d
Merge branch 'master' into fix-s3-cluster
2022-04-21 13:39:52 +02:00
Sergei Trifonov
34842f209c
merge master and fix conflicts
2022-04-20 18:35:56 +02:00
Kruglov Pavel
813e228fcc
Merge branch 'master' into improve-globs
2022-04-20 16:31:47 +02:00
avogar
1f252cedfe
Make better
2022-04-19 19:16:47 +00:00
avogar
9a81b67ba5
Fix bug in s3Cluster schema inference
2022-04-19 18:23:04 +00:00
avogar
1c065f8c7a
Some refactoring around schema inference with globs
2022-04-13 17:02:48 +00:00
kssenii
ef2b238444
Fix
2022-04-08 12:30:24 +02:00
Sergei Trifonov
01425df758
fix build
2022-04-07 16:07:12 +02:00
Sergei Trifonov
755d5c55f9
resolve conflict
2022-04-07 11:57:45 +02:00
Sergei Trifonov
9c2804b924
merge from master + resolve conflicts
2022-04-07 11:22:01 +02:00
kssenii
cbd7f4acfb
Fix
2022-04-05 19:24:10 +02:00
kssenii
4e50da722b
Better s3 settings
2022-04-04 16:14:56 +02:00
Alexey Milovidov
5c6b84c398
Merge pull request #35586 from kssenii/fix-virtual-columns-s3
...
Fix s3 engine getting virtual columns
2022-04-01 04:43:32 +03:00
kssenii
099fa0e668
Merge master
2022-03-30 17:33:32 +02:00
Antonio Andelic
a3ad99be03
Create CallbackRunner in Storage
2022-03-30 11:24:50 +00:00
Antonio Andelic
1173eab15e
Use same IOThreadPool for URL and S3
2022-03-30 08:15:20 +00:00
kssenii
4374415bd7
Fix clang-tidy, style check
2022-03-29 14:20:21 +02:00
kssenii
a2013ebe2b
Better
2022-03-28 22:40:27 +02:00
Antonio Andelic
d9d826c813
Address PR review
2022-03-28 08:19:23 +00:00
kssenii
c6706b8bfa
Better
2022-03-25 19:30:13 +01:00
Antonio Andelic
394682b983
Merge branch 'master' into parallel-s3-downloading
2022-03-25 13:00:44 +00:00
kssenii
e91e30aaa4
Fix
2022-03-24 17:10:27 +01:00
Antonio Andelic
62a495a5fc
Small refactor
2022-03-24 09:45:30 +00:00
Sergei Trifonov
bf9c6be6c6
move threadPoolCallbackRunner from Common into Interpreters folder to fix split build issue
2022-03-23 17:57:19 +01:00
Antonio Andelic
1a5f5c32bb
Remove commented out code
2022-03-23 14:36:25 +00:00
Antonio Andelic
93ad209910
Attach memory tracker
2022-03-23 11:52:31 +00:00
Antonio Andelic
131b3a091c
Refactor StorageS3
2022-03-23 08:57:01 +00:00
Antonio Andelic
6785ad165a
Fix issue for mutliple download threads
2022-03-22 14:06:30 +00:00
Antonio Andelic
04e6e6ab5e
Add ParallelReadBuffer for S3
2022-03-21 14:52:26 +00:00
mergify[bot]
7ac606fa65
Merge branch 'master' into parallel-multipart-upload-for-s3storage
2022-03-20 18:25:28 +00:00
Sergei Trifonov
bbe657ba24
allow parallelization of multipart S3 storage upload fixes #34350
2022-03-16 19:41:41 +01:00
Anton Popov
0ba78c3c3a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-16 15:28:09 +00:00
Saad Ur Rahman
679abaafdc
[Storages] bugfix
...
Extracting Version Id for S3::URI.
2022-03-14 22:25:40 -04:00
Saad Ur Rahman
fb450b7068
[Storages] bugfix
...
Corrected order of parameters by swapping bucket and versionId.
2022-03-14 22:20:02 -04:00
Saad Ur Rahman
171260f40b
[Storages] Added Version ID parameter for S3.
2022-03-14 22:03:58 -04:00
Maksim Kita
2fdcf53a76
Fix clang-tidy warnings in Server, Storages folders
2022-03-14 18:17:35 +00:00
Anton Popov
df3b07fe7c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-03 22:25:28 +00:00
Anton Popov
fcdebea925
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-25 13:41:30 +03:00
kssenii
cfad79bf74
Remove redundant
2022-02-25 09:37:48 +01:00
kssenii
003b807b00
Fix
2022-02-23 20:33:05 +01:00
Kruglov Pavel
dd863ca2a0
Merge branch 'master' into fix-url-globs
2022-02-16 12:45:31 +03:00
Dmitry Novik
50131fd732
Fix cancelation for S3 and HDFS
2022-02-11 18:14:55 +00:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
avogar
84087c1554
Improve schema inference with globs
2022-02-09 19:16:15 +03:00
alesapin
f6e7cea1f8
Add settings for S3 multipart upload parts size
2022-02-08 19:38:04 +03:00
feng lv
6325d4d9b0
continue of #34317
...
fix
fix
2022-02-06 08:59:17 +00:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
alesapin
5230c59f05
Smaller scope
2022-01-31 14:03:06 +03:00
alesapin
1adc7538fe
Fix benign race condition for storage HDFS, S3, URL
2022-01-31 13:50:28 +03:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
avogar
4efadfad3c
Fix tests
2022-01-18 22:26:13 +03:00
avogar
af54ca4866
Fix partition write
2022-01-14 21:21:36 +03:00
Kruglov Pavel
a7df9cd53a
Merge branch 'master' into formats-with-suffixes
2022-01-14 21:03:49 +03:00
avogar
89a181bd19
Make better
2022-01-14 18:16:18 +03:00
avogar
8884c4a3fe
Fix tests and build
2022-01-14 16:27:57 +03:00
avogar
2d7b1bfa5e
Detect format in S3/HDFS/URL table engines
2022-01-13 16:14:18 +03:00
avogar
97788b9c21
Allow to create new files on insert for File/S3/HDFS engines
2021-12-29 21:19:13 +03:00
avogar
aaf9f85c67
Add more tests and fixes
2021-12-29 12:18:56 +03:00
avogar
8112a71233
Implement schema inference for most input formats
2021-12-29 12:18:56 +03:00
kssenii
1f6ca619b7
Allow some killing
2021-12-27 22:42:56 +03:00
avogar
58966e90e9
Don't allow to write into S3 if path contains globs
2021-12-24 17:13:35 +03:00
Anton Popov
ccd78e3838
Merge remote-tracking branch 'upstream/master' into HEAD
2021-11-22 17:19:35 +03:00
avogar
e5661b9b62
Minor change
2021-11-11 21:09:21 +03:00
Anton Popov
a20922b2d3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-11-09 15:36:25 +03:00
avogar
2dd4393ca1
Small refactoring in formats
2021-11-03 20:07:05 +03:00
Kseniia Sumarokova
065106c299
Merge pull request #30690 from kssenii/partitioned-write
...
Partitioned write for file, hdfs, url storages and table functions
2021-11-02 22:58:16 +03:00
kssenii
20772fde04
For storage
2021-10-27 10:04:17 +03:00
kssenii
2ba3ee830e
Partitioned write part 2
2021-10-26 14:00:41 +03:00
kssenii
52296e8b04
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-25 00:22:32 +03:00
kssenii
e04e37916e
Review fixes
2021-10-23 18:20:31 +03:00
kssenii
36d0f67b38
Merge branch 'master' of github.com:ClickHouse/ClickHouse into remote-connection
2021-10-23 17:12:46 +03:00
kssenii
39b9e9c258
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-22 15:40:41 +03:00
kssenii
4a0bdd296d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-16 18:35:48 +03:00
Nikolai Kochetov
a08c98d760
Move some files.
2021-10-16 17:03:50 +03:00
Nikolai Kochetov
fd14faeae2
Remove DataStreams folder.
2021-10-15 23:18:20 +03:00
kssenii
7b81766168
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into remote-connection
2021-10-15 12:36:44 +00:00
kssenii
881ae8617e
Merge branch 'master' of github.com:ClickHouse/ClickHouse into disk-async-read
2021-10-15 15:09:56 +03:00
Nikolai Kochetov
ab28c6c855
Remove BlockInputStream interfaces.
2021-10-14 13:25:43 +03:00
Nikolai Kochetov
2957971ee3
Remove some last streams.
2021-10-13 21:22:02 +03:00
Nikolai Kochetov
ec18340351
Remove streams from formats.
2021-10-11 19:11:50 +03:00
kssenii
448d63a0fc
Range read with upper bound offset
2021-10-11 01:53:51 +03:00
kssenii
5a87b6eb4f
Predefined configuration for table function remote
2021-10-06 05:55:09 +00:00
Nikolai Kochetov
998d29ebc7
Merge branch 'master' into rewrite-pushing-to-views
2021-09-23 13:10:27 +03:00
Nikolai Kochetov
341553febd
Fix build.
2021-09-16 20:40:42 +03:00
kssenii
1650c92407
Better
2021-09-16 11:38:26 +03:00
kssenii
221c09589c
Review fixes
2021-09-15 21:11:49 +03:00
Nikolai Kochetov
b997214620
Rename QueryPipeline to QueryPipelineBuilder.
2021-09-14 20:48:18 +03:00
Nikolai Kochetov
f569a3e3f7
Merge branch 'master' into rewrite-pushing-to-views
2021-09-09 20:30:23 +03:00
Anton Popov
4c388e3d84
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-09-09 14:10:16 +03:00
kssenii
e9721804b6
Fix tests
2021-09-09 12:34:47 +03:00
kssenii
3deb9a0ecb
s3, url, http headers
2021-09-07 14:17:25 +03:00
Nikolai Kochetov
66a76ab70f
Rewrite PushingToViewsBlockOutputStream part 6
2021-09-03 20:29:36 +03:00
Nikita Mikhaylov
e41d1cff1d
Fix test
2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
e5593f594d
After rebase
2021-08-31 12:54:06 +00:00
Nikita Mikhaylov
ae241fc129
done
2021-08-31 12:54:06 +00:00
vdimir
6fe63a80bc
Fix validateKey/Bucket for S3
2021-08-23 09:22:15 +03:00
vdimir
6ba6577978
Do not allow slashes in bucket formatted from PARTITION BY
2021-08-23 09:22:15 +03:00
vdimir
c4c42b61c4
Validate uft8 in partition key from PARTITION BY for s3
2021-08-23 09:22:15 +03:00
Vladimir Chebotarev
f6cb82aba5
Minor fix.
2021-08-23 09:22:15 +03:00
Vladimir Chebotarev
e203c330c2
Validation.
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
6e10d28c78
Fix.
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
ea67849fd9
Typo.
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
119172543b
Better fix.
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
0b75c66378
Revert "Update src/Storages/StorageS3.cpp
"
...
This reverts commit bbbfe1891bfdc868878cfc8d52c254babd4ca181.
2021-08-23 09:22:14 +03:00
Vladimir C
067cc763cc
Update src/Storages/StorageS3.cpp
...
StringRef in fmt::format throws `fmt::v7::format_error: cannot switch from manual to automatic argument indexing`
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
d8bc601dbf
Minor.
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
4b1cc38056
Fixes.
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
cebe1e7123
Style fix.
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
b34f60ad46
Fixed find
.
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
be343766a6
Update src/Storages/StorageS3.cpp
...
Co-authored-by: Vladimir C <vdimir@yandex-team.ru>
2021-08-23 09:22:14 +03:00
Vladimir Chebotarev
24fcf152ed
Minor fix.
2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
5ce9e266b4
Less strict validation.
2021-08-23 09:22:13 +03:00
vdimir
4c9fb7ff58
Fix error code in StorageS3.cpp
2021-08-23 09:22:13 +03:00
vdimir
eb31b194b9
Add validation to insert partition by key to s3
2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
d1efe3afa7
Review fixes.
2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
e99433e094
Better functionality.
2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
328213f5d9
Fixes.
2021-08-23 09:22:13 +03:00
Vladimir Chebotarev
1146efacff
Partitioned writes for S3.
2021-08-23 09:22:13 +03:00
Anton Popov
61239343e3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-08-20 16:33:30 +03:00
Vladimir Chebotarev
c2410920d3
Safer ReadBufferFromS3
for merges and backports.
2021-08-04 09:14:20 +03:00
Anton Popov
e36736b50c
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-08-02 22:52:02 +03:00
Nikolai Kochetov
9b5a816b43
Merge branch 'master' into output-streams-to-processors
2021-07-26 18:03:11 +03:00
Nikolai Kochetov
9c92f43359
Update storages.
2021-07-23 22:33:59 +03:00
Nikolai Kochetov
f38de35b14
Rename some constants.
2021-07-21 19:13:17 +03:00
Nikolai Kochetov
179ec05a72
Remove some streams.
2021-07-20 21:18:43 +03:00
Anton Popov
3ed7f5a6cc
dynamic subcolumns: add snapshot for storage
2021-07-09 06:15:41 +03:00
Kseniia Sumarokova
55ce7de248
Remove trailing spaces -- style check fix
2021-07-02 22:39:21 +03:00
Kseniia Sumarokova
ed687f45a1
Merge branch 'master' into truncate
2021-06-30 20:08:42 +03:00
Raúl Marín
bfc122df64
Fix some typos in Storage classes
2021-06-28 19:03:56 +02:00
kssenii
ac0f86cdbf
Truncate for s3
2021-06-21 15:44:36 +00:00
Nikita Mikhaylov
0055a924f6
better
2021-06-01 14:23:46 +00:00
Nikita Mikhaylov
3f18bde246
Merge branch 'master' of github.com:ClickHouse/ClickHouse into merging-s3-backoff
2021-06-01 13:39:09 +00:00
Nikita Mikhaylov
f33b70c54a
better
2021-05-20 00:42:25 +03:00
kssenii
9b8df78fdd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-17 17:42:05 +03:00
Vladimir Chebotarev
bdf03387ab
Better S3 read retries. Renamed s3_max_single_read_retries
-> s3_single_read_retry_attempts
, added sleeps before next attempts.
2021-05-17 10:38:32 +03:00
feng lv
9035e3a643
fix test and conflict
2021-05-11 16:22:24 +00:00
kssenii
393b026f2e
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-11 16:49:23 +03:00
Vladimir
d02d14fa00
Merge pull request #23846 from excitoon-favorites/s3regions
2021-05-10 12:14:08 +03:00
kssenii
266567e04c
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-08 17:49:05 +03:00
kssenii
02288359c5
Less manual concatenation of paths
2021-05-08 13:59:55 +03:00
Azat Khuzhin
e398635cb3
Make StorageS3.cpp self compilable (missing AbstractConfiguration.h)
2021-05-07 10:07:44 +03:00
feng lv
39f68bf5ff
fix conflict
2021-05-02 16:33:45 +00:00
Vladimir Chebotarev
a6ab040b07
Added region
parameter for S3 storage and disk.
2021-05-01 20:45:35 +03:00
Azat Khuzhin
a4036a8199
Correctly stop ParallelFormattingOutputFormat from StorageS3BlockOutputStream
2021-04-25 12:49:57 +03:00
feng lv
0f5c05854c
fix conflict
2021-04-23 12:22:51 +00:00
feng lv
4ffe199d39
Implement table comments
2021-04-23 12:18:23 +00:00
Nikita Mikhaylov
f55eacee7d
Merge pull request #22988 from excitoon-favorites/s3connectionresetretries
...
Retries on HTTP connection drops during reads from S3
2021-04-20 14:59:21 +03:00
Vladimir Chebotarev
9d4295f261
Removed infinite retries.
2021-04-19 11:02:36 +03:00
Vladimir Chebotarev
f59d35efd8
Rebase fix.
2021-04-15 10:53:00 +03:00
Vladimir Chebotarev
c0ea3ce08a
Removed STS client.
2021-04-15 10:01:43 +03:00
Vladimir Chebotarev
3a467ffc50
Attempt to make credentials provider which uses internal credentials provider chain.
2021-04-15 10:01:13 +03:00
Vladimir Chebotarev
037a4de854
Added STS GetCallerIdentity credentials provider.
2021-04-15 10:00:18 +03:00
Vladimir Chebotarev
10bf56e540
Attempt to add retries to S3 reads.
2021-04-15 09:47:19 +03:00
Nikita Mikhaylov
2a86d76ccd
style
2021-04-13 23:19:04 +03:00
Nikita Mikhaylov
ec35a878d3
simplify storages3cluster
2021-04-13 23:17:25 +03:00
Nikita Mikhaylov
024374a2ec
review fixes
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
a743442a17
build fixes
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
507cb8514a
review fixes
2021-04-13 22:39:43 +03:00
Nikita Mikhaylov
7a68820342
style
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7b95ff579d
fix S3 test
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
4465a0627f
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
c333c3dedb
review fixes
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
36a8419f60
style
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
7276b40556
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
587fbdd10d
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
3ab17233cf
cleanup
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
b3094412b1
better
2021-04-13 22:39:42 +03:00
Nikita Mikhaylov
64b4cd0e63
save
2021-04-13 22:39:41 +03:00
Nikita Mikhaylov
44ca65a9a4
save
2021-04-13 22:39:41 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Nikita Mikhaylov
7d7a2bc724
Merge pull request #21630 from nikitamikhaylov/pf-local
...
Parallel formatting for clickhouse-local
2021-03-16 14:37:40 +03:00
Nikita Mikhaylov
1f92c8ce58
parallel formatting everywhere
2021-03-15 14:51:24 +03:00
vdimir
1194d50e48
Try to fix test_storage_s3: crash in WriteBufferFromS3
2021-03-11 15:21:13 +03:00
Pervakov Grigorii
eb87afc4d4
recreate S3 client if credentials changed
2021-03-10 13:36:57 +03:00
alexey-milovidov
a299790522
Merge pull request #19748 from excitoon-favorites/s3ssec
...
Added Server Side Encryption Customer Keys support in S3 client
2021-02-15 15:23:14 +03:00
Vladimir Chebotarev
d729aacb09
Fixed table function S3 auto
compression mode.
2021-01-29 07:54:52 +03:00
Vladimir Chebotarev
579f8da573
Added SSE-C support in S3 client.
2021-01-28 09:32:41 +03:00
alexey-milovidov
7c240d0ca7
Merge pull request #17870 from excitoon-favorites/s3testiostream
...
Update of AWS C++ SDK
2021-01-28 07:12:52 +03:00
Vladimir Chebotarev
afdea5eff0
Changed handling of PocoHTTPClientConfiguration
in order to ensure AWS SDK is initialized.
2021-01-27 09:38:35 +03:00
Vladimir Chebotarev
8ba8e8efd5
Added prefix-based S3 endpoint settings.
2021-01-25 06:40:57 +03:00
Vladimir Chebotarev
5cbc25c647
Added global setting s3_max_connections
.
2021-01-05 13:34:55 +03:00
Nikita Mikhailov
c5f92e5096
better formatfactory
2020-12-30 06:07:30 +03:00
Nikita Mikhailov
d293ae5e8e
style
2020-12-28 20:18:53 +03:00
Nikita Mikhaylov
e0addac6fc
save changes
2020-12-15 00:56:46 +03:00
Pavel Kovalenko
59a5e63417
Merge remote-tracking branch 'origin/master' into s3-adaptive-single-part-upload
...
# Conflicts:
# src/Disks/S3/DiskS3.cpp
# src/Disks/S3/DiskS3.h
# src/Disks/S3/registerDiskS3.cpp
# src/IO/WriteBufferFromS3.cpp
# src/IO/WriteBufferFromS3.h
2020-12-11 18:28:41 +03:00
Vladimir Chebotarev
39d867e56d
Added proper authentication for S3 client ( #16856 )
2020-12-10 12:19:42 +03:00
Pavel Kovalenko
db7a87089b
Adaptive choose of single/multi part upload in WriteBufferFromS3.
2020-12-09 17:09:04 +03:00
alesapin
45f09fe293
Merge pull request #17220 from ianton-ru/s3_max_redirects
...
S3 max redirects
2020-11-24 12:13:10 +03:00
Anton Ivashkin
aa208a9d9f
Use only 's3_max_redirect' in params instead of all settings
2020-11-23 14:02:17 +03:00
Amos Bird
1d9d586e20
Make global_context consistent.
2020-11-20 18:23:14 +08:00
Anton Ivashkin
15b5d3a55a
Merge master
2020-11-13 19:32:35 +03:00
Anton Ivashkin
026f7e0a27
Add 's3_max_redirects' setting
2020-11-13 19:31:51 +03:00
Alexander Tokmakov
b94cc5c4e5
remove more stringstreams
2020-11-10 21:22:26 +03:00
Alexander Tokmakov
5cdfcfb307
remove other stringstreams
2020-11-09 22:12:44 +03:00
alexey-milovidov
f4ba5f1f9a
Merge pull request #16772 from ClickHouse/fix-stringstream
...
Fix "server failed to start" error
2020-11-08 14:27:08 +03:00
Alexey Milovidov
5314185e25
Merge branch 'master' into azat-optimize_skip_unused_shards-optimization
2020-11-08 00:17:59 +03:00
Alexey Milovidov
fd84d16387
Fix "server failed to start" error
2020-11-07 03:14:53 +03:00
Amos Bird
47fcd8bffb
Fix some problems introduced by refactoring.
2020-10-12 17:58:09 +08:00
Amos Bird
d2dcfc3f0d
Refactor processors.
2020-10-12 17:30:05 +08:00
Azat Khuzhin
b838214a35
Pass non-const SelectQueryInfo (and drop mutable qualifiers)
2020-10-02 22:42:35 +03:00
alesapin
17b86dec96
Recursive defaults
2020-10-02 15:38:50 +03:00
Vladimir Chebotarev
a38c7ff2a8
Somehow uncompressed PUT works and gzipped PUT doesn't, in S3 storage.
2020-09-30 15:04:42 +03:00
Vladimir Chebotarev
698768add0
Fixed format
argument in S3 storage when compression method is selected.
2020-09-30 13:55:44 +03:00
alexey-milovidov
eeefcd6493
Merge pull request #13337 from ClickHouse/refactor-pipes-3
...
Refactor Pipe
2020-08-14 04:12:22 +03:00
Vladimir Chebotarev
7371cf6bd1
Proper remote host checking in S3 redirects ( #13404 )
...
* Proper remote host checking in S3 redirects.
Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-08-11 09:09:08 +03:00
Nikolai Kochetov
2cca4d5fcf
Refactor Pipe [part 2].
2020-08-03 16:54:14 +03:00