thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
123,253 Commits 9,326 Branches 1,783 Tags 2.1 GiB
e67c002cb0
Commit Graph

315 Commits

Author SHA1 Message Date
Ziy1-Tan
1bb0d1519e Fix style 
Signed-off-by: Ziy1-Tan <ajb459684460@gmail.com>
 2023-05-02 16:54:14 +08:00
Ziy1-Tan
c93ceedbef Fix style 
Signed-off-by: Ziy1-Tan <ajb459684460@gmail.com>
 2023-05-02 10:38:37 +08:00
Ziy1-Tan
2c159061ed Support _path and _file virtual columns for table function url. 2023-05-01 21:40:30 +08:00
Kruglov Pavel
75a3b6c322
Fix build 2023-04-24 21:08:53 +02:00
Kruglov Pavel
8ff864cd8b
Fix 2023-04-24 19:12:50 +02:00
avogar
c503f6532c Add more finalize() to avoid terminate 2023-04-24 15:11:36 +00:00
avogar
0097230611 Better 2023-04-21 17:35:17 +00:00
avogar
0805b517ee Fix parsing failover options 2023-04-21 17:28:14 +00:00
avogar
944f54aadf Finish urlCluster, refactor code, reduce code duplication 2023-04-21 17:24:37 +00:00
avogar
c949f0ebf5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster 2023-04-21 14:13:33 +02:00
avogar
86686fbbc3 Fix conflicts 2023-04-21 14:11:18 +02:00
Michael Kolupaev
87be78e6de Better 2023-04-17 04:58:32 +00:00
Michael Kolupaev
e133633359 Parallel decoding with one row group per thread 2023-04-17 04:58:32 +00:00
Michael Kolupaev
683077890f Highly questionable refactoring (getInputMultistream() nonsense) 2023-04-17 04:58:32 +00:00
Michael Kolupaev
2d4fe85513 Something 2023-04-17 04:58:32 +00:00
kssenii
bb0beb7449 Merge remote-tracking branch 'upstream/master' into named-collections-finish 2023-03-17 13:02:36 +01:00
Antonio Andelic
a70ca31884 Merge branch 'master' into fix-url-progress-bar 2023-03-09 10:17:33 +00:00
kssenii
8f2d75cef8 Fix tests 2023-03-05 12:56:00 +01:00
Konstantin Bogdanov
1bbf5acd47
Pass headers from StorageURL to WriteBufferFromHTTP (#46996) 
* Pass headers from StorageURL to WriteBufferFromHTTP

* Add a test

* Lint

* `time.sleep(1)`

* Start echo server earlier

* Add proper handling for mock server start

* Automatic style fix

---------

Co-authored-by: robot-clickhouse <robot-clickhouse@users.noreply.github.com>
 2023-03-03 13:55:52 +01:00
Antonio Andelic
f540f7f6f9 Fix some tests 2023-03-01 12:45:00 +00:00
Antonio Andelic
45dc5dc25d No progress bar if no size 2023-02-28 15:25:28 +00:00
Antonio Andelic
56a126f7af Fix progress bar with URL 2023-02-24 14:49:14 +00:00
Robert Schulze
10af0b3e49
Reduce redundancies 2023-02-07 12:27:23 +00:00
Robert Schulze
84b9ff450f
Fix terribly broken, fragile and potentially cyclic linking 
Sorry for the clickbaity title. This is about static method
ConnectionTimeouts::getHTTPTimeouts(). It was be declared in header
IO/ConnectionTimeouts.h, and defined in header
IO/ConnectionTimeoutsContext.h (!). This is weird and caused issues with
linking on s390x (##45520). There was an attempt to fix some
inconsistencies (#45848) but neither did @Algunenano nor me at first
really understand why the definition is in the header.

Turns out that ConnectionTimeoutsContext.h is only #include'd from
source files which are part of the normal server build BUT NOT part of
the keeper standalone build (which must be enabled via CMake
-DBUILD_STANDALONE_KEEPER=1). This dependency was not documented and as
a result, some misguided workarounds were introduced earlier, e.g.
0341c6c54b

The deeper cause was that getHTTPTimeouts() is passed a "Context". This
class is part of the "dbms" libary which is deliberately not linked by
the standalone build of clickhouse-keeper. The context is only used to
read the settings and the "Settings" class is part of the
clickhouse_common library which is linked by clickhouse-keeper already.

To resolve this mess, this PR

- creates source file IO/ConnectionTimeouts.cpp and moves all
  ConnectionTimeouts definitions into it, including getHTTPTimeouts().

- breaks the wrong dependency by passing "Settings" instead of "Context"
  into getHTTPTimeouts().

- resolves the previous hacks
 2023-02-05 20:49:34 +00:00
attack204
7ed6bad097
Merge branch 'master' into urlCluster 2023-02-02 21:12:20 +08:00
Anton Popov
5c0307bc6a fix race in StorageURL and StorageHDFS 2023-01-24 12:34:43 +00:00
attack204
1f4139718a fix:style 2023-01-19 16:19:39 +08:00
attack204
f549380867 fix:style 2023-01-19 16:10:59 +08:00
attack204
7bd0010c50 fix conflict 2023-01-19 13:18:07 +08:00
attack204
31fa490822 fix conflict 2023-01-19 10:45:14 +08:00
attack204
e312cfa794 feature:urlCluster 2023-01-19 10:19:04 +08:00
Alexander Tokmakov
2d7773fccc Merge branch 'master' into text_log_add_pattern 2023-01-13 20:33:46 +01:00
kssenii
fcb042d80c Fix cland tidy 2023-01-05 12:04:07 +01:00
kssenii
67509aa2d5 Merge remote-tracking branch 'upstream/master' into use-new-named-collections-code-2 2023-01-03 16:41:30 +01:00
Kruglov Pavel
966f57ef68
Merge pull request #42777 from Avogar/improve-streaming-engines 
Refactor and Improve streaming engines Kafka/RabbitMQ/NATS and data formats
 2023-01-02 15:59:06 +01:00
Alexander Tokmakov
ca989e9212 less runtime format strings 2022-12-23 19:50:34 +01:00
kssenii
8d9bf77588 Fix tests 2022-12-19 13:56:23 +01:00
kssenii
326747b555 Fix tests 2022-12-19 12:16:50 +01:00
kssenii
0c63ce9731 Remove some more old code 2022-12-17 00:34:29 +01:00
kssenii
30547d2dcd Replace old named collections code for url 2022-12-17 00:24:05 +01:00
Kruglov Pavel
c5b2e4cc23
Merge branch 'master' into improve-streaming-engines 2022-12-15 18:44:35 +01:00
Alexey Milovidov
127631ee47
Merge branch 'master' into perf_experiment 2022-11-12 18:58:25 +01:00
Kseniia Sumarokova
ed44b20694
Merge pull request #42224 from kssenii/fit-http-buffer-retries 
Fix http buffer retries
 2022-11-04 11:50:17 +01:00
Kruglov Pavel
b124875257
Merge branch 'master' into improve-streaming-engines 2022-11-03 13:22:06 +01:00
kssenii
4fe4a07600 Add test 2022-11-03 12:14:08 +01:00
avogar
8e13d1f1ec Improve and refactor Kafka/StorageMQ/NATS and data formats 2022-10-28 16:41:10 +00:00
Raúl Marín
6e0a9452e7 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-25 15:25:06 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Raúl Marín
e60415d07d Make clang-tidy happy 2022-10-18 11:40:12 +02:00
kssenii
0a801dad2a Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-23 19:39:07 +02:00
kssenii
420ac4eb43 s3 header auth in ast 2022-09-13 15:13:28 +02:00
kssenii
e51313b6b3 Get rid of static threadpools 2022-09-07 17:48:11 +02:00
Frank Chen
f17d56b528 Merge branch 'master' into tracing_context_propagation 2022-08-30 14:24:36 +08:00
Alexey Milovidov
6e564b18bf
Merge pull request #40600 from FrankChen021/check_url_arg 
Validate the CompressionMethod parameter of URL table engine
 2022-08-27 19:29:55 +03:00
Frank Chen
2e874b384d
Update src/Storages/StorageURL.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-08-26 10:34:46 +08:00
avogar
cef2972af4 Better format detection for url table function/engine 2022-08-25 18:31:06 +00:00
Frank Chen
92f7ca3616 Move TracingContextOnThread::current() out of class for simplicity 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:23:56 +08:00
Frank Chen
bb00dcc19b Remove using namespace from header 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:20:13 +08:00
Frank Chen
c9ea4f9f77 Change compression_method from String to CompressionMethod 2022-08-25 19:18:04 +08:00
Frank Chen
4c1fe0d623 validate parameter 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 14:49:17 +08:00
Frank Chen
99c37ce6c6
Merge branch 'master' into tracing_context_propagation 2022-08-25 10:07:16 +08:00
Frank Chen
cd19366b44 Move classes into DB::OpenTelemetry namespace 2022-08-24 16:41:40 +08:00
avogar
5ab87f1da4 Small refactoring 2022-08-19 16:42:23 +00:00
Frank Chen
a3b6ad2a65
Merge branch 'master' into tracing_context_propagation 2022-08-18 20:59:07 +08:00
avogar
8dd54c043d Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache 2022-08-17 11:47:40 +00:00
avogar
c4ff3ffeea Rename settings 2022-08-15 12:45:18 +00:00
avogar
9b1a267203 Refactor, remove TTL, add size limit, add system table and system query 2022-08-05 16:20:15 +00:00
Frank Chen
7d8a8f26cd Fix build check 2022-08-02 13:31:36 +08:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache 2022-07-21 18:59:14 +02:00
avogar
6b541aa98f Fix WriteBuffer finalize when cancel insert into function 2022-07-21 12:18:37 +00:00
Kruglov Pavel
92995a832b
Revert "Fix WriteBuffer finalize in destructor when cacnel query" 2022-07-21 01:45:16 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache 2022-07-20 13:30:42 +02:00
avogar
5c16d6b553 Fix WriteBuffer finalize in destructor when cacnel query 2022-07-19 19:21:30 +00:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache 2022-07-14 12:29:54 +02:00
Frank Chen
93dc109e36 Fix code 2022-07-07 17:44:19 +08:00
Frank Chen
57a7e4a7c9 Remove old API reference 2022-07-07 17:42:35 +08:00
avogar
106f92dcdb Fix tests 2022-06-28 16:13:42 +00:00
avogar
5155262a16 Add some additional information to cache keys 2022-06-27 12:43:24 +00:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache 2022-06-24 16:10:25 +02:00
avogar
59c1c472cb Better exception messages on wrong table engines/functions argument types 2022-06-23 20:04:06 +00:00
Kseniia Sumarokova
af2150a68d
Merge pull request #38176 from kssenii/url-headers-in-ast 
Add ability to pass headers to url table function / storage via sql
 2022-06-23 10:15:07 +02:00
kssenii
0acc95fe22 Review fixes 2022-06-22 16:55:47 +02:00
avogar
c14364e3d9 Check last modification time for URL function too 2022-06-21 17:18:14 +00:00
avogar
d37ad2e6de Implement cache for schema inference for file/s3/hdfs/url 2022-06-21 13:02:48 +00:00
kssenii
4656dd3de3 Headers for url in ast 2022-06-21 14:25:05 +02:00
Alexey Milovidov
73709b0488
Revert "Revert "Add a setting to use more memory for zstd decompression"" 2022-06-18 15:55:35 +03:00
alesapin
16e8b85fbf
Revert "Add a setting to use more memory for zstd decompression" 2022-06-18 14:08:14 +02:00
Alexey Milovidov
e20259e9ca
Merge pull request #37015 from wuxiaobai24/zstd_window_log_max 
Add a setting to use more memory for zstd decompression
 2022-06-18 04:19:27 +03:00
Nikolai Kochetov
8991f39412 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 17:00:08 +00:00
Azat Khuzhin
545a56ce45 Fix sinks with onException() handler 
It is possible to call onException() even after onFinish(), in case of
onFinish() throws, and in this case onException() should be no-op for
such sinks.

Also there can be caveats with PartitionedSync.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-01 21:50:30 +03:00
Azat Khuzhin
02af58f41d Fix possible "Cannot write to finalized buffer" 
It is still possible to get this error since onException does not
finalize format correctly.

Here is an example of such error, that was found by CI [1]:

<details>

    [ 2686 ] {fa01bf02-73f6-4f7f-b14f-e725de6d7f9b} <Fatal> : Logical error: 'Cannot write to finalized buffer'.
    [ 34577 ] {} <Fatal> BaseDaemon: ########################################
    [ 34577 ] {} <Fatal> BaseDaemon: (version 22.6.1.1, build id: AB8040A6769E01A0) (from thread 2686) (query_id: fa01bf02-73f6-4f7f-b14f-e725de6d7f9b) (query: insert into test_02302 select number from numbers(10) settings s3_truncate_on_insert=1;) Received signal Aborted (6)
    [ 34577 ] {} <Fatal> BaseDaemon:
    [ 34577 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fcbaa5a703b 0x7fcbaa586859 0xfad9bab 0xfad9e05 0xfaf6a3b 0x24a48c7f 0x258fb9b9 0x258f2004 0x258b88f4 0x258b863b 0x2581773d 0x258177ce 0x24bb5e98 0xfad01d6 0xfad0105 0x2419b11d 0xfad01d6 0xfad0105 0x2215afbb 0x2215aa48 0xfad01d6 0xfad0105 0xfcc265d 0x225cc546 0x249a1c40 0x249bc1b6 0x2685902c 0x26859505 0x269d7767 0x269d504c 0x7fcbaa75e609 0x7fcbaa683163
    [ 34577 ] {} <Fatal> BaseDaemon: 3. raise @ 0x7fcbaa5a703b in ?
    [ 34577 ] {} <Fatal> BaseDaemon: 4. abort @ 0x7fcbaa586859 in ?
    [ 34577 ] {} <Fatal> BaseDaemon: 5. ./build_docker/../src/Common/Exception.cpp:47: DB::abortOnFailedAssertion(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfad9bab in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 6. ./build_docker/../src/Common/Exception.cpp:70: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xfad9e05 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 7. ./build_docker/../src/IO/WriteBuffer.h:0: DB::WriteBuffer::write(char const*, unsigned long) @ 0xfaf6a3b in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 8. ./build_docker/../src/Processors/Formats/Impl/ArrowBufferedStreams.cpp:47: DB::ArrowBufferedOutputStream::Write(void const*, long) @ 0x24a48c7f in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 9. long parquet::ThriftSerializer::Serialize<parquet::format::FileMetaData>(parquet::format::FileMetaData const*, arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) @ 0x258fb9b9 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 10. parquet::FileMetaData::FileMetaDataImpl::WriteTo(arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) const @ 0x258f2004 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 11. parquet::WriteFileMetaData(parquet::FileMetaData const&, arrow::io::OutputStream*) @ 0x258b88f4 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 12. parquet::ParquetFileWriter::~ParquetFileWriter() @ 0x258b863b in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 13. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x2581773d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 14. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x258177ce in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 15. ./build_docker/../src/Processors/Formats/Impl/ParquetBlockOutputFormat.h:27: DB::ParquetBlockOutputFormat::~ParquetBlockOutputFormat() @ 0x24bb5e98 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 16. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 17. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:312: std::__1::unique_ptr<DB::WriteBuffer, std::__1::default_delete<DB::WriteBuffer> >::reset(DB::WriteBuffer*)
    [ 34577 ] {} <Fatal> BaseDaemon: 18.2. inlined from ../contrib/libcxx/include/__memory/unique_ptr.h:269: ~unique_ptr
    [ 34577 ] {} <Fatal> BaseDaemon: 18. ../src/Storages/StorageS3.cpp:566: DB::StorageS3Sink::~StorageS3Sink() @ 0x2419b11d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 19. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 20. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 21. ./build_docker/../contrib/abseil-cpp/absl/container/internal/raw_hash_set.h:1662: absl::lts_20211102::container_internal::raw_hash_set<absl::lts_20211102::container_internal::FlatHashMapPolicy<StringRef, std::__1::shared_ptr<DB::SinkToStorage> >, absl::lts_20211102::hash_internal::Hash<StringRef>, std::__1::equal_to<StringRef>, std::__1::allocator<std::__1::pair<StringRef const, std::__1::shared_ptr<DB::SinkToStorage> > > >::destroy_slots() @ 0x2215afbb in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./build_docker/../contrib/libcxx/include/string:1445: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long() const
    [ 34577 ] {} <Fatal> BaseDaemon: 22.2. inlined from ../contrib/libcxx/include/string:2231: ~basic_string
    [ 34577 ] {} <Fatal> BaseDaemon: 22. ../src/Storages/PartitionedSink.h:14: DB::PartitionedSink::~PartitionedSink() @ 0x2215aa48 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 23. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 24. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 25. ./build_docker/../contrib/libcxx/include/vector:802: std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >::__base_destruct_at_end(std::__1::shared_ptr<DB::IProcessor>*) @ 0xfcc265d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 26.1. inlined from ./build_docker/../contrib/libcxx/include/vector:402: ~vector
    [ 34577 ] {} <Fatal> BaseDaemon: 26.2. inlined from ../src/QueryPipeline/QueryPipeline.cpp:29: ~QueryPipeline
    [ 34577 ] {} <Fatal> BaseDaemon: 26. ../src/QueryPipeline/QueryPipeline.cpp:535: DB::QueryPipeline::reset() @ 0x225cc546 in /usr/bin/clickhouse
    [ 614 ] {} <Fatal> Application: Child process was terminated by signal 6.

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37542/8a224239c1d922158b4dc9f5d6609dca836dfd06/stress_test__undefined__actions_.html

Follow-up for: #36979

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-01 21:50:30 +03:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
avogar
f782fa31c6 Merge branch 'master' of github.com:ClickHouse/ClickHouse into check-format-on-storage-creation 2022-05-25 08:42:54 +00:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
avogar
37b66c8a9e Check format name on storage creation 2022-05-23 12:48:48 +00:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names 2022-05-23 12:14:20 +02:00
Nikolai Kochetov
56feef01e7 Move some resources 2022-05-20 19:49:31 +00:00
avogar
78069765bb Try to remove code duplication 2022-05-20 13:20:52 +00:00
avogar
2d4b4b9008 Fix inserting defaults for missing values in columnar formats 2022-05-16 14:19:44 +00:00