Kruglov Pavel
c5b2e4cc23
Merge branch 'master' into improve-streaming-engines
2022-12-15 18:44:35 +01:00
pufit
5c52f26823
ya fix.
2022-12-12 00:39:08 -05:00
pufit
6979dc9f2f
dummy fix, additional test
2022-12-11 17:36:30 -05:00
pufit
1d6e77a29a
Move reader selection logic back to StorageFile
.
2022-12-11 16:15:41 -05:00
pufit
e38a93c45a
Fix UB, fix test.
2022-12-10 22:26:07 -05:00
pufit
2d87cc1a6c
Add storage_file_read_method
setting.
2022-12-08 18:02:29 -05:00
pufit
76401ad0b9
Test and codestyle fix.
2022-12-07 23:17:10 -05:00
pufit
9b46baa17d
Rewrite StorageFile
buffer creation with createReadBufferFromFileBase
.
...
Add file descriptor support for `createReadBufferFromFileBase`.
Fix file_size overflow in `createReadBufferFromFileBase`.
Fix `MMapReadBufferFromFileWithCache` file_size definition.
2022-12-07 22:31:32 -05:00
pufit
084e465d84
Use mmap only on regular files.
2022-12-04 23:39:23 -05:00
pufit
bc7a76a486
Added mmap for StorageFile
2022-12-04 17:27:28 -05:00
kssenii
5e01441f61
Show progress bar while reading from s3 table function
2022-11-21 17:56:02 +01:00
Kruglov Pavel
b124875257
Merge branch 'master' into improve-streaming-engines
2022-11-03 13:22:06 +01:00
avogar
8e13d1f1ec
Improve and refactor Kafka/StorageMQ/NATS and data formats
2022-10-28 16:41:10 +00:00
SmitaRKulkarni
96c8260230
Merge branch 'master' into 36316_Support_glob_for_recursive_directory_traversal
2022-10-24 18:34:19 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Smita Kulkarni
91433e5b9c
Added ** glob support for recursive directory traversal to filesystem and S3.
...
Implementation:
* Updated parseGlob to not add ‘/‘ restriction when ** is used.
* Updated S3 & filesystem to fetch files and not use regex match if glob is **.
Testing:
* Added a test for filesystem tests/queries/0_stateless/02459_glob_for_recursive_directory_traversal.sh
2022-10-17 09:04:25 +02:00
Alexey Milovidov
ab4db2d0c4
Fix 5/6 of trash
2022-09-19 08:50:53 +02:00
avogar
5ab87f1da4
Small refactoring
2022-08-19 16:42:23 +00:00
avogar
8dd54c043d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache
2022-08-17 11:47:40 +00:00
avogar
c4ff3ffeea
Rename settings
2022-08-15 12:45:18 +00:00
Kseniia Sumarokova
ec4a4d31ea
Fix style check
2022-08-08 11:23:57 +02:00
Kseniia Sumarokova
895639644e
Update src/Storages/StorageFile.cpp
2022-08-07 14:17:42 +02:00
flynn
4fa1762f96
Merge branch 'master' into file
2022-08-07 14:22:08 +08:00
flynn
384a7ae901
Fix read of StorageFile with virtual columns
2022-08-06 17:29:33 +00:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache
2022-07-21 18:59:14 +02:00
avogar
6b541aa98f
Fix WriteBuffer finalize when cancel insert into function
2022-07-21 12:18:37 +00:00
Kruglov Pavel
92995a832b
Revert "Fix WriteBuffer finalize in destructor when cacnel query"
2022-07-21 01:45:16 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache
2022-07-20 13:30:42 +02:00
avogar
5c16d6b553
Fix WriteBuffer finalize in destructor when cacnel query
2022-07-19 19:21:30 +00:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache
2022-07-14 12:29:54 +02:00
avogar
106f92dcdb
Fix tests
2022-06-28 16:13:42 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache
2022-06-24 16:10:25 +02:00
avogar
59c1c472cb
Better exception messages on wrong table engines/functions argument types
2022-06-23 20:04:06 +00:00
avogar
c14364e3d9
Check last modification time for URL function too
2022-06-21 17:18:14 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Alexey Milovidov
73709b0488
Revert "Revert "Add a setting to use more memory for zstd decompression""
2022-06-18 15:55:35 +03:00
alesapin
16e8b85fbf
Revert "Add a setting to use more memory for zstd decompression"
2022-06-18 14:08:14 +02:00
Alexey Milovidov
e20259e9ca
Merge pull request #37015 from wuxiaobai24/zstd_window_log_max
...
Add a setting to use more memory for zstd decompression
2022-06-18 04:19:27 +03:00
Nikolai Kochetov
8991f39412
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-06-02 17:00:08 +00:00
Azat Khuzhin
545a56ce45
Fix sinks with onException() handler
...
It is possible to call onException() even after onFinish(), in case of
onFinish() throws, and in this case onException() should be no-op for
such sinks.
Also there can be caveats with PartitionedSync.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-01 21:50:30 +03:00
Azat Khuzhin
02af58f41d
Fix possible "Cannot write to finalized buffer"
...
It is still possible to get this error since onException does not
finalize format correctly.
Here is an example of such error, that was found by CI [1]:
<details>
[ 2686 ] {fa01bf02-73f6-4f7f-b14f-e725de6d7f9b} <Fatal> : Logical error: 'Cannot write to finalized buffer'.
[ 34577 ] {} <Fatal> BaseDaemon: ########################################
[ 34577 ] {} <Fatal> BaseDaemon: (version 22.6.1.1, build id: AB8040A6769E01A0) (from thread 2686) (query_id: fa01bf02-73f6-4f7f-b14f-e725de6d7f9b) (query: insert into test_02302 select number from numbers(10) settings s3_truncate_on_insert=1;) Received signal Aborted (6)
[ 34577 ] {} <Fatal> BaseDaemon:
[ 34577 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fcbaa5a703b 0x7fcbaa586859 0xfad9bab 0xfad9e05 0xfaf6a3b 0x24a48c7f 0x258fb9b9 0x258f2004 0x258b88f4 0x258b863b 0x2581773d 0x258177ce 0x24bb5e98 0xfad01d6 0xfad0105 0x2419b11d 0xfad01d6 0xfad0105 0x2215afbb 0x2215aa48 0xfad01d6 0xfad0105 0xfcc265d 0x225cc546 0x249a1c40 0x249bc1b6 0x2685902c 0x26859505 0x269d7767 0x269d504c 0x7fcbaa75e609 0x7fcbaa683163
[ 34577 ] {} <Fatal> BaseDaemon: 3. raise @ 0x7fcbaa5a703b in ?
[ 34577 ] {} <Fatal> BaseDaemon: 4. abort @ 0x7fcbaa586859 in ?
[ 34577 ] {} <Fatal> BaseDaemon: 5. ./build_docker/../src/Common/Exception.cpp:47: DB::abortOnFailedAssertion(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfad9bab in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 6. ./build_docker/../src/Common/Exception.cpp:70: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xfad9e05 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 7. ./build_docker/../src/IO/WriteBuffer.h:0: DB::WriteBuffer::write(char const*, unsigned long) @ 0xfaf6a3b in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 8. ./build_docker/../src/Processors/Formats/Impl/ArrowBufferedStreams.cpp:47: DB::ArrowBufferedOutputStream::Write(void const*, long) @ 0x24a48c7f in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 9. long parquet::ThriftSerializer::Serialize<parquet::format::FileMetaData>(parquet::format::FileMetaData const*, arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) @ 0x258fb9b9 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 10. parquet::FileMetaData::FileMetaDataImpl::WriteTo(arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) const @ 0x258f2004 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 11. parquet::WriteFileMetaData(parquet::FileMetaData const&, arrow::io::OutputStream*) @ 0x258b88f4 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 12. parquet::ParquetFileWriter::~ParquetFileWriter() @ 0x258b863b in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 13. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x2581773d in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 14. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x258177ce in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 15. ./build_docker/../src/Processors/Formats/Impl/ParquetBlockOutputFormat.h:27: DB::ParquetBlockOutputFormat::~ParquetBlockOutputFormat() @ 0x24bb5e98 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 16. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 17. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:312: std::__1::unique_ptr<DB::WriteBuffer, std::__1::default_delete<DB::WriteBuffer> >::reset(DB::WriteBuffer*)
[ 34577 ] {} <Fatal> BaseDaemon: 18.2. inlined from ../contrib/libcxx/include/__memory/unique_ptr.h:269: ~unique_ptr
[ 34577 ] {} <Fatal> BaseDaemon: 18. ../src/Storages/StorageS3.cpp:566: DB::StorageS3Sink::~StorageS3Sink() @ 0x2419b11d in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 19. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 20. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 21. ./build_docker/../contrib/abseil-cpp/absl/container/internal/raw_hash_set.h:1662: absl::lts_20211102::container_internal::raw_hash_set<absl::lts_20211102::container_internal::FlatHashMapPolicy<StringRef, std::__1::shared_ptr<DB::SinkToStorage> >, absl::lts_20211102::hash_internal::Hash<StringRef>, std::__1::equal_to<StringRef>, std::__1::allocator<std::__1::pair<StringRef const, std::__1::shared_ptr<DB::SinkToStorage> > > >::destroy_slots() @ 0x2215afbb in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./build_docker/../contrib/libcxx/include/string:1445: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long() const
[ 34577 ] {} <Fatal> BaseDaemon: 22.2. inlined from ../contrib/libcxx/include/string:2231: ~basic_string
[ 34577 ] {} <Fatal> BaseDaemon: 22. ../src/Storages/PartitionedSink.h:14: DB::PartitionedSink::~PartitionedSink() @ 0x2215aa48 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 23. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 24. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 25. ./build_docker/../contrib/libcxx/include/vector:802: std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >::__base_destruct_at_end(std::__1::shared_ptr<DB::IProcessor>*) @ 0xfcc265d in /usr/bin/clickhouse
[ 34577 ] {} <Fatal> BaseDaemon: 26.1. inlined from ./build_docker/../contrib/libcxx/include/vector:402: ~vector
[ 34577 ] {} <Fatal> BaseDaemon: 26.2. inlined from ../src/QueryPipeline/QueryPipeline.cpp:29: ~QueryPipeline
[ 34577 ] {} <Fatal> BaseDaemon: 26. ../src/QueryPipeline/QueryPipeline.cpp:535: DB::QueryPipeline::reset() @ 0x225cc546 in /usr/bin/clickhouse
[ 614 ] {} <Fatal> Application: Child process was terminated by signal 6.
</details>
[1]: https://s3.amazonaws.com/clickhouse-test-reports/37542/8a224239c1d922158b4dc9f5d6609dca836dfd06/stress_test__undefined__actions_.html
Follow-up for: #36979
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-01 21:50:30 +03:00
Nikolai Kochetov
86fbb74703
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-31 18:07:47 +00:00
Nikolai Kochetov
1b85f2c1d6
Merge branch 'master' into refactor-read-metrics-and-callbacks
2022-05-25 16:27:40 +02:00
avogar
f782fa31c6
Merge branch 'master' of github.com:ClickHouse/ClickHouse into check-format-on-storage-creation
2022-05-25 08:42:54 +00:00
Nikolai Kochetov
3d84aae0ab
Better.
2022-05-24 20:06:08 +00:00
avogar
37b66c8a9e
Check format name on storage creation
2022-05-23 12:48:48 +00:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names
2022-05-23 12:14:20 +02:00
Nikolai Kochetov
56feef01e7
Move some resources
2022-05-20 19:49:31 +00:00
avogar
2d4b4b9008
Fix inserting defaults for missing values in columnar formats
2022-05-16 14:19:44 +00:00
avogar
68bb07d166
Better naming
2022-05-13 18:39:19 +00:00
avogar
b17fec659a
Improve performance and memory usage for select of subset of columns for some formats
2022-05-13 13:51:28 +00:00
wuxiaobai24
f3e573bfde
Merge remote-tracking branch 'upstream/master' into zstd_window_log_max
2022-05-13 20:42:54 +08:00
wuxiaobai24
4cd7e65d97
fix style check
2022-05-13 18:24:50 +08:00
wuxiaobai24
ab5636a46a
fix
2022-05-13 17:52:39 +08:00
kssenii
c84e0d13fa
Fix test
2022-05-11 14:36:34 +02:00
kssenii
61f2737e17
Merge master
2022-05-10 19:31:22 +02:00
kssenii
0feda67ec4
Fix progress for insert select
2022-05-10 15:04:08 +02:00
Anton Popov
e911900054
remove last mentions of data streams
2022-05-09 19:15:24 +00:00
wuxiaobai24
58755cb156
add zstd_window_log_max setting
2022-05-08 14:02:40 +08:00
avogar
02bd5f6542
Finalize write buffers in case of exception
2022-05-06 17:30:18 +00:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying
...
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
previously allowed.
Hence, this change
- removes shared_ptr_helper and as a result all inherited create() methods,
- instead, Storage objects are now created using make_shared<>() by the
caller (for that to work, many constructors had to be made public), and
- all Storage classes were marked as noncopyable using boost::noncopyable.
In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
2022-05-02 08:46:52 +02:00
Anton Popov
866a51a948
reduce number of stat calls in storage File
2022-04-25 22:13:15 +00:00
Kruglov Pavel
34c342fdd3
Merge pull request #36205 from Avogar/improve-globs
...
Some refactoring around schema inference with globs
2022-04-25 13:14:46 +02:00
avogar
1f252cedfe
Make better
2022-04-19 19:16:47 +00:00
Anton Popov
2de6668b3f
show names of erroneous files
2022-04-16 00:10:47 +00:00
avogar
1c065f8c7a
Some refactoring around schema inference with globs
2022-04-13 17:02:48 +00:00
xiedeyantu
9441111833
fix INSERT INTO table FROM INFILE does not display progress bar ( #35429 )
...
* fix INSERT INTO table FROM INFILE does not display progress bar
* Update StorageFile.cpp
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-23 14:24:59 +01:00
Anton Popov
fcdebea925
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-25 13:41:30 +03:00
Kruglov Pavel
dd863ca2a0
Merge branch 'master' into fix-url-globs
2022-02-16 12:45:31 +03:00
Maksim Kita
e2c8ba9ab2
Added performance test
2022-02-12 16:05:35 +00:00
avogar
bfa96463ca
Fix possible error 'file_size: Operation not supported'
2022-02-10 09:23:27 +03:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
avogar
84087c1554
Improve schema inference with globs
2022-02-09 19:16:15 +03:00
feng lv
7bf2243431
use LowCardinality for _file and _path virtual columns in StorageFile
...
fix
2022-02-04 14:15:56 +00:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
Kruglov Pavel
a9d0beb7ae
Fix data race in StorageFile ( #34113 )
...
* Fix data race in StorageFile
* Update StorageFile.h
* Fix
2022-01-31 11:58:40 +03:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
Kruglov Pavel
7873b4475f
Merge branch 'master' into autodetect-format
2022-01-25 10:56:52 +03:00
avogar
a6740d2f9a
Detect format and schema for stdin in clickhouse-local
2022-01-25 10:25:37 +03:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
avogar
af54ca4866
Fix partition write
2022-01-14 21:21:36 +03:00
avogar
253035a5df
Fix
2022-01-14 19:17:06 +03:00
avogar
89a181bd19
Make better
2022-01-14 18:16:18 +03:00
avogar
97788b9c21
Allow to create new files on insert for File/S3/HDFS engines
2021-12-29 21:19:13 +03:00
avogar
8112a71233
Implement schema inference for most input formats
2021-12-29 12:18:56 +03:00
Anton Popov
99ebabd822
Merge remote-tracking branch 'upstream/master' into HEAD
2021-12-17 19:02:29 +03:00
Alexey Milovidov
71926a3a97
Fix surprisingly bad code in function "file"
2021-12-13 07:57:54 +03:00
Anton Popov
6f4d9a53b2
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-12-01 15:54:33 +03:00
Raúl Marín
051dddd8df
Reduce dependencies on ASTIdentifier.h
...
Goes from rebuilding 483 objects to 165 when it's modified
2021-11-26 16:49:40 +01:00
Anton Popov
ccd78e3838
Merge remote-tracking branch 'upstream/master' into HEAD
2021-11-22 17:19:35 +03:00
Kruglov Pavel
d9c1a0c8ec
Merge branch 'master' into fix-write-buffers
2021-11-20 17:48:24 +03:00
Kruglov Pavel
1bbe0df945
Fix perf test
2021-11-16 11:41:50 +03:00
avogar
8f7be52ac4
Update test, remove commented code
2021-11-15 14:58:18 +03:00
avogar
e5661b9b62
Minor change
2021-11-11 21:09:21 +03:00
avogar
c521a9131a
Small refactoring of WriteBiffer-s
2021-11-11 02:11:18 +03:00
Anton Popov
a20922b2d3
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-11-09 15:36:25 +03:00
avogar
2dd4393ca1
Small refactoring in formats
2021-11-03 20:07:05 +03:00
kssenii
03d7476864
One more fix
2021-10-29 00:50:59 +03:00
kssenii
d39840376f
Review fixes
2021-10-28 15:44:12 +03:00
kssenii
1d743b9259
Partitoned write
2021-10-25 19:23:44 +03:00
Kseniia Sumarokova
c692155c7e
Merge pull request #30309 from kssenii/fix-file-symlinks
...
Fix symlinks in file table function
2021-10-20 11:09:51 +03:00
Kseniia Sumarokova
5633865df1
Update src/Storages/StorageFile.cpp
...
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
2021-10-17 14:16:40 +03:00
kssenii
059fc1de69
Allow symlinks in file storage
2021-10-17 09:11:06 +00:00
Nikolai Kochetov
a08c98d760
Move some files.
2021-10-16 17:03:50 +03:00
Nikolai Kochetov
ab28c6c855
Remove BlockInputStream interfaces.
2021-10-14 13:25:43 +03:00
Nikolai Kochetov
2957971ee3
Remove some last streams.
2021-10-13 21:22:02 +03:00
Nikolai Kochetov
ec18340351
Remove streams from formats.
2021-10-11 19:11:50 +03:00
Nikolai Kochetov
341553febd
Fix build.
2021-09-16 20:40:42 +03:00
Nikolai Kochetov
b997214620
Rename QueryPipeline to QueryPipelineBuilder.
2021-09-14 20:48:18 +03:00
Nikolai Kochetov
f569a3e3f7
Merge branch 'master' into rewrite-pushing-to-views
2021-09-09 20:30:23 +03:00
Anton Popov
4c388e3d84
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-09-09 14:10:16 +03:00
Nikolai Kochetov
66a76ab70f
Rewrite PushingToViewsBlockOutputStream part 6
2021-09-03 20:29:36 +03:00
kssenii
c5ee335c68
Fix tests
2021-08-22 21:41:09 +03:00
kssenii
14afb3c509
File progress for local fix
2021-08-22 20:33:55 +03:00
Anton Popov
e36736b50c
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-08-02 22:52:02 +03:00
Nikolai Kochetov
9b5a816b43
Merge branch 'master' into output-streams-to-processors
2021-07-26 18:03:11 +03:00
Nikolai Kochetov
0eb563dc1b
Fix more tests.
2021-07-26 17:47:29 +03:00
Nikolai Kochetov
fa1c223269
Fix some tests.
2021-07-26 13:08:40 +03:00
Alexey Milovidov
2f2dce399a
Fix style
2021-07-25 01:26:21 +03:00
Alexey Milovidov
215505b985
Remove unused error code
2021-07-24 19:51:56 +03:00
Alexey Milovidov
774f6d2617
Allow to read file descriptor multiple times in File storage
2021-07-24 19:50:03 +03:00
BoloniniD
2fa02061b7
Fix parallel reads
2021-07-24 15:31:35 +03:00
BoloniniD
07c57edbfc
Merge branch 'master' of github.com:ClickHouse/ClickHouse into pipe_reading
2021-07-23 23:39:38 +03:00
Nikolai Kochetov
9c92f43359
Update storages.
2021-07-23 22:33:59 +03:00
Nikolai Kochetov
2dc5c89b66
Update Storage::write
2021-07-23 17:25:35 +03:00
Nikolai Kochetov
f38de35b14
Rename some constants.
2021-07-21 19:13:17 +03:00
Nikolai Kochetov
179ec05a72
Remove some streams.
2021-07-20 21:18:43 +03:00
BoloniniD
49ee24e824
What if we replace unique_lock with shared?
2021-07-20 19:22:41 +03:00
Anton Popov
f99374cca6
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-07-20 18:20:21 +03:00
Nikita Mikhaylov
4d3f828beb
Merge pull request #26314 from kssenii/fix-hdfs-crash
...
Fix hdfs crash
2021-07-20 15:01:56 +03:00
BoloniniD
a9deeb6a78
Trying to fix freeze on union all
2021-07-18 15:55:24 +03:00
Alexey Milovidov
261a220227
Remove some code
2021-07-17 21:06:46 +03:00
BoloniniD
19b673fb3b
Style fix for error message
2021-07-17 15:30:03 +03:00
kssenii
61652acede
Add test
2021-07-14 22:49:16 +03:00
BoloniniD
407652457f
Added fstat for table fd + more fixes
2021-07-09 12:20:11 +03:00
Anton Popov
3ed7f5a6cc
dynamic subcolumns: add snapshot for storage
2021-07-09 06:15:41 +03:00
Anton Popov
072e65b728
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-07-07 17:20:38 +03:00
BoloniniD
0f584c3e76
Fix style and add check for seekability
2021-07-07 14:21:32 +03:00
BoloniniD
077c147e8f
First changes after review
2021-07-06 13:23:39 +03:00
BoloniniD
e988f35d86
Added read counter to checker
2021-07-04 21:15:40 +03:00
BoloniniD
bbe33f061e
Merge branch 'master' of github.com:ClickHouse/ClickHouse into pipe_reading
2021-07-04 15:36:49 +03:00
BoloniniD
6f5b4ba49e
First look
2021-07-01 17:04:10 +03:00
Raúl Marín
bfc122df64
Fix some typos in Storage classes
2021-06-28 19:03:56 +02:00
Anton Popov
205a23282b
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-06-07 19:38:22 +03:00
kssenii
de277f8ac4
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into progress-bar
2021-06-01 07:58:19 +00:00
Anton Popov
0bdf9d207c
Merge remote-tracking branch 'origin/sparse-serialization' into HEAD
2021-05-26 00:25:47 +03:00
kssenii
31107816bb
fs::absolute to fs::canonical
2021-05-24 20:37:54 +03:00
kssenii
9b8df78fdd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into poco-file-to-std-fs
2021-05-17 17:42:05 +03:00
kssenii
e9c3bc1836
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into progress-bar
2021-05-16 20:48:35 +00:00
kssenii
c25be65ebf
Refactor progress bar, adjust progress indication for reading from file
2021-05-14 06:16:31 +00:00
kssenii
02288359c5
Less manual concatenation of paths
2021-05-08 13:59:55 +03:00
Anton Popov
e44706911e
dynamic columns: better getting of sample block
2021-05-05 02:02:54 +03:00
kssenii
c33f358a9a
Better way to track progress, display progress for ch-client
2021-04-28 19:12:28 +00:00
kssenii
563f8a97c6
Fix build, typos check
2021-04-27 17:01:41 +00:00
kssenii
eeb71672a0
Change in Storages/*
2021-04-27 16:49:37 +03:00
kssenii
3a0372535e
Show progress only with --progress option
2021-04-27 13:21:42 +00:00
kssenii
1135c47082
Fixes
2021-04-27 11:47:57 +00:00
kssenii
5efd075ad2
Slightly bette
2021-04-27 00:24:04 +00:00
kssenii
ffcfe5f9a1
Progress bar for file table engine
2021-04-26 21:02:52 +00:00
feng lv
4ffe199d39
Implement table comments
2021-04-23 12:18:23 +00:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr ( #22297 )
...
* Replace all Context references with std::weak_ptr
* Fix shared context captured by value
* Fix build
* Fix Context with named sessions
* Fix copy context
* Fix gcc build
* Merge with master and fix build
* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Nikita Mikhailov
71a2c85d9a
better
2021-04-06 22:23:16 +03:00
Nikita Mikhailov
af85ce2fd1
better
2021-04-06 22:23:16 +03:00
Nikita Mikhailov
37f48d13b4
add test
2021-04-06 22:23:16 +03:00
Nikita Mikhailov
282775b10c
save
2021-04-06 22:23:16 +03:00
keenwolf
1a4961499b
the Native format is not supported after test
2021-04-06 22:23:16 +03:00
keenwolf
b95f54edce
Small fix
2021-04-06 22:23:16 +03:00
keenwolf
139bb678a9
Branch with Format isColumnOriented() or not
2021-04-06 22:23:15 +03:00
keenwolf
de365a5852
fix
2021-04-06 22:23:15 +03:00
keenwolf
6f9bad682f
Only read needed columns for formats as parquet etc
2021-04-06 22:23:15 +03:00
M0r64n
2f7d0ba926
Replace direct truncate with O_TRUNC flag
2021-02-20 03:27:23 +04:00
M0r64n
75117389ec
Add a couple of QOL file engine settings
2021-02-16 18:50:11 +04:00
Alexey Milovidov
1a90c569cb
Merge branch 'master' into revert-15259-revert-14962-log-avoid-deadlock
2021-01-01 21:39:08 +03:00
Nikita Mikhailov
ec0f70b8ef
cleanup
2020-12-30 08:39:08 +03:00
Nikita Mikhailov
60b4a36c4a
arcadia fix + live view fix + cleanup
2020-12-30 07:50:58 +03:00
Nikita Mikhailov
c5f92e5096
better formatfactory
2020-12-30 06:07:30 +03:00
alexey-milovidov
69719f743d
Merge branch 'master' into revert-15259-revert-14962-log-avoid-deadlock
2020-12-22 00:41:52 +03:00
nikitamikhaylov
12e624fd9a
fix tests
2020-12-15 00:56:48 +03:00
nikitamikhaylov
f7ac8bf542
rebase and fix tests
2020-12-15 00:56:48 +03:00
Nikita Mikhaylov
5a47928431
save
2020-12-15 00:56:47 +03:00
Nikita Mikhaylov
f40f3ced2a
fix JSONEachRowArray
2020-12-15 00:56:46 +03:00
Nikita Mikhaylov
e0addac6fc
save changes
2020-12-15 00:56:46 +03:00
Alexander Tokmakov
04ceaa66f2
implement ATTACH TABLE name FROM 'path/to/data/
2020-12-08 17:15:23 +03:00
tavplubix
67099f28ac
Merge pull request #16591 from ClickHouse/aku/create-file
...
Support `SETTINGS` clause for File engine
2020-11-09 14:15:42 +03:00
Alexey Milovidov
5314185e25
Merge branch 'master' into azat-optimize_skip_unused_shards-optimization
2020-11-08 00:17:59 +03:00
Alexander Kuzmenkov
3c60f6cec2
make a separate settings collection + some cleanup
2020-11-07 11:53:39 +03:00
Alexander Kuzmenkov
2efbcbaa14
Merge remote-tracking branch 'origin/master' into HEAD
2020-11-05 14:28:56 +03:00
Alexander Kuzmenkov
14f31f5117
add support of settings to URL, fix url and file table functions
2020-11-05 14:28:20 +03:00
Alexander Tokmakov
ac32809b6a
fix #16482
2020-11-02 19:40:39 +03:00
Alexander Kuzmenkov
99ee127620
Support SETTINGS
clause for File engine
...
Accept the usual user settings related to file formats.
Most of the diff are the mechanistic code changes required to allow
providing the required FormatSettings to the format factory. The File
engine then extracts these settings from the `CREATE` query, and specifies
them when creating the format parser.
2020-11-02 10:50:38 +03:00
Alexander Tokmakov
72b1339656
Revert "Revert "Write structure of table functions to metadata""
...
This reverts commit c65d1e5c70
.
2020-10-14 15:19:29 +03:00
tavplubix
c65d1e5c70
Revert "Write structure of table functions to metadata"
2020-10-14 13:59:29 +03:00
alexey-milovidov
f60ccb4edf
Merge pull request #14295 from ClickHouse/write_structure_of_table_functions
...
Write structure of table functions to metadata
2020-10-13 23:56:09 +03:00
Alexey Milovidov
5b482f4191
Cleanups
2020-10-10 19:31:10 +03:00
Azat Khuzhin
b838214a35
Pass non-const SelectQueryInfo (and drop mutable qualifiers)
2020-10-02 22:42:35 +03:00
alesapin
17b86dec96
Recursive defaults
2020-10-02 15:38:50 +03:00
Alexander Tokmakov
b0d99217fb
Merge branch 'master' into write_structure_of_table_functions
2020-09-27 14:26:47 +03:00
alexey-milovidov
47d150b116
Revert "Revert "Avoid deadlocks in Log/TinyLog""
2020-09-25 02:29:16 +03:00