Kruglov Pavel
a2805f8f44
Merge branch 'master' into formats-with-subcolumns
2023-07-04 23:27:03 +02:00
avogar
98aa6b317f
Support reading subcolumns from file/s3/hdfs/url/azureBlobStorage table functions
2023-07-04 21:17:26 +00:00
avogar
f88c1fdf44
Fixes
2023-06-29 11:59:09 +00:00
avogar
4eeb431003
Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-progress-bar-2
2023-06-28 18:53:08 +00:00
Andrey Zvonov
ae26ff27a1
Merge branch 'master' into zvonand-issue-49290
2023-06-28 12:13:05 +02:00
Alexey Milovidov
b8e6bd3299
Merge branch 'master' into refactor-subqueries-for-in
2023-06-26 06:05:12 +03:00
Azat Khuzhin
a7b14f87e0
Throw an error instead of silenty ignore storage_file_read_method=mmap in server
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-24 23:42:31 +03:00
Andrey Zvonov
0b6688ffe4
Merge branch 'master' into zvonand-issue-49290
2023-06-24 12:36:27 +02:00
avogar
c679dd400e
Make better
2023-06-23 13:43:40 +00:00
avogar
24fab7bfde
Remove old includes
2023-06-22 18:48:15 +00:00
avogar
cf082f2f9a
Use read_bytes/total_bytes_to_read for progress bar in s3/file/url/... table functions
2023-06-22 17:24:43 +00:00
Andrey Zvonov
18e7b029d5
Merge branch 'master' into zvonand-issue-49290
2023-06-22 17:20:24 +02:00
Sema Checherinda
d0bb985061
fix other classes based on SinkToStorage
2023-06-22 14:33:25 +02:00
Sema Checherinda
95349a405b
release buffers with exception context
2023-06-22 13:00:13 +02:00
Nikolai Kochetov
a940031878
Merge branch 'master' into refactor-subqueries-for-in
2023-06-22 12:18:48 +02:00
zvonand
b16c30e521
upd for review
2023-06-19 23:06:08 +02:00
avogar
d492acbcd2
Fix tests
2023-06-19 13:36:29 +00:00
Nikolai Kochetov
afa74f697c
Refactor a bit.
2023-06-16 19:38:50 +00:00
zvonand
1c10578f14
update to master
2023-06-16 21:30:47 +02:00
avogar
56789af0fa
Remove loggig
2023-06-16 15:55:05 +00:00
avogar
3209ebe34b
Improve progress bar for file/s3/hdfs/url table functions. Step 1
2023-06-16 15:51:18 +00:00
Andrey Zvonov
02cf8a1bad
Update comment StorageFile.cpp
2023-06-15 14:09:38 +03:00
Andrey Zvonov
2f572b7211
Merge branch 'master' into zvonand-issue-49290
2023-06-14 14:22:32 +02:00
zvonand
1d8013074d
fix style & black
2023-06-13 18:47:02 +02:00
avogar
2e1f56ae33
Address comments
2023-06-13 14:43:50 +00:00
Kruglov Pavel
e4838725e3
Merge branch 'master' into allow-skip-empty-files
2023-06-12 20:03:23 +02:00
Kruglov Pavel
873cee9451
Merge pull request #49626 from alekseygolub/renamefile
...
Added option to rename files, loaded via TableFunctionFile, after successful processing
2023-06-12 15:01:22 +02:00
zvonand
3e6d393e17
remove debug cerr
2023-06-12 12:06:21 +02:00
zvonand
eb9cdbcf7d
fix File test being flaky
2023-06-12 11:41:36 +02:00
zvonand
2c97a94892
fix hdfs + style update
2023-06-11 01:50:17 +02:00
Kruglov Pavel
bf28074d32
Merge branch 'master' into allow-skip-empty-files
2023-06-08 12:36:18 +02:00
Antonio Andelic
b11f744252
Correctly disable async insert with deduplication when it's not needed ( #50663 )
...
* Correctly disable async insert when it's not used
* Better
* Add comment
* Better
* Fix tests
---------
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
2023-06-07 20:33:08 +02:00
Kruglov Pavel
b83b057045
Merge branch 'master' into renamefile
2023-06-06 19:38:05 +02:00
zvonand
1a361ef306
works for file
2023-06-05 03:21:43 +02:00
Kruglov Pavel
0beca0336d
Merge pull request #49112 from ClickHouse/Avogar-patch-3
...
Fix possible terminate called for uncaught exception in some places
2023-05-31 16:55:43 +02:00
avogar
d4efbbfbd3
Allow to skip empty files in file/s3/url/hdfs table functions
2023-05-30 19:32:24 +00:00
Kruglov Pavel
f863dee8e7
Merge branch 'master' into renamefile
2023-05-30 12:26:40 +02:00
nikitakeba
f604fb82b2
Merge branch 'master' into add-reading-from-archives-support
2023-05-29 23:34:19 +03:00
Nikita Keba
c18bff58b3
fix style
2023-05-29 20:08:18 +00:00
Nikita Keba
564691e25b
add reading from archives
2023-05-25 00:00:32 +00:00
alekseygolub
2b68a6a22a
Fix style
2023-05-19 16:03:22 +00:00
alekseygolub
c85c3afa1f
Added option to rename files, loaded via TableFunctionFile, after success processing
2023-05-19 16:03:22 +00:00
SmitaRKulkarni
a91c793684
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs
2023-05-18 09:24:25 +02:00
Smita Kulkarni
fd58eac75a
Fixed max_threads datatype issue for builds
2023-05-13 10:22:37 +02:00
Smita Kulkarni
792565d858
Updated to ULL
2023-05-12 17:23:37 +02:00
Smita Kulkarni
ef1100bb90
Added include to fix build issue
2023-05-11 14:44:36 +02:00
SmitaRKulkarni
5c030c428c
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs
2023-05-10 09:51:38 +02:00
Alexey Milovidov
a2c4b8e23d
Disable mmap for server
2023-05-10 03:16:52 +02:00
Michael Kolupaev
3bd1489f18
Propagate input_format_parquet_preserve_order to parallelizeOutputAfterReading()
2023-05-05 04:20:27 +00:00
Michael Kolupaev
eb3b774ad0
Better control over Parquet row group size
2023-05-04 14:59:55 -07:00
Smita Kulkarni
8205398f31
Fixed comment
2023-05-02 16:31:39 +02:00
Smita Kulkarni
a5d47ea489
Fixed build issues
2023-04-30 19:01:06 +02:00
Smita Kulkarni
307aa127d4
Updated to calculate and send max_parsing_threads
2023-04-25 13:27:20 +02:00
Smita Kulkarni
b70878aa0e
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs
2023-04-24 19:39:36 +02:00
avogar
c503f6532c
Add more finalize() to avoid terminate
2023-04-24 15:11:36 +00:00
Igor Nikonov
8603807b57
Use generic way to parallelize output for file()
...
+ disable parallelization for storage Null
2023-04-15 12:35:24 +00:00
Igor Nikonov
1187534545
Simpler way to resize pipeline
2023-04-09 21:26:39 +00:00
Igor Nikonov
78038a3c2c
Fix: do not resize pipeline when there is no files to process (globs expands to empty set)
2023-04-07 11:34:04 +00:00
Igor Nikonov
96213fa464
Fix header
2023-04-06 22:17:09 +00:00
Igor Nikonov
2e139c21d2
Parallel reading in FROM file()
2023-04-06 21:57:03 +00:00
SmitaRKulkarni
d9c67a3380
Merge branch 'master' into 42192_Lower_parallel_parsing_threads_with_globs
2023-03-30 09:35:03 +02:00
Azat Khuzhin
33b13549ad
Separate out DirectoryMonitorSource as DistributedAsyncInsertSource
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-28 22:33:36 +01:00
Smita Kulkarni
3e9ab1276a
Updated to reduce parallel parsing based on number of files - When reading from multiple files reduce parallel parsing
2023-02-22 14:56:44 +01:00
Smita Kulkarni
3a6dea5e16
When reading from multiple files displace parallel parsing
...
Implementation:
* Added a new parameter to getInput & getInputFormat to disable parallel parsing.
* Currently this is used only by StorageFile as we have not seen degradation for other storages reading from multiple paths.
2023-02-21 17:03:00 +01:00
Sergei Trifonov
0d1ea05ff6
Merge pull request #45007 from ClickHouse/cancellable-mutex-integration
...
Fast shared mutex integration
2023-01-25 11:15:46 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Sergei Trifonov
0fbfa17863
Merge branch 'master' into cancellable-mutex-integration
2023-01-23 12:44:09 +01:00
Anton Popov
f40fd7a151
Add checks for compilation of regexps ( #45356 )
2023-01-17 23:46:04 +01:00
serxa
693489a8ad
review fixes
2023-01-12 15:51:04 +00:00
Kruglov Pavel
29240ef380
Merge pull request #43927 from pufit/mmap-for-storage-file
...
Added mmap for StorageFile
2023-01-11 21:25:02 +01:00
Kruglov Pavel
ce6962614d
Merge branch 'master' into mmap-for-storage-file
2023-01-10 17:34:01 +01:00
Maksim Kita
fbba28b31e
Analyzer aggregation without column fix
2023-01-10 16:49:55 +01:00
pufit
2d942af7b4
Fix codestyle, fix test.
2022-12-16 11:55:50 -05:00
pufit
b7df684762
Enum settings, fix else branch.
2022-12-15 18:08:19 -05:00
Kruglov Pavel
c5b2e4cc23
Merge branch 'master' into improve-streaming-engines
2022-12-15 18:44:35 +01:00
pufit
5c52f26823
ya fix.
2022-12-12 00:39:08 -05:00
pufit
6979dc9f2f
dummy fix, additional test
2022-12-11 17:36:30 -05:00
pufit
1d6e77a29a
Move reader selection logic back to StorageFile
.
2022-12-11 16:15:41 -05:00
pufit
e38a93c45a
Fix UB, fix test.
2022-12-10 22:26:07 -05:00
pufit
2d87cc1a6c
Add storage_file_read_method
setting.
2022-12-08 18:02:29 -05:00
pufit
76401ad0b9
Test and codestyle fix.
2022-12-07 23:17:10 -05:00
pufit
9b46baa17d
Rewrite StorageFile
buffer creation with createReadBufferFromFileBase
.
...
Add file descriptor support for `createReadBufferFromFileBase`.
Fix file_size overflow in `createReadBufferFromFileBase`.
Fix `MMapReadBufferFromFileWithCache` file_size definition.
2022-12-07 22:31:32 -05:00
pufit
084e465d84
Use mmap only on regular files.
2022-12-04 23:39:23 -05:00
pufit
bc7a76a486
Added mmap for StorageFile
2022-12-04 17:27:28 -05:00
kssenii
5e01441f61
Show progress bar while reading from s3 table function
2022-11-21 17:56:02 +01:00
Kruglov Pavel
b124875257
Merge branch 'master' into improve-streaming-engines
2022-11-03 13:22:06 +01:00
avogar
8e13d1f1ec
Improve and refactor Kafka/StorageMQ/NATS and data formats
2022-10-28 16:41:10 +00:00
SmitaRKulkarni
96c8260230
Merge branch 'master' into 36316_Support_glob_for_recursive_directory_traversal
2022-10-24 18:34:19 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Smita Kulkarni
91433e5b9c
Added ** glob support for recursive directory traversal to filesystem and S3.
...
Implementation:
* Updated parseGlob to not add ‘/‘ restriction when ** is used.
* Updated S3 & filesystem to fetch files and not use regex match if glob is **.
Testing:
* Added a test for filesystem tests/queries/0_stateless/02459_glob_for_recursive_directory_traversal.sh
2022-10-17 09:04:25 +02:00
Alexey Milovidov
ab4db2d0c4
Fix 5/6 of trash
2022-09-19 08:50:53 +02:00
avogar
5ab87f1da4
Small refactoring
2022-08-19 16:42:23 +00:00
avogar
8dd54c043d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache
2022-08-17 11:47:40 +00:00
avogar
c4ff3ffeea
Rename settings
2022-08-15 12:45:18 +00:00
Kseniia Sumarokova
ec4a4d31ea
Fix style check
2022-08-08 11:23:57 +02:00
Kseniia Sumarokova
895639644e
Update src/Storages/StorageFile.cpp
2022-08-07 14:17:42 +02:00
flynn
4fa1762f96
Merge branch 'master' into file
2022-08-07 14:22:08 +08:00
flynn
384a7ae901
Fix read of StorageFile with virtual columns
2022-08-06 17:29:33 +00:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache
2022-07-21 18:59:14 +02:00