avogar
d11cd0dc30
Fix tests
2023-07-05 17:56:03 +00:00
avogar
98aa6b317f
Support reading subcolumns from file/s3/hdfs/url/azureBlobStorage table functions
2023-07-04 21:17:26 +00:00
Kseniia Sumarokova
a8e2eb9686
Merge branch 'master' into minor_improve_hdfs_read_buffer
2023-07-01 13:19:21 +02:00
Kseniia Sumarokova
349083e673
Update ReadBufferFromHDFS.cpp
2023-06-30 13:54:57 +02:00
avogar
f88c1fdf44
Fixes
2023-06-29 11:59:09 +00:00
avogar
4eeb431003
Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-progress-bar-2
2023-06-28 18:53:08 +00:00
李扬
0e15266170
Merge branch 'master' into improve_geohash
2023-06-26 14:54:44 +08:00
taiyang-li
769169f820
fix heap overflow in read buffer from hdfs
2023-06-25 18:26:53 +08:00
Andrey Zvonov
0b6688ffe4
Merge branch 'master' into zvonand-issue-49290
2023-06-24 12:36:27 +02:00
avogar
938a9ee3c1
Better
2023-06-23 14:03:59 +00:00
avogar
c679dd400e
Make better
2023-06-23 13:43:40 +00:00
avogar
24fab7bfde
Remove old includes
2023-06-22 18:48:15 +00:00
avogar
cf082f2f9a
Use read_bytes/total_bytes_to_read for progress bar in s3/file/url/... table functions
2023-06-22 17:24:43 +00:00
Andrey Zvonov
18e7b029d5
Merge branch 'master' into zvonand-issue-49290
2023-06-22 17:20:24 +02:00
Sema Checherinda
d0bb985061
fix other classes based on SinkToStorage
2023-06-22 14:33:25 +02:00
Sema Checherinda
95349a405b
release buffers with exception context
2023-06-22 13:00:13 +02:00
Sema Checherinda
01de36f1fa
Merge pull request #50395 from CheSema/better-log
...
require `finalize()` call before d-tor for all writes buffers
2023-06-21 21:12:02 +02:00
avogar
1d80072c21
Fix tests and builds
2023-06-20 13:01:47 +00:00
zvonand
4e61f93fde
upd
2023-06-19 23:44:59 +02:00
zvonand
b25555d445
update style
2023-06-17 13:49:15 +02:00
zvonand
1c10578f14
update to master
2023-06-16 21:30:47 +02:00
Sema Checherinda
e8d4dedbcc
fix build and tests
2023-06-16 19:52:52 +02:00
avogar
3209ebe34b
Improve progress bar for file/s3/hdfs/url table functions. Step 1
2023-06-16 15:51:18 +00:00
avogar
870f3d1270
Fix comments
2023-06-15 12:59:46 +00:00
Andrey Zvonov
2f572b7211
Merge branch 'master' into zvonand-issue-49290
2023-06-14 14:22:32 +02:00
zvonand
1d8013074d
fix style & black
2023-06-13 18:47:02 +02:00
avogar
2e1f56ae33
Address comments
2023-06-13 14:43:50 +00:00
Andrey Zvonov
cd1a3916a6
cleanup HDFS
2023-06-12 13:28:17 +02:00
zvonand
7d7bd5bb66
update comment describing workflow
2023-06-12 11:50:23 +02:00
zvonand
2c97a94892
fix hdfs + style update
2023-06-11 01:50:17 +02:00
Kruglov Pavel
bf28074d32
Merge branch 'master' into allow-skip-empty-files
2023-06-08 12:36:18 +02:00
Antonio Andelic
b11f744252
Correctly disable async insert with deduplication when it's not needed ( #50663 )
...
* Correctly disable async insert when it's not used
* Better
* Add comment
* Better
* Fix tests
---------
Co-authored-by: Nikita Mikhaylov <mikhaylovnikitka@gmail.com>
2023-06-07 20:33:08 +02:00
Kruglov Pavel
f99a7366da
Fix tests
2023-06-01 16:56:18 +02:00
Kruglov Pavel
0beca0336d
Merge pull request #49112 from ClickHouse/Avogar-patch-3
...
Fix possible terminate called for uncaught exception in some places
2023-05-31 16:55:43 +02:00
avogar
d4efbbfbd3
Allow to skip empty files in file/s3/url/hdfs table functions
2023-05-30 19:32:24 +00:00
Sergei Trifonov
0d1f2e297b
Unify priorities: lower value means higher priority ( #50205 )
2023-05-26 15:55:30 +02:00
kevinyhzou
15b847b410
init file_size as 0
2023-05-24 09:49:12 +08:00
Kruglov Pavel
b5cad024e0
Merge branch 'master' into urlCluster
2023-05-22 14:59:34 +02:00
taiyang-li
1f91a75b54
fix bugs of geoHashEncode
2023-05-22 15:52:58 +08:00
kevinyhzou
e6d6d41f22
minor improve read buffer of hdfs
2023-05-22 12:23:31 +08:00
kssenii
3e42ee7f2b
Get rid of finalize callback in object storages
2023-05-19 17:29:37 +02:00
avogar
334f062fa0
fix style
2023-05-15 16:39:26 +00:00
avogar
f9e0eb47d7
Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster
2023-05-15 16:35:24 +00:00
Alexey Milovidov
5a44dc26e7
Fixes for clang-17
2023-05-13 02:57:31 +02:00
avogar
70a8fd2c50
Fix schema inference with named collection, refactor Cluster table functions
2023-05-12 13:58:45 +00:00
avogar
3ee8de792c
Merge branch 'master' of github.com:ClickHouse/ClickHouse into urlCluster
2023-05-11 12:46:20 +00:00
kssenii
ecfbf1e304
Remove dependency from DB::Context in readers
2023-05-02 21:45:27 +02:00
avogar
c503f6532c
Add more finalize() to avoid terminate
2023-04-24 15:11:36 +00:00
avogar
2949ceced1
Fix adding structure to cluster table functions, make it better
2023-04-24 13:20:04 +00:00
Alexey Milovidov
31000978f4
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards
2023-04-23 16:18:48 +03:00
avogar
447189a6ca
Better
2023-04-21 17:54:09 +00:00
avogar
944f54aadf
Finish urlCluster, refactor code, reduce code duplication
2023-04-21 17:24:37 +00:00
Kruglov Pavel
2ad161d2b7
Merge branch 'master' into non-blocking-connect
2023-04-19 13:39:40 +02:00
Alexey Milovidov
4d74e3da4d
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards
2023-04-12 22:13:52 +03:00
Azat Khuzhin
79b83c4fd2
Remove superfluous includes of logger_userful.h from headers
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-04-10 17:59:30 +02:00
Alexey Milovidov
db61140607
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards
2023-04-09 08:03:23 +03:00
Kruglov Pavel
3ee12e21fb
Merge branch 'master' into non-blocking-connect
2023-03-23 20:53:44 +01:00
avogar
38e44861ae
Fix possible race conditions
2023-03-21 16:01:54 +00:00
Robert Schulze
5b036a1a3b
More preparation for libcxx(abi), llvm, clang-tidy 16 (follow-up to #47722 )
2023-03-20 12:55:03 +00:00
Alexander Tokmakov
ed08f8f5c5
Merge branch 'master' into revert_25674
2023-03-12 02:33:25 +03:00
Alexander Tokmakov
7b1b238d0b
Revert "Merge pull request #25674 from amosbird/distributedreturnconnection"
...
This reverts commit 5ffd99dfd42796aa333f
2023-03-11 19:09:47 +01:00
Maksim Kita
0358cb36d8
Fixed tests
2023-03-11 11:51:54 +01:00
zk_kiger
4f05eeeff9
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards
2023-02-27 17:15:02 +08:00
kigerzhang
f74f3718a0
add skip_unavailable_shards setting for table function cluste
2023-02-23 17:05:51 +08:00
kssenii
3067c1d723
Merge remote-tracking branch 'upstream/master' into resubmit-prefetches
2023-02-11 11:36:23 +01:00
Sergei Trifonov
2931c3bbe1
Merge pull request #45711 from ClickHouse/io-scheduler-integration
...
Integrate IO scheduler with buffers for remote reads and writes
2023-02-11 10:26:58 +01:00
serxa
8c3f85e58d
more review fixes
2023-02-10 17:39:28 +00:00
serxa
acabee8554
review fixes
2023-02-09 20:17:46 +00:00
kssenii
b0b865c32e
Resubmit prefetches
2023-02-08 21:26:24 +01:00
Sergei Trifonov
b0fc551c27
Merge branch 'master' into io-scheduler-integration
2023-02-08 18:16:00 +01:00
Robert Schulze
84b9ff450f
Fix terribly broken, fragile and potentially cyclic linking
...
Sorry for the clickbaity title. This is about static method
ConnectionTimeouts::getHTTPTimeouts(). It was be declared in header
IO/ConnectionTimeouts.h, and defined in header
IO/ConnectionTimeoutsContext.h (!). This is weird and caused issues with
linking on s390x (##45520). There was an attempt to fix some
inconsistencies (#45848 ) but neither did @Algunenano nor me at first
really understand why the definition is in the header.
Turns out that ConnectionTimeoutsContext.h is only #include'd from
source files which are part of the normal server build BUT NOT part of
the keeper standalone build (which must be enabled via CMake
-DBUILD_STANDALONE_KEEPER=1). This dependency was not documented and as
a result, some misguided workarounds were introduced earlier, e.g.
0341c6c54b
2023-02-05 20:49:34 +00:00
Raúl Marín
7c31cb7adc
Proper includes for ConnectionTimeoutsContext.h
2023-01-31 16:11:32 +01:00
Sergei Trifonov
5c2686fbf0
Merge branch 'master' into io-scheduler-integration
2023-01-27 19:56:23 +01:00
serxa
21e871a0a4
integrate IO scheduler with buffers
2023-01-27 18:47:22 +00:00
Anton Popov
b58b73b0e7
Merge pull request #45529 from CurtizJ/fix-storage-s3-race
...
Try to fix test `test_storage_s3/test.py::test_wrong_s3_syntax` (race in `StorageS3`)
2023-01-26 14:21:32 +01:00
Kruglov Pavel
cd3d3fdd68
Merge pull request #45483 from Avogar/fix-hdfs-cluster
...
Fix schema inference from insertion table in hdfsCluster
2023-01-25 20:24:51 +01:00
avogar
07ad750366
Fix tests
2023-01-24 15:08:19 +00:00
Anton Popov
5c0307bc6a
fix race in StorageURL and StorageHDFS
2023-01-24 12:34:43 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Alexander Gololobov
91fb1eab8c
Merge pull request #45373 from ClickHouse/throttler-metrics
...
Add detailed profile events for throttling
2023-01-22 11:20:45 +01:00
avogar
eed1db7e07
Fix schema inference in hdfsCluster
2023-01-20 21:17:35 +00:00
Anton Popov
f40fd7a151
Add checks for compilation of regexps ( #45356 )
2023-01-17 23:46:04 +01:00
serxa
ce7e22b87b
add detailed profile events for throttling
2023-01-17 18:29:24 +00:00
Maksim Kita
fbba28b31e
Analyzer aggregation without column fix
2023-01-10 16:49:55 +01:00
Kseniia Sumarokova
b22b51213f
Merge pull request #44868 from kssenii/improve-async-buffer-from-remote
...
Small perf improvement for async buffer from remote
2023-01-09 13:31:15 +01:00
Nikita Mikhaylov
857799fbca
Parallel distributed insert select with s3Cluster [3] ( #44955 )
...
* Revert "Revert "Resurrect parallel distributed insert select with s3Cluster (#41535 )""
This reverts commit b8d9066004
2023-01-09 13:30:32 +01:00
kssenii
bda8f0d410
Better async buffer from remote
2023-01-03 15:27:37 +01:00
xiedeyantu
ec6698395e
fix skip_unavailable_shards does not work using hdfsCluster table function
2022-11-15 13:25:15 +08:00
Kruglov Pavel
21d50f76ea
Merge pull request #41979 from Avogar/s3-cluster-schema-inference
...
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-11-01 14:00:21 +01:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Kruglov Pavel
df96acc80f
Fix style
2022-10-19 21:48:45 +02:00
avogar
2c2f977096
Make better
2022-10-18 13:03:55 +00:00
avogar
75c4ef0be7
Merge branch 'master' of github.com:ClickHouse/ClickHouse into s3-cluster-schema-inference
2022-10-18 11:58:12 +00:00
Kruglov Pavel
3d9f46a1e7
Merge branch 'master' into s3-cluster-schema-inference
2022-10-14 22:07:54 +02:00
Ilya Yatsishin
3361b27965
less
2022-10-14 14:27:52 +02:00
Ilya Yatsishin
bbf88b66a0
more
2022-10-13 11:42:35 +02:00
Ilya Yatsishin
e7ca30cd11
fix typo
2022-10-13 10:10:13 +02:00
Ilya Yatsishin
3201c08bb5
style
2022-10-11 13:54:03 +02:00
Ilya Yatsishin
28c7a22428
address review comments
2022-10-11 11:59:48 +02:00
Ilya Yatsishin
95c5c8a744
Fail fast on empty URL
2022-10-10 14:50:09 +02:00
Alexander Tokmakov
b8d9066004
Revert "Resurrect parallel distributed insert select with s3Cluster ( #41535 )"
...
This reverts commit 860e34e760
2022-10-07 15:53:30 +02:00
Robert Schulze
78be400ac0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup
2022-10-06 12:22:43 +00:00
Nikita Mikhaylov
860e34e760
Resurrect parallel distributed insert select with s3Cluster ( #41535 )
2022-10-06 13:47:32 +02:00
avogar
c74b5c8126
Fix schema inference in s3Cluster and improve in hdfsCluster
2022-09-30 16:59:17 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
kssenii
ab702e43fc
Merge remote-tracking branch 'upstream/master' into fix-thread-status
2022-09-23 15:21:33 +02:00
kssenii
52ef3758c4
Merge remote-tracking branch 'upstream/master' into fix-thread-status
2022-09-13 16:34:31 +02:00
Azat Khuzhin
4e3135383b
Fix clang-tidy warnings (from clang-15)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-10 21:38:35 +02:00
kssenii
e51313b6b3
Get rid of static threadpools
2022-09-07 17:48:11 +02:00
kssenii
4fa291cf59
Fix hdfs read buffer too
2022-09-01 16:46:46 +02:00
Alexander Tokmakov
f9f85a0e8b
Revert "Parallel distributed insert select from *Cluster table functions ( #39107 )"
...
This reverts commit d3cc234986
2022-08-24 15:17:15 +03:00
Kruglov Pavel
72f02bd6eb
Merge pull request #40414 from Avogar/improve-schema-inference-cache
...
Improve schema inference cache, respect format settings that can change the schema
2022-08-23 17:04:58 +02:00
Alexey Milovidov
74e1f4dc61
Fix clang-tidy
2022-08-20 17:09:20 +02:00
avogar
5ab87f1da4
Small refactoring
2022-08-19 16:42:23 +00:00
avogar
8dd54c043d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache
2022-08-17 11:47:40 +00:00
avogar
c4ff3ffeea
Rename settings
2022-08-15 12:45:18 +00:00
Nikita Mikhaylov
d3cc234986
Parallel distributed insert select from *Cluster table functions ( #39107 )
2022-08-15 12:41:17 +02:00
Alexey Milovidov
ddd79e6c05
Merge pull request #40038 from ClickHouse/non-significant-changes
...
Non-significant changes
2022-08-10 07:19:03 +03:00
Alexey Milovidov
78d3079f0c
Non-significant changes
2022-08-09 21:32:42 +02:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
Yakov Olkhovskiy
23037daf17
Merge branch 'master' into fix-quota-key
2022-08-04 12:14:49 -04:00
Yakov Olkhovskiy
2e34b384c1
update tcp protocol, add quota_key
2022-08-03 15:44:08 -04:00
Kruglov Pavel
ccd1e1bdb8
Merge branch 'master' into fix-cancel-insert-into-function
2022-07-29 20:27:32 +02:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache
2022-07-27 11:35:36 +02:00
Anton Popov
c1a5514e46
fix reading from async read buffer with direct io
2022-07-22 16:58:30 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache
2022-07-21 18:59:14 +02:00
avogar
52b3a87ed1
Fix typo
2022-07-21 12:38:18 +00:00
avogar
6b541aa98f
Fix WriteBuffer finalize when cancel insert into function
2022-07-21 12:18:37 +00:00
Kruglov Pavel
92995a832b
Revert "Fix WriteBuffer finalize in destructor when cacnel query"
2022-07-21 01:45:16 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache
2022-07-20 13:30:42 +02:00
avogar
5c16d6b553
Fix WriteBuffer finalize in destructor when cacnel query
2022-07-19 19:21:30 +00:00
Sergei Trifonov
b2317ff710
renaming to be consistent
2022-07-14 17:33:22 +02:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache
2022-07-14 12:29:54 +02:00
Sergei Trifonov
15ab3bc99f
use context->getWriteSettings()
2022-07-13 19:48:57 +02:00
Sergei Trifonov
f854507729
merge master and resolve conflicts
2022-07-13 19:07:28 +02:00
kssenii
6f29eada80
Merge master
2022-07-12 19:28:18 +02:00
Sergei Trifonov
6ed777d2a8
add azure support
2022-07-11 19:19:57 +02:00
Sergei Trifonov
add0d1698e
respect new limits in HDFS
2022-07-11 17:30:23 +02:00
kssenii
bf1c99daab
Refactoring of object storages, add LocalObjectStorage
2022-07-09 12:39:31 +02:00
avogar
ee54c4f9b7
Add some fixes and add settings in docs
2022-06-30 12:41:56 +00:00
mergify[bot]
ea416c877f
Merge branch 'master' into better-exception-messages
2022-06-28 11:13:26 +00:00
Yakov Olkhovskiy
d5f65ece9b
Merge pull request #38105 from arenadata/ADQM-419
...
Add kerberosInit function as a replacement for kinit executable calls in Kafka and HDFS
2022-06-27 14:19:24 -04:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache
2022-06-24 16:10:25 +02:00
avogar
59c1c472cb
Better exception messages on wrong table engines/functions argument types
2022-06-23 20:04:06 +00:00
avogar
13a05adcb9
Some fixes
2022-06-21 17:34:08 +00:00
avogar
c14364e3d9
Check last modification time for URL function too
2022-06-21 17:18:14 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
kssenii
5dd1bb2fd8
improvements for getFileSize
2022-06-20 15:22:56 +02:00
Alexey Milovidov
7700c26076
Merge pull request #38196 from ClickHouse/revert-38194-revert-37015-zstd_window_log_max
...
Revert "Revert "Add a setting to use more memory for zstd decompression""
2022-06-19 19:58:03 +03:00
