thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
135,450 Commits 9,328 Branches 1,783 Tags 2.1 GiB
0b90076a36
Commit Graph

513 Commits

Author SHA1 Message Date
avogar
447189a6ca Better 2023-04-21 17:54:09 +00:00
avogar
944f54aadf Finish urlCluster, refactor code, reduce code duplication 2023-04-21 17:24:37 +00:00
Kruglov Pavel
2ad161d2b7
Merge branch 'master' into non-blocking-connect 2023-04-19 13:39:40 +02:00
Alexey Milovidov
4d74e3da4d
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards 2023-04-12 22:13:52 +03:00
Azat Khuzhin
79b83c4fd2 Remove superfluous includes of logger_userful.h from headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-10 17:59:30 +02:00
Alexey Milovidov
db61140607
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards 2023-04-09 08:03:23 +03:00
Kruglov Pavel
3ee12e21fb
Merge branch 'master' into non-blocking-connect 2023-03-23 20:53:44 +01:00
avogar
38e44861ae Fix possible race conditions 2023-03-21 16:01:54 +00:00
Robert Schulze
5b036a1a3b
More preparation for libcxx(abi), llvm, clang-tidy 16 (follow-up to #47722) 2023-03-20 12:55:03 +00:00
Alexander Tokmakov
ed08f8f5c5
Merge branch 'master' into revert_25674 2023-03-12 02:33:25 +03:00
Alexander Tokmakov
7b1b238d0b Revert "Merge pull request #25674 from amosbird/distributedreturnconnection" 
This reverts commit 5ffd99dfd4, reversing
changes made to 2796aa333f.
 2023-03-11 19:09:47 +01:00
Maksim Kita
0358cb36d8 Fixed tests 2023-03-11 11:51:54 +01:00
zk_kiger
4f05eeeff9
Merge branch 'master' into improve_cluster_table_funcion_skip_unavailable_shards 2023-02-27 17:15:02 +08:00
kigerzhang
f74f3718a0 add skip_unavailable_shards setting for table function cluste 2023-02-23 17:05:51 +08:00
kssenii
3067c1d723 Merge remote-tracking branch 'upstream/master' into resubmit-prefetches 2023-02-11 11:36:23 +01:00
Sergei Trifonov
2931c3bbe1
Merge pull request #45711 from ClickHouse/io-scheduler-integration 
Integrate IO scheduler with buffers for remote reads and writes
 2023-02-11 10:26:58 +01:00
serxa
8c3f85e58d more review fixes 2023-02-10 17:39:28 +00:00
serxa
acabee8554 review fixes 2023-02-09 20:17:46 +00:00
kssenii
b0b865c32e Resubmit prefetches 2023-02-08 21:26:24 +01:00
Sergei Trifonov
b0fc551c27
Merge branch 'master' into io-scheduler-integration 2023-02-08 18:16:00 +01:00
Robert Schulze
84b9ff450f
Fix terribly broken, fragile and potentially cyclic linking 
Sorry for the clickbaity title. This is about static method
ConnectionTimeouts::getHTTPTimeouts(). It was be declared in header
IO/ConnectionTimeouts.h, and defined in header
IO/ConnectionTimeoutsContext.h (!). This is weird and caused issues with
linking on s390x (##45520). There was an attempt to fix some
inconsistencies (#45848) but neither did @Algunenano nor me at first
really understand why the definition is in the header.

Turns out that ConnectionTimeoutsContext.h is only #include'd from
source files which are part of the normal server build BUT NOT part of
the keeper standalone build (which must be enabled via CMake
-DBUILD_STANDALONE_KEEPER=1). This dependency was not documented and as
a result, some misguided workarounds were introduced earlier, e.g.
0341c6c54b

The deeper cause was that getHTTPTimeouts() is passed a "Context". This
class is part of the "dbms" libary which is deliberately not linked by
the standalone build of clickhouse-keeper. The context is only used to
read the settings and the "Settings" class is part of the
clickhouse_common library which is linked by clickhouse-keeper already.

To resolve this mess, this PR

- creates source file IO/ConnectionTimeouts.cpp and moves all
  ConnectionTimeouts definitions into it, including getHTTPTimeouts().

- breaks the wrong dependency by passing "Settings" instead of "Context"
  into getHTTPTimeouts().

- resolves the previous hacks
 2023-02-05 20:49:34 +00:00
Raúl Marín
7c31cb7adc Proper includes for ConnectionTimeoutsContext.h 2023-01-31 16:11:32 +01:00
Sergei Trifonov
5c2686fbf0
Merge branch 'master' into io-scheduler-integration 2023-01-27 19:56:23 +01:00
serxa
21e871a0a4 integrate IO scheduler with buffers 2023-01-27 18:47:22 +00:00
Anton Popov
b58b73b0e7
Merge pull request #45529 from CurtizJ/fix-storage-s3-race 
Try to fix test `test_storage_s3/test.py::test_wrong_s3_syntax` (race in `StorageS3`)
 2023-01-26 14:21:32 +01:00
Kruglov Pavel
cd3d3fdd68
Merge pull request #45483 from Avogar/fix-hdfs-cluster 
Fix schema inference from insertion table in hdfsCluster
 2023-01-25 20:24:51 +01:00
avogar
07ad750366 Fix tests 2023-01-24 15:08:19 +00:00
Anton Popov
5c0307bc6a fix race in StorageURL and StorageHDFS 2023-01-24 12:34:43 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Alexander Gololobov
91fb1eab8c
Merge pull request #45373 from ClickHouse/throttler-metrics 
Add detailed profile events for throttling
 2023-01-22 11:20:45 +01:00
avogar
eed1db7e07 Fix schema inference in hdfsCluster 2023-01-20 21:17:35 +00:00
Anton Popov
f40fd7a151
Add checks for compilation of regexps (#45356) 2023-01-17 23:46:04 +01:00
serxa
ce7e22b87b add detailed profile events for throttling 2023-01-17 18:29:24 +00:00
Maksim Kita
fbba28b31e Analyzer aggregation without column fix 2023-01-10 16:49:55 +01:00
Kseniia Sumarokova
b22b51213f
Merge pull request #44868 from kssenii/improve-async-buffer-from-remote 
Small perf improvement for async buffer from remote
 2023-01-09 13:31:15 +01:00
Nikita Mikhaylov
857799fbca
Parallel distributed insert select with s3Cluster [3] (#44955) 
* Revert "Revert "Resurrect parallel distributed insert select with s3Cluster (#41535)""

This reverts commit b8d9066004.

* Fix build

* Better

* Fix test

* Automatic style fix

Co-authored-by: robot-clickhouse <robot-clickhouse@users.noreply.github.com>
 2023-01-09 13:30:32 +01:00
kssenii
bda8f0d410 Better async buffer from remote 2023-01-03 15:27:37 +01:00
xiedeyantu
ec6698395e fix skip_unavailable_shards does not work using hdfsCluster table function 2022-11-15 13:25:15 +08:00
Kruglov Pavel
21d50f76ea
Merge pull request #41979 from Avogar/s3-cluster-schema-inference 
Fix schema inference in s3Cluster and improve in hdfsCluster
 2022-11-01 14:00:21 +01:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32 
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:19 +02:00
Kruglov Pavel
df96acc80f
Fix style 2022-10-19 21:48:45 +02:00
avogar
2c2f977096 Make better 2022-10-18 13:03:55 +00:00
avogar
75c4ef0be7 Merge branch 'master' of github.com:ClickHouse/ClickHouse into s3-cluster-schema-inference 2022-10-18 11:58:12 +00:00
Kruglov Pavel
3d9f46a1e7
Merge branch 'master' into s3-cluster-schema-inference 2022-10-14 22:07:54 +02:00
Ilya Yatsishin
3361b27965 less 2022-10-14 14:27:52 +02:00
Ilya Yatsishin
bbf88b66a0 more 2022-10-13 11:42:35 +02:00
Ilya Yatsishin
e7ca30cd11 fix typo 2022-10-13 10:10:13 +02:00
Ilya Yatsishin
3201c08bb5 style 2022-10-11 13:54:03 +02:00
Ilya Yatsishin
28c7a22428 address review comments 2022-10-11 11:59:48 +02:00
Ilya Yatsishin
95c5c8a744 Fail fast on empty URL 2022-10-10 14:50:09 +02:00
Alexander Tokmakov
b8d9066004 Revert "Resurrect parallel distributed insert select with s3Cluster (#41535)" 
This reverts commit 860e34e760.
 2022-10-07 15:53:30 +02:00
Robert Schulze
78be400ac0
Merge remote-tracking branch 'origin/master' into generated-file-cleanup 2022-10-06 12:22:43 +00:00
Nikita Mikhaylov
860e34e760
Resurrect parallel distributed insert select with s3Cluster (#41535) 2022-10-06 13:47:32 +02:00
avogar
c74b5c8126 Fix schema inference in s3Cluster and improve in hdfsCluster 2022-09-30 16:59:17 +00:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH} 
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
 2022-09-28 12:48:26 +00:00
kssenii
ab702e43fc Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-23 15:21:33 +02:00
kssenii
52ef3758c4 Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-13 16:34:31 +02:00
Azat Khuzhin
4e3135383b Fix clang-tidy warnings (from clang-15) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
kssenii
e51313b6b3 Get rid of static threadpools 2022-09-07 17:48:11 +02:00
kssenii
4fa291cf59 Fix hdfs read buffer too 2022-09-01 16:46:46 +02:00
Alexander Tokmakov
f9f85a0e8b Revert "Parallel distributed insert select from *Cluster table functions (#39107)" 
This reverts commit d3cc234986.
 2022-08-24 15:17:15 +03:00
Kruglov Pavel
72f02bd6eb
Merge pull request #40414 from Avogar/improve-schema-inference-cache 
Improve schema inference cache, respect format settings that can change the schema
 2022-08-23 17:04:58 +02:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
avogar
5ab87f1da4 Small refactoring 2022-08-19 16:42:23 +00:00
avogar
8dd54c043d Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache 2022-08-17 11:47:40 +00:00
avogar
c4ff3ffeea Rename settings 2022-08-15 12:45:18 +00:00
Nikita Mikhaylov
d3cc234986
Parallel distributed insert select from *Cluster table functions (#39107) 2022-08-15 12:41:17 +02:00
Alexey Milovidov
ddd79e6c05
Merge pull request #40038 from ClickHouse/non-significant-changes 
Non-significant changes
 2022-08-10 07:19:03 +03:00
Alexey Milovidov
78d3079f0c Non-significant changes 2022-08-09 21:32:42 +02:00
avogar
9b1a267203 Refactor, remove TTL, add size limit, add system table and system query 2022-08-05 16:20:15 +00:00
Yakov Olkhovskiy
23037daf17
Merge branch 'master' into fix-quota-key 2022-08-04 12:14:49 -04:00
Yakov Olkhovskiy
2e34b384c1 update tcp protocol, add quota_key 2022-08-03 15:44:08 -04:00
Kruglov Pavel
ccd1e1bdb8
Merge branch 'master' into fix-cancel-insert-into-function 2022-07-29 20:27:32 +02:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache 2022-07-27 11:35:36 +02:00
Anton Popov
c1a5514e46 fix reading from async read buffer with direct io 2022-07-22 16:58:30 +00:00
Kruglov Pavel
9252f42b4c
Merge branch 'master' into schema-inference-cache 2022-07-21 18:59:14 +02:00
avogar
52b3a87ed1 Fix typo 2022-07-21 12:38:18 +00:00
avogar
6b541aa98f Fix WriteBuffer finalize when cancel insert into function 2022-07-21 12:18:37 +00:00
Kruglov Pavel
92995a832b
Revert "Fix WriteBuffer finalize in destructor when cacnel query" 2022-07-21 01:45:16 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache 2022-07-20 13:30:42 +02:00
avogar
5c16d6b553 Fix WriteBuffer finalize in destructor when cacnel query 2022-07-19 19:21:30 +00:00
Sergei Trifonov
b2317ff710 renaming to be consistent 2022-07-14 17:33:22 +02:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache 2022-07-14 12:29:54 +02:00
Sergei Trifonov
15ab3bc99f use context->getWriteSettings() 2022-07-13 19:48:57 +02:00
Sergei Trifonov
f854507729 merge master and resolve conflicts 2022-07-13 19:07:28 +02:00
kssenii
6f29eada80 Merge master 2022-07-12 19:28:18 +02:00
Sergei Trifonov
6ed777d2a8 add azure support 2022-07-11 19:19:57 +02:00
Sergei Trifonov
add0d1698e respect new limits in HDFS 2022-07-11 17:30:23 +02:00
kssenii
bf1c99daab Refactoring of object storages, add LocalObjectStorage 2022-07-09 12:39:31 +02:00
avogar
ee54c4f9b7 Add some fixes and add settings in docs 2022-06-30 12:41:56 +00:00
mergify[bot]
ea416c877f
Merge branch 'master' into better-exception-messages 2022-06-28 11:13:26 +00:00
Yakov Olkhovskiy
d5f65ece9b
Merge pull request #38105 from arenadata/ADQM-419 
Add kerberosInit function as a replacement for kinit executable calls in Kafka and HDFS
 2022-06-27 14:19:24 -04:00
avogar
5155262a16 Add some additional information to cache keys 2022-06-27 12:43:24 +00:00
Kruglov Pavel
86e8f31ad4
Merge branch 'master' into schema-inference-cache 2022-06-24 16:10:25 +02:00
avogar
59c1c472cb Better exception messages on wrong table engines/functions argument types 2022-06-23 20:04:06 +00:00
avogar
13a05adcb9 Some fixes 2022-06-21 17:34:08 +00:00
avogar
c14364e3d9 Check last modification time for URL function too 2022-06-21 17:18:14 +00:00
avogar
d37ad2e6de Implement cache for schema inference for file/s3/hdfs/url 2022-06-21 13:02:48 +00:00
kssenii
5dd1bb2fd8 improvements for getFileSize 2022-06-20 15:22:56 +02:00
Alexey Milovidov
7700c26076
Merge pull request #38196 from ClickHouse/revert-38194-revert-37015-zstd_window_log_max 
Revert "Revert "Add a setting to use more memory for zstd decompression""
 2022-06-19 19:58:03 +03:00
Alexey Milovidov
04b7a34870 Maybe fix error 2022-06-19 05:21:59 +02:00
Alexey Milovidov
73709b0488
Revert "Revert "Add a setting to use more memory for zstd decompression"" 2022-06-18 15:55:35 +03:00
alesapin
16e8b85fbf
Revert "Add a setting to use more memory for zstd decompression" 2022-06-18 14:08:14 +02:00
Alexey Milovidov
e20259e9ca
Merge pull request #37015 from wuxiaobai24/zstd_window_log_max 
Add a setting to use more memory for zstd decompression
 2022-06-18 04:19:27 +03:00
Roman Vasin
ed3fe84b63 Fix runKinit() is called only for USE_KRB5 2022-06-16 14:45:27 +03:00
Roman Vasin
6e28275569 Add warnings about using krb5 parameters 2022-06-16 14:21:04 +03:00
Roman Vasin
d93fd3bd2d Add complilation support for case when krb5 is not used 2022-06-16 09:30:40 +00:00
Roman Vasin
344fbe8de4 Fix code style 2022-06-15 20:26:42 +03:00
Roman Vasin
dd5b0ee065 Add kerberosInit() function to call KeberosInit 2022-06-15 17:02:53 +03:00
Roman Vasin
9bf6b9d491 Add kinit presence handling in StorageKafka; Cleanup code in HDFSCommon 2022-06-15 11:37:02 +03:00
Roman Vasin
d1d6d87432 Cleanup code in KerberosInit 2022-06-09 11:51:15 +03:00
Roman Vasin
3cfea6e76f Cleanup code in KerberosInit, HDFSCommon and StorageKafka; update English and Russian documentation. 2022-06-08 17:57:45 +03:00
Roman Vasin
2b76d0c6a9 Add new integration test for kerberized Kafka; remove old kinit code from HDFSCommon 2022-06-08 12:26:35 +03:00
taiyang-li
57b6cf6c09 fix build error 2022-06-08 09:58:09 +08:00
Roman Vasin
a156a77890 Add KerberosInit into StorageKafka 2022-06-07 14:59:46 +03:00
Roman Vasin
cb53aa15ec Fix HDFSCommon and test_storage_kerberized_hdfs to make running integration tests 2022-06-07 12:06:22 +03:00
taiyang-li
73a484256e Merge branch 'master' into async_hdfs_read_buffer 2022-06-07 12:16:46 +08:00
taiyang-li
c65c56fd48 fix typo 2022-06-07 09:58:29 +08:00
taiyang-li
b36d9f8143 refactor readinto 2022-06-06 12:58:22 +08:00
Roman Vasin
8b5bf02927 Add support of cache commands in KerberosInit 2022-06-03 18:07:18 +03:00
Nikita Mikhaylov
fe477355df
Added support for schema inference for hdfsCluster [2] (#35812) 2022-06-03 15:54:29 +02:00
taiyang-li
f202c35311 Merge branch 'master' into async_hdfs_read_buffer 2022-06-03 17:52:09 +08:00
Roman Vasin
82ce2d76c3 Add KerberosInit class; add kerberos_init console example; modify HDFSCommon.cpp 2022-06-03 12:06:31 +03:00
Nikolai Kochetov
8991f39412 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 17:00:08 +00:00
Azat Khuzhin
545a56ce45 Fix sinks with onException() handler 
It is possible to call onException() even after onFinish(), in case of
onFinish() throws, and in this case onException() should be no-op for
such sinks.

Also there can be caveats with PartitionedSync.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-01 21:50:30 +03:00
Azat Khuzhin
02af58f41d Fix possible "Cannot write to finalized buffer" 
It is still possible to get this error since onException does not
finalize format correctly.

Here is an example of such error, that was found by CI [1]:

<details>

    [ 2686 ] {fa01bf02-73f6-4f7f-b14f-e725de6d7f9b} <Fatal> : Logical error: 'Cannot write to finalized buffer'.
    [ 34577 ] {} <Fatal> BaseDaemon: ########################################
    [ 34577 ] {} <Fatal> BaseDaemon: (version 22.6.1.1, build id: AB8040A6769E01A0) (from thread 2686) (query_id: fa01bf02-73f6-4f7f-b14f-e725de6d7f9b) (query: insert into test_02302 select number from numbers(10) settings s3_truncate_on_insert=1;) Received signal Aborted (6)
    [ 34577 ] {} <Fatal> BaseDaemon:
    [ 34577 ] {} <Fatal> BaseDaemon: Stack trace: 0x7fcbaa5a703b 0x7fcbaa586859 0xfad9bab 0xfad9e05 0xfaf6a3b 0x24a48c7f 0x258fb9b9 0x258f2004 0x258b88f4 0x258b863b 0x2581773d 0x258177ce 0x24bb5e98 0xfad01d6 0xfad0105 0x2419b11d 0xfad01d6 0xfad0105 0x2215afbb 0x2215aa48 0xfad01d6 0xfad0105 0xfcc265d 0x225cc546 0x249a1c40 0x249bc1b6 0x2685902c 0x26859505 0x269d7767 0x269d504c 0x7fcbaa75e609 0x7fcbaa683163
    [ 34577 ] {} <Fatal> BaseDaemon: 3. raise @ 0x7fcbaa5a703b in ?
    [ 34577 ] {} <Fatal> BaseDaemon: 4. abort @ 0x7fcbaa586859 in ?
    [ 34577 ] {} <Fatal> BaseDaemon: 5. ./build_docker/../src/Common/Exception.cpp:47: DB::abortOnFailedAssertion(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xfad9bab in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 6. ./build_docker/../src/Common/Exception.cpp:70: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xfad9e05 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 7. ./build_docker/../src/IO/WriteBuffer.h:0: DB::WriteBuffer::write(char const*, unsigned long) @ 0xfaf6a3b in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 8. ./build_docker/../src/Processors/Formats/Impl/ArrowBufferedStreams.cpp:47: DB::ArrowBufferedOutputStream::Write(void const*, long) @ 0x24a48c7f in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 9. long parquet::ThriftSerializer::Serialize<parquet::format::FileMetaData>(parquet::format::FileMetaData const*, arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) @ 0x258fb9b9 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 10. parquet::FileMetaData::FileMetaDataImpl::WriteTo(arrow::io::OutputStream*, std::__1::shared_ptr<parquet::Encryptor> const&) const @ 0x258f2004 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 11. parquet::WriteFileMetaData(parquet::FileMetaData const&, arrow::io::OutputStream*) @ 0x258b88f4 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 12. parquet::ParquetFileWriter::~ParquetFileWriter() @ 0x258b863b in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 13. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x2581773d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 14. parquet::arrow::FileWriterImpl::~FileWriterImpl() @ 0x258177ce in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 15. ./build_docker/../src/Processors/Formats/Impl/ParquetBlockOutputFormat.h:27: DB::ParquetBlockOutputFormat::~ParquetBlockOutputFormat() @ 0x24bb5e98 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 16. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 17. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./build_docker/../contrib/libcxx/include/__memory/unique_ptr.h:312: std::__1::unique_ptr<DB::WriteBuffer, std::__1::default_delete<DB::WriteBuffer> >::reset(DB::WriteBuffer*)
    [ 34577 ] {} <Fatal> BaseDaemon: 18.2. inlined from ../contrib/libcxx/include/__memory/unique_ptr.h:269: ~unique_ptr
    [ 34577 ] {} <Fatal> BaseDaemon: 18. ../src/Storages/StorageS3.cpp:566: DB::StorageS3Sink::~StorageS3Sink() @ 0x2419b11d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 19. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 20. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 21. ./build_docker/../contrib/abseil-cpp/absl/container/internal/raw_hash_set.h:1662: absl::lts_20211102::container_internal::raw_hash_set<absl::lts_20211102::container_internal::FlatHashMapPolicy<StringRef, std::__1::shared_ptr<DB::SinkToStorage> >, absl::lts_20211102::hash_internal::Hash<StringRef>, std::__1::equal_to<StringRef>, std::__1::allocator<std::__1::pair<StringRef const, std::__1::shared_ptr<DB::SinkToStorage> > > >::destroy_slots() @ 0x2215afbb in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./build_docker/../contrib/libcxx/include/string:1445: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long() const
    [ 34577 ] {} <Fatal> BaseDaemon: 22.2. inlined from ../contrib/libcxx/include/string:2231: ~basic_string
    [ 34577 ] {} <Fatal> BaseDaemon: 22. ../src/Storages/PartitionedSink.h:14: DB::PartitionedSink::~PartitionedSink() @ 0x2215aa48 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 23. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:173: std::__1::__shared_count::__release_shared() @ 0xfad01d6 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 24. ./build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:216: std::__1::__shared_weak_count::__release_shared() @ 0xfad0105 in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 25. ./build_docker/../contrib/libcxx/include/vector:802: std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >::__base_destruct_at_end(std::__1::shared_ptr<DB::IProcessor>*) @ 0xfcc265d in /usr/bin/clickhouse
    [ 34577 ] {} <Fatal> BaseDaemon: 26.1. inlined from ./build_docker/../contrib/libcxx/include/vector:402: ~vector
    [ 34577 ] {} <Fatal> BaseDaemon: 26.2. inlined from ../src/QueryPipeline/QueryPipeline.cpp:29: ~QueryPipeline
    [ 34577 ] {} <Fatal> BaseDaemon: 26. ../src/QueryPipeline/QueryPipeline.cpp:535: DB::QueryPipeline::reset() @ 0x225cc546 in /usr/bin/clickhouse
    [ 614 ] {} <Fatal> Application: Child process was terminated by signal 6.

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37542/8a224239c1d922158b4dc9f5d6609dca836dfd06/stress_test__undefined__actions_.html

Follow-up for: #36979

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-01 21:50:30 +03:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00
Kruglov Pavel
0615866aea
Merge pull request #37450 from Avogar/check-format-on-storage-creation 
Check format name on storage creation
 2022-05-30 14:23:20 +02:00
Nikolai Kochetov
5b4658aa5e Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-30 09:47:35 +00:00
taiyang-li
dbb8a09825 merge master and solve conflict 2022-05-30 10:47:04 +08:00
taiyang-li
51a893c8be add some metrics 2022-05-30 10:05:20 +08:00
taiyang-li
73d2c889c6 fix log level 2022-05-27 19:23:58 +08:00
taiyang-li
ea450b86cb add some prefetch metric codes 2022-05-27 18:06:40 +08:00
taiyang-li
561c87222d add prefetch for hive text 2022-05-26 11:04:35 +08:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
msaf1980
fda6ddeffa cleanup StorageHDFS (unused variables) 2022-05-25 19:23:05 +05:00
taiyang-li
a7a816dcb6 fix build error 2022-05-25 19:55:11 +08:00
taiyang-li
1d9f65a7d4 Merge branch 'master' into async_hdfs_read_buffer 2022-05-25 17:10:22 +08:00
avogar
f782fa31c6 Merge branch 'master' of github.com:ClickHouse/ClickHouse into check-format-on-storage-creation 2022-05-25 08:42:54 +00:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
avogar
37b66c8a9e Check format name on storage creation 2022-05-23 12:48:48 +00:00
taiyang-li
29e2157469 change as request 2022-05-23 18:42:54 +08:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names 2022-05-23 12:14:20 +02:00
Nikolai Kochetov
56feef01e7 Move some resources 2022-05-20 19:49:31 +00:00
taiyang-li
14ab7eb5a3 merge master and solve conflict 2022-05-17 16:28:08 +08:00
avogar
2d4b4b9008 Fix inserting defaults for missing values in columnar formats 2022-05-16 14:19:44 +00:00
avogar
68bb07d166 Better naming 2022-05-13 18:39:19 +00:00
avogar
b17fec659a Improve performance and memory usage for select of subset of columns for some formats 2022-05-13 13:51:28 +00:00
wuxiaobai24
f3e573bfde Merge remote-tracking branch 'upstream/master' into zstd_window_log_max 2022-05-13 20:42:54 +08:00
wuxiaobai24
ab5636a46a fix 2022-05-13 17:52:39 +08:00
wuxiaobai24
58755cb156 add zstd_window_log_max setting 2022-05-08 14:02:40 +08:00
avogar
02bd5f6542 Finalize write buffers in case of exception 2022-05-06 17:30:18 +00:00
Robert Schulze
e583099158
Fix build, pt. V 2022-05-04 15:50:52 +02:00
mergify[bot]
64084b5e32
Merge branch 'master' into shared_ptr_helper3 2022-05-03 20:46:16 +00:00
Dmitry Novik
5ba7a55c18
Merge pull request #36650 from bigo-sg/hive_text_parallel_parsing 
Parallel parsing of hive text format
 2022-05-03 15:56:28 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable 
... IStorage has deleted copy ctor / assignment already
 2022-05-03 09:07:08 +02:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
kssenii
9d364cdce2 Refactor 2022-04-26 15:33:53 +02:00
taiyang-li
b7cc344d62 remove useless codes 2022-04-26 14:42:43 +08:00
taiyang-li
99dee35b6e parallel parsing of hive text format 2022-04-26 14:33:10 +08:00
Kruglov Pavel
34c342fdd3
Merge pull request #36205 from Avogar/improve-globs 
Some refactoring around schema inference with globs
 2022-04-25 13:14:46 +02:00
Amos Bird
e73d7fea31
Rename narrowBlockInputStream to narrowPipe 2022-04-24 18:33:48 +08:00
taiyang-li
f040c8ae65 merge master and solve conflict 2022-04-24 11:42:36 +08:00
taiyang-li
957f3f13f7 fix build error 2022-04-21 20:19:29 +08:00
taiyang-li
ad9e683356 fix code syle 2022-04-21 18:12:15 +08:00
taiyang-li
94d0358b15 fix code style 2022-04-21 17:40:55 +08:00
taiyang-li
169dae2a35 ready for review 2022-04-21 17:37:12 +08:00
taiyang-li
fb34a4ec01 ready for review 2022-04-21 17:27:41 +08:00
taiyang-li
1ea96a8732 merge master and solve conflict 2022-04-21 17:16:07 +08:00
Kruglov Pavel
813e228fcc
Merge branch 'master' into improve-globs 2022-04-20 16:31:47 +02:00
taiyang-li
3b722eea7a profileing 2022-04-20 20:59:36 +08:00
taiyang-li
d533b569ad debugging 2022-04-20 19:58:31 +08:00
taiyang-li
56fe6fa608 finish dev 2022-04-20 17:49:53 +08:00
taiyang-li
fb6a56d4b0 finish debug 2022-04-20 16:24:18 +08:00
avogar
1f252cedfe Make better 2022-04-19 19:16:47 +00:00
Anton Popov
bee4ca9b62 add more tests for error diagnostics in files 2022-04-19 15:56:34 +00:00
taiyang-li
e2c51d581c Merge branch 'fix_hdfs_seek' into async_hdfs_read_buffer 2022-04-19 15:50:01 +08:00
zhanglistar
5670dcf750 Fix crash in debug mode of ReadBufferFromHDFS. 2022-04-18 17:52:59 +08:00
taiyang-li
27fa2a019a commit again 2022-04-18 15:56:49 +08:00
taiyang-li
cd83fd5f8a tobe debug 2022-04-16 18:41:18 +08:00
taiyang-li
f20e80c318 wip: deving 2022-04-16 16:19:48 +08:00
zhanglistar
fe93fd83f5
Merge branch 'ClickHouse:master' into fixhdfsreadbuffer 2022-04-15 16:27:20 +08:00
zhanglistar
e22a175afb Fix ReadBufferFromHDFS crash in debug mode 2022-04-15 14:04:30 +08:00
avogar
1c065f8c7a Some refactoring around schema inference with globs 2022-04-13 17:02:48 +00:00
taiyang-li
b4cd8561a5 wip 2022-04-13 16:06:19 +08:00
taiyang-li
9d31c44d39 fix bug of read buffer from hdfs 2022-04-12 12:01:51 +08:00
lgbo-ustc
759e4a9a05 Merge remote-tracking branch 'ck/master' into s3_cluster_join 2022-04-08 09:25:30 +08:00
shuchaome
a565a93740 reduce mutex scope when setenv LIBHDFS3_CONF 2022-04-02 06:52:39 +08:00
Alexey Milovidov
5c6b84c398
Merge pull request #35586 from kssenii/fix-virtual-columns-s3 
Fix s3 engine getting virtual columns
 2022-04-01 04:43:32 +03:00
Nikita Mikhaylov
5f27f690f4 Revert "Added support for schema inference for hdfsCluster (#35602)" 
This reverts commit f6bfdcc0c9.
 2022-03-31 14:19:56 +02:00
Nikita Mikhaylov
f6bfdcc0c9
Added support for schema inference for hdfsCluster (#35602) 2022-03-31 12:47:36 +02:00
lgbo-ustc
08e1fdc6b8 Fixed :a elimate AMBIGUOUS_COLUMN_NAME exception 
elimate AMBIGUOUS_COLUMN_NAME exception in hdfs/s3 cluster when run a query with join
 2022-03-30 17:06:38 +08:00
kssenii
4374415bd7 Fix clang-tidy, style check 2022-03-29 14:20:21 +02:00
kssenii
a2013ebe2b Better 2022-03-28 22:40:27 +02:00
taiyang-li
fbadd8fc0e only deduce absolute path of hdfs config when it's relative path do not exists 2022-03-25 22:28:47 +08:00
taiyang-li
c0eb9f73a9 remote std::cout 2022-03-24 21:17:43 +08:00
taiyang-li
6eccd09e2b deduce hdfs config path 2022-03-24 18:04:29 +08:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00