thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-19 21:03:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
84,987 Commits 9,338 Branches 1,784 Tags 2.2 GiB
ae3d55c6a2
Commit Graph

23855 Commits

Author SHA1 Message Date
Kruglov Pavel
906b68dbcb
Merge pull request #35291 from Avogar/segfault-in-schema-inference 
Fix possible segfault in JSONEachRow schema inference
 2022-03-16 12:24:35 +01:00
Kruglov Pavel
ec6968c17e
Merge pull request #35313 from kssenii/fix-pg-db 
Fix possible segfault in DatabasePostgreSQL::getCreateTableQuery
 2022-03-16 12:23:25 +01:00
Maksim Kita
ed74a464bd Fixed tests 2022-03-16 11:21:18 +00:00
Vitaly Baranov
e690d28fef Update src/Access/AccessControl.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-03-16 12:03:09 +01:00
Nikolai Kochetov
bcd3c76296
Merge pull request #35214 from ClickHouse/add-database_replicated_allow_only_replicated_engine-setting 
Add database_replicated_allow_only_replicated_engine setting.
 2022-03-16 11:24:32 +01:00
zhangyuli1
3cba1177ee remove wchc from four_letter_word_white_list 2022-03-16 17:40:49 +08:00
Robert Schulze
0d2ece6d91
Merge branch 'ClickHouse:master' into protobuf-batch-write 2022-03-16 09:43:33 +01:00
Kseniia Sumarokova
b4e03eda8b
Update DatabasePostgreSQL.cpp 2022-03-16 01:04:40 +01:00
kssenii
3a784fe611 Fix 2022-03-16 00:52:51 +01:00
Kruglov Pavel
2b7e89e171
Merge pull request #35300 from Avogar/fix-bzip2 
Fix possible assertion while using bzip2 compression
 2022-03-15 22:17:37 +01:00
Kruglov Pavel
20e17a6d3e
Merge pull request #35296 from Avogar/fix-lz4 
Fix possible segfault while using lz4 compression
 2022-03-15 22:17:14 +01:00
Kruglov Pavel
6fd1a8281f
Merge pull request #35295 from Avogar/fix-lzma 
Fix possible Assertion failed while using lzma compression
 2022-03-15 22:14:35 +01:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts 
Reload remote_url_allow_hosts after config update.
 2022-03-15 22:07:16 +01:00
Nikolai Kochetov
cf260b8508
Merge pull request #35305 from ClickHouse/try-to-fix-using-deleted-memory-tracker-inside-s3-disk 
Maybe fix use-after-free inside S3 upload thread
 2022-03-15 22:06:17 +01:00
Vladimir C
e022245a33
Merge pull request #35193 from vdimir/issue-34776 2022-03-15 19:56:09 +01:00
vdimir
695708d57c
[wip] merge join duplicate rows 2022-03-15 18:51:04 +00:00
mergify[bot]
a5e93ea30b
Merge branch 'master' into fix-bzip2 2022-03-15 18:42:26 +00:00
mergify[bot]
381a5e4f6e
Merge branch 'master' into fix-lz4 2022-03-15 18:42:14 +00:00
Maksim Kita
8bdb906acb Fixed tests 2022-03-15 15:43:39 +00:00
Maksim Kita
08bb39d869 Fixed performance tests 2022-03-15 15:43:39 +00:00
Maksim Kita
fa4508bcb9 Fixed tests 2022-03-15 15:35:25 +00:00
Nikolai Kochetov
b4aed421de Reset THreadPool's thread memory tracker parent to global after S3 upload task is finished. 2022-03-15 15:33:43 +00:00
Maksim Kita
5ffb12eb8c
Merge pull request #35289 from bigo-sg/getmacro_const 
return const for function getMacro if not distributed query
 2022-03-15 16:32:07 +01:00
avogar
ec3c5f86ad Fix possible assertion while using bzip2 compression 2022-03-15 14:23:52 +00:00
Nikolai Kochetov
d6710b3354 Always clear hosts. 2022-03-15 14:08:56 +00:00
Kseniia Sumarokova
afec02fe18
Merge pull request #35264 from kssenii/test-mode-for-local 
Support test mode for clickhouse-local
 2022-03-15 14:53:01 +01:00
avogar
0cfaf48e1d Fix possible segfault while using lz4 compression 2022-03-15 13:46:14 +00:00
Vladimir C
02414f1e88
Merge branch 'master' into issue-34776 2022-03-15 14:24:51 +01:00
Kruglov Pavel
24ac217be3
Remove redundant include 2022-03-15 14:07:31 +01:00
avogar
90ce1709d7 Fix possible Assertion failed while using lzma compression 2022-03-15 13:00:53 +00:00
Nikolai Kochetov
97aa6c82ce Reload remote_url_allow_hosts after config update. 2022-03-15 13:00:31 +00:00
Kruglov Pavel
2e2ee2856e
Merge pull request #35281 from Avogar/fix-brotli 
Fix possible Abort while using Brotli compression
 2022-03-15 13:58:16 +01:00
Anton Popov
b5a04ff938
Merge pull request #35207 from bigo-sg/add_distributed_depth_in_querylog 
Add column  distributed_depth for system.query_log and system.processes
 2022-03-15 13:38:21 +01:00
avogar
e2d1e643f2 Fix possible segfault in JSONEachRow schema inference 2022-03-15 11:44:15 +00:00
alesapin
bfe2872ab3
Merge pull request #35278 from kssenii/add-cache-optimization 
Add remote fs cache optimization
 2022-03-15 11:50:25 +01:00
Robert Schulze
23122cb327
Fix review comments 
ParquetBlockOutputFormat.cpp:
- undo unrelated formatting

ProtobufSerializer.cpp:
- undef debug tracing
- simplify logic in writeRow()

ProtobufSchemas.cpp:
- restore original search in cache by message type
 2022-03-15 11:27:17 +01:00
Vladimir C
af88d3f133
Merge pull request #35173 from bigo-sg/dump_ast_in_dot 2022-03-15 11:18:41 +01:00
taiyang-li
c08e3c0dd2 return const for function getMacro if not distributed query 2022-03-15 17:06:16 +08:00
Kseniia Sumarokova
427ae808ba
Merge pull request #35191 from DevTeamBK/RemoteHostFilter_improvement 
Added RemoteHostFilter check for MYSQL and postgresSQL
 2022-03-15 09:58:08 +01:00
mergify[bot]
a7298a98b9
Merge branch 'master' into fix-logical-error-in-cache 2022-03-15 08:29:16 +00:00
mergify[bot]
9cdd6ec7ab
Merge branch 'master' into test-mode-for-local 2022-03-15 08:28:40 +00:00
mergify[bot]
d613ad9445
Merge branch 'master' into fix-brotli 2022-03-15 08:25:12 +00:00
taiyang-li
3688ff10e0 fix bug 2022-03-15 12:19:08 +08:00
lgbo-ustc
abfaa82bca fixed hive query bugs 2022-03-15 12:01:34 +08:00
Anton Popov
ccbddd53a3 fix mutations in tables with enabled sparse columns 2022-03-15 01:48:21 +00:00
mergify[bot]
ee3f4fb0c3
Merge branch 'master' into add-cache-optimization 2022-03-14 20:58:52 +00:00
alesapin
fbb1ebd9b8
Merge pull request #35274 from CurtizJ/fix-check-table-sparse-columns 
Fix check table in case when there exist sparse columns
 2022-03-14 21:56:04 +01:00
Vitaly Baranov
1eb2e8693e Fix code style and other minor corrections after implementing allow_no_password. 2022-03-14 20:55:34 +01:00
Nikolai Kochetov
7526c36f30 Remove comment 2022-03-14 19:38:03 +00:00
Nikolai Kochetov
ce94b560bb Add test. 2022-03-14 19:30:52 +00:00
avogar
4b12c6e6b3 Fix possible Abort while using Brotli compression 2022-03-14 18:58:05 +00:00
Maksim Kita
282dd98779 Fix build 2022-03-14 18:17:36 +00:00
Maksim Kita
3ca40d5d2e Fixed style check 2022-03-14 18:17:35 +00:00
Maksim Kita
5cf0e0c0f3 Fixed build 2022-03-14 18:17:35 +00:00
Maksim Kita
2fdcf53a76 Fix clang-tidy warnings in Server, Storages folders 2022-03-14 18:17:35 +00:00
Maksim Kita
2665724301 Fix clang-tidy warnings in Parsers, Processors, QueryPipeline folders 2022-03-14 18:17:35 +00:00
Maksim Kita
e30117a3d6 Fix clang-tidy warnings in Interpreters, IO folders 2022-03-14 18:17:35 +00:00
Maksim Kita
538f8cbaad Fix clang-tidy warnings in Disks, Formats, Functions folders 2022-03-14 18:17:35 +00:00
Maksim Kita
1d674123a9 Fix clang-tidy warnings in Databases, DataTypes, Dictionaries folders 2022-03-14 18:17:35 +00:00
Maksim Kita
65c52298b6 Fix clang-tidy warnings in Compression, Coordination, Core folders 2022-03-14 18:17:35 +00:00
Maksim Kita
e7772ed434 Fix clang-tidy warnings in Common folder 2022-03-14 18:17:35 +00:00
Maksim Kita
07b0845110 Fix clang-tidy warnings in Backups, Client, Columns folders 2022-03-14 18:17:35 +00:00
Maksim Kita
5147da3355 Fix clang-tidy warnings in AggregateFunction folder 2022-03-14 18:17:35 +00:00
Maksim Kita
e14cfd5dcd Fix clang-tidy warnings in Access folder 2022-03-14 18:17:35 +00:00
Anton Popov
063917786e minor fixes 2022-03-14 17:29:18 +00:00
kssenii
53d23d901e Fix 2022-03-14 18:11:59 +01:00
kssenii
fc9c6eab9d Add optimization 2022-03-14 17:58:19 +01:00
Anton Popov
36ec379aeb Merge remote-tracking branch 'upstream/master' into HEAD 2022-03-14 16:28:35 +00:00
Vitaly Baranov
4af61fb9d3
Merge pull request #34738 from DevTeamBK/Issue-33953 
New setting in Server Configuration to on/off AuthType Plaintext_password and No_password
 2022-03-14 17:09:46 +01:00
Anton Popov
428bbd6377 fix check table in case when there exist sparse columns 2022-03-14 15:22:23 +00:00
Heena Bansal
c774458f96
Update src/Storages/StoragePostgreSQL.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-14 10:38:12 -04:00
Heena Bansal
01c8b2f71e
Update src/Storages/StorageMySQL.cpp 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-14 10:37:59 -04:00
alesapin
b13b415d3f
Merge pull request #35257 from azat/fix-s3-pool-shutdown 
Wait for IDiskRemote thread pool properly
 2022-03-14 13:56:37 +01:00
vdimir
db46c2ada8
Fix reading port from config 2022-03-14 11:58:47 +00:00
Nikolai Kochetov
ff09076524 A little bit better. 2022-03-14 11:31:24 +00:00
Antonio Andelic
ed3d71d83f
Merge pull request #35072 from azat/buffer-memory-tracker-leak 
Avoid MEMORY_LIMIT_EXCEEDED during INSERT into Buffer with AggregateFunction
 2022-03-14 12:31:04 +01:00
kssenii
199188be08 Support test mode for clickhouse-local 2022-03-14 12:02:06 +01:00
alesapin
98144bc1f5
Merge pull request #35031 from ClickHouse/standalone_keeper_build 
Building small keeper binary
 2022-03-14 11:27:25 +01:00
Antonio Andelic
ebf72bf61d
Merge pull request #35145 from bigo-sg/lower-column-name 
add setting to lower column case when reading parquet/orc file
 2022-03-14 11:25:03 +01:00
alesapin
b149f28cd3
Merge pull request #35209 from kssenii/cache-fix 
Cache fix
 2022-03-14 11:07:40 +01:00
Maksim Kita
ad6b3693e1
Merge pull request #35123 from zhanghuajieHIT/fix_build_fail_with_gcc 
fix build fail with gcc
 2022-03-14 10:36:15 +01:00
Kseniia Sumarokova
58a2d2b458
Merge pull request #35118 from zzsmdfj/issue/#31469_MaterializedMysql_mysqlDate2CkDate32 
to #31469_MaterializedMysql_mysqlDate2CkDate32
 2022-03-14 10:32:33 +01:00
Maksim Kita
36dc7f03a4 Fixed tests 2022-03-14 09:14:41 +00:00
Azat Khuzhin
d13fc9ec44 Wait for IDiskRemote thread pool properly 
ASan report [1]:

    WARNING: ThreadSanitizer: heap-use-after-free (pid=611)
      Read of size 4 at 0x7b48006f0250 by thread T273:
        0 MemoryTracker::allocImpl() obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:105:49 (clickhouse+0xa05d0ef)
        1 MemoryTracker::allocImpl() obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:239:22 (clickhouse+0xa05d580)
        2 MemoryTracker::alloc(long) obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:246:5 (clickhouse+0xa05dea7)
        3 DB::ThreadStatus::~ThreadStatus() obj-x86_64-linux-gnu/../src/Common/ThreadStatus.cpp:154:28 (clickhouse+0xa059b01)
        4 ThreadFromGlobalPool::ThreadFromGlobalPool<>()::'lambda'()::operator()() obj-x86_64-linux-gnu/../s>

      Previous write of size 8 at 0x7b48006f0250 by main thread:
        8 std::__1::unordered_map<std::__1::basic_string<>, DB::ProcessListForUser, std::__1::hash<>, std::__1>
        9 DB::ProcessList::~ProcessList() obj-x86_64-linux-gnu/../src/Interpreters/ProcessList.h:275:7 (clickhouse+0x17b71c9c)
        10 DB::ContextSharedPart::~ContextSharedPart() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:304:5 (clickhouse+0x17b70712)
        13 DB::SharedContextHolder::reset() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:453:44 (clickhouse+0x17b3daf9)
        14 DB::Server::main()::$_8::o>

      Thread T273 (tid=946, running) created by thread T223 at:
        0 pthread_create <null> (clickhouse+0x9f7d49d)
        15 DB::WriteBufferFromS3::makeSinglepartUpload() obj-x86_64-linux-gnu/../src/IO/WriteBufferFromS3.cpp:332:9 (clickhouse+0x17a81b6a)

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/35072/19be9c8c6433ffd1bbf8eb08975d8afbaebf3d43/stress_test__thread__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-14 12:07:47 +03:00
Azat Khuzhin
9f7fae56b0 WriteBufferFromS3: remove unused headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-14 12:07:47 +03:00
Robert Schulze
514d4d2187
Implement ProtobufList - fixes ClickHouse#16436 
Introduce IO format "ProtobufList" with protobuf schema

    // schemafile.proto
    message Envelope {
      message MessageType {
        uint32 colA = 1;
        string colB = 2;
      }
      repeated MessageType mt = 1;
    }

where "Envelope" is a hard-coded/expected top-level message and
"MessageType" is a message with user-provided name containing the table
fields to export/import, e.g.

    SELECT * FROM db1.tab1 FORMAT ProtobufList SETTINGS format_schema =
    'schemafile:MessageType'

As a result, the new format wraps a list of messages (one per row) into
a single, containing message. Compare that to the schema of the existing
IO formats "Protobuf" and "ProtobufSingle":

    message MessageType {
      uint32 colA = 1;
      string colB = 2;
    }

The new format does not save space compared to the existing formats, but
it is conceptually a bit more beautiful and also more convenenient.

Implementation details:

- Created new files ProtobufList(Input|Output)Format which use the
  existing ProtobufSerializer mechanism. The goal was to reuse as much
  code as possible and avoid copypasta.

- I was torn between inheriting from I(Input|Output)Format vs.
  IRow(Input|Output)Format for ProtobufList(Input|Output)Format. The
  former is chunk-based which can be better for performance. Since the
  ProtobufSerializer mechanism is row-based but data is generally passed
  around in chunks, I decided for the latter to leverage the existing
  chunk <--> row mapping code in IRow(InputOutput)Format.

- A new ProtobufSerializer called ProtobufSerializerEnvelope was
  introduced (--> ProtobufSerializer.cpp). It represents the top-level
  message which encloses the list of inner nested messages, i.e. the
  rows.

- With the new format, parsing the schema file and matching the fields in
  the schema file to table column works like for the old formats. The only
  difference is that parsing starts one level below the "Envelope" (-->
  ProtobufSchema.cpp). This is more natural than forcing customers to
  have table columns start with "Envelope".

- Creation of the ProtobufSerializer tree also works like before. What
  is different is that we finally add a ProtobufSerializerEnvelope as
  new root of the tree. It's only purpose is to write/read the top-level
  message for the first/last row to write/read.

Caveats:

- The low-level serialization code in ProtobufWriter uses an internal
  buffer which is flushed to the output file only in endMessage().
  In the existing "Protobuf" format, this happens once per row, in the
  new format this happens only at the end of the serialization
  since row-level messages now call start/endNestedMessage(). As a
  future TODO to, the buffer should be flushed also in
  start/endNestedMessage() to reduce memory consumption.
 2022-03-14 08:04:58 +01:00
Maksim Kita
ce0c8e5597
Update JSONRowOutputFormat.cpp 2022-03-14 00:58:36 +01:00
Azat Khuzhin
19be9c8c64 Add a comment for ColumnAggregateFunction::force_data_ownership 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-13 23:28:38 +03:00
Robert Schulze
f0ba39b071
Clean up some header includes and make formatting more consistent 2022-03-13 20:24:12 +01:00
Maksim Kita
0dd807d19d
Merge pull request #34750 from kitaisreal/merge-tree-improve-insert-performance 
MergeTree improve insert performance
 2022-03-13 13:39:18 +01:00
Kseniia Sumarokova
35e5b4e8a5
Update CachedReadBufferFromRemoteFS.cpp 2022-03-13 12:37:00 +01:00
Alexey Milovidov
978877a9c0
Merge pull request #35212 from rschu1ze/cpp14-trait-aliases 
Use C++14 aliases for some type traits
 2022-03-12 22:20:14 +03:00
Alexey Milovidov
3385275003
Merge pull request #35226 from ClickHouse/timezone-in-config 
Change timezone example in server config
 2022-03-12 21:58:29 +03:00
Maksim Kita
b67f756a43 Fixed performance tests 2022-03-12 18:04:08 +00:00
Maksim Kita
3a2b3ce503 Standardize behaviour of CAST into IPv4, IPv6, toIPv4, toIPv6 functions 2022-03-12 17:12:05 +00:00
Robert Schulze
6fc6d3d452
Remove runtime conditional using constexpr if 2022-03-12 10:41:15 +01:00
zzsmdfj
88560c3917 to #35128_add_mysql_error__detail 2022-03-12 11:10:26 +08:00
Alexey Milovidov
df906dfbd4 Change comments 2022-03-11 23:46:02 +01:00
Alexey Milovidov
38fa55fff0
Merge pull request #35182 from vdimir/regex-replace-35117 
Fix replaceRegexpAll
 2022-03-12 01:36:18 +03:00
Alexey Milovidov
e4e058d667
Update ReplaceRegexpImpl.h 2022-03-12 01:35:37 +03:00
Maksim Kita
f1d2f2a9e1 Updated tests 2022-03-11 21:16:25 +00:00
Nikolai Kochetov
cdc529fa2a Add database_replicated_allow_only_replicated_engine setting. 2022-03-11 16:03:05 +00:00
HeenaBansal2009
09be30ac26 Added suggestion from review comments 2022-03-11 07:33:23 -08:00
alexey-milovidov
a3a8a1055d
Merge pull request #34919 from azat/fix-glibc2.35-build 
Fix alignment for aligned_alloc() to fix build for glibc2.35+
 2022-03-11 17:59:49 +03:00
alexey-milovidov
003619b18c
Merge pull request #34771 from azat/fix-glibc2.34-build 
Fix undefined __pthread_mutex_lock/unlock for glibc 2.34+/DISABLE_HERMETIC_BUILD
 2022-03-11 17:58:37 +03:00
alexey-milovidov
b6f511506b
Update ThreadFuzzer.cpp 2022-03-11 17:58:27 +03:00
Robert Schulze
85ce4a9bea
Use C++14 aliases for some type traits 2022-03-11 15:54:29 +01:00
alesapin
96c0e9fddf Better cmake 2022-03-11 15:47:07 +01:00
zhanghuajie
53a8987b3b fix build fail with gcc --fix warnings without disabling some parameters 2022-03-11 21:59:19 +08:00
kssenii
a18bb606a2 Update 2022-03-11 14:24:02 +01:00
alesapin
33ff2f76e5 Merge branch 'master' into standalone_keeper_build 2022-03-11 13:58:44 +01:00
kssenii
d70d7aac41 Merge master 2022-03-11 12:24:03 +01:00
kssenii
2b478e2501 Fix 2022-03-11 12:17:17 +01:00
Maksim Kita
5f8900cee6
Merge pull request #35079 from qieqieplus/master 
fix #33798
 2022-03-11 11:55:49 +01:00
Maksim Kita
6d40963cad
Merge pull request #35198 from kitaisreal/replicated-mysql-protocol-example-fix 
MaterializedMySQL protocol example fix
 2022-03-11 11:53:14 +01:00
taiyang-li
1f9e050152 add column distributed_depth for system.query_log and system.processes 2022-03-11 17:57:34 +08:00
shuchaome
7a3623d216 fix bug 2022-03-11 17:26:13 +08:00
Nikolai Kochetov
47f4bd30cd
Merge pull request #35186 from amosbird/fixwithtotalemptychunk 
Fix empty chunk in with total transform
 2022-03-11 10:24:19 +01:00
metahys
ff934cf0c2
Fix unexpected result when use -state type aggregate function in window frame (#34999) 
* Fix unexpected result when use -state type aggregate function in window frame

* fix style

* fix style

* fix test

* fix flaky test

* fix flaky test

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-03-11 11:54:17 +03:00
Kseniia Sumarokova
2bd99a4c70
Update FileSegment.cpp 2022-03-11 09:52:50 +01:00
shuchaome
46cb4483a6 Optimise by lowering schema on the beginning. Add a functional test. 2022-03-11 14:34:46 +08:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs 
Local cache for remote filesystem
 2022-03-11 07:23:10 +01:00
HeenaBansal2009
3f031df225 Code refactoring 2022-03-10 22:22:51 -08:00
Nikita Mikhaylov
909a7309ca
Delete nodes with attributes remove or replace if they didn't merge (#35141) 2022-03-10 23:40:55 +01:00
Maksim Kita
e5b85953e8 Added unit tests 2022-03-10 21:45:31 +00:00
Maksim Kita
6b916c7bb5 Fixed tests 2022-03-10 21:45:31 +00:00
Maksim Kita
5b2be4d3b8 Fixed tests 2022-03-10 21:45:31 +00:00
Maksim Kita
cbe059f4bd Updated IColumn interface to support getting stable permutation 2022-03-10 21:45:31 +00:00
Maksim Kita
765cd09d06 MergeTree improve insert performance 2022-03-10 21:45:31 +00:00
Maksim Kita
d70b7a1a41 ReplicatedMySQL protocol example fix 2022-03-10 21:26:39 +00:00
Anton Popov
37efe2ddb5
Apply suggestions from code review 
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
 2022-03-10 22:24:19 +01:00
alesapin
c0d8ccc91b
Merge pull request #35178 from Varinara/master 
Added disk_name to system.part_log
 2022-03-10 22:22:37 +01:00
alexey-milovidov
58e53b06a6
Merge pull request #35134 from azat/optimize_skip_unused_shards_rewrite_in-signed 
Fix `optimize_skip_unused_shards_rewrite_in` for signed columns and negative values
 2022-03-10 23:12:05 +03:00
vdimir
c66fb7cf6e
Attempt to fix replaceRegexpAll 2022-03-10 19:57:13 +00:00
Kseniia Sumarokova
5149bd2d29
Merge pull request #35172 from kssenii/add-non-retriable-errors-for-url 
Add non-retriable errors check for HTTP read buffer
 2022-03-10 20:28:37 +01:00
kssenii
c86955deb2 Fix typo 2022-03-10 20:22:39 +01:00
kssenii
ddcb020d26 Fix test 2022-03-10 19:16:26 +01:00
mergify[bot]
059682a0cd
Merge branch 'master' into asof_ftw 2022-03-10 17:29:38 +00:00
HeenaBansal2009
d981463b05 Added RemoteHostFilter check for MYSQL and postgresSQL 2022-03-10 08:58:48 -08:00
mergify[bot]
9f4ebc313b
Merge branch 'master' into master 2022-03-10 16:55:27 +00:00
Maksim Kita
83de2f66d1
Merge pull request #35032 from ClickHouse/fix-query-profiler-on-bad-linux-kernels 
Avoid division by zero in Query Profiler if Linux kernel has a bug
 2022-03-10 17:49:27 +01:00
Maksim Kita
493169910b
Merge pull request #35174 from zhangyifan27/fix_typo 
fix typos
 2022-03-10 17:10:44 +01:00
Kseniia Sumarokova
3fc399b6e9
Merge pull request #35158 from kssenii/fix-materialized-postgresql 
Fix materialised postrgesql adding new table after manually removing it
 2022-03-10 17:02:32 +01:00
Kseniia Sumarokova
e30b0c5d57
Merge pull request #35162 from kssenii/fix-materialized-postgresql-table-override 
Fix materialised postgres `table overrides` for partition by, etc
 2022-03-10 17:01:24 +01:00
Varinara
f5523f7ff0 added disk_name to system.part_log 2022-03-10 18:44:19 +03:00
mergify[bot]
8434dd424a
Merge branch 'master' into fix_typo 2022-03-10 15:30:06 +00:00
Amos Bird
a1b61dabfd
Fix empty chunk in with total transform. 2022-03-10 23:27:36 +08:00
Kruglov Pavel
a506120646
Fix bug in schema inference in s3 table function (#35176) 2022-03-10 15:16:07 +01:00
mergify[bot]
df01290e73
Merge branch 'master' into fix_typo 2022-03-10 13:35:04 +00:00
Miel Donkers
4a95e6d602
Parsing YAML config to XML leads to incorrect structures (#35135) 2022-03-10 13:09:48 +01:00
kssenii
1dc3f36a11 Better 2022-03-10 12:19:20 +01:00
Vladimir C
84af08b1a1
Merge pull request #35116 from bigo-sg/snappy_bug 2022-03-10 11:47:37 +01:00
kssenii
787a0805a5 Merge master 2022-03-10 11:42:19 +01:00
kssenii
050802b99b Add non-retriable errors 2022-03-10 11:38:46 +01:00
Kseniia Sumarokova
e6ee891c9c
Merge pull request #34957 from bigo-sg/hive_random_access_file_cache 
Optimization for first time to read a random access readbuffer in hive
 2022-03-10 11:36:22 +01:00
taiyang-li
ee8fc2730e remote comment 2022-03-10 18:29:52 +08:00
zhangyifan27
e6fa9f699a fix typo 2022-03-10 18:29:42 +08:00
taiyang-li
382417d893 fix bug 2022-03-10 18:29:00 +08:00
kssenii
3cd1da1e11 Fix 2022-03-10 11:11:59 +01:00
taiyang-li
2cd469f113 support print ast in dot format 2022-03-10 18:04:35 +08:00
kssenii
bd68d1d6d0 Update test 2022-03-10 11:02:12 +01:00
HeenaBansal2009
3ce9397246 Added Suggestions from Code review 2022-03-09 20:35:01 -08:00
lgbo-ustc
fdd423a3da fixed code style 2022-03-10 12:13:19 +08:00
lgbo-ustc
e4883f31b7 update tests 
1. fixed code style in src/IO/tests/gtest_hadoop_snappy_decoder.cpp
2. enable tests 01060_avro.sh
 2022-03-10 09:46:43 +08:00
kssenii
af9d8d278e Fix 2022-03-09 19:25:43 +01:00
kssenii
d089b0ac86 Fix race 2022-03-09 18:14:28 +01:00
mergify[bot]
78c8029aeb
Merge branch 'master' into stress-test 2022-03-09 16:22:45 +00:00
mergify[bot]
c326ebd67f
Merge branch 'master' into issue/#34890_fix_input_format_null_as_default_bug 2022-03-09 15:59:04 +00:00
Vladimir C
ce266b5a3e
Merge pull request #35146 from amosbird/fixpartitionprunerin 2022-03-09 13:23:45 +01:00
Nikita Mikhaylov
d749295222
Fix hardcoded page size (#35129) 2022-03-09 12:35:23 +01:00
shuchaome
b7cd85df6b remove unused column_names in ORCBlockInputFormat 2022-03-09 18:16:22 +08:00
kssenii
6571c7faa0 Clean up code 2022-03-09 10:36:52 +01:00
shuchaome
bb50133424
Apply suggestions from code review 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-03-09 17:32:27 +08:00
shuchaome
9647818adc add unlikely for performance 2022-03-09 17:02:07 +08:00
kssenii
96789fa5ce Fix test, disable cache for DiskCacheWrapper 2022-03-09 09:46:04 +01:00
shuchaome
8027bb1e32 modify code style 2022-03-09 16:32:18 +08:00
Nikolai Kochetov
6bfee7aca2
Merge pull request #35132 from azat/parallel_distributed_insert_select-view 
Support view() for parallel_distributed_insert_select
 2022-03-09 09:10:34 +01:00
Nikolai Kochetov
c364908061
Merge pull request #35094 from amosbird/getridofredundantplan 
Get rid of duplicate query planing.
 2022-03-09 09:10:20 +01:00
shuchaome
56795b831d add setting to lower column case when reading parquet/orc file 2022-03-09 16:07:02 +08:00
zzsmdfj
00794fd4a4 to #31469_MaterializedMysql_mysqlDate2CkDate32-add MySQLDataTypesSupport setting 2022-03-09 15:58:43 +08:00
Azat Khuzhin
e2960e1a52 Avoid MEMORY_LIMIT_EXCEEDED during INSERT into Buffer with AggregateFunction 
In case of Buffer table has columns of AggregateFunction type,
aggregate states for such columns will be allocated from the query
context but those states can be destroyed from the server context (in
case of background flush), and thus memory will be leaked from the query
since aggregate states can be shared, and eventually this will lead to
MEMORY_LIMIT_EXCEEDED error.

To avoid this, prohibit sharing the aggregate states.

But note, that this problem only about memory accounting, not memory
usage itself.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-09 10:57:49 +03:00
Azat Khuzhin
4118b065e6 Export IColumn::ensureOwnership() interface 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-09 10:57:49 +03:00
Amos Bird
a19224bc9b
Fix partition pruner: non-monotonic function IN 2022-03-09 15:48:42 +08:00
Azat Khuzhin
ec6bae2b2a Fix optimize_skip_unused_shards_rewrite_in for signed columns 
Fixes: #35131
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-09 08:29:28 +03:00
lgbo-ustc
8dc8c87fd1 add a test case 2022-03-09 10:03:04 +08:00
kssenii
16e84c265b Fix build, update test 2022-03-08 21:41:58 +01:00
Azat Khuzhin
3a5a39a9df Do not delay final part writing by default 
For async s3 writes final part flushing was defered until all the INSERT
block was processed, however in case of too many partitions/columns you
may exceed max_memory_usage limit (since each stream has overhead).

Introduce max_insert_delayed_streams_for_parallel_writes (with default
to 1000 for S3, 0 otherwise), to avoid this.

This should "Memory limit exceeded" errors in performance tests.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:17:36 +03:00
Azat Khuzhin
4200b56f3a Introduce IDisk::supportParallelWrite() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:17:36 +03:00
Azat Khuzhin
4843e210c3 Support view() for parallel_distributed_insert_select 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:05:57 +03:00
Azat Khuzhin
75da778d10 Tiny cmake refactoring 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:05:57 +03:00
Azat Khuzhin
fd3f7347f3 Remove unused DBMS_COMMON_LIBRARIES 
Fixes: 4f8438bb34 ("cmake: do not allow internal libstdc++ usage")
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:02:18 +03:00
Azat Khuzhin
132bbce29c Add ability to get SELECT query from TableFunctionView 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 22:02:18 +03:00
Azat Khuzhin
ced34dea84 Take flush_time into account for scheduling background flush of the Buffer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 21:58:10 +03:00
kssenii
5260822964 Merge master 2022-03-08 18:21:28 +01:00
kssenii
e231c3a3e0 Fix split build 2022-03-08 18:05:55 +01:00
Azat Khuzhin
a871036361
Fix parallel_reading_from_replicas with clickhouse-bechmark (#34751) 
* Use INITIAL_QUERY for clickhouse-benchmark

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix parallel_reading_from_replicas with clickhouse-bechmark

Before it produces the following error:

    $ clickhouse-benchmark --stacktrace -i1 --query "select * from remote('127.1', default.data_mt) limit 10" --allow_experimental_parallel_reading_from_replicas=1 --max_parallel_replicas=3
    Loaded 1 queries.
    Logical error: 'Coordinator for parallel reading from replicas is not initialized'.
    Aborted (core dumped)

Since it uses the same code, i.e RemoteQueryExecutor ->
MultiplexedConnections, which enables coordinator if it was requested
from settings, but it should be done only for non-initial queries, i.e.
when server send connection to another server.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix 02226_parallel_reading_from_replicas_benchmark for older shellcheck

By shellcheck 0.8 does not complains, while on CI shellcheck 0.7.0 and
it does complains [1]:

    In 02226_parallel_reading_from_replicas_benchmark.sh line 17:
        --allow_experimental_parallel_reading_from_replicas=1
        ^-- SC2191: The = here is literal. To assign by index, use ( [index]=value ) with no spaces. To keep as literal, quote it.

    Did you mean:
        "--allow_experimental_parallel_reading_from_replicas=1"

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/34751/d883af711822faf294c876b017cbf745b1cda1b3/style_check__actions_/shellcheck_output.txt

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 16:42:29 +01:00
zhanghuajie
11dde7c127 fix build fail with gcc 2022-03-08 22:34:51 +08:00
Azat Khuzhin
c4b6342853
Improvements for parallel_distributed_insert_select (and related) (#34728) 
* Add a warning if parallel_distributed_insert_select was ignored

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Respect max_distributed_depth for parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Print warning for non applied parallel_distributed_insert_select only for initial query

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Remove Cluster::getHashOfAddresses()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Forbid parallel_distributed_insert_select for remote()/cluster() with different addresses

Before it uses empty cluster name (getClusterName()) which is not
correct, compare all addresses instead.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix max_distributed_depth check

max_distributed_depth=1 must mean not more then one distributed query,
not two, since max_distributed_depth=0 means no limit, and
distribute_depth is 0 for the first query.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Fix INSERT INTO remote()/cluster() with parallel_distributed_insert_select

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Add a test for parallel_distributed_insert_select with cluster()/remote()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Return <remote> instead of empty cluster name in Distributed engine

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Make user with sharding_key and w/o in remote()/cluster() identical

Before with sharding_key the user was "default", while w/o it it was
empty.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-03-08 15:24:39 +01:00
kssenii
f3cba6f270 Remove already fixed crutch 2022-03-08 14:30:21 +01:00
lgbo-ustc
7f89a1bcf3 add some usage test 2022-03-08 20:00:39 +08:00