Maksim Kita
cca121f0da
Fixed minSampleSize functions
2022-03-17 15:30:59 +00:00
Maksim Kita
9bec85b6fe
Merge pull request #35358 from kitaisreal/function-h3-get-res-0-indexes-crash-fix
...
Function h3GetRes0Indexes crash fix
2022-03-17 16:29:33 +01:00
Antonio Andelic
b789a41662
Fix ASAN problems
2022-03-17 15:18:20 +00:00
Anton Popov
11ab24acb1
Merge pull request #35356 from CurtizJ/fix-clang-tidy
...
Fix clang-tidy
2022-03-17 16:17:23 +01:00
Vitaly Baranov
98e48d7b54
SHOW CREATE ROW POLICY now always shows AS permissive/restrictive if there is TO section.
2022-03-17 15:28:53 +01:00
Vitaly Baranov
3e950d79b3
Add new setting rbac_version.
2022-03-17 15:28:51 +01:00
Vitaly Baranov
470582e262
Change type of RowPolicyKind: bool -> enum.
2022-03-17 15:28:05 +01:00
mergify[bot]
28734562bb
Merge branch 'master' into issue/#34890_fix_input_format_null_as_default_bug
2022-03-17 13:24:34 +00:00
Antonio Andelic
607f785e48
Revert "Merge pull request #35145 from bigo-sg/lower-column-name"
...
This reverts commit ebf72bf61df1b812bdc1
2022-03-17 12:31:43 +00:00
Sergei Trifonov
5655045854
add comments in threadPoolCallbackRunner.h
2022-03-17 13:21:32 +01:00
Maksim Kita
3f45755822
Merge pull request #34354 from achimbab/minSampleSize
...
Implement functions for minimum-sample-size-calculators
2022-03-17 12:47:53 +01:00
Maksim Kita
1bf3ff67b2
Function h3GetRes0Indexes crash fix
2022-03-17 11:45:25 +00:00
Nikolai Kochetov
ee9c2ec735
Merge pull request #34780 from azat/mt-delayed-part-flush
...
Do not delay final part writing by default (fixes possible Memory limit exceeded during INSERT)
2022-03-17 12:30:51 +01:00
Anton Popov
fe2dc85788
fix clang-tidy
2022-03-17 11:16:32 +00:00
mergify[bot]
5295dd0ef2
Merge branch 'master' into stress-test
2022-03-17 11:10:37 +00:00
Maksim Kita
9e88f3b4b9
Merge pull request #34568 from bharatnc/ncb/h3-misc-funcs-3
...
add remaining h3 miscellaneous functions
2022-03-17 11:55:58 +01:00
alesapin
bf5b3a856d
Rename some variables in keeper
2022-03-17 11:55:15 +01:00
kssenii
a2cd165d38
Add remote host filter
2022-03-17 11:48:42 +01:00
alesapin
6ab309b25d
Merge pull request #35320 from zhangyuli1/remove_wchs_from_default_4lw
...
remove wchc from four_letter_word_white_list
2022-03-17 11:44:30 +01:00
Robert Schulze
6e1d7a31bc
Fix build + typo
2022-03-17 11:41:20 +01:00
alesapin
92ab33f560
Merge pull request #35348 from ClickHouse/fix_yet_another_zero_copy_bug
...
Fix race in zero copy replication
2022-03-17 11:34:38 +01:00
alesapin
bb251938dc
Merge pull request #35344 from ClickHouse/changelog-22.3
...
Changelog 22.3
2022-03-17 11:25:36 +01:00
Antonio Andelic
103a3fa140
Merge branch 'master' into parallel-downloading-url-engine
2022-03-17 10:11:58 +00:00
Antonio Andelic
44de5a581c
Fix split build
2022-03-17 10:09:16 +00:00
alesapin
457fa0d1ba
Merge pull request #23932 from CurtizJ/dynamic-columns
...
Support of dynamic subcolumns (JSON data type)
2022-03-17 10:45:46 +01:00
Maksim Kita
4d8c2b2009
Merge pull request #35347 from rschu1ze/enable-if-to-concepts
...
Replace a few uses of enable_if for SFINAE by concepts
2022-03-17 10:34:08 +01:00
Maksim Kita
f837f609f3
Merge pull request #35240 from kitaisreal/cast-to-ip-address-refactoring
...
Standardize behaviour of CAST into IPv4, IPv6, toIPv4, toIPv6 functions
2022-03-17 10:26:32 +01:00
Antonio Andelic
cc623bbf5e
Fix redirect_url tests
2022-03-17 08:03:22 +00:00
Alexey Milovidov
68ef49ea51
Fix something stupid
2022-03-17 05:57:13 +01:00
Alexey Milovidov
edbdefeb83
Merge remote-tracking branch 'origin/master' into changelog-22.3
2022-03-17 04:33:00 +01:00
Anton Popov
416c7f2820
fix clang-tidy
2022-03-17 01:37:54 +00:00
Alexey Milovidov
b70dd1fc36
More features
2022-03-17 02:33:29 +01:00
alesapin
03ff6d52f6
Fix race in zero copy replication
2022-03-16 21:51:33 +01:00
Anton Popov
de2cc23e15
fix race
2022-03-16 20:16:59 +00:00
Maksim Kita
346c6b9d79
Fixed tests
2022-03-16 19:59:31 +00:00
Robert Schulze
b56c28d841
Replace a few uses of enable_if for SFINAE by concepts
...
- enable_if is usually regarded as fragile and unreadable
- C++20 concepts are much easier to read and produce more expressive
error messages
2022-03-16 19:51:38 +01:00
Sergei Trifonov
bbe657ba24
allow parallelization of multipart S3 storage upload fixes #34350
2022-03-16 19:41:41 +01:00
mergify[bot]
3e072da80e
Merge branch 'master' into remove_wchs_from_default_4lw
2022-03-16 18:31:02 +00:00
Vitaly Baranov
39614e6e15
Merge pull request #35276 from vitlibar/fix-code-style-and-minor-corrections-after-allow-no-password
...
Fix code style and other minor corrections after implementing allow_no_password.
2022-03-16 18:55:05 +01:00
Anton Popov
2ced42ed41
add experimental settings for Object type
2022-03-16 16:51:23 +00:00
Maksim Kita
9ba53ae037
Merge pull request #34888 from kitaisreal/performance-tests-fix
...
Performance tests fix
2022-03-16 17:19:16 +01:00
Maksim Kita
dc31a4175f
Fixed style check
2022-03-16 16:16:08 +00:00
Antonio Andelic
0979155f2f
Address PR comments
2022-03-16 15:45:17 +00:00
Anton Popov
0ba78c3c3a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-16 15:28:09 +00:00
Kseniia Sumarokova
663c8e9e4a
Merge pull request #35275 from kssenii/fix-logical-error-in-cache
...
Fix logical error in remote fs cache
2022-03-16 15:01:41 +01:00
avogar
f7c5fe14e4
Fix schema inference for TSKV format while using small max_read_buffer_size
2022-03-16 13:53:50 +00:00
Antonio Andelic
84f86bf3da
Limit number of bytes read per worker
2022-03-16 12:34:14 +00:00
Nikita Mikhaylov
96c6325fc3
Merging #33398 (proportionsZTest) ( #35157 )
2022-03-16 13:17:07 +01:00
Kruglov Pavel
4a579768ef
Merge pull request #35311 from vdimir/pmj-dup-rows
...
Attempt to fix merge join duplicate rows
2022-03-16 12:48:29 +01:00
Kruglov Pavel
906b68dbcb
Merge pull request #35291 from Avogar/segfault-in-schema-inference
...
Fix possible segfault in JSONEachRow schema inference
2022-03-16 12:24:35 +01:00
Kruglov Pavel
ec6968c17e
Merge pull request #35313 from kssenii/fix-pg-db
...
Fix possible segfault in DatabasePostgreSQL::getCreateTableQuery
2022-03-16 12:23:25 +01:00
Maksim Kita
ed74a464bd
Fixed tests
2022-03-16 11:21:18 +00:00
Vitaly Baranov
e690d28fef
Update src/Access/AccessControl.cpp
...
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-03-16 12:03:09 +01:00
Nikolai Kochetov
bcd3c76296
Merge pull request #35214 from ClickHouse/add-database_replicated_allow_only_replicated_engine-setting
...
Add database_replicated_allow_only_replicated_engine setting.
2022-03-16 11:24:32 +01:00
Antonio Andelic
2291b7c1e1
Merge branch 'master' into parallel-downloading-url-engine
2022-03-16 09:59:52 +00:00
zhangyuli1
3cba1177ee
remove wchc from four_letter_word_white_list
2022-03-16 17:40:49 +08:00
Antonio Andelic
3465664871
Attach thread pool threads to counters
2022-03-16 09:26:01 +00:00
Robert Schulze
0d2ece6d91
Merge branch 'ClickHouse:master' into protobuf-batch-write
2022-03-16 09:43:33 +01:00
Kseniia Sumarokova
b4e03eda8b
Update DatabasePostgreSQL.cpp
2022-03-16 01:04:40 +01:00
kssenii
3a784fe611
Fix
2022-03-16 00:52:51 +01:00
Kruglov Pavel
2b7e89e171
Merge pull request #35300 from Avogar/fix-bzip2
...
Fix possible assertion while using bzip2 compression
2022-03-15 22:17:37 +01:00
Kruglov Pavel
20e17a6d3e
Merge pull request #35296 from Avogar/fix-lz4
...
Fix possible segfault while using lz4 compression
2022-03-15 22:17:14 +01:00
Kruglov Pavel
6fd1a8281f
Merge pull request #35295 from Avogar/fix-lzma
...
Fix possible Assertion failed while using lzma compression
2022-03-15 22:14:35 +01:00
Nikolai Kochetov
a380aa6b8a
Merge pull request #35294 from ClickHouse/reload-remote_url_allow_hosts
...
Reload remote_url_allow_hosts after config update.
2022-03-15 22:07:16 +01:00
Nikolai Kochetov
cf260b8508
Merge pull request #35305 from ClickHouse/try-to-fix-using-deleted-memory-tracker-inside-s3-disk
...
Maybe fix use-after-free inside S3 upload thread
2022-03-15 22:06:17 +01:00
Vladimir C
e022245a33
Merge pull request #35193 from vdimir/issue-34776
2022-03-15 19:56:09 +01:00
vdimir
695708d57c
[wip] merge join duplicate rows
2022-03-15 18:51:04 +00:00
mergify[bot]
a5e93ea30b
Merge branch 'master' into fix-bzip2
2022-03-15 18:42:26 +00:00
mergify[bot]
381a5e4f6e
Merge branch 'master' into fix-lz4
2022-03-15 18:42:14 +00:00
Maksim Kita
8bdb906acb
Fixed tests
2022-03-15 15:43:39 +00:00
Maksim Kita
08bb39d869
Fixed performance tests
2022-03-15 15:43:39 +00:00
Maksim Kita
fa4508bcb9
Fixed tests
2022-03-15 15:35:25 +00:00
Nikolai Kochetov
b4aed421de
Reset THreadPool's thread memory tracker parent to global after S3 upload task is finished.
2022-03-15 15:33:43 +00:00
Maksim Kita
5ffb12eb8c
Merge pull request #35289 from bigo-sg/getmacro_const
...
return const for function getMacro if not distributed query
2022-03-15 16:32:07 +01:00
avogar
ec3c5f86ad
Fix possible assertion while using bzip2 compression
2022-03-15 14:23:52 +00:00
Nikolai Kochetov
d6710b3354
Always clear hosts.
2022-03-15 14:08:56 +00:00
Kseniia Sumarokova
afec02fe18
Merge pull request #35264 from kssenii/test-mode-for-local
...
Support test mode for clickhouse-local
2022-03-15 14:53:01 +01:00
avogar
0cfaf48e1d
Fix possible segfault while using lz4 compression
2022-03-15 13:46:14 +00:00
Vladimir C
02414f1e88
Merge branch 'master' into issue-34776
2022-03-15 14:24:51 +01:00
Kruglov Pavel
24ac217be3
Remove redundant include
2022-03-15 14:07:31 +01:00
avogar
90ce1709d7
Fix possible Assertion failed while using lzma compression
2022-03-15 13:00:53 +00:00
Nikolai Kochetov
97aa6c82ce
Reload remote_url_allow_hosts after config update.
2022-03-15 13:00:31 +00:00
Kruglov Pavel
2e2ee2856e
Merge pull request #35281 from Avogar/fix-brotli
...
Fix possible Abort while using Brotli compression
2022-03-15 13:58:16 +01:00
Anton Popov
b5a04ff938
Merge pull request #35207 from bigo-sg/add_distributed_depth_in_querylog
...
Add column distributed_depth for system.query_log and system.processes
2022-03-15 13:38:21 +01:00
avogar
e2d1e643f2
Fix possible segfault in JSONEachRow schema inference
2022-03-15 11:44:15 +00:00
alesapin
bfe2872ab3
Merge pull request #35278 from kssenii/add-cache-optimization
...
Add remote fs cache optimization
2022-03-15 11:50:25 +01:00
Robert Schulze
23122cb327
Fix review comments
...
ParquetBlockOutputFormat.cpp:
- undo unrelated formatting
ProtobufSerializer.cpp:
- undef debug tracing
- simplify logic in writeRow()
ProtobufSchemas.cpp:
- restore original search in cache by message type
2022-03-15 11:27:17 +01:00
Vladimir C
af88d3f133
Merge pull request #35173 from bigo-sg/dump_ast_in_dot
2022-03-15 11:18:41 +01:00
taiyang-li
c08e3c0dd2
return const for function getMacro if not distributed query
2022-03-15 17:06:16 +08:00
Kseniia Sumarokova
427ae808ba
Merge pull request #35191 from DevTeamBK/RemoteHostFilter_improvement
...
Added RemoteHostFilter check for MYSQL and postgresSQL
2022-03-15 09:58:08 +01:00
mergify[bot]
a7298a98b9
Merge branch 'master' into fix-logical-error-in-cache
2022-03-15 08:29:16 +00:00
mergify[bot]
9cdd6ec7ab
Merge branch 'master' into test-mode-for-local
2022-03-15 08:28:40 +00:00
mergify[bot]
d613ad9445
Merge branch 'master' into fix-brotli
2022-03-15 08:25:12 +00:00
Antonio Andelic
067b79b00b
Merge branch 'master' into parallel-downloading-url-engine
2022-03-15 07:55:41 +00:00
taiyang-li
3688ff10e0
fix bug
2022-03-15 12:19:08 +08:00
lgbo-ustc
abfaa82bca
fixed hive query bugs
2022-03-15 12:01:34 +08:00
Anton Popov
ccbddd53a3
fix mutations in tables with enabled sparse columns
2022-03-15 01:48:21 +00:00
mergify[bot]
ee3f4fb0c3
Merge branch 'master' into add-cache-optimization
2022-03-14 20:58:52 +00:00
alesapin
fbb1ebd9b8
Merge pull request #35274 from CurtizJ/fix-check-table-sparse-columns
...
Fix check table in case when there exist sparse columns
2022-03-14 21:56:04 +01:00
Vitaly Baranov
1eb2e8693e
Fix code style and other minor corrections after implementing allow_no_password.
2022-03-14 20:55:34 +01:00
Nikolai Kochetov
7526c36f30
Remove comment
2022-03-14 19:38:03 +00:00
Nikolai Kochetov
ce94b560bb
Add test.
2022-03-14 19:30:52 +00:00
avogar
4b12c6e6b3
Fix possible Abort while using Brotli compression
2022-03-14 18:58:05 +00:00
Maksim Kita
282dd98779
Fix build
2022-03-14 18:17:36 +00:00
Maksim Kita
3ca40d5d2e
Fixed style check
2022-03-14 18:17:35 +00:00
Maksim Kita
5cf0e0c0f3
Fixed build
2022-03-14 18:17:35 +00:00
Maksim Kita
2fdcf53a76
Fix clang-tidy warnings in Server, Storages folders
2022-03-14 18:17:35 +00:00
Maksim Kita
2665724301
Fix clang-tidy warnings in Parsers, Processors, QueryPipeline folders
2022-03-14 18:17:35 +00:00
Maksim Kita
e30117a3d6
Fix clang-tidy warnings in Interpreters, IO folders
2022-03-14 18:17:35 +00:00
Maksim Kita
538f8cbaad
Fix clang-tidy warnings in Disks, Formats, Functions folders
2022-03-14 18:17:35 +00:00
Maksim Kita
1d674123a9
Fix clang-tidy warnings in Databases, DataTypes, Dictionaries folders
2022-03-14 18:17:35 +00:00
Maksim Kita
65c52298b6
Fix clang-tidy warnings in Compression, Coordination, Core folders
2022-03-14 18:17:35 +00:00
Maksim Kita
e7772ed434
Fix clang-tidy warnings in Common folder
2022-03-14 18:17:35 +00:00
Maksim Kita
07b0845110
Fix clang-tidy warnings in Backups, Client, Columns folders
2022-03-14 18:17:35 +00:00
Maksim Kita
5147da3355
Fix clang-tidy warnings in AggregateFunction folder
2022-03-14 18:17:35 +00:00
Maksim Kita
e14cfd5dcd
Fix clang-tidy warnings in Access folder
2022-03-14 18:17:35 +00:00
Anton Popov
063917786e
minor fixes
2022-03-14 17:29:18 +00:00
kssenii
53d23d901e
Fix
2022-03-14 18:11:59 +01:00
kssenii
fc9c6eab9d
Add optimization
2022-03-14 17:58:19 +01:00
Anton Popov
36ec379aeb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-14 16:28:35 +00:00
Vitaly Baranov
4af61fb9d3
Merge pull request #34738 from DevTeamBK/Issue-33953
...
New setting in Server Configuration to on/off AuthType Plaintext_password and No_password
2022-03-14 17:09:46 +01:00
Antonio Andelic
5f8d9885b3
Ignore read bytes
2022-03-14 15:50:30 +00:00
Anton Popov
428bbd6377
fix check table in case when there exist sparse columns
2022-03-14 15:22:23 +00:00
Heena Bansal
c774458f96
Update src/Storages/StoragePostgreSQL.cpp
...
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
2022-03-14 10:38:12 -04:00
Heena Bansal
01c8b2f71e
Update src/Storages/StorageMySQL.cpp
...
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
2022-03-14 10:37:59 -04:00
alesapin
b13b415d3f
Merge pull request #35257 from azat/fix-s3-pool-shutdown
...
Wait for IDiskRemote thread pool properly
2022-03-14 13:56:37 +01:00
Antonio Andelic
d3353f3f0c
Merge branch 'master' into parallel-downloading-url-engine
2022-03-14 12:47:23 +00:00
Antonio Andelic
556fe2bcc5
Don't send RANGE with HEAD call
2022-03-14 12:46:22 +00:00
vdimir
db46c2ada8
Fix reading port from config
2022-03-14 11:58:47 +00:00
Nikolai Kochetov
ff09076524
A little bit better.
2022-03-14 11:31:24 +00:00
Antonio Andelic
ed3d71d83f
Merge pull request #35072 from azat/buffer-memory-tracker-leak
...
Avoid MEMORY_LIMIT_EXCEEDED during INSERT into Buffer with AggregateFunction
2022-03-14 12:31:04 +01:00
kssenii
199188be08
Support test mode for clickhouse-local
2022-03-14 12:02:06 +01:00
alesapin
98144bc1f5
Merge pull request #35031 from ClickHouse/standalone_keeper_build
...
Building small keeper binary
2022-03-14 11:27:25 +01:00
Antonio Andelic
ebf72bf61d
Merge pull request #35145 from bigo-sg/lower-column-name
...
add setting to lower column case when reading parquet/orc file
2022-03-14 11:25:03 +01:00
alesapin
b149f28cd3
Merge pull request #35209 from kssenii/cache-fix
...
Cache fix
2022-03-14 11:07:40 +01:00
Maksim Kita
ad6b3693e1
Merge pull request #35123 from zhanghuajieHIT/fix_build_fail_with_gcc
...
fix build fail with gcc
2022-03-14 10:36:15 +01:00
Kseniia Sumarokova
58a2d2b458
Merge pull request #35118 from zzsmdfj/issue/#31469_MaterializedMysql_mysqlDate2CkDate32
...
to #31469_MaterializedMysql_mysqlDate2CkDate32
2022-03-14 10:32:33 +01:00
Antonio Andelic
9dda2863d3
Split download threads when multiple URLs are used
2022-03-14 09:27:09 +00:00
Maksim Kita
36dc7f03a4
Fixed tests
2022-03-14 09:14:41 +00:00
Azat Khuzhin
d13fc9ec44
Wait for IDiskRemote thread pool properly
...
ASan report [1]:
WARNING: ThreadSanitizer: heap-use-after-free (pid=611)
Read of size 4 at 0x7b48006f0250 by thread T273:
0 MemoryTracker::allocImpl() obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:105:49 (clickhouse+0xa05d0ef)
1 MemoryTracker::allocImpl() obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:239:22 (clickhouse+0xa05d580)
2 MemoryTracker::alloc(long) obj-x86_64-linux-gnu/../src/Common/MemoryTracker.cpp:246:5 (clickhouse+0xa05dea7)
3 DB::ThreadStatus::~ThreadStatus() obj-x86_64-linux-gnu/../src/Common/ThreadStatus.cpp:154:28 (clickhouse+0xa059b01)
4 ThreadFromGlobalPool::ThreadFromGlobalPool<>()::'lambda'()::operator()() obj-x86_64-linux-gnu/../s>
Previous write of size 8 at 0x7b48006f0250 by main thread:
8 std::__1::unordered_map<std::__1::basic_string<>, DB::ProcessListForUser, std::__1::hash<>, std::__1>
9 DB::ProcessList::~ProcessList() obj-x86_64-linux-gnu/../src/Interpreters/ProcessList.h:275:7 (clickhouse+0x17b71c9c)
10 DB::ContextSharedPart::~ContextSharedPart() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:304:5 (clickhouse+0x17b70712)
13 DB::SharedContextHolder::reset() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:453:44 (clickhouse+0x17b3daf9)
14 DB::Server::main()::$_8::o>
Thread T273 (tid=946, running) created by thread T223 at:
0 pthread_create <null> (clickhouse+0x9f7d49d)
15 DB::WriteBufferFromS3::makeSinglepartUpload() obj-x86_64-linux-gnu/../src/IO/WriteBufferFromS3.cpp:332:9 (clickhouse+0x17a81b6a)
[1]: https://s3.amazonaws.com/clickhouse-test-reports/35072/19be9c8c6433ffd1bbf8eb08975d8afbaebf3d43/stress_test__thread__actions_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-14 12:07:47 +03:00
Azat Khuzhin
9f7fae56b0
WriteBufferFromS3: remove unused headers
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-14 12:07:47 +03:00
Robert Schulze
514d4d2187
Implement ProtobufList - fixes ClickHouse#16436
...
Introduce IO format "ProtobufList" with protobuf schema
// schemafile.proto
message Envelope {
message MessageType {
uint32 colA = 1;
string colB = 2;
}
repeated MessageType mt = 1;
}
where "Envelope" is a hard-coded/expected top-level message and
"MessageType" is a message with user-provided name containing the table
fields to export/import, e.g.
SELECT * FROM db1.tab1 FORMAT ProtobufList SETTINGS format_schema =
'schemafile:MessageType'
As a result, the new format wraps a list of messages (one per row) into
a single, containing message. Compare that to the schema of the existing
IO formats "Protobuf" and "ProtobufSingle":
message MessageType {
uint32 colA = 1;
string colB = 2;
}
The new format does not save space compared to the existing formats, but
it is conceptually a bit more beautiful and also more convenenient.
Implementation details:
- Created new files ProtobufList(Input|Output)Format which use the
existing ProtobufSerializer mechanism. The goal was to reuse as much
code as possible and avoid copypasta.
- I was torn between inheriting from I(Input|Output)Format vs.
IRow(Input|Output)Format for ProtobufList(Input|Output)Format. The
former is chunk-based which can be better for performance. Since the
ProtobufSerializer mechanism is row-based but data is generally passed
around in chunks, I decided for the latter to leverage the existing
chunk <--> row mapping code in IRow(InputOutput)Format.
- A new ProtobufSerializer called ProtobufSerializerEnvelope was
introduced (--> ProtobufSerializer.cpp). It represents the top-level
message which encloses the list of inner nested messages, i.e. the
rows.
- With the new format, parsing the schema file and matching the fields in
the schema file to table column works like for the old formats. The only
difference is that parsing starts one level below the "Envelope" (-->
ProtobufSchema.cpp). This is more natural than forcing customers to
have table columns start with "Envelope".
- Creation of the ProtobufSerializer tree also works like before. What
is different is that we finally add a ProtobufSerializerEnvelope as
new root of the tree. It's only purpose is to write/read the top-level
message for the first/last row to write/read.
Caveats:
- The low-level serialization code in ProtobufWriter uses an internal
buffer which is flushed to the output file only in endMessage().
In the existing "Protobuf" format, this happens once per row, in the
new format this happens only at the end of the serialization
since row-level messages now call start/endNestedMessage(). As a
future TODO to, the buffer should be flushed also in
start/endNestedMessage() to reduce memory consumption.
2022-03-14 08:04:58 +01:00
Maksim Kita
ce0c8e5597
Update JSONRowOutputFormat.cpp
2022-03-14 00:58:36 +01:00
Azat Khuzhin
19be9c8c64
Add a comment for ColumnAggregateFunction::force_data_ownership
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-13 23:28:38 +03:00
Robert Schulze
f0ba39b071
Clean up some header includes and make formatting more consistent
2022-03-13 20:24:12 +01:00
Maksim Kita
0dd807d19d
Merge pull request #34750 from kitaisreal/merge-tree-improve-insert-performance
...
MergeTree improve insert performance
2022-03-13 13:39:18 +01:00
Kseniia Sumarokova
35e5b4e8a5
Update CachedReadBufferFromRemoteFS.cpp
2022-03-13 12:37:00 +01:00
Alexey Milovidov
978877a9c0
Merge pull request #35212 from rschu1ze/cpp14-trait-aliases
...
Use C++14 aliases for some type traits
2022-03-12 22:20:14 +03:00
Alexey Milovidov
3385275003
Merge pull request #35226 from ClickHouse/timezone-in-config
...
Change timezone example in server config
2022-03-12 21:58:29 +03:00
Maksim Kita
b67f756a43
Fixed performance tests
2022-03-12 18:04:08 +00:00
Maksim Kita
3a2b3ce503
Standardize behaviour of CAST into IPv4, IPv6, toIPv4, toIPv6 functions
2022-03-12 17:12:05 +00:00
Robert Schulze
6fc6d3d452
Remove runtime conditional using constexpr if
2022-03-12 10:41:15 +01:00
zzsmdfj
88560c3917
to #35128_add_mysql_error__detail
2022-03-12 11:10:26 +08:00
Alexey Milovidov
df906dfbd4
Change comments
2022-03-11 23:46:02 +01:00
Alexey Milovidov
38fa55fff0
Merge pull request #35182 from vdimir/regex-replace-35117
...
Fix replaceRegexpAll
2022-03-12 01:36:18 +03:00
Alexey Milovidov
e4e058d667
Update ReplaceRegexpImpl.h
2022-03-12 01:35:37 +03:00
Maksim Kita
f1d2f2a9e1
Updated tests
2022-03-11 21:16:25 +00:00
Nikolai Kochetov
cdc529fa2a
Add database_replicated_allow_only_replicated_engine setting.
2022-03-11 16:03:05 +00:00
HeenaBansal2009
09be30ac26
Added suggestion from review comments
2022-03-11 07:33:23 -08:00
alexey-milovidov
a3a8a1055d
Merge pull request #34919 from azat/fix-glibc2.35-build
...
Fix alignment for aligned_alloc() to fix build for glibc2.35+
2022-03-11 17:59:49 +03:00
alexey-milovidov
003619b18c
Merge pull request #34771 from azat/fix-glibc2.34-build
...
Fix undefined __pthread_mutex_lock/unlock for glibc 2.34+/DISABLE_HERMETIC_BUILD
2022-03-11 17:58:37 +03:00
alexey-milovidov
b6f511506b
Update ThreadFuzzer.cpp
2022-03-11 17:58:27 +03:00
Robert Schulze
85ce4a9bea
Use C++14 aliases for some type traits
2022-03-11 15:54:29 +01:00
alesapin
96c0e9fddf
Better cmake
2022-03-11 15:47:07 +01:00
zhanghuajie
53a8987b3b
fix build fail with gcc --fix warnings without disabling some parameters
2022-03-11 21:59:19 +08:00
Antonio Andelic
f5d3a8a31d
Polishing
2022-03-11 13:38:19 +00:00
kssenii
a18bb606a2
Update
2022-03-11 14:24:02 +01:00
alesapin
33ff2f76e5
Merge branch 'master' into standalone_keeper_build
2022-03-11 13:58:44 +01:00
kssenii
d70d7aac41
Merge master
2022-03-11 12:24:03 +01:00
kssenii
2b478e2501
Fix
2022-03-11 12:17:17 +01:00
Maksim Kita
5f8900cee6
Merge pull request #35079 from qieqieplus/master
...
fix #33798
2022-03-11 11:55:49 +01:00
Maksim Kita
6d40963cad
Merge pull request #35198 from kitaisreal/replicated-mysql-protocol-example-fix
...
MaterializedMySQL protocol example fix
2022-03-11 11:53:14 +01:00
taiyang-li
1f9e050152
add column distributed_depth for system.query_log and system.processes
2022-03-11 17:57:34 +08:00
shuchaome
7a3623d216
fix bug
2022-03-11 17:26:13 +08:00
Nikolai Kochetov
47f4bd30cd
Merge pull request #35186 from amosbird/fixwithtotalemptychunk
...
Fix empty chunk in with total transform
2022-03-11 10:24:19 +01:00
metahys
ff934cf0c2
Fix unexpected result when use -state type aggregate function in window frame ( #34999 )
...
* Fix unexpected result when use -state type aggregate function in window frame
* fix style
* fix style
* fix test
* fix flaky test
* fix flaky test
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-11 11:54:17 +03:00
Kseniia Sumarokova
2bd99a4c70
Update FileSegment.cpp
2022-03-11 09:52:50 +01:00
Antonio Andelic
28e9508c4e
Improve shared pool and add settings
2022-03-11 08:33:34 +00:00
Antonio Andelic
cce318273b
Use a shared IO thread pool
2022-03-11 08:33:34 +00:00
Antonio Andelic
adaaaaf061
Unblock readers before stoping
2022-03-11 08:33:34 +00:00
Antonio Andelic
df0f5e20d0
Refactor
2022-03-11 08:33:34 +00:00
Antonio Andelic
7b526b25a8
Initial implementation with Arena
2022-03-11 08:33:34 +00:00
Antonio Andelic
16c20332cf
Fix deadlocks
2022-03-11 08:33:34 +00:00
Antonio Andelic
c6664bbbdf
Add seekable ParallelReadBuffer
2022-03-11 08:33:34 +00:00
Antonio Andelic
e051587fc2
Format code
2022-03-11 08:33:34 +00:00
Antonio Andelic
29c32ed831
Refactor code
2022-03-11 08:33:34 +00:00
Antonio Andelic
0a1a3a230e
Add support for parallel http range requests
2022-03-11 08:33:34 +00:00
Antonio Andelic
58557b9bec
Test check for ranges support
2022-03-11 08:33:34 +00:00
Antonio Andelic
aad87879cf
Small refactor
2022-03-11 08:33:34 +00:00
shuchaome
46cb4483a6
Optimise by lowering schema on the beginning. Add a functional test.
2022-03-11 14:34:46 +08:00
Kseniia Sumarokova
818459b9f0
Merge pull request #33717 from kssenii/local-cache-for-remote-fs
...
Local cache for remote filesystem
2022-03-11 07:23:10 +01:00
HeenaBansal2009
3f031df225
Code refactoring
2022-03-10 22:22:51 -08:00
Nikita Mikhaylov
909a7309ca
Delete nodes with attributes remove or replace if they didn't merge ( #35141 )
2022-03-10 23:40:55 +01:00
Maksim Kita
e5b85953e8
Added unit tests
2022-03-10 21:45:31 +00:00
Maksim Kita
6b916c7bb5
Fixed tests
2022-03-10 21:45:31 +00:00
Maksim Kita
5b2be4d3b8
Fixed tests
2022-03-10 21:45:31 +00:00
Maksim Kita
cbe059f4bd
Updated IColumn interface to support getting stable permutation
2022-03-10 21:45:31 +00:00
Maksim Kita
765cd09d06
MergeTree improve insert performance
2022-03-10 21:45:31 +00:00
Maksim Kita
d70b7a1a41
ReplicatedMySQL protocol example fix
2022-03-10 21:26:39 +00:00
Anton Popov
37efe2ddb5
Apply suggestions from code review
...
Co-authored-by: Kseniia Sumarokova <54203879+kssenii@users.noreply.github.com>
2022-03-10 22:24:19 +01:00
alesapin
c0d8ccc91b
Merge pull request #35178 from Varinara/master
...
Added disk_name to system.part_log
2022-03-10 22:22:37 +01:00
alexey-milovidov
58e53b06a6
Merge pull request #35134 from azat/optimize_skip_unused_shards_rewrite_in-signed
...
Fix `optimize_skip_unused_shards_rewrite_in` for signed columns and negative values
2022-03-10 23:12:05 +03:00
vdimir
c66fb7cf6e
Attempt to fix replaceRegexpAll
2022-03-10 19:57:13 +00:00
Kseniia Sumarokova
5149bd2d29
Merge pull request #35172 from kssenii/add-non-retriable-errors-for-url
...
Add non-retriable errors check for HTTP read buffer
2022-03-10 20:28:37 +01:00
kssenii
c86955deb2
Fix typo
2022-03-10 20:22:39 +01:00
kssenii
ddcb020d26
Fix test
2022-03-10 19:16:26 +01:00
mergify[bot]
059682a0cd
Merge branch 'master' into asof_ftw
2022-03-10 17:29:38 +00:00
HeenaBansal2009
d981463b05
Added RemoteHostFilter check for MYSQL and postgresSQL
2022-03-10 08:58:48 -08:00
mergify[bot]
9f4ebc313b
Merge branch 'master' into master
2022-03-10 16:55:27 +00:00
Maksim Kita
83de2f66d1
Merge pull request #35032 from ClickHouse/fix-query-profiler-on-bad-linux-kernels
...
Avoid division by zero in Query Profiler if Linux kernel has a bug
2022-03-10 17:49:27 +01:00
Maksim Kita
493169910b
Merge pull request #35174 from zhangyifan27/fix_typo
...
fix typos
2022-03-10 17:10:44 +01:00
Kseniia Sumarokova
3fc399b6e9
Merge pull request #35158 from kssenii/fix-materialized-postgresql
...
Fix materialised postrgesql adding new table after manually removing it
2022-03-10 17:02:32 +01:00
Kseniia Sumarokova
e30b0c5d57
Merge pull request #35162 from kssenii/fix-materialized-postgresql-table-override
...
Fix materialised postgres `table overrides` for partition by, etc
2022-03-10 17:01:24 +01:00
Varinara
f5523f7ff0
added disk_name to system.part_log
2022-03-10 18:44:19 +03:00
mergify[bot]
8434dd424a
Merge branch 'master' into fix_typo
2022-03-10 15:30:06 +00:00
Amos Bird
a1b61dabfd
Fix empty chunk in with total transform.
2022-03-10 23:27:36 +08:00
Kruglov Pavel
a506120646
Fix bug in schema inference in s3 table function ( #35176 )
2022-03-10 15:16:07 +01:00
mergify[bot]
df01290e73
Merge branch 'master' into fix_typo
2022-03-10 13:35:04 +00:00
Miel Donkers
4a95e6d602
Parsing YAML config to XML leads to incorrect structures ( #35135 )
2022-03-10 13:09:48 +01:00
kssenii
1dc3f36a11
Better
2022-03-10 12:19:20 +01:00
Vladimir C
84af08b1a1
Merge pull request #35116 from bigo-sg/snappy_bug
2022-03-10 11:47:37 +01:00
kssenii
787a0805a5
Merge master
2022-03-10 11:42:19 +01:00
kssenii
050802b99b
Add non-retriable errors
2022-03-10 11:38:46 +01:00
Kseniia Sumarokova
e6ee891c9c
Merge pull request #34957 from bigo-sg/hive_random_access_file_cache
...
Optimization for first time to read a random access readbuffer in hive
2022-03-10 11:36:22 +01:00
taiyang-li
ee8fc2730e
remote comment
2022-03-10 18:29:52 +08:00
zhangyifan27
e6fa9f699a
fix typo
2022-03-10 18:29:42 +08:00
taiyang-li
382417d893
fix bug
2022-03-10 18:29:00 +08:00
kssenii
3cd1da1e11
Fix
2022-03-10 11:11:59 +01:00
taiyang-li
2cd469f113
support print ast in dot format
2022-03-10 18:04:35 +08:00
kssenii
bd68d1d6d0
Update test
2022-03-10 11:02:12 +01:00
HeenaBansal2009
3ce9397246
Added Suggestions from Code review
2022-03-09 20:35:01 -08:00
lgbo-ustc
fdd423a3da
fixed code style
2022-03-10 12:13:19 +08:00
lgbo-ustc
e4883f31b7
update tests
...
1. fixed code style in src/IO/tests/gtest_hadoop_snappy_decoder.cpp
2. enable tests 01060_avro.sh
2022-03-10 09:46:43 +08:00
kssenii
af9d8d278e
Fix
2022-03-09 19:25:43 +01:00
kssenii
d089b0ac86
Fix race
2022-03-09 18:14:28 +01:00
mergify[bot]
78c8029aeb
Merge branch 'master' into stress-test
2022-03-09 16:22:45 +00:00
mergify[bot]
c326ebd67f
Merge branch 'master' into issue/#34890_fix_input_format_null_as_default_bug
2022-03-09 15:59:04 +00:00
Vladimir C
ce266b5a3e
Merge pull request #35146 from amosbird/fixpartitionprunerin
2022-03-09 13:23:45 +01:00
Nikita Mikhaylov
d749295222
Fix hardcoded page size ( #35129 )
2022-03-09 12:35:23 +01:00
shuchaome
b7cd85df6b
remove unused column_names in ORCBlockInputFormat
2022-03-09 18:16:22 +08:00
kssenii
6571c7faa0
Clean up code
2022-03-09 10:36:52 +01:00
shuchaome
bb50133424
Apply suggestions from code review
...
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-03-09 17:32:27 +08:00
shuchaome
9647818adc
add unlikely for performance
2022-03-09 17:02:07 +08:00
kssenii
96789fa5ce
Fix test, disable cache for DiskCacheWrapper
2022-03-09 09:46:04 +01:00
shuchaome
8027bb1e32
modify code style
2022-03-09 16:32:18 +08:00
Nikolai Kochetov
6bfee7aca2
Merge pull request #35132 from azat/parallel_distributed_insert_select-view
...
Support view() for parallel_distributed_insert_select
2022-03-09 09:10:34 +01:00
Nikolai Kochetov
c364908061
Merge pull request #35094 from amosbird/getridofredundantplan
...
Get rid of duplicate query planing.
2022-03-09 09:10:20 +01:00
shuchaome
56795b831d
add setting to lower column case when reading parquet/orc file
2022-03-09 16:07:02 +08:00
zzsmdfj
00794fd4a4
to #31469_MaterializedMysql_mysqlDate2CkDate32-add MySQLDataTypesSupport setting
2022-03-09 15:58:43 +08:00
Azat Khuzhin
e2960e1a52
Avoid MEMORY_LIMIT_EXCEEDED during INSERT into Buffer with AggregateFunction
...
In case of Buffer table has columns of AggregateFunction type,
aggregate states for such columns will be allocated from the query
context but those states can be destroyed from the server context (in
case of background flush), and thus memory will be leaked from the query
since aggregate states can be shared, and eventually this will lead to
MEMORY_LIMIT_EXCEEDED error.
To avoid this, prohibit sharing the aggregate states.
But note, that this problem only about memory accounting, not memory
usage itself.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-09 10:57:49 +03:00
