Kruglov Pavel
6fc12dd922
Merge pull request #41703 from Avogar/json-object-each-row
...
Add setting to obtain object name as column value in JSONObjectEachRow format
2022-10-14 20:11:04 +02:00
avogar
52427e6028
Remove code duplication
2022-10-14 18:07:02 +00:00
Kruglov Pavel
ff11904850
Merge branch 'master' into improve-combinators
2022-10-14 17:19:31 +02:00
Alexander Tokmakov
4175f8cde6
abort instead of __builtin_unreachable in debug builds
2022-10-07 21:49:08 +02:00
Anton Popov
6e61cf92f5
Merge remote-tracking branch 'upstream/master' into HEAD
2022-10-03 13:16:57 +00:00
Robert Schulze
db5ef7b3cb
Merge branch 'master' into generated-file-cleanup
2022-10-02 23:13:18 +02:00
Vitaly Baranov
f65d3ff95a
Fix parallel parsing: segmentator now checks max_block_size.
2022-09-30 22:34:03 +02:00
Robert Schulze
cc92a2d174
Merge branch 'master' into generated-file-cleanup
2022-09-30 09:56:31 +02:00
vdimir
0f1a7c252d
better TemporaryDataOnDisk
2022-09-29 09:51:46 +00:00
vdimir
efe0f99658
Fix reading block info in NativeReader with header in ctor
2022-09-29 09:51:44 +00:00
vdimir
ac39bbb3f1
[wip] Common interface for temporary data on disk
2022-09-29 09:51:40 +00:00
avogar
e16cfd361b
Improve DateTime type inference for text formats
2022-09-28 16:55:42 +00:00
Robert Schulze
09c62f6728
Consolidate config_formats.h into config.h
...
Less duplication, less confusion ...
2022-09-28 12:59:05 +00:00
Kruglov Pavel
6340369c2a
Merge branch 'master' into improve-combinators
2022-09-28 14:55:30 +02:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Kruglov Pavel
f1ac2d66be
Merge branch 'master' into json-object-each-row
2022-09-28 14:15:02 +02:00
Robert Schulze
06507c40de
${ConfigIncludePath} --> ${CONFIG_INCLUDE_PATH}
2022-09-28 08:28:47 +00:00
Robert Schulze
1885bb0524
Make comment consistent accross generated files
2022-09-28 08:11:09 +00:00
avogar
76be0d2ee1
Infer Object type only when allow_experimental_object_type is enabled
2022-09-27 23:07:36 +00:00
Kruglov Pavel
3dc54272ed
Merge branch 'master' into improve-combinators
2022-09-26 13:03:32 +02:00
avogar
d3d06251a3
Add setting to obtain object name as column value in JSONObjectEachRow format
2022-09-22 16:48:54 +00:00
Alexey Milovidov
45afacdae4
Merge pull request #41186 from ClickHouse/fix-three-fourth-of-trash
...
Fix more than half of the trash
2022-09-22 07:28:26 +03:00
Kruglov Pavel
22e11aef2d
Merge pull request #40910 from Avogar/new-json-formats
...
Add new JSON formats, add improvements and refactoring
2022-09-21 14:19:08 +02:00
avogar
868ce8bc16
Fix comments, make better naming, add docs, add setting output_format_json_quote_64bit_floats
2022-09-20 13:49:17 +00:00
Kruglov Pavel
47f6f09ce0
Merge branch 'master' into improve-combinators
2022-09-19 14:31:12 +02:00
Alexey Milovidov
84f42e0874
Fix 3/4 of trash
2022-09-19 08:50:53 +02:00
Alexey Milovidov
2f0684b97c
Fix trash in schema inference
2022-09-17 23:11:33 +02:00
Alexey Milovidov
47167494d9
Fix trash in schema inference
2022-09-17 22:53:41 +02:00
avogar
0101cc2e56
Support complex combinators in window transform, arrayReduce*, initializeAggregation and Aggregate functons versionning
2022-09-16 19:07:36 +00:00
Alexey Milovidov
da01982652
Merge pull request #41046 from azat/build/llvm-15
...
Switch to llvm/clang 15
2022-09-16 07:31:06 +03:00
Azat Khuzhin
e8d7403a38
Suppress warning in FormatFactory::getFormatFromFileDescriptor() for FreeBSD
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-10 21:38:35 +02:00
zhenjial
bd9fabc3f7
code optimization, add test
2022-09-09 23:27:42 +08:00
avogar
ad68b7be0f
Better
2022-09-09 15:01:45 +00:00
avogar
46a0318a36
Support JSONColumnsWithMetadata input format
2022-09-08 17:58:44 +00:00
zhenjial
469ceaa156
code optimization
2022-09-09 00:47:43 +08:00
avogar
c380decbbb
Make better, add new settings
2022-09-08 16:07:20 +00:00
Anton Popov
86b29b7f1a
fix serilization of Object inside other types
2022-09-08 15:16:39 +00:00
zhenjial
0f788d98f5
new implementation
2022-09-06 20:39:54 +08:00
avogar
b94e896c1c
Remove logs
2022-09-01 19:01:27 +00:00
avogar
afc34dca41
Add new JSON formats, add improvements and refactoring
2022-09-01 19:00:24 +00:00
avogar
acf87c1d10
Fix nested JSON Objects schema inference
2022-08-31 14:10:29 +00:00
vdimir
0349c85017
Use getCompressedBytes in BufferingToFileTransform and TemporaryFileStream
2022-08-24 16:14:10 +00:00
vdimir
51c44424cc
More metrics for temp files
2022-08-24 16:14:09 +00:00
avogar
29a887578b
Fix
2022-08-23 11:42:57 +00:00
avogar
5ab87f1da4
Small refactoring
2022-08-19 16:42:23 +00:00
avogar
612ffaffde
Make schema inference cache better, respect format settings that can change the schema
2022-08-19 16:39:13 +00:00
Nikolai Kochetov
5a85531ef7
Merge pull request #38286 from Avogar/schema-inference-cache
...
Add schema inference cache for s3/hdfs/file/url
2022-08-18 13:07:50 +02:00
avogar
8dd54c043d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache
2022-08-17 11:47:40 +00:00
avogar
e1ff996ec3
Allow to specify structure hints in schema inference
2022-08-16 09:46:57 +00:00
Kruglov Pavel
088e8cf9bd
Merge branch 'master' into numbers-schema-inference
2022-08-09 14:00:36 +02:00
avogar
1304e3487c
Add comments, remove unneded stuff
2022-08-08 13:43:14 +00:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
Kruglov Pavel
6b2186bfeb
Merge branch 'master' into numbers-schema-inference
2022-08-02 19:34:53 +02:00
Anton Popov
a333cc4146
Merge remote-tracking branch 'upstream/master' into HEAD
2022-08-02 12:57:43 +00:00
Kruglov Pavel
a0d51601bf
Update EscapingRuleUtils.cpp
2022-08-01 13:07:48 +02:00
Alexey Milovidov
4828be7fc4
Fix double escaping in the metadata of FORMAT JSON
2022-07-30 23:56:41 +02:00
Anton Popov
57e4fb2e30
Merge remote-tracking branch 'upstream/master' into HEAD
2022-07-29 11:42:11 +00:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache
2022-07-27 11:35:36 +02:00
Kruglov Pavel
5aae0a2e04
Fix style
2022-07-25 17:20:01 +02:00
Anton Popov
49627aa554
Merge remote-tracking branch 'upstream/master' into HEAD
2022-07-22 17:16:06 +00:00
avogar
794aa691bc
Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-protobuf-capnp-empty-message
2022-07-21 17:04:37 +00:00
Anton Popov
e0d2c8fb37
fix json type with sparse columns
2022-07-21 14:47:19 +00:00
avogar
17a271ec30
Fix error codes
2022-07-20 14:33:46 +00:00
Kruglov Pavel
46da17ca8c
Merge branch 'master' into numbers-schema-inference
2022-07-20 13:32:39 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache
2022-07-20 13:30:42 +02:00
Kruglov Pavel
3fb3015375
Merge pull request #39340 from Avogar/better-exception-messages
...
Better exception messages in schema inference
2022-07-20 13:29:15 +02:00
avogar
784ee11594
Add settings to skip fields with unsupported types in Protobuf/CapnProto schema inference
2022-07-20 11:16:25 +00:00
Kruglov Pavel
88d59520a2
Fix
2022-07-19 15:20:56 +02:00
Kruglov Pavel
1513285166
Fix typo
2022-07-18 20:54:13 +02:00
Kruglov Pavel
24c9467641
Fix
2022-07-18 19:55:14 +02:00
avogar
3f81aadb60
Fix schema inference in case of empty messages in Protobuf/CapnProto formats
2022-07-18 17:53:33 +00:00
avogar
2367f40b70
Better exception messages in schema inference
2022-07-18 15:36:33 +00:00
Kruglov Pavel
857290b586
Fix style
2022-07-18 15:40:28 +02:00
Kruglov Pavel
0f6044e50f
Fix style
2022-07-18 15:39:53 +02:00
avogar
9291d33080
Pass const std::string_view & by value, not by reference
2022-07-14 16:11:57 +00:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache
2022-07-14 12:29:54 +02:00
avogar
2b7c6b7ecd
Remove logging
2022-07-13 15:59:04 +00:00
avogar
7cde9d3b40
Add new features in schema inference
2022-07-13 15:57:55 +00:00
avogar
5b0fd31c64
Put column names in quotes
2022-06-30 16:14:30 +00:00
avogar
ee54c4f9b7
Add some fixes and add settings in docs
2022-06-30 12:41:56 +00:00
mergify[bot]
9482c99ab8
Merge branch 'master' into sql-insert-format
2022-06-29 11:03:07 +00:00
Robert Schulze
c22038d48b
More clang-tidy fixes
2022-06-28 11:50:05 +00:00
avogar
9bb68bc6de
Add SQLInsert output format
2022-06-27 18:31:57 +00:00
avogar
b0c9d1a25d
Fix style
2022-06-27 14:04:28 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Alexey Milovidov
5e9e5a4eaf
Merge pull request #37525 from Avogar/avro-structs
...
Support Maps and Records, allow to insert null as default in Avro format
2022-06-15 00:04:29 +03:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros
...
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.
__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
2022-06-10 10:22:31 +02:00
Kruglov Pavel
6f17ba17ba
Revert "Revert "Fix possible segfault in schema inference""
2022-06-02 13:28:27 +02:00
Alexander Tokmakov
4baae59252
Revert "Fix possible segfault in schema inference"
2022-06-02 14:04:28 +03:00
avogar
4abfd54dd6
Fix possible segfault in schema inference
2022-06-01 16:53:37 +00:00
Kruglov Pavel
7cc87d9a65
Merge pull request #37537 from Avogar/skip-first-lines
...
Allow to skip some of the first lines in CSV/TSV formats
2022-05-31 14:26:21 +02:00
Kruglov Pavel
0615866aea
Merge pull request #37450 from Avogar/check-format-on-storage-creation
...
Check format name on storage creation
2022-05-30 14:23:20 +02:00
Alexey Milovidov
c50791dd3b
Fix clang-tidy-14, part 1
2022-05-27 22:52:14 +02:00
avogar
4c9812d4c1
Allow to skip some of the first rows in CSV/TSV formats
2022-05-25 15:00:11 +00:00
avogar
038a422aeb
Add setting to insert null as default
2022-05-25 12:56:59 +00:00
avogar
f782fa31c6
Merge branch 'master' of github.com:ClickHouse/ClickHouse into check-format-on-storage-creation
2022-05-25 08:42:54 +00:00
avogar
37b66c8a9e
Check format name on storage creation
2022-05-23 12:48:48 +00:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names
2022-05-23 12:14:20 +02:00
Kruglov Pavel
ce48e8e102
Merge pull request #36975 from Avogar/json-columns-formats
...
Add columnar JSON formats
2022-05-23 12:11:28 +02:00
avogar
a4cf07708c
Fix comments
2022-05-20 14:57:27 +00:00
avogar
566d1b15fd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into formats-with-names
2022-05-20 13:54:52 +00:00
avogar
44726122bb
Join JSON registration
2022-05-20 12:09:51 +00:00
avogar
a6a430c5ee
Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats
2022-05-20 11:08:30 +00:00
mergify[bot]
1ac4199e78
Merge branch 'master' into arrow-strings
2022-05-20 10:43:33 +00:00
avogar
cd6a29897e
Apply input_format_max_rows_to_read_for_schema_inference for all files in globs in total
2022-05-18 17:56:36 +00:00
avogar
a0369fb9a6
Allow to use String type instead of Binary in Arrow/Parquet/ORC formats
2022-05-18 14:51:21 +00:00
Kruglov Pavel
134821eff8
Fix build
2022-05-18 12:44:20 +02:00
avogar
12010a81b7
Make better
2022-05-18 09:25:26 +00:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies
2022-05-16 10:12:50 +02:00
avogar
68bb07d166
Better naming
2022-05-13 18:39:19 +00:00
avogar
cef13c2c02
Allow to skip unknown columns in Native format
2022-05-13 14:27:15 +00:00
avogar
b17fec659a
Improve performance and memory usage for select of subset of columns for some formats
2022-05-13 13:51:28 +00:00
avogar
f6b16880bd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats
2022-05-10 12:57:18 +00:00
Anton Popov
e911900054
remove last mentions of data streams
2022-05-09 19:15:24 +00:00
avogar
04fdd75c56
Make JSONColumns frormats mono block by default
2022-05-09 11:13:44 +00:00
Robert Schulze
f2b1748c48
Enable clang-tidy bugprone-suspicious-semicolon
...
Official docs:
Finds most instances of stray semicolons that unexpectedly alter the
meaning of the code.
2022-05-08 19:13:37 +02:00
avogar
62a7ba3f26
Add columnar JSON formats
2022-05-06 16:48:48 +00:00
Kruglov Pavel
77e55c344c
Merge pull request #36667 from Avogar/mysqldump-format
...
Add MySQLDump input format
2022-05-04 19:49:48 +02:00
Dmitry Novik
5ba7a55c18
Merge pull request #36650 from bigo-sg/hive_text_parallel_parsing
...
Parallel parsing of hive text format
2022-05-03 15:56:28 +02:00
Kruglov Pavel
d613f7eab0
Merge branch 'master' into mysqldump-format
2022-05-02 13:31:57 +02:00
Antonio Andelic
a1a22b0007
Merge pull request #35149 from ContentSquare/nullables_with_proto3
...
Nullables with proto3 using Google wrappers
2022-05-02 09:49:37 +02:00
Robert Schulze
89aa9ae00f
Fixed clang-tidy check "bugprone-branch-clone"
...
The check is currently *not* part of .clang-tidy. It complains about:
(1) "switch has multiple consecutive identical branches"
(2) "repeated branch in conditional chain"
About (1): Lots of findings in switches were about redundant
"[[fallthrough]]" in places where the compiler would not warn anyways. I
have cleaned these up.
About (2): In if-else_if-else chains, fixing the warning would usually
mean concatenating multiple if-conditions. As this would reduce
readability in most cases, I did not fix these places.
Because of (2), I also refrained from adding "bugprone-branch-clone" to
.clang-tidy.
2022-04-30 19:40:28 +02:00
Jakub Kuklis
a1f2dd6d34
Adding two settings in place of one, improvements to the test clarity
2022-04-29 10:01:51 +02:00
Jakub Kuklis
e73fa271a2
Minor improvements
2022-04-29 10:01:51 +02:00
Jakub Kuklis
5ca095c779
Pass the setting to buildFieldSerializer to fix undeclared
2022-04-29 10:01:51 +02:00
Jakub Kuklis
e705425374
Minor improvements
2022-04-29 10:01:51 +02:00
Jakub Kuklis
5c34585a00
Improve the test clarity
2022-04-29 10:01:51 +02:00
Jakub Kuklis
f19e473482
Remove local change
2022-04-29 10:01:51 +02:00
Jakub Kuklis
507ba1042c
Adding a setting to enable Google wrappers special treatment
2022-04-29 10:01:51 +02:00
Jakub Kuklis
6d5c1e2fc0
Adding a setting to enable special treatment of google wrappers
2022-04-29 10:01:50 +02:00
Jakub Kuklis
b7a8acc302
Alternative design for output, mory messy, but the default value inside Google wrapper is not serialized
2022-04-29 10:01:50 +02:00
Jakub Kuklis
53e2454800
Corrected the behaviour for Proto Nullable output
2022-04-29 10:01:50 +02:00
Jakub Kuklis
10425c17b2
Write empty values for Google wrappers
2022-04-29 10:01:50 +02:00
Jakub Kuklis
ff49fad1f1
Another const keyword corrections for debug build
2022-04-29 10:01:50 +02:00
Jakub Kuklis
08ee7470f0
const keyword corrections for debug build
2022-04-29 10:01:50 +02:00
Jakub Kuklis
c0acc4dfa0
Fixing assert
2022-04-29 10:01:50 +02:00
Jakub Kuklis
7a78197746
Style corrections
2022-04-29 10:01:50 +02:00
Jakub Kuklis
ae1194bf9c
Nullables detection in protobuf using Google wrappers
2022-04-29 10:01:50 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common
2022-04-29 10:26:35 +08:00
avogar
d295de1689
Fix comments and test
2022-04-28 14:59:35 +00:00
Kruglov Pavel
4d08587559
Merge branch 'master' into mysqldump-format
2022-04-28 15:58:18 +02:00
Vladimir C
1cbdc1ef3a
Merge pull request #36206 from vdimir/output-format-prometheus
2022-04-28 12:09:53 +02:00
vdimir
be0aa06958
Add output format Prometheus
2022-04-26 14:57:35 +00:00
avogar
b666b4e1c9
Fix possible heap-use-after-free in schema inference
2022-04-26 14:36:16 +00:00
avogar
33d845dade
Add MySQLDump input format
2022-04-26 10:42:56 +00:00
taiyang-li
b7cc344d62
remove useless codes
2022-04-26 14:42:43 +08:00
taiyang-li
99dee35b6e
parallel parsing of hive text format
2022-04-26 14:33:10 +08:00
Kruglov Pavel
34c342fdd3
Merge pull request #36205 from Avogar/improve-globs
...
Some refactoring around schema inference with globs
2022-04-25 13:14:46 +02:00
avogar
80eacc8533
Merge branch 'master' of github.com:ClickHouse/ClickHouse into improve-json-schema-inference
2022-04-22 17:18:44 +00:00
avogar
a093181b4f
Fix comments
2022-04-21 11:48:17 +00:00
Kruglov Pavel
813e228fcc
Merge branch 'master' into improve-globs
2022-04-20 16:31:47 +02:00
avogar
f31f019252
Fix
2022-04-19 19:25:41 +00:00
avogar
1f252cedfe
Make better
2022-04-19 19:16:47 +00:00
Kruglov Pavel
ec4e1cb6d8
Merge pull request #36211 from Avogar/insert-select-all-formats
...
Allow insert select for files with formats without schema inference
2022-04-19 14:25:59 +02:00
avogar
ae88549c4f
Allow insert select for files with formats without schema inference
2022-04-13 20:02:52 +00:00
avogar
8b60aeb7bc
Improve schema inference for json objects
2022-04-13 19:13:40 +00:00
avogar
1c065f8c7a
Some refactoring around schema inference with globs
2022-04-13 17:02:48 +00:00
avogar
348cae0d16
Fix possible segfault in schema inference for JSON formats
2022-04-13 12:34:40 +00:00
avogar
d2017a63b1
Merge branch 'master' of github.com:ClickHouse/ClickHouse into improve-schema-inference
2022-04-07 11:36:40 +00:00
Kruglov Pavel
f3f8f27db5
Merge pull request #35735 from Avogar/allow-read-bools-as-numbers
...
Allow to infer and parse bools as numbers in JSON input formats
2022-04-07 13:20:49 +02:00
taiyang-li
2ef316801c
Merge branch 'master' into use_minmax_index
2022-04-07 10:53:25 +08:00
Kruglov Pavel
ec2213493f
Merge branch 'master' into allow-read-bools-as-numbers
2022-04-06 14:53:02 +02:00
Kruglov Pavel
9141066de3
Merge branch 'master' into improve-schema-inference
2022-04-06 13:51:07 +02:00
taiyang-li
acb9f1632e
suppoort skip splits in orc and parquet
2022-04-06 16:40:22 +08:00
Maksim Kita
e6c9a36ac7
Merge pull request #35733 from kitaisreal/ipv6-invalid-insert-test
...
Added test for insert of invalid IPv6 value
2022-04-04 12:28:16 +02:00
mergify[bot]
1e43e26fa1
Merge branch 'master' into fix-order
2022-04-02 12:00:29 +00:00
avogar
ab2a963287
Merge branch 'master' of github.com:ClickHouse/ClickHouse into allow-read-bools-as-numbers
2022-03-31 14:09:43 +00:00
mergify[bot]
24ade25d61
Merge branch 'master' into improve-schema-inference
2022-03-31 13:42:47 +00:00
Kruglov Pavel
564a77c462
Fix build
2022-03-31 12:49:23 +02:00
Maksim Kita
371cdc956a
Added input format settings for parsing invalid IPv4, IPv6 addresses as default values
2022-03-30 12:54:19 +02:00
avogar
3fc36627b3
Allow to infer and parse bools as numbers in JSON input formats
2022-03-29 17:37:31 +00:00
avogar
ce97ccbfb9
Improve schema inference for JSONEachRow and TSKV formats
2022-03-29 14:47:51 +00:00
Kruglov Pavel
a2fd09e031
Fix style
2022-03-29 16:34:07 +02:00
Antonio Andelic
9990abb76a
Use compile-time check for Exception messages, fix wrong messages
2022-03-29 13:16:11 +00:00
mergify[bot]
343588de2c
Merge branch 'master' into improve-schema-inference
2022-03-29 13:06:00 +00:00
Anton Popov
9610139477
Merge pull request #35629 from CurtizJ/dynamic-columns-5
...
Support schema inference for type `Object` in format `JSONEachRow`
2022-03-29 14:17:09 +02:00
Anton Popov
d677635cd8
Merge pull request #35592 from CurtizJ/dynamic-columns-4
...
Add parallel parsing and schema inference for format `JSONAsObject`
2022-03-28 19:29:55 +02:00
Anton Popov
a6450be8b6
fix schema inference
2022-03-26 01:33:10 +00:00
Anton Popov
67195bfdd5
support schema inference for type Object in format JSONEachRow
2022-03-25 21:51:53 +00:00
Kruglov Pavel
d45143ffe0
Merge branch 'master' into improve-schema-inference
2022-03-25 12:05:40 +01:00
Kruglov Pavel
1823cac89d
Update src/Formats/EscapingRuleUtils.h
...
Co-authored-by: Vladimir C <vdimir@clickhouse.com>
2022-03-24 19:19:32 +01:00
Anton Popov
78100abc5f
add parallel parsing and schema inference for type Object
2022-03-24 17:51:35 +00:00
avogar
abc020a502
Clean up
2022-03-24 13:08:58 +00:00
avogar
557edbd172
Add some improvements and fixes in schema inference
2022-03-24 12:54:12 +00:00
Antonio Andelic
0c23cd7b94
Add support for case insensitive column matching in arrow
2022-03-22 10:55:10 +00:00
Antonio Andelic
29d2bf7d1a
Merge branch 'master' into case-insensitive-column-matching
2022-03-21 08:17:27 +00:00
Antonio Andelic
f75b054255
Allow case insensitive column matching
2022-03-21 07:47:37 +00:00
Kruglov Pavel
aa3c05e9d4
Merge pull request #35152 from rschu1ze/protobuf-batch-write
...
ProtobufList
2022-03-18 13:24:34 +01:00
Antonio Andelic
607f785e48
Revert "Merge pull request #35145 from bigo-sg/lower-column-name"
...
This reverts commit ebf72bf61d
, reversing
changes made to f1b812bdc1
.
2022-03-17 12:31:43 +00:00
Robert Schulze
6e1d7a31bc
Fix build + typo
2022-03-17 11:41:20 +01:00
Anton Popov
2ced42ed41
add experimental settings for Object type
2022-03-16 16:51:23 +00:00
Anton Popov
0ba78c3c3a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-16 15:28:09 +00:00
Robert Schulze
0d2ece6d91
Merge branch 'ClickHouse:master' into protobuf-batch-write
2022-03-16 09:43:33 +01:00
avogar
e2d1e643f2
Fix possible segfault in JSONEachRow schema inference
2022-03-15 11:44:15 +00:00
Robert Schulze
23122cb327
Fix review comments
...
ParquetBlockOutputFormat.cpp:
- undo unrelated formatting
ProtobufSerializer.cpp:
- undef debug tracing
- simplify logic in writeRow()
ProtobufSchemas.cpp:
- restore original search in cache by message type
2022-03-15 11:27:17 +01:00
Maksim Kita
538f8cbaad
Fix clang-tidy warnings in Disks, Formats, Functions folders
2022-03-14 18:17:35 +00:00
Anton Popov
36ec379aeb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-14 16:28:35 +00:00
Antonio Andelic
ebf72bf61d
Merge pull request #35145 from bigo-sg/lower-column-name
...
add setting to lower column case when reading parquet/orc file
2022-03-14 11:25:03 +01:00
Robert Schulze
514d4d2187
Implement ProtobufList - fixes ClickHouse#16436
...
Introduce IO format "ProtobufList" with protobuf schema
// schemafile.proto
message Envelope {
message MessageType {
uint32 colA = 1;
string colB = 2;
}
repeated MessageType mt = 1;
}
where "Envelope" is a hard-coded/expected top-level message and
"MessageType" is a message with user-provided name containing the table
fields to export/import, e.g.
SELECT * FROM db1.tab1 FORMAT ProtobufList SETTINGS format_schema =
'schemafile:MessageType'
As a result, the new format wraps a list of messages (one per row) into
a single, containing message. Compare that to the schema of the existing
IO formats "Protobuf" and "ProtobufSingle":
message MessageType {
uint32 colA = 1;
string colB = 2;
}
The new format does not save space compared to the existing formats, but
it is conceptually a bit more beautiful and also more convenenient.
Implementation details:
- Created new files ProtobufList(Input|Output)Format which use the
existing ProtobufSerializer mechanism. The goal was to reuse as much
code as possible and avoid copypasta.
- I was torn between inheriting from I(Input|Output)Format vs.
IRow(Input|Output)Format for ProtobufList(Input|Output)Format. The
former is chunk-based which can be better for performance. Since the
ProtobufSerializer mechanism is row-based but data is generally passed
around in chunks, I decided for the latter to leverage the existing
chunk <--> row mapping code in IRow(InputOutput)Format.
- A new ProtobufSerializer called ProtobufSerializerEnvelope was
introduced (--> ProtobufSerializer.cpp). It represents the top-level
message which encloses the list of inner nested messages, i.e. the
rows.
- With the new format, parsing the schema file and matching the fields in
the schema file to table column works like for the old formats. The only
difference is that parsing starts one level below the "Envelope" (-->
ProtobufSchema.cpp). This is more natural than forcing customers to
have table columns start with "Envelope".
- Creation of the ProtobufSerializer tree also works like before. What
is different is that we finally add a ProtobufSerializerEnvelope as
new root of the tree. It's only purpose is to write/read the top-level
message for the first/last row to write/read.
Caveats:
- The low-level serialization code in ProtobufWriter uses an internal
buffer which is flushed to the output file only in endMessage().
In the existing "Protobuf" format, this happens once per row, in the
new format this happens only at the end of the serialization
since row-level messages now call start/endNestedMessage(). As a
future TODO to, the buffer should be flushed also in
start/endNestedMessage() to reduce memory consumption.
2022-03-14 08:04:58 +01:00
zhanghuajie
53a8987b3b
fix build fail with gcc --fix warnings without disabling some parameters
2022-03-11 21:59:19 +08:00
shuchaome
46cb4483a6
Optimise by lowering schema on the beginning. Add a functional test.
2022-03-11 14:34:46 +08:00
shuchaome
56795b831d
add setting to lower column case when reading parquet/orc file
2022-03-09 16:07:02 +08:00
Anton Popov
0bc57da238
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-07 14:46:08 +00:00
Azat Khuzhin
c426eef07d
Fix generating USE_* for system.build_options
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-03-04 15:31:32 +03:00
Anton Popov
df3b07fe7c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-03-03 22:25:28 +00:00
Maksim Kita
f1b1baf56e
Merge pull request #34982 from Cai-Yao/master
...
date_time_input_format = 'best_effort_us'
2022-03-03 09:22:57 +01:00
Maksim Kita
b1a956c5f1
clang-tidy check performance-move-const-arg fix
2022-03-02 18:15:27 +00:00
cwkyaoyao
72194bbaf3
Add date_time_input_format = best_effort_us
2022-03-02 16:00:06 +08:00
avogar
a7c6d11532
Fix schema inference for unquoted dates in CSV
2022-03-01 11:03:26 +00:00
Anton Popov
18940b8637
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-09 23:38:38 +03:00
Kruglov Pavel
15d85682e8
Fix style
2022-02-07 18:29:22 +03:00
avogar
a4c7ecde87
Make better
2022-02-07 17:51:26 +03:00
avogar
77b42bb9ff
Support UUID in MsgPack format
2022-02-07 17:11:44 +03:00
Anton Popov
836a348a9c
Merge remote-tracking branch 'upstream/master' into HEAD
2022-02-01 15:23:07 +03:00
Maksim Kita
5ef83deaa6
Update sort to pdqsort
2022-01-30 19:49:48 +00:00
Anton Popov
78b9f15abb
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-30 03:24:37 +03:00
Kruglov Pavel
7873b4475f
Merge branch 'master' into autodetect-format
2022-01-25 10:56:52 +03:00
avogar
a6740d2f9a
Detect format and schema for stdin in clickhouse-local
2022-01-25 10:25:37 +03:00
avogar
1f49acc164
Better naming
2022-01-24 16:28:36 +03:00
Anton Popov
e8ce091e68
Merge remote-tracking branch 'upstream/master' into HEAD
2022-01-21 20:11:18 +03:00
Kruglov Pavel
7bfb1231b9
Merge branch 'master' into formats-with-suffixes
2022-01-20 14:47:17 +03:00
Azat Khuzhin
91e3ceeea9
Remove unbundled capnp support
2022-01-20 10:01:58 +03:00
Azat Khuzhin
a30ef87d65
Remove unbundled msgpack support
2022-01-20 10:01:58 +03:00
Azat Khuzhin
788cb6b2b0
Remove unbundled protobuf support
2022-01-20 08:47:16 +03:00
Azat Khuzhin
1145e32af6
Remove unbundled snappy support
2022-01-20 08:47:16 +03:00
Azat Khuzhin
ab8cdb198f
Remove unbundled orc support
2022-01-20 08:47:16 +03:00
Azat Khuzhin
d1b2bd5fbe
Remove unbundled avro support
2022-01-20 08:47:16 +03:00
Azat Khuzhin
b4ad324a88
Remove unbundled parquet/arrow support
2022-01-20 08:47:16 +03:00
Kruglov Pavel
a7df9cd53a
Merge branch 'master' into formats-with-suffixes
2022-01-14 21:03:49 +03:00
avogar
253035a5df
Fix
2022-01-14 19:17:06 +03:00
Kruglov Pavel
d2e9f37bee
Merge branch 'master' into format-by-extention
2022-01-14 18:36:23 +03:00
avogar
89a181bd19
Make better
2022-01-14 18:16:18 +03:00
avogar
817a314263
Fix tests and style
2022-01-14 17:46:24 +03:00
Kruglov Pavel
5a908e8edd
Merge branch 'master' into formats-with-suffixes
2022-01-14 16:45:20 +03:00
Kseniia Sumarokova
5da673c3a5
Merge pull request #31104 from bigo-sg/hive_table
...
Implement hive table engine
2022-01-14 09:39:17 +03:00
avogar
2d7b1bfa5e
Detect format in S3/HDFS/URL table engines
2022-01-13 16:14:18 +03:00
Kruglov Pavel
305d58a762
Merge pull request #33524 from Avogar/stacktrace-in-client
...
Don't print exception twice in client in case of exception in parallel parsing
2022-01-13 15:50:42 +03:00
avogar
8390e9ad60
Detect format by file name in file/hdfs/s3/url table functions
2022-01-12 18:29:31 +03:00
taiyang-li
66813a3aa9
merge master
2022-01-12 16:56:29 +08:00
avogar
0ae0aa712b
Don't print exception twice in client in case of exception in parallel parsing
2022-01-11 18:37:07 +03:00
zhongyuankai
99279c1443
INTO OUTFILE / FROM INFILE: autodetect FORMAT by file extension
2022-01-11 21:26:14 +08:00
zhongyuankai
878e44eb97
auto format by file extension
2022-01-08 21:47:14 +08:00
taiyang-li
1e102bc1b2
merge master
2022-01-01 09:01:06 +08:00
alesapin
16c36d72b1
Merge pull request #33296 from ClickHouse/fix_clang_tidy_3
...
Fix clang tidy 3
2021-12-29 22:43:42 +03:00
avogar
97788b9c21
Allow to create new files on insert for File/S3/HDFS engines
2021-12-29 21:19:13 +03:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference
...
Automatic schema inference for input formats
2021-12-29 21:03:48 +03:00
alesapin
34145c47da
Fix clang tidy
2021-12-29 18:36:42 +03:00
avogar
78b522fd51
Fix fasttest build
2021-12-29 12:21:01 +03:00
avogar
d718a2e220
Clean up
2021-12-29 12:21:01 +03:00