avogar
b0c9d1a25d
Fix style
2022-06-27 14:04:28 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Alexey Milovidov
5e9e5a4eaf
Merge pull request #37525 from Avogar/avro-structs
...
Support Maps and Records, allow to insert null as default in Avro format
2022-06-15 00:04:29 +03:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros
...
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.
__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
2022-06-10 10:22:31 +02:00
Kruglov Pavel
6f17ba17ba
Revert "Revert "Fix possible segfault in schema inference""
2022-06-02 13:28:27 +02:00
Alexander Tokmakov
4baae59252
Revert "Fix possible segfault in schema inference"
2022-06-02 14:04:28 +03:00
avogar
4abfd54dd6
Fix possible segfault in schema inference
2022-06-01 16:53:37 +00:00
Kruglov Pavel
7cc87d9a65
Merge pull request #37537 from Avogar/skip-first-lines
...
Allow to skip some of the first lines in CSV/TSV formats
2022-05-31 14:26:21 +02:00
Kruglov Pavel
0615866aea
Merge pull request #37450 from Avogar/check-format-on-storage-creation
...
Check format name on storage creation
2022-05-30 14:23:20 +02:00
Alexey Milovidov
c50791dd3b
Fix clang-tidy-14, part 1
2022-05-27 22:52:14 +02:00
avogar
4c9812d4c1
Allow to skip some of the first rows in CSV/TSV formats
2022-05-25 15:00:11 +00:00
avogar
038a422aeb
Add setting to insert null as default
2022-05-25 12:56:59 +00:00
avogar
f782fa31c6
Merge branch 'master' of github.com:ClickHouse/ClickHouse into check-format-on-storage-creation
2022-05-25 08:42:54 +00:00
avogar
37b66c8a9e
Check format name on storage creation
2022-05-23 12:48:48 +00:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names
2022-05-23 12:14:20 +02:00
Kruglov Pavel
ce48e8e102
Merge pull request #36975 from Avogar/json-columns-formats
...
Add columnar JSON formats
2022-05-23 12:11:28 +02:00
avogar
a4cf07708c
Fix comments
2022-05-20 14:57:27 +00:00
avogar
566d1b15fd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into formats-with-names
2022-05-20 13:54:52 +00:00
avogar
44726122bb
Join JSON registration
2022-05-20 12:09:51 +00:00
avogar
a6a430c5ee
Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats
2022-05-20 11:08:30 +00:00
mergify[bot]
1ac4199e78
Merge branch 'master' into arrow-strings
2022-05-20 10:43:33 +00:00
avogar
cd6a29897e
Apply input_format_max_rows_to_read_for_schema_inference for all files in globs in total
2022-05-18 17:56:36 +00:00
avogar
a0369fb9a6
Allow to use String type instead of Binary in Arrow/Parquet/ORC formats
2022-05-18 14:51:21 +00:00
Kruglov Pavel
134821eff8
Fix build
2022-05-18 12:44:20 +02:00
avogar
12010a81b7
Make better
2022-05-18 09:25:26 +00:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies
2022-05-16 10:12:50 +02:00
avogar
68bb07d166
Better naming
2022-05-13 18:39:19 +00:00
avogar
cef13c2c02
Allow to skip unknown columns in Native format
2022-05-13 14:27:15 +00:00
avogar
b17fec659a
Improve performance and memory usage for select of subset of columns for some formats
2022-05-13 13:51:28 +00:00
avogar
f6b16880bd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats
2022-05-10 12:57:18 +00:00
Anton Popov
e911900054
remove last mentions of data streams
2022-05-09 19:15:24 +00:00
avogar
04fdd75c56
Make JSONColumns frormats mono block by default
2022-05-09 11:13:44 +00:00
Robert Schulze
f2b1748c48
Enable clang-tidy bugprone-suspicious-semicolon
...
Official docs:
Finds most instances of stray semicolons that unexpectedly alter the
meaning of the code.
2022-05-08 19:13:37 +02:00
avogar
62a7ba3f26
Add columnar JSON formats
2022-05-06 16:48:48 +00:00
Kruglov Pavel
77e55c344c
Merge pull request #36667 from Avogar/mysqldump-format
...
Add MySQLDump input format
2022-05-04 19:49:48 +02:00
Dmitry Novik
5ba7a55c18
Merge pull request #36650 from bigo-sg/hive_text_parallel_parsing
...
Parallel parsing of hive text format
2022-05-03 15:56:28 +02:00
Kruglov Pavel
d613f7eab0
Merge branch 'master' into mysqldump-format
2022-05-02 13:31:57 +02:00
Antonio Andelic
a1a22b0007
Merge pull request #35149 from ContentSquare/nullables_with_proto3
...
Nullables with proto3 using Google wrappers
2022-05-02 09:49:37 +02:00
Robert Schulze
89aa9ae00f
Fixed clang-tidy check "bugprone-branch-clone"
...
The check is currently *not* part of .clang-tidy. It complains about:
(1) "switch has multiple consecutive identical branches"
(2) "repeated branch in conditional chain"
About (1): Lots of findings in switches were about redundant
"[[fallthrough]]" in places where the compiler would not warn anyways. I
have cleaned these up.
About (2): In if-else_if-else chains, fixing the warning would usually
mean concatenating multiple if-conditions. As this would reduce
readability in most cases, I did not fix these places.
Because of (2), I also refrained from adding "bugprone-branch-clone" to
.clang-tidy.
2022-04-30 19:40:28 +02:00
Jakub Kuklis
a1f2dd6d34
Adding two settings in place of one, improvements to the test clarity
2022-04-29 10:01:51 +02:00
Jakub Kuklis
e73fa271a2
Minor improvements
2022-04-29 10:01:51 +02:00
Jakub Kuklis
5ca095c779
Pass the setting to buildFieldSerializer to fix undeclared
2022-04-29 10:01:51 +02:00
Jakub Kuklis
e705425374
Minor improvements
2022-04-29 10:01:51 +02:00
Jakub Kuklis
5c34585a00
Improve the test clarity
2022-04-29 10:01:51 +02:00
Jakub Kuklis
f19e473482
Remove local change
2022-04-29 10:01:51 +02:00
Jakub Kuklis
507ba1042c
Adding a setting to enable Google wrappers special treatment
2022-04-29 10:01:51 +02:00
Jakub Kuklis
6d5c1e2fc0
Adding a setting to enable special treatment of google wrappers
2022-04-29 10:01:50 +02:00
Jakub Kuklis
b7a8acc302
Alternative design for output, mory messy, but the default value inside Google wrapper is not serialized
2022-04-29 10:01:50 +02:00
Jakub Kuklis
53e2454800
Corrected the behaviour for Proto Nullable output
2022-04-29 10:01:50 +02:00
Jakub Kuklis
10425c17b2
Write empty values for Google wrappers
2022-04-29 10:01:50 +02:00
Jakub Kuklis
ff49fad1f1
Another const keyword corrections for debug build
2022-04-29 10:01:50 +02:00
Jakub Kuklis
08ee7470f0
const keyword corrections for debug build
2022-04-29 10:01:50 +02:00
Jakub Kuklis
c0acc4dfa0
Fixing assert
2022-04-29 10:01:50 +02:00
Jakub Kuklis
7a78197746
Style corrections
2022-04-29 10:01:50 +02:00
Jakub Kuklis
ae1194bf9c
Nullables detection in protobuf using Google wrappers
2022-04-29 10:01:50 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common
2022-04-29 10:26:35 +08:00
avogar
d295de1689
Fix comments and test
2022-04-28 14:59:35 +00:00
Kruglov Pavel
4d08587559
Merge branch 'master' into mysqldump-format
2022-04-28 15:58:18 +02:00
Vladimir C
1cbdc1ef3a
Merge pull request #36206 from vdimir/output-format-prometheus
2022-04-28 12:09:53 +02:00
vdimir
be0aa06958
Add output format Prometheus
2022-04-26 14:57:35 +00:00
avogar
b666b4e1c9
Fix possible heap-use-after-free in schema inference
2022-04-26 14:36:16 +00:00
avogar
33d845dade
Add MySQLDump input format
2022-04-26 10:42:56 +00:00
taiyang-li
b7cc344d62
remove useless codes
2022-04-26 14:42:43 +08:00
taiyang-li
99dee35b6e
parallel parsing of hive text format
2022-04-26 14:33:10 +08:00
Kruglov Pavel
34c342fdd3
Merge pull request #36205 from Avogar/improve-globs
...
Some refactoring around schema inference with globs
2022-04-25 13:14:46 +02:00
avogar
80eacc8533
Merge branch 'master' of github.com:ClickHouse/ClickHouse into improve-json-schema-inference
2022-04-22 17:18:44 +00:00
avogar
a093181b4f
Fix comments
2022-04-21 11:48:17 +00:00
Kruglov Pavel
813e228fcc
Merge branch 'master' into improve-globs
2022-04-20 16:31:47 +02:00
avogar
f31f019252
Fix
2022-04-19 19:25:41 +00:00
avogar
1f252cedfe
Make better
2022-04-19 19:16:47 +00:00
Kruglov Pavel
ec4e1cb6d8
Merge pull request #36211 from Avogar/insert-select-all-formats
...
Allow insert select for files with formats without schema inference
2022-04-19 14:25:59 +02:00
avogar
ae88549c4f
Allow insert select for files with formats without schema inference
2022-04-13 20:02:52 +00:00
avogar
8b60aeb7bc
Improve schema inference for json objects
2022-04-13 19:13:40 +00:00
avogar
1c065f8c7a
Some refactoring around schema inference with globs
2022-04-13 17:02:48 +00:00
avogar
348cae0d16
Fix possible segfault in schema inference for JSON formats
2022-04-13 12:34:40 +00:00
avogar
d2017a63b1
Merge branch 'master' of github.com:ClickHouse/ClickHouse into improve-schema-inference
2022-04-07 11:36:40 +00:00
Kruglov Pavel
f3f8f27db5
Merge pull request #35735 from Avogar/allow-read-bools-as-numbers
...
Allow to infer and parse bools as numbers in JSON input formats
2022-04-07 13:20:49 +02:00
taiyang-li
2ef316801c
Merge branch 'master' into use_minmax_index
2022-04-07 10:53:25 +08:00
Kruglov Pavel
ec2213493f
Merge branch 'master' into allow-read-bools-as-numbers
2022-04-06 14:53:02 +02:00
Kruglov Pavel
9141066de3
Merge branch 'master' into improve-schema-inference
2022-04-06 13:51:07 +02:00
taiyang-li
acb9f1632e
suppoort skip splits in orc and parquet
2022-04-06 16:40:22 +08:00
Maksim Kita
e6c9a36ac7
Merge pull request #35733 from kitaisreal/ipv6-invalid-insert-test
...
Added test for insert of invalid IPv6 value
2022-04-04 12:28:16 +02:00
mergify[bot]
1e43e26fa1
Merge branch 'master' into fix-order
2022-04-02 12:00:29 +00:00
avogar
ab2a963287
Merge branch 'master' of github.com:ClickHouse/ClickHouse into allow-read-bools-as-numbers
2022-03-31 14:09:43 +00:00
mergify[bot]
24ade25d61
Merge branch 'master' into improve-schema-inference
2022-03-31 13:42:47 +00:00
Kruglov Pavel
564a77c462
Fix build
2022-03-31 12:49:23 +02:00
Maksim Kita
371cdc956a
Added input format settings for parsing invalid IPv4, IPv6 addresses as default values
2022-03-30 12:54:19 +02:00
avogar
3fc36627b3
Allow to infer and parse bools as numbers in JSON input formats
2022-03-29 17:37:31 +00:00
avogar
ce97ccbfb9
Improve schema inference for JSONEachRow and TSKV formats
2022-03-29 14:47:51 +00:00
Kruglov Pavel
a2fd09e031
Fix style
2022-03-29 16:34:07 +02:00
Antonio Andelic
9990abb76a
Use compile-time check for Exception messages, fix wrong messages
2022-03-29 13:16:11 +00:00
mergify[bot]
343588de2c
Merge branch 'master' into improve-schema-inference
2022-03-29 13:06:00 +00:00
Anton Popov
9610139477
Merge pull request #35629 from CurtizJ/dynamic-columns-5
...
Support schema inference for type `Object` in format `JSONEachRow`
2022-03-29 14:17:09 +02:00
Anton Popov
d677635cd8
Merge pull request #35592 from CurtizJ/dynamic-columns-4
...
Add parallel parsing and schema inference for format `JSONAsObject`
2022-03-28 19:29:55 +02:00
Anton Popov
a6450be8b6
fix schema inference
2022-03-26 01:33:10 +00:00
Anton Popov
67195bfdd5
support schema inference for type Object in format JSONEachRow
2022-03-25 21:51:53 +00:00
Kruglov Pavel
d45143ffe0
Merge branch 'master' into improve-schema-inference
2022-03-25 12:05:40 +01:00
Kruglov Pavel
1823cac89d
Update src/Formats/EscapingRuleUtils.h
...
Co-authored-by: Vladimir C <vdimir@clickhouse.com>
2022-03-24 19:19:32 +01:00
Anton Popov
78100abc5f
add parallel parsing and schema inference for type Object
2022-03-24 17:51:35 +00:00
