Anton Popov
2ae3cfa9e0
Merge branch 'master' into dynamic-columns-14
2022-10-31 16:15:19 +01:00
avogar
fe0aea2e3a
Support parallel parsing for LineAsString input format
2022-10-28 21:56:09 +00:00
Kruglov Pavel
781a27edb3
Remove write callback defenition
2022-10-28 19:46:52 +02:00
avogar
8e13d1f1ec
Improve and refactor Kafka/StorageMQ/NATS and data formats
2022-10-28 16:41:10 +00:00
Kruglov Pavel
e099817449
Merge branch 'master' into Avogar-patch-3
2022-10-27 12:46:18 +02:00
Azat Khuzhin
56bc85746f
Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32
...
Conflicts:
- src/Interpreters/ProcessList.cpp
2022-10-22 16:49:08 +02:00
Azat Khuzhin
4e76629aaf
Fixes for -Wshorten-64-to-32
...
- lots of static_cast
- add safe_cast
- types adjustments
- config
- IStorage::read/watch
- ...
- some TODO's (to convert types in future)
P.S. That was quite a journey...
v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Alexey Milovidov
ff26251477
Merge branch 'master' into fix-race-condition-finish-cancel
2022-10-21 04:14:21 +02:00
Kruglov Pavel
867bcdbb1c
Fix typo in setting name that led to bad usage of schema inference cache
2022-10-20 16:46:25 +02:00
Alexander Tokmakov
68c18abfbb
Merge pull request #42406 from ClickHouse/template_format_better_error
...
Better error message for unsupported delimiters in custom formats
2022-10-20 15:52:08 +03:00
Alexey Milovidov
dfa202a15d
Merge branch 'master' into fix-race-condition-finish-cancel
2022-10-19 02:35:42 +02:00
Kruglov Pavel
25e13bdd2f
Merge pull request #41107 from Avogar/improve-combinators
...
Support all combinators combination in WindowTransform/arratReduce*/initializeAggregation/aggregate functions versioning
2022-10-18 15:24:49 +02:00
Kruglov Pavel
8af95a6fc2
Merge pull request #41912 from Avogar/better-datetime-inference
...
Improve DateTime type inference for text formats
2022-10-18 15:23:59 +02:00
Alexander Tokmakov
fffecbb9ad
better error message for unsupported delimiters in custom formats
2022-10-17 18:08:52 +02:00
Alexey Milovidov
f88ed8195b
Fix trash
2022-10-17 04:21:08 +02:00
Kruglov Pavel
6fc12dd922
Merge pull request #41703 from Avogar/json-object-each-row
...
Add setting to obtain object name as column value in JSONObjectEachRow format
2022-10-14 20:11:04 +02:00
avogar
52427e6028
Remove code duplication
2022-10-14 18:07:02 +00:00
Kruglov Pavel
ff11904850
Merge branch 'master' into improve-combinators
2022-10-14 17:19:31 +02:00
Alexander Tokmakov
4175f8cde6
abort instead of __builtin_unreachable in debug builds
2022-10-07 21:49:08 +02:00
Anton Popov
6e61cf92f5
Merge remote-tracking branch 'upstream/master' into HEAD
2022-10-03 13:16:57 +00:00
Robert Schulze
db5ef7b3cb
Merge branch 'master' into generated-file-cleanup
2022-10-02 23:13:18 +02:00
Vitaly Baranov
f65d3ff95a
Fix parallel parsing: segmentator now checks max_block_size.
2022-09-30 22:34:03 +02:00
Robert Schulze
cc92a2d174
Merge branch 'master' into generated-file-cleanup
2022-09-30 09:56:31 +02:00
vdimir
0f1a7c252d
better TemporaryDataOnDisk
2022-09-29 09:51:46 +00:00
vdimir
efe0f99658
Fix reading block info in NativeReader with header in ctor
2022-09-29 09:51:44 +00:00
vdimir
ac39bbb3f1
[wip] Common interface for temporary data on disk
2022-09-29 09:51:40 +00:00
avogar
e16cfd361b
Improve DateTime type inference for text formats
2022-09-28 16:55:42 +00:00
Robert Schulze
09c62f6728
Consolidate config_formats.h into config.h
...
Less duplication, less confusion ...
2022-09-28 12:59:05 +00:00
Kruglov Pavel
6340369c2a
Merge branch 'master' into improve-combinators
2022-09-28 14:55:30 +02:00
Robert Schulze
78fc36ca49
Generate config.h into ${CONFIG_INCLUDE_PATH}
...
This makes the target location consistent with other auto-generated
files like config_formats.h, config_core.h, and config_functions.h and
simplifies the build of clickhouse_common.
2022-09-28 12:48:26 +00:00
Kruglov Pavel
f1ac2d66be
Merge branch 'master' into json-object-each-row
2022-09-28 14:15:02 +02:00
Robert Schulze
06507c40de
${ConfigIncludePath} --> ${CONFIG_INCLUDE_PATH}
2022-09-28 08:28:47 +00:00
Robert Schulze
1885bb0524
Make comment consistent accross generated files
2022-09-28 08:11:09 +00:00
avogar
76be0d2ee1
Infer Object type only when allow_experimental_object_type is enabled
2022-09-27 23:07:36 +00:00
Kruglov Pavel
3dc54272ed
Merge branch 'master' into improve-combinators
2022-09-26 13:03:32 +02:00
avogar
d3d06251a3
Add setting to obtain object name as column value in JSONObjectEachRow format
2022-09-22 16:48:54 +00:00
Alexey Milovidov
45afacdae4
Merge pull request #41186 from ClickHouse/fix-three-fourth-of-trash
...
Fix more than half of the trash
2022-09-22 07:28:26 +03:00
Kruglov Pavel
22e11aef2d
Merge pull request #40910 from Avogar/new-json-formats
...
Add new JSON formats, add improvements and refactoring
2022-09-21 14:19:08 +02:00
avogar
868ce8bc16
Fix comments, make better naming, add docs, add setting output_format_json_quote_64bit_floats
2022-09-20 13:49:17 +00:00
Kruglov Pavel
47f6f09ce0
Merge branch 'master' into improve-combinators
2022-09-19 14:31:12 +02:00
Alexey Milovidov
84f42e0874
Fix 3/4 of trash
2022-09-19 08:50:53 +02:00
Alexey Milovidov
2f0684b97c
Fix trash in schema inference
2022-09-17 23:11:33 +02:00
Alexey Milovidov
47167494d9
Fix trash in schema inference
2022-09-17 22:53:41 +02:00
avogar
0101cc2e56
Support complex combinators in window transform, arrayReduce*, initializeAggregation and Aggregate functons versionning
2022-09-16 19:07:36 +00:00
Alexey Milovidov
da01982652
Merge pull request #41046 from azat/build/llvm-15
...
Switch to llvm/clang 15
2022-09-16 07:31:06 +03:00
Azat Khuzhin
e8d7403a38
Suppress warning in FormatFactory::getFormatFromFileDescriptor() for FreeBSD
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-10 21:38:35 +02:00
zhenjial
bd9fabc3f7
code optimization, add test
2022-09-09 23:27:42 +08:00
avogar
ad68b7be0f
Better
2022-09-09 15:01:45 +00:00
avogar
46a0318a36
Support JSONColumnsWithMetadata input format
2022-09-08 17:58:44 +00:00
zhenjial
469ceaa156
code optimization
2022-09-09 00:47:43 +08:00
avogar
c380decbbb
Make better, add new settings
2022-09-08 16:07:20 +00:00
Anton Popov
86b29b7f1a
fix serilization of Object inside other types
2022-09-08 15:16:39 +00:00
zhenjial
0f788d98f5
new implementation
2022-09-06 20:39:54 +08:00
avogar
b94e896c1c
Remove logs
2022-09-01 19:01:27 +00:00
avogar
afc34dca41
Add new JSON formats, add improvements and refactoring
2022-09-01 19:00:24 +00:00
avogar
acf87c1d10
Fix nested JSON Objects schema inference
2022-08-31 14:10:29 +00:00
vdimir
0349c85017
Use getCompressedBytes in BufferingToFileTransform and TemporaryFileStream
2022-08-24 16:14:10 +00:00
vdimir
51c44424cc
More metrics for temp files
2022-08-24 16:14:09 +00:00
avogar
29a887578b
Fix
2022-08-23 11:42:57 +00:00
avogar
5ab87f1da4
Small refactoring
2022-08-19 16:42:23 +00:00
avogar
612ffaffde
Make schema inference cache better, respect format settings that can change the schema
2022-08-19 16:39:13 +00:00
Nikolai Kochetov
5a85531ef7
Merge pull request #38286 from Avogar/schema-inference-cache
...
Add schema inference cache for s3/hdfs/file/url
2022-08-18 13:07:50 +02:00
avogar
8dd54c043d
Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache
2022-08-17 11:47:40 +00:00
avogar
e1ff996ec3
Allow to specify structure hints in schema inference
2022-08-16 09:46:57 +00:00
Kruglov Pavel
088e8cf9bd
Merge branch 'master' into numbers-schema-inference
2022-08-09 14:00:36 +02:00
avogar
1304e3487c
Add comments, remove unneded stuff
2022-08-08 13:43:14 +00:00
avogar
9b1a267203
Refactor, remove TTL, add size limit, add system table and system query
2022-08-05 16:20:15 +00:00
Kruglov Pavel
6b2186bfeb
Merge branch 'master' into numbers-schema-inference
2022-08-02 19:34:53 +02:00
Anton Popov
a333cc4146
Merge remote-tracking branch 'upstream/master' into HEAD
2022-08-02 12:57:43 +00:00
Kruglov Pavel
a0d51601bf
Update EscapingRuleUtils.cpp
2022-08-01 13:07:48 +02:00
Alexey Milovidov
4828be7fc4
Fix double escaping in the metadata of FORMAT JSON
2022-07-30 23:56:41 +02:00
Anton Popov
57e4fb2e30
Merge remote-tracking branch 'upstream/master' into HEAD
2022-07-29 11:42:11 +00:00
Kruglov Pavel
381ea139c2
Merge branch 'master' into schema-inference-cache
2022-07-27 11:35:36 +02:00
Kruglov Pavel
5aae0a2e04
Fix style
2022-07-25 17:20:01 +02:00
Anton Popov
49627aa554
Merge remote-tracking branch 'upstream/master' into HEAD
2022-07-22 17:16:06 +00:00
avogar
794aa691bc
Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-protobuf-capnp-empty-message
2022-07-21 17:04:37 +00:00
Anton Popov
e0d2c8fb37
fix json type with sparse columns
2022-07-21 14:47:19 +00:00
avogar
17a271ec30
Fix error codes
2022-07-20 14:33:46 +00:00
Kruglov Pavel
46da17ca8c
Merge branch 'master' into numbers-schema-inference
2022-07-20 13:32:39 +02:00
Kruglov Pavel
3046cd6d29
Merge branch 'master' into schema-inference-cache
2022-07-20 13:30:42 +02:00
Kruglov Pavel
3fb3015375
Merge pull request #39340 from Avogar/better-exception-messages
...
Better exception messages in schema inference
2022-07-20 13:29:15 +02:00
avogar
784ee11594
Add settings to skip fields with unsupported types in Protobuf/CapnProto schema inference
2022-07-20 11:16:25 +00:00
Kruglov Pavel
88d59520a2
Fix
2022-07-19 15:20:56 +02:00
Kruglov Pavel
1513285166
Fix typo
2022-07-18 20:54:13 +02:00
Kruglov Pavel
24c9467641
Fix
2022-07-18 19:55:14 +02:00
avogar
3f81aadb60
Fix schema inference in case of empty messages in Protobuf/CapnProto formats
2022-07-18 17:53:33 +00:00
avogar
2367f40b70
Better exception messages in schema inference
2022-07-18 15:36:33 +00:00
Kruglov Pavel
857290b586
Fix style
2022-07-18 15:40:28 +02:00
Kruglov Pavel
0f6044e50f
Fix style
2022-07-18 15:39:53 +02:00
avogar
9291d33080
Pass const std::string_view & by value, not by reference
2022-07-14 16:11:57 +00:00
Kruglov Pavel
b38241b08a
Merge branch 'master' into schema-inference-cache
2022-07-14 12:29:54 +02:00
avogar
2b7c6b7ecd
Remove logging
2022-07-13 15:59:04 +00:00
avogar
7cde9d3b40
Add new features in schema inference
2022-07-13 15:57:55 +00:00
avogar
5b0fd31c64
Put column names in quotes
2022-06-30 16:14:30 +00:00
avogar
ee54c4f9b7
Add some fixes and add settings in docs
2022-06-30 12:41:56 +00:00
mergify[bot]
9482c99ab8
Merge branch 'master' into sql-insert-format
2022-06-29 11:03:07 +00:00
Robert Schulze
c22038d48b
More clang-tidy fixes
2022-06-28 11:50:05 +00:00
avogar
9bb68bc6de
Add SQLInsert output format
2022-06-27 18:31:57 +00:00
avogar
b0c9d1a25d
Fix style
2022-06-27 14:04:28 +00:00
avogar
5155262a16
Add some additional information to cache keys
2022-06-27 12:43:24 +00:00
avogar
d37ad2e6de
Implement cache for schema inference for file/s3/hdfs/url
2022-06-21 13:02:48 +00:00
Alexey Milovidov
5e9e5a4eaf
Merge pull request #37525 from Avogar/avro-structs
...
Support Maps and Records, allow to insert null as default in Avro format
2022-06-15 00:04:29 +03:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros
...
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.
__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
2022-06-10 10:22:31 +02:00
Kruglov Pavel
6f17ba17ba
Revert "Revert "Fix possible segfault in schema inference""
2022-06-02 13:28:27 +02:00
Alexander Tokmakov
4baae59252
Revert "Fix possible segfault in schema inference"
2022-06-02 14:04:28 +03:00
avogar
4abfd54dd6
Fix possible segfault in schema inference
2022-06-01 16:53:37 +00:00
Kruglov Pavel
7cc87d9a65
Merge pull request #37537 from Avogar/skip-first-lines
...
Allow to skip some of the first lines in CSV/TSV formats
2022-05-31 14:26:21 +02:00
Kruglov Pavel
0615866aea
Merge pull request #37450 from Avogar/check-format-on-storage-creation
...
Check format name on storage creation
2022-05-30 14:23:20 +02:00
Alexey Milovidov
c50791dd3b
Fix clang-tidy-14, part 1
2022-05-27 22:52:14 +02:00
avogar
4c9812d4c1
Allow to skip some of the first rows in CSV/TSV formats
2022-05-25 15:00:11 +00:00
avogar
038a422aeb
Add setting to insert null as default
2022-05-25 12:56:59 +00:00
avogar
f782fa31c6
Merge branch 'master' of github.com:ClickHouse/ClickHouse into check-format-on-storage-creation
2022-05-25 08:42:54 +00:00
avogar
37b66c8a9e
Check format name on storage creation
2022-05-23 12:48:48 +00:00
Kruglov Pavel
f539fb835d
Merge branch 'master' into formats-with-names
2022-05-23 12:14:20 +02:00
Kruglov Pavel
ce48e8e102
Merge pull request #36975 from Avogar/json-columns-formats
...
Add columnar JSON formats
2022-05-23 12:11:28 +02:00
avogar
a4cf07708c
Fix comments
2022-05-20 14:57:27 +00:00
avogar
566d1b15fd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into formats-with-names
2022-05-20 13:54:52 +00:00
avogar
44726122bb
Join JSON registration
2022-05-20 12:09:51 +00:00
avogar
a6a430c5ee
Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats
2022-05-20 11:08:30 +00:00
mergify[bot]
1ac4199e78
Merge branch 'master' into arrow-strings
2022-05-20 10:43:33 +00:00
avogar
cd6a29897e
Apply input_format_max_rows_to_read_for_schema_inference for all files in globs in total
2022-05-18 17:56:36 +00:00
avogar
a0369fb9a6
Allow to use String type instead of Binary in Arrow/Parquet/ORC formats
2022-05-18 14:51:21 +00:00
Kruglov Pavel
134821eff8
Fix build
2022-05-18 12:44:20 +02:00
avogar
12010a81b7
Make better
2022-05-18 09:25:26 +00:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies
2022-05-16 10:12:50 +02:00
avogar
68bb07d166
Better naming
2022-05-13 18:39:19 +00:00
avogar
cef13c2c02
Allow to skip unknown columns in Native format
2022-05-13 14:27:15 +00:00
avogar
b17fec659a
Improve performance and memory usage for select of subset of columns for some formats
2022-05-13 13:51:28 +00:00
avogar
f6b16880bd
Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-columns-formats
2022-05-10 12:57:18 +00:00
Anton Popov
e911900054
remove last mentions of data streams
2022-05-09 19:15:24 +00:00
avogar
04fdd75c56
Make JSONColumns frormats mono block by default
2022-05-09 11:13:44 +00:00
Robert Schulze
f2b1748c48
Enable clang-tidy bugprone-suspicious-semicolon
...
Official docs:
Finds most instances of stray semicolons that unexpectedly alter the
meaning of the code.
2022-05-08 19:13:37 +02:00
avogar
62a7ba3f26
Add columnar JSON formats
2022-05-06 16:48:48 +00:00
Kruglov Pavel
77e55c344c
Merge pull request #36667 from Avogar/mysqldump-format
...
Add MySQLDump input format
2022-05-04 19:49:48 +02:00
Dmitry Novik
5ba7a55c18
Merge pull request #36650 from bigo-sg/hive_text_parallel_parsing
...
Parallel parsing of hive text format
2022-05-03 15:56:28 +02:00
Kruglov Pavel
d613f7eab0
Merge branch 'master' into mysqldump-format
2022-05-02 13:31:57 +02:00
Antonio Andelic
a1a22b0007
Merge pull request #35149 from ContentSquare/nullables_with_proto3
...
Nullables with proto3 using Google wrappers
2022-05-02 09:49:37 +02:00
Robert Schulze
89aa9ae00f
Fixed clang-tidy check "bugprone-branch-clone"
...
The check is currently *not* part of .clang-tidy. It complains about:
(1) "switch has multiple consecutive identical branches"
(2) "repeated branch in conditional chain"
About (1): Lots of findings in switches were about redundant
"[[fallthrough]]" in places where the compiler would not warn anyways. I
have cleaned these up.
About (2): In if-else_if-else chains, fixing the warning would usually
mean concatenating multiple if-conditions. As this would reduce
readability in most cases, I did not fix these places.
Because of (2), I also refrained from adding "bugprone-branch-clone" to
.clang-tidy.
2022-04-30 19:40:28 +02:00
Jakub Kuklis
a1f2dd6d34
Adding two settings in place of one, improvements to the test clarity
2022-04-29 10:01:51 +02:00
Jakub Kuklis
e73fa271a2
Minor improvements
2022-04-29 10:01:51 +02:00
Jakub Kuklis
5ca095c779
Pass the setting to buildFieldSerializer to fix undeclared
2022-04-29 10:01:51 +02:00
Jakub Kuklis
e705425374
Minor improvements
2022-04-29 10:01:51 +02:00
Jakub Kuklis
5c34585a00
Improve the test clarity
2022-04-29 10:01:51 +02:00
Jakub Kuklis
f19e473482
Remove local change
2022-04-29 10:01:51 +02:00
Jakub Kuklis
507ba1042c
Adding a setting to enable Google wrappers special treatment
2022-04-29 10:01:51 +02:00
Jakub Kuklis
6d5c1e2fc0
Adding a setting to enable special treatment of google wrappers
2022-04-29 10:01:50 +02:00
Jakub Kuklis
b7a8acc302
Alternative design for output, mory messy, but the default value inside Google wrapper is not serialized
2022-04-29 10:01:50 +02:00
Jakub Kuklis
53e2454800
Corrected the behaviour for Proto Nullable output
2022-04-29 10:01:50 +02:00
Jakub Kuklis
10425c17b2
Write empty values for Google wrappers
2022-04-29 10:01:50 +02:00
Jakub Kuklis
ff49fad1f1
Another const keyword corrections for debug build
2022-04-29 10:01:50 +02:00
