Michael Kolupaev
d3a514d221
Compress marks in memory
2023-03-13 16:29:00 -07:00
Alexey Milovidov
f331b9b398
Fix errors and add tests
2023-03-13 23:49:28 +01:00
Alexey Milovidov
14647525f8
Merge branch 'fix-bson-bug' of github.com:Avogar/ClickHouse into fuzzer-of-data-formats
2023-03-13 22:45:00 +01:00
avogar
4213ec609f
Proper fix for bug in parquet, revert reverted #45878
2023-03-13 18:22:09 +00:00
Alexey Milovidov
1fd24c212b
Update comment
2023-03-13 07:42:58 +01:00
Alexey Milovidov
02f7ef4723
Update comment
2023-03-13 05:28:06 +01:00
Alexey Milovidov
43b938d303
Update the fuzzer
2023-03-13 05:21:48 +01:00
Alexey Milovidov
f33b651686
Add fuzzer for data formats
2023-03-13 04:51:50 +01:00
avogar
5a18acde90
Revert #45878 and add a test
2023-03-11 21:15:14 +00:00
Kruglov Pavel
f387e6013a
Merge pull request #46990 from Avogar/native-types-conversions
...
Allow types conversion in Native input format
2023-03-10 16:55:16 +01:00
Alexey Milovidov
6f35d46ac8
Update SchemaInferenceUtils.cpp
2023-03-10 05:01:06 +03:00
avogar
46979e383f
Fix big numbers inference in CSV
2023-03-09 18:21:47 +00:00
Kruglov Pavel
fe973f3d6f
Merge branch 'master' into native-types-conversions
2023-03-09 13:03:25 +01:00
Kruglov Pavel
71b6d6c6ae
Merge pull request #47114 from Avogar/parquet-compression
...
Improve working with compression methods in Parquet/ORC/Arrow formats
2023-03-09 13:02:18 +01:00
Mike Kot
9920a52c51
use std::lerp, constexpr hex.h
2023-03-07 22:50:17 +00:00
Kruglov Pavel
69a1309ade
Merge branch 'master' into native-types-conversions
2023-03-07 20:06:17 +01:00
Kruglov Pavel
479cd9b90b
Merge pull request #46972 from Avogar/json-date-int-inference
...
Fix date and int inference from string in JSON
2023-03-06 20:40:38 +01:00
Kruglov Pavel
3de905bb7c
Merge pull request #46616 from Avogar/fix-ipv4-ipv6-formats
...
Fix IPv4/IPv6 serialization/deserialization in binary formats
2023-03-06 19:40:29 +01:00
avogar
5ab5902f38
Allow control compression in Parquet/ORC/Arrow output formats, support more compression for input formats
2023-03-01 21:27:46 +00:00
Kruglov Pavel
65f06fc9b1
Merge branch 'master' into json-date-int-inference
2023-02-28 14:31:57 +01:00
avogar
ab899bf2f3
Allow types conversion in Native input format
2023-02-27 19:28:19 +00:00
avogar
2e921e3d6b
Fix date and int inference from string in JSON
2023-02-27 16:00:19 +00:00
Kruglov Pavel
443dedddca
Merge branch 'master' into use-parquet-2
2023-02-27 14:31:43 +01:00
Kruglov Pavel
47f9ca2166
Merge branch 'master' into fix-ipv4-ipv6-formats
2023-02-23 20:32:43 +01:00
avogar
eec6051a50
style
2023-02-23 16:16:08 +00:00
avogar
54622566df
Add setting to change parquet version
2023-02-23 16:14:10 +00:00
Kruglov Pavel
ef0d6becba
Merge branch 'master' into null-as-default-all-formats
2023-02-21 16:52:39 +01:00
Kruglov Pavel
b0424c1021
Merge pull request #46171 from Avogar/insert-null-as-default
...
Use default of column type in `insert_null_as_default` if column DEFAULT values is not specified
2023-02-20 21:45:02 +01:00
Kruglov Pavel
9866ecfe8b
Merge branch 'master' into null-as-default-all-formats
2023-02-20 20:49:30 +01:00
avogar
8da3594cd8
Fix IPv4/IPv6 serialization/deserialization in binary formats
2023-02-20 17:42:56 +00:00
Alexey Milovidov
d8cda3dbb8
Remove PVS-Studio
2023-02-19 23:30:05 +01:00
Kruglov Pavel
9fd2226c4c
Update NativeReader.h
2023-02-15 15:13:04 +01:00
Geoff Genz
be8bf3a6a3
Merge branch 'master' into http_client_version
2023-02-13 08:43:59 -07:00
avogar
d1efd02480
Extend setting input_format_null_as_default for more formats
2023-02-10 16:41:09 +00:00
Geoff Genz
99c3ff53c5
Merge remote-tracking branch 'origin/master' into http_client_version
...
# Conflicts:
# src/Interpreters/Context.cpp
# src/Interpreters/Context.h
2023-02-10 04:35:53 -07:00
Geoff Genz
7ed8ed0284
Add support for client_protocol_version sent with HTTP
2023-02-10 03:47:06 -07:00
avogar
c3e8dd8984
Fix low cardinality case
2023-02-08 19:14:28 +00:00
Kruglov Pavel
4e2918cee3
Merge branch 'master' into parquet-fixed-binary
2023-02-08 12:31:13 +01:00
Antonio Andelic
a39e4e24c6
Merge branch 'master' into optimize_parquet_reader
2023-02-02 14:18:00 +01:00
Vladimir C
7c6281c446
Merge pull request #45581 from Avogar/fix-date-inference
2023-02-01 13:04:12 +01:00
liuneng
17fc22a21e
add parquet max_block_size setting
2023-02-01 18:29:20 +08:00
Alexey Milovidov
04078dbed3
Remove trash
2023-01-29 22:43:36 +01:00
Kruglov Pavel
96700abbe1
Merge pull request #45678 from azat/formats/json-parse-tupels
...
Add ability to ignore unknown keys in JSON object for named tuples
2023-01-27 21:11:05 +01:00
Azat Khuzhin
1a8437f2c9
Add ability to ignore unknown keys in JSON object for named tuples
...
This can be useful in case your input JSON is complex, while you need
only few fields in it.
This behaviour is controlled by the
input_format_json_ignore_unknown_keys_in_named_tuple setting name, that
is turned OFF by default.
This will, almost, allow to parse gharchive dataset without jq. "almost"
because of two things:
- Tuple cannot be Nullable, so such keys with Tuple type in ClickHouse
cannot be `null` in JSON
- You cannot use dot.dot notation to extract columns for file() engine,
only tupleElement()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-27 10:01:08 +01:00
Alexander Tokmakov
d1baa7300c
reformat ParsingException
2023-01-24 23:21:29 +01:00
Alexander Tokmakov
dd57215934
Merge branch 'master' into exception_message_patterns4
2023-01-24 17:03:12 +01:00
Kruglov Pavel
23c12ac8ee
Merge branch 'master' into parquet-fixed-binary
2023-01-24 16:51:05 +01:00
avogar
7eeb2a0bc7
Change comment
2023-01-24 15:46:32 +00:00
avogar
159f49266e
Don't infer Dates from 8 digit numbers
2023-01-24 15:45:27 +00:00
Kruglov Pavel
cd1cd904a7
Merge branch 'master' into tsv-csv-detect-header
2023-01-23 23:49:56 +01:00
Alexander Tokmakov
3f6594f4c6
forbid old ctor of Exception
2023-01-23 22:18:05 +01:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages ( #45449 )
...
* save format string for NetException
* format exceptions
* format exceptions 2
* format exceptions 3
* format exceptions 4
* format exceptions 5
* format exceptions 6
* fix
* format exceptions 7
* format exceptions 8
* Update MergeTreeIndexGin.cpp
* Update AggregateFunctionMap.cpp
* Update AggregateFunctionMap.cpp
* fix
2023-01-24 00:13:58 +03:00
Kruglov Pavel
9820beae68
Apply suggestions from code review
...
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2023-01-19 16:11:13 +01:00
avogar
5bf4704e7a
Support FixedSizeBinary type in Parquet/Arrow
2023-01-16 21:01:31 +00:00
Kruglov Pavel
e9d6590926
Merge branch 'master' into tsv-csv-detect-header
2023-01-16 17:50:24 +01:00
avogar
87b934c472
Insert default values in case of missing tuple elements in JSONEachRow
2023-01-12 16:36:44 +00:00
avogar
b461935374
Better
2023-01-12 13:11:04 +00:00
Kruglov Pavel
05a11ff4a4
Merge branch 'master' into tsv-csv-detect-header
2023-01-12 12:35:18 +01:00
avogar
26cd56d113
Fix tests, make better
2023-01-11 22:52:15 +00:00
Kruglov Pavel
50eb9fca67
Merge pull request #44696 from Avogar/schema-inference-uint
...
Infer UInt64 in case of Int64 overflow
2023-01-11 14:24:42 +01:00
Alexey Milovidov
0d39d26a34
Don't fix parallel formatting
2023-01-09 06:15:20 +01:00
Anton Popov
1f32ffedf8
Merge pull request #43221 from ClickHouse/refactoring-ip-types
...
Replace domain IP types (IPv4, IPv6) with native
2023-01-07 12:01:21 +01:00
avogar
ee72799121
Fix tests, make better
2023-01-06 20:46:43 +00:00
Anton Popov
b25f875674
Merge pull request #44875 from ClickHouse/fix-another-one-cannot-read-all-data-for-lc-dict-error
...
Fix right offset for reading LowCardinality dictionary from remote fs
2023-01-06 15:24:36 +01:00
avogar
7fcdb08ec6
Detect header in CSV/TSV/CustomSeparated files automatically
2023-01-05 22:57:25 +00:00
Yakov Olkhovskiy
7a5a36cbed
Merge branch 'master' into refactoring-ip-types
2023-01-04 11:11:06 -05:00
avogar
1f3d75cbf2
Better
2023-01-04 14:58:17 +00:00
Kruglov Pavel
7062054d60
Merge branch 'master' into schema-inference-uint
2023-01-04 14:50:01 +01:00
Nikolai Kochetov
da26f62a9b
Fix right offset for reading LowCardinality dictionary from remote fs in case if right mark was in the middle of compressed block.
2023-01-03 18:19:51 +00:00
Alexey Milovidov
e855d3519a
Merge branch 'master' into refactoring-ip-types
2023-01-02 21:58:53 +03:00
avogar
73fecae5ff
Fix comments
2023-01-02 15:31:07 +00:00
Kruglov Pavel
0a43976977
Merge branch 'master' into validate-types
2023-01-02 16:10:14 +01:00
Kruglov Pavel
69b9842bc6
Merge branch 'master' into schema-inference-uint
2022-12-30 18:16:00 +01:00
Kruglov Pavel
4982d132fb
Merge branch 'master' into validate-types
2022-12-30 17:52:13 +01:00
Kruglov Pavel
894726bd8f
Merge branch 'master' into improve-streaming-engines
2022-12-29 22:59:45 +01:00
Kruglov Pavel
150a699dda
Merge pull request #44546 from Avogar/better-object-as-string-inference
...
Improve json object as string inference
2022-12-29 21:58:46 +01:00
avogar
1ce69371fb
Infer UInt64 in case of Int64 overflow
2022-12-28 21:46:08 +00:00
Raúl Marín
5de11979ce
Unify query elapsed time measurements ( #43455 )
...
* Unify query elapsed time reporting
* add-test: Make shell tests executable
* Add some tests around query elapsed time
* Style and ubsan
2022-12-28 21:01:41 +01:00
avogar
411f98306a
Merge branch 'master' of github.com:ClickHouse/ClickHouse into validate-types
2022-12-27 19:24:15 +00:00
Kruglov Pavel
819e7a3008
Merge pull request #44550 from Avogar/better-json-tuples-to-arrays-inference
...
Improve inferring arrays with nulls in JSON formats
2022-12-27 18:22:13 +01:00
Kruglov Pavel
ac162a2c49
Merge pull request #44522 from Avogar/zero-numbers
...
Infer numbers starting from zero as strings in TSV
2022-12-27 17:00:10 +01:00
avogar
798c3111ed
Improve inferring arrays with nulls in JSON formats
2022-12-24 00:21:48 +00:00
avogar
331f4bfee1
Fix
2022-12-23 19:58:50 +00:00
avogar
f15bf1839a
Add missed settings into additional cache info
2022-12-23 19:52:54 +00:00
avogar
8dfe90a6c1
Improve json object as string inference
2022-12-23 19:44:13 +00:00
avogar
123392c996
Fix tests
2022-12-23 14:42:38 +00:00
Vladimir C
7482ea54ab
Merge pull request #43972 from ClickHouse/vdimir/tmp-data-in-fs-cache-2
2022-12-23 11:59:27 +01:00
avogar
f555048ae5
Infer numbers starting from zero as strings in TSV
2022-12-22 21:55:39 +00:00
Dmitry Novik
cff882d506
Merge remote-tracking branch 'origin/master' into refector-function-node
2022-12-22 21:34:29 +00:00
Kruglov Pavel
6a017a6586
Merge pull request #43379 from Avogar/better-capn-proto
...
Add small improvements in CapnProto format
2022-12-22 14:50:10 +01:00
vdimir
d30d25dbbe
Temporary files evict fs cache
2022-12-22 10:22:49 +00:00
Yakov Olkhovskiy
a8cb29da4b
Merge branch 'master' into refactoring-ip-types
2022-12-21 23:56:24 -05:00
avogar
4ab3e90382
Validate types in table function arguments/CAST function arguments/JSONAsObject schema inference
2022-12-21 21:21:30 +00:00
Kruglov Pavel
5e01a3d74e
Merge branch 'master' into improve-streaming-engines
2022-12-21 10:51:50 +01:00
Kruglov Pavel
09ab5832b1
Merge pull request #44382 from Avogar/fix-bson-object-id
...
Fix reading ObjectId in BSON schema inference
2022-12-21 10:48:50 +01:00
Dmitry Novik
4793412887
Merge branch 'master' into refector-function-node
2022-12-20 18:26:19 +01:00
Kruglov Pavel
c0b17ca0af
Merge branch 'master' into fix-bson-object-id
2022-12-20 17:18:10 +01:00
avogar
21cdf6e6ae
Fix reading ObjectId in BSON schema inference
2022-12-19 14:13:42 +00:00
Dmitry Novik
875a24a650
Merge remote-tracking branch 'origin/master' into refector-function-node
2022-12-16 16:07:30 +00:00
avogar
4a51bdce86
Fix comments
2022-12-16 13:58:54 +00:00
Kruglov Pavel
3fad5c7f1f
Merge branch 'master' into refactor-schema-inference
2022-12-16 14:24:51 +01:00
avogar
cfcb444699
Merge branch 'master' of github.com:ClickHouse/ClickHouse into better-capn-proto
2022-12-15 20:04:43 +00:00
Kruglov Pavel
25f199dd89
Merge pull request #43332 from Avogar/csv-custom-delimiter
...
Improve reading CSV field in CustomSeparated/Template format
2022-12-15 21:03:29 +01:00
Kruglov Pavel
c5b2e4cc23
Merge branch 'master' into improve-streaming-engines
2022-12-15 18:44:35 +01:00
avogar
f19afbc03e
Fix fasttest
2022-12-13 12:59:27 +00:00
avogar
739ad23b1f
Make better, fix bugs, improve error messages
2022-12-12 22:00:45 +00:00
Dmitry Novik
3d2fccab87
Merge branch 'master' into refector-function-node
2022-12-12 21:36:39 +01:00
avogar
f3e37c2c9b
Merge branch 'refactor-schema-inference' of github.com:Avogar/ClickHouse into refactor-schema-inference
2022-12-12 14:47:04 +00:00
Kruglov Pavel
a03549df28
Apply suggestions from code review
...
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2022-12-12 15:46:03 +01:00
avogar
cd4fa00d2c
Merge branch 'master' of github.com:ClickHouse/ClickHouse into refactor-schema-inference
2022-12-09 14:45:10 +00:00
avogar
1ec5f8451b
Merge branch 'master' of github.com:ClickHouse/ClickHouse into csv-custom-delimiter
2022-12-08 19:17:42 +00:00
avogar
d0f9bb2ec2
Allow to parse JSON objects into Strings
2022-12-08 18:58:18 +00:00
Yakov Olkhovskiy
0641066183
Merge branch 'master' into refactoring-ip-types
2022-12-08 11:12:05 -05:00
Kruglov Pavel
26ed850b2d
Fix typo
2022-12-07 23:00:11 +01:00
Yakov Olkhovskiy
bf9194f405
review suggestions
2022-12-07 21:29:17 +00:00
avogar
7375a7d429
Refactor and improve schema inference for text formats
2022-12-07 21:19:27 +00:00
Dmitry Novik
15b8c48ca9
Cleanup code
2022-12-02 19:15:26 +00:00
Dmitry Novik
2c70dbc76a
Refactor FunctionNode
2022-12-02 19:15:26 +00:00
Vladimir C
7d6950d397
Revert "Temporary files evict fs cache"
2022-12-02 14:50:56 +01:00
Kruglov Pavel
c35b2a6495
Add a limit for string size in RowBinary format ( #43842 )
2022-12-02 13:57:11 +01:00
vdimir
816af3dc16
wip: temporary files evict fs cache
2022-12-01 11:49:25 +00:00
vdimir
98fe3c6c02
Temporary files evict fs cache
2022-12-01 11:49:17 +00:00
Anton Popov
fe5fff0347
Merge pull request #43329 from xiedeyantu/support_nested_column
...
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-29 22:27:19 +01:00
Yakov Olkhovskiy
770b520ded
Merge branch 'master' into refactoring-ip-types
2022-11-28 08:50:19 -05:00
xiedeyantu
304b6ebf3a
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-23 23:36:12 +08:00
Kruglov Pavel
98d6b96c82
Merge pull request #42033 from mark-polokhov/BSONEachRow
...
Add BSONEachRow input/output format
2022-11-22 14:45:21 +01:00
Kruglov Pavel
49eed2a07c
Merge branch 'master' into better-capn-proto
2022-11-22 14:11:53 +01:00
avogar
db8126f9c5
Merge branch 'master' of github.com:ClickHouse/ClickHouse into csv-custom-delimiter
2022-11-21 13:49:14 +00:00
avogar
37e14dc091
Fix tests
2022-11-21 13:46:15 +00:00
avogar
ecdeff622b
Add small improvements in CapnProto format
2022-11-18 20:13:00 +00:00
avogar
fcfdd73d17
Improve reading CSV field in CustomSeparated/Template format
2022-11-17 15:36:56 +00:00
Vitaly Baranov
ce81166c7e
Fix style.
2022-11-16 01:35:11 +01:00
Yakov Olkhovskiy
813cb7fb0d
merge master
2022-11-15 22:46:05 +00:00
avogar
4d993e653a
Fix build and style
2022-11-15 13:06:24 +00:00
avogar
842d25c358
Minor improvements, better docs
2022-11-14 20:05:01 +00:00
Vitaly Baranov
8e99f5fea3
Move maskSensitiveInfoInQueryForLogging() to src/Parsers/
2022-11-14 18:55:19 +01:00
Yakov Olkhovskiy
9aeebf3bdf
Merge branch 'master' into refactoring-ip-types
2022-11-14 09:21:54 -05:00
Yakov Olkhovskiy
4d144be39c
replace domain IP types (IPv4, IPv6) with native
2022-11-14 14:17:17 +00:00
avogar
564d83bbc7
Better handle uint64
2022-11-11 13:24:12 +00:00
avogar
88636b0f5b
Fix style
2022-11-11 12:41:16 +00:00
avogar
4f05045726
Fix build
2022-11-11 11:41:14 +00:00
avogar
cd36caf013
Fix style
2022-11-10 20:37:24 +00:00
avogar
9e89af28c6
Refactor BSONEachRow format, fix bugs, support more data types, support parallel parsing and schema inference
2022-11-10 20:15:14 +00:00
Kruglov Pavel
b124875257
Merge branch 'master' into improve-streaming-engines
2022-11-03 13:22:06 +01:00
avogar
7cc87679e4
Merge branch 'master' of github.com:ClickHouse/ClickHouse into BSONEachRow
2022-11-02 19:47:42 +00:00
Mark Polokhov
2fff4887ac
Add BSON input/output format
2022-11-02 19:39:14 +00:00
avogar
774a86021f
Fix datetime schema inference in case of empty string
2022-11-02 19:18:34 +00:00
Kruglov Pavel
38124b6533
Merge pull request #42780 from Avogar/parallel-parsing
...
Support parallel parsing for LineAsString input format
2022-11-02 13:21:53 +01:00
Anton Popov
876dca48da
Merge pull request #36969 from CurtizJ/dynamic-columns-14
...
Support `Object` type inside other types
2022-11-01 15:20:02 +01:00
Anton Popov
2ae3cfa9e0
Merge branch 'master' into dynamic-columns-14
2022-10-31 16:15:19 +01:00
