Kruglov Pavel
7220797637
Fix style
2024-03-26 15:26:42 +01:00
avogar
dc87c483dd
Add a setting to allow saving bad escape sequences in JSON input formats
2024-03-25 21:58:53 +00:00
Alexey Milovidov
3e5ddddb35
Merge branch 'master' into dont-cut-single-value
2024-03-24 00:51:10 +01:00
Alexey Milovidov
4cbecd0bbd
Add a setting
2024-03-23 04:20:52 +01:00
Alexey Milovidov
a2e89c8be7
Fix wrong cases of numbers pretty printing
...
Add a test
Revert changes from another branch
Add a test
Better test
Revert wrong changes
2024-03-23 03:33:03 +01:00
shuai-xu
9d5cabb26d
fix csv format not support tuple
2024-03-22 16:51:58 +08:00
Raúl Marín
de855ca917
Reduce header dependencies
2024-03-19 17:04:29 +01:00
Shaun Struwig
01919f0bd3
Merge branch 'ClickHouse:master' into 56257_parse_crlf_with_TSV_files
2024-03-17 20:32:39 +01:00
Alexey Milovidov
01136bbc3b
Limit backtracking in parser
2024-03-17 19:54:45 +01:00
Alexey Milovidov
0a3e42401c
Fix fuzzers
2024-03-17 15:44:36 +01:00
avogar
feda83a7c8
Merge branch 'master' of github.com:ClickHouse/ClickHouse into 56257_parse_crlf_with_TSV_files
2024-03-14 17:44:38 +00:00
Shaun Struwig
f251a6d262
Merge branch 'ClickHouse:master' into 59557_form_input_format
2024-03-11 18:52:28 +01:00
Raúl Marín
9bada70f45
Remove a bunch of transitive dependencies
2024-03-11 14:52:32 +01:00
Mikhail Koviazin
490efd2efa
Fixes addressing review comments
2024-03-06 14:35:48 +02:00
Blargian
2ad8ab2a57
Fix linker errors
2024-03-05 19:13:20 +01:00
Shaun Struwig
beb0d08bdb
Merge branch 'ClickHouse:master' into 56257_parse_crlf_with_TSV_files
2024-03-05 14:09:01 +01:00
Alexey Milovidov
570692fe83
Merge branch 'master' into json-ambg-tuple-inference
2024-03-05 04:50:39 +03:00
Kruglov Pavel
4bdafed801
Merge pull request #60420 from HowePa/format_case_insensitive
...
Make all format names case insensitive.
2024-03-04 19:09:10 +01:00
avogar
9a05461680
Better exception message
2024-03-04 17:49:33 +00:00
avogar
70abdf7a41
Small improvements in JSON schema inference
2024-03-04 17:32:22 +00:00
Alexey Milovidov
cbf5443585
Remove old code
2024-03-04 00:11:55 +01:00
Shaun Struwig
b54286bed4
Merge branch 'master' into 56257_parse_crlf_with_TSV_files
2024-03-01 18:21:05 +01:00
Mikhail Koviazin
11371e886c
Update protobuf to v25.1
...
The new version deprecates `syntax()` and makes it inaccessible. Instead, the
attributes corresponding to a feature should be used. This commit addresses
this.
2024-03-01 17:03:10 +02:00
Robert Schulze
4ee1aa8c7c
Fixing more headers
2024-02-29 15:40:30 +00:00
Alexey Milovidov
e6dffb1f2d
Merge pull request #60379 from rogeryk/improve-pretty-format
...
Improve pretty format if a block consists of a single numeric value and exceeds one million.
2024-02-29 02:20:42 +03:00
Alexey Milovidov
fe42d8ecfc
Merge pull request #60050 from ClickHouse/less-memory-usage-primary-key-2
...
Less memory usage in primary key, variant 2
2024-02-28 19:22:46 +03:00
Alexey Milovidov
c192a448d0
Update to clang-19
2024-02-27 14:37:21 +01:00
豪肥肥
6f9cb058a6
Update FormatFactory.cpp
2024-02-27 07:59:09 +08:00
豪肥肥
24155c80c9
Update src/Formats/FormatFactory.cpp
...
Co-authored-by: Kruglov Pavel <48961922+Avogar@users.noreply.github.com>
2024-02-27 07:50:04 +08:00
HowePa
dbd8d35f01
use lower case in dict
2024-02-27 00:48:34 +08:00
HowePa
0b72f7b182
Make all format names case insensitive.
2024-02-26 22:46:51 +08:00
rogeryk
7a92f542b4
Add setting output_format_pretty_single_large_number_tip_threshold
2024-02-26 20:19:53 +08:00
Alexey Milovidov
209a151e00
Merge branch 'master' into less-memory-usage-primary-key-2
2024-02-23 16:02:29 +01:00
Shaun Struwig
4a5761ce1f
Merge branch 'ClickHouse:master' into 56257_parse_crlf_with_TSV_files
2024-02-20 22:18:01 +01:00
Kruglov Pavel
5fd2582e83
Merge pull request #59500 from Avogar/exponent-floats-inference
...
Don't infer floats in exponential notation by default
2024-02-19 13:51:07 +01:00
Alexey Milovidov
df48106cd5
Merge pull request #60015 from azat/values-quote-escape
...
Fix INSERT into SQLite with single quote (by escaping single quotes with a quote instead of backslash)
2024-02-19 10:37:45 +01:00
Alexey Milovidov
04b2f085bf
Revert changes where they were not correct
2024-02-17 05:14:43 +01:00
Shaun Struwig
1d440f0399
Merge branch 'ClickHouse:master' into 56257_parse_crlf_with_TSV_files
2024-02-16 19:48:33 +01:00
Kruglov Pavel
4d6f167e0c
Merge pull request #59092 from Avogar/auto-format-detection
...
Try to detect file format automatically during schema inference if it's unknown
2024-02-16 14:32:18 +01:00
Azat Khuzhin
bbe38a3fe4
Add ability to escape quotes in Values format with single quote
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-02-15 12:47:29 +01:00
Alexey Milovidov
bbd7acd7f9
Merge branch 'master' into exponent-floats-inference
2024-02-15 01:46:51 +01:00
Shaun Struwig
525a5188e4
Merge branch 'master' into 56257_parse_crlf_with_TSV_files
2024-02-14 17:40:08 +01:00
Kruglov Pavel
db2c15c0a6
Apply suggestions from code review
...
Co-authored-by: Sergei Trifonov <svtrifonov@gmail.com>
2024-02-14 12:24:10 +01:00
Blargian
91d681693c
add failing test and Form format skeleton code
2024-02-14 10:19:06 +01:00
Blargian
0ff4202452
fix up test and modify skipRowEndDelimiter so that crlf works with .tsv
2024-02-08 07:08:17 +01:00
Blargian
ab384f8652
add support_crlf for TSV format
2024-02-04 15:29:57 +01:00
avogar
ff21aa9a19
Don't infer floats in exponential notation by default
2024-02-01 19:47:05 +00:00
Shaun Struwig
e081a4f059
Merge branch 'master' into #31363_format_template_configure_in_settings
2024-01-29 21:29:22 +01:00
Blargian
4a8a7208f2
rename of settings, add setting for resultset, extend test, fix documentation and add to SettingsChanges log
2024-01-29 21:25:58 +01:00
avogar
377937d415
Merge branch 'master' of github.com:ClickHouse/ClickHouse into auto-format-detection
2024-01-29 15:45:18 +00:00
Kruglov Pavel
6858d2f4ca
Merge pull request #58047 from Avogar/variant-data-type
...
Implement Variant data type
2024-01-29 11:36:08 +01:00
Alexey Milovidov
aec3f28ccb
Support backups for compressed in-memory tables
2024-01-28 23:06:50 +01:00
Blargian
8421f23975
#56257 - add failing test and new setting for parsing TSV files with crlf
2024-01-28 22:56:47 +01:00
avogar
5833641fa5
Merge branch 'master' of github.com:ClickHouse/ClickHouse into variant-data-type
2024-01-26 16:54:02 +00:00
Kruglov Pavel
46a6b84a5a
Merge branch 'master' into auto-format-detection
2024-01-25 22:11:07 +01:00
Shaun Struwig
e6844a5412
Merge branch 'ClickHouse:master' into #31363_format_template_configure_in_settings
2024-01-25 20:06:45 +01:00
Maksim Kita
2a327107b6
Updated implementation
2024-01-25 14:31:49 +03:00
avogar
11f1ea50d7
Fix tests
2024-01-24 17:55:31 +00:00
avogar
93fbe1d9c8
Fixes
2024-01-23 18:59:40 +00:00
avogar
5e4796ae16
Fix heap-use-after-free
2024-01-23 18:59:40 +00:00
avogar
eaca40c53e
Update tests
2024-01-23 18:59:40 +00:00
avogar
f05174e441
Fix style
2024-01-23 18:59:40 +00:00
avogar
617cc514b7
Try to detect file format automatically during schema inference if it's unknown
2024-01-23 18:59:39 +00:00
Blargian
7b235fe643
#31363 - remove schema delimiter setting and add test 00937_format_schema_rows_template.sh and reference
2024-01-22 22:59:59 +02:00
Blargian
eae39ff545
#31363 - modified TemplateBlockOutputFormat to work with added format_schema_rows_template setting
2024-01-21 21:51:06 +02:00
Blargian
f1749217ee
added format_schema_rows_template setting
2024-01-18 21:53:56 +02:00
Kruglov Pavel
5444cde408
Merge branch 'master' into variant-data-type
2024-01-18 18:31:27 +01:00
Kruglov Pavel
6d064512e1
Merge pull request #58614 from Blargian/58363_disable_ansi_pretty_automatically
...
58363 Automatically disable ANSI escape sequences in Pretty formats if the output is not a terminal
2024-01-17 13:45:41 +01:00
Blargian
5f500522a4
#58363 - added setting is_writing_to_terminal to FormatSettings.h, modified PrettyBlockOutputFormat to use this, which is set in FormatFactory.cpp getOutputFormat and getOutputFormatParallelIfPossible
2024-01-15 16:32:51 +02:00
Alexey Milovidov
afb50f03d9
Merge pull request #58519 from Avogar/control-arrow-dict-indexes-type
...
Add settings for better control of indexes type in Arrow dictionary
2024-01-13 20:00:40 +01:00
Blargian
ce012d217f
#58363 - fix style check failing attempt no.2
2024-01-13 13:21:36 +02:00
Blargian
0fdba3b83d
#58363 - fix failing style check
2024-01-13 12:14:54 +02:00
Alexey Milovidov
d112492c56
Remove some code
2024-01-13 03:48:04 +01:00
Blargian
72b5cf5993
#58363 - removed switch from PrettyBlockOutputFormat and modified BlockOutputFormats to use color variable. Updated english and russian documentation. Updated test 00405 reference file.
2024-01-12 19:46:03 +02:00
avogar
fbfdde60a7
Add settings for better control of indexes type in Arrow dictionary. Use signed integer type for indexes by default
2024-01-12 13:06:51 +00:00
Blargian
aa8876a611
#58363 - Changes based on review of draft PR - changed output_format_pretty_color to use UInt64Auto. Added isWritingToTerminal function to IO/WriteHelpers.h and updated test
2024-01-12 12:31:57 +02:00
Blargian
b65adbecc1
minor fixes. Doesnt seem to be using ANSI escapes anymore
2024-01-08 23:52:25 +02:00
Kruglov Pavel
b947609b8e
Merge branch 'master' into variant-data-type
2024-01-08 15:04:51 +01:00
Blargian
a15b573315
#58363 - fix formatting issues and change ON, OFF, AUTO to 0, 1, auto
2024-01-08 15:25:14 +02:00
avogar
7e5ba62017
Allow to read Bool values into String in JSON input formats
2024-01-05 20:33:30 +00:00
Blargian
459946035c
#58363 🚧 modified Pretty in FormatSettings.h to have PrettyColor which can be 0,1 or auto. modified output_format_pretty_color in FormatFactory.cpp to make use of this, added the default to Settings.h. Implemented the logic for enabling/disabling based on output_format_pretty_color in PrettyBlockOutputFormat.h
2024-01-04 16:10:36 +02:00
Kruglov Pavel
4d8cf71ba7
Merge branch 'master' into variant-data-type
2024-01-03 15:21:23 +01:00
Kruglov Pavel
c03e36e012
Merge branch 'master' into better-parsing-exceptions
2023-12-29 18:07:32 +01:00
avogar
fee2eadaf0
Fix parallel parsing for JSONCompactEachRow
2023-12-27 16:16:41 +00:00
Alexey Milovidov
ef66714bf2
Merge pull request #58196 from ClickHouse/strange-code
...
Looking at strange code
2023-12-24 03:36:41 +01:00
Alexey Milovidov
6bb181ce55
Looking at strange code
2023-12-23 13:06:34 +01:00
Alexey Milovidov
ff6419361a
Merge pull request #58181 from ClickHouse/remove-parallel-parsing-json-compact-each-row
...
Remove parallel parsing for JSONCompactEachRow
2023-12-23 10:16:40 +01:00
Alexey Milovidov
468b5e2813
Fix use-after-move
2023-12-23 08:23:15 +01:00
Alexey Milovidov
6a23fe034f
Remove parallel parsing for JSONCompactEachRow
2023-12-23 06:17:47 +01:00
avogar
319ae440b6
Implement Variant data type
2023-12-19 16:45:15 +00:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files ( #57642 )
2023-12-14 12:47:11 +01:00
Kruglov Pavel
ee39dca8c7
Merge branch 'master' into better-parsing-exceptions
2023-12-12 17:10:38 +01:00
Kruglov Pavel
6567fb2c08
Merge pull request #56859 from Avogar/csv-infer-numbers-from-strings
...
Allow to infer numbers from strings in CSV format
2023-12-12 17:09:02 +01:00
Kruglov Pavel
fd08c6f06a
Merge pull request #57751 from Avogar/better-json-inference-unnamed-tuples
...
Slightly better inference of unnamed tupes in JSON formats
2023-12-12 17:08:34 +01:00
Kruglov Pavel
8a447bf57c
Merge pull request #55892 from Avogar/schema-inference-union
...
Add 'union' mode for schema inference
2023-12-12 15:02:06 +01:00
Kruglov Pavel
9dba7aa13d
Merge pull request #57364 from Avogar/better-json-fallback
...
Better JSON -> JSONEachRow fallback without catching exceptions
2023-12-11 19:03:50 +01:00
Kruglov Pavel
20510cde34
Merge pull request #57006 from Avogar/save-errors-better
...
Fix early stop while parsing file with skipping lots of errors
2023-12-11 19:03:14 +01:00
avogar
c3a76fcc08
Allow to infer numbers from strings in CSV format
2023-12-11 18:02:05 +00:00
avogar
a87a8e91cf
Slightly better inference of unnamed tupes in JSON formats
2023-12-11 14:46:12 +00:00
avogar
ee7af95bc0
Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union
2023-12-08 20:29:28 +00:00
Kruglov Pavel
c6fecfb1af
Merge pull request #56901 from KevinyhZou/Fix_allow_cr_end_of_csv_line
...
Fix allow cr end of line for csv
2023-11-29 20:57:58 +01:00
avogar
b493ce2385
Better JSON -> JSONEachRow fallback without catching exceptions
2023-11-29 14:19:38 +00:00
János Benjamin Antal
ab935e3dd7
Use the google proto files when importing protobuf schemas
2023-11-22 12:39:41 +00:00
avogar
7e392eec50
Better exception messages in input formats
2023-11-21 13:13:42 +00:00
kevinyhzou
3adc8fdf78
Fix ci
2023-11-21 11:22:12 +08:00
avogar
ffa90628f0
Make input format errors logger a bit better
2023-11-20 17:22:49 +00:00
avogar
081fa9f3de
Address comments
2023-11-20 15:53:28 +00:00
avogar
872556a5d4
Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union
2023-11-20 14:03:36 +00:00
avogar
6366819f12
Fix generating deep nested columns in CapnProto/Protobuf schemas
2023-11-17 16:52:20 +00:00
yariks5s
181231d500
init
2023-11-07 17:56:02 +00:00
kevinyhzou
2a50daf5dd
Allow cr at end of csv line
2023-11-06 12:21:42 +08:00
kevinyhzou
ef30e6723d
bug fix csv read while end of line is not crlf
2023-11-06 12:21:42 +08:00
Kruglov Pavel
754ab9fa6c
Merge pull request #55974 from Avogar/fix-protobuf-auto-schema
...
Fix autogenerated Protobuf schema with fields with underscore
2023-11-01 18:17:09 +01:00
Kruglov Pavel
bf77ce691c
Merge pull request #55982 from yariks5s/npy_input_format
...
New input format Npy
2023-11-01 14:26:22 +01:00
yariks5s
6c4bf59021
fix suggestions and enhance tests
2023-10-31 18:10:55 +00:00
yariks5s
9a2d89e3e4
removed getSize() and enhanced docs
2023-10-30 12:42:19 +00:00
yariks5s
e14a7f066a
fix typos
2023-10-28 01:46:59 +00:00
yariks5s
894724bfb3
suggested changes
2023-10-28 01:17:25 +00:00
yariks5s
23635352f1
fixed due to review
2023-10-27 15:43:03 +00:00
Kruglov Pavel
bb4b95e891
Merge branch 'master' into schema-inference-union
2023-10-27 14:53:58 +02:00
Kruglov Pavel
570b66f027
Merge branch 'master' into schema-inference-union
2023-10-26 19:26:00 +02:00
zvonand
0766c73aab
Rename date_time_overflow_mode -> date_time_overflow_behavior, moved it to format settings
2023-10-25 23:11:13 +02:00
李扬
465962df7f
Support orc filter push down (file + stripe + rowgroup level) ( #55330 )
...
* support orc filter push down
* update orc lib version
* replace setqueryinfo with setkeycondition
* fix issue https://github.com/ClickHouse/ClickHouse/issues/53536
* refactor source with key condition
* fix building error
* remove std::cout
* update orc
* update orc version
* fix bugs
* improve code
* upgrade orc lib
* fix code style
* change as requested
* add performance tests for orc filter push down
* add performance tests for orc filter push down
* fix all bugs
* fix default as null issue
* add uts for null as default issues
* upgrade orc lib
* fix failed orc lib uts and fix typo
* fix failed uts
* fix failed uts
* fix ast fuzzer tests
* fix bug of uint64 overflow in https://s3.amazonaws.com/clickhouse-test-reports/55330/de22fdcaea2e12c96f300e95f59beba84401712d/fuzzer_astfuzzerubsan/report.html
* fix asan fatal caused by reused column vector batch in native orc input format. refer to https://s3.amazonaws.com/clickhouse-test-reports/55330/be39d23af2d7e27f5ec7f168947cf75aeaabf674/stateless_tests__asan__[4_4].htm
* fix wrong performance tests
* disable 02892_orc_filter_pushdown on aarch64. https://s3.amazonaws.com/clickhouse-test-reports/55330/be39d23af2d7e27f5ec7f168947cf75aeaabf674/stateless_tests__aarch64_.html
* add some comments
* add some comments
* inline range::equals and range::less
* fix data race of key condition
* trigger ci
2023-10-24 12:08:17 -07:00
avogar
b2c72f95b2
Fix autogenerated Protobuf schema with fields with underscore
2023-10-24 13:08:06 +00:00
taiyang-li
9c186d18a8
retrigger ci
2023-10-24 16:13:53 +08:00
taiyang-li
a02c49e16f
allow skip null values when serailize tuple to json objects
2023-10-24 11:47:46 +08:00
avogar
544b217d91
Fix style
2023-10-20 21:05:26 +00:00
Kruglov Pavel
6f61ccfe28
Merge branch 'master' into schema-inference-union
2023-10-20 22:54:11 +02:00
avogar
6934e27e8b
Add union mode for schema inference to infer union schema of files with different schemas
2023-10-20 20:46:41 +00:00
Raúl Marín
e500dc22e4
Respect default format when using http_write_exception_in_output_format
2023-10-17 14:14:58 +02:00
Michael Kolupaev
ce7eca0615
DWARF input format ( #55450 )
...
* Add ReadBufferFromFileBase::isRegularLocalFile()
* DWARF input format
* Review comments
* Changed things around ENABLE_EMBEDDED_COMPILER build setting
* Added 'ranges' column
* no-msan no-ubsan
2023-10-16 17:00:07 -07:00
yariks5s
9ae025d7e6
mid commit
2023-10-12 17:37:59 +00:00
Azat Khuzhin
2cbb069b68
Add ability to ignore data after semicolon in Values format
...
This is required for client, to handle comments in multiquery mode.
v0: separate context for input format
v2: cannot use separate context since params and stuff are changed in global context
v3: do not sent this setting to the server (breaks queries for readonly profiles)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-10-12 14:55:26 +02:00
Robert Schulze
9d04d3c3ad
Merge remote-tracking branch 'rschu1ze/master' into better-use-mysql-types-in-show-columns
2023-10-11 15:04:03 +00:00
Robert Schulze
bd43b84bf8
Make use_mysql_types_in_show_columns affect only SHOW COLUMNS
2023-10-10 23:09:49 +00:00
Kruglov Pavel
5ded0005a3
Merge pull request #55064 from AVMusorin/system-drop-format-cache
...
Allow drop cache for Protobuf format
2023-10-10 14:23:11 +02:00
Aleksandr Musorin
8d0c961af0
Allow drop cache for protobuf format
...
Before it was impossible to update Protobuf schema without server
restart. With this commit, it is enough to send query `SYSTEM DROP
SCHEMA FORMAT CACHE [FOR Protobuf]`.
2023-10-09 10:41:15 +02:00
avogar
c68e008f87
Apply suggestions
2023-09-27 11:18:39 +00:00
avogar
6b3dbc4403
Apply suggestions
2023-09-26 16:41:35 +00:00
Kruglov Pavel
bea80ab5b7
Merge branch 'master' into json-object-as-tuple-inference
2023-09-26 15:23:08 +02:00
Kruglov Pavel
69a17bbef6
Merge pull request #52853 from Avogar/http-valid-json-on-exception
...
Output valid JSON/XML on excetpion during HTTP query execution
2023-09-26 14:25:55 +02:00
Kruglov Pavel
b6863a9f52
Fix comments
2023-09-26 14:13:34 +02:00
avogar
cabb3ddaae
Fix tests
2023-09-25 21:45:11 +00:00
avogar
95d50fd7de
Fix tests
2023-09-25 18:47:33 +00:00
avogar
4d4e3db84a
Fix style and build
2023-09-25 17:49:56 +00:00
avogar
9e75825515
Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-object-as-tuple-inference
2023-09-25 17:24:36 +00:00
avogar
33a896ee6c
Small fixes
2023-09-25 15:52:17 +00:00
avogar
42ca897f2d
Better schema inference for JSON formats
2023-09-25 15:42:59 +00:00
robot-ch-test-poll2
37f732f622
Merge pull request #54808 from ClickHouse/eof
...
Prevent parquet schema inference reading the first 1 MB of the file unnecessarily
2023-09-20 09:49:42 +02:00
Michael Kolupaev
7271cfd187
Prevent parquet schema inference reading the first 1 MB of the file unnecessarily
2023-09-19 21:58:12 +00:00
