Azat Khuzhin
d2d49972f1
Bump CI (after non-restartable inner CI issue)
...
This reverts commit d199961e6e
2020-07-08 21:18:49 +03:00
tavplubix
33c3545125
Merge pull request #12197 from nikitamikhaylov/csv-file-bugfix
...
Headers for CSVWithNames in StorageFile
2020-07-08 20:42:49 +03:00
alexey-milovidov
d199961e6e
Rerun tests
2020-07-08 15:12:23 +03:00
Azat Khuzhin
fd5e08884a
Bump arrow to 0.17 (and flatbuffers to v1.12, required by arrow)
...
MOTIVATION:
- remove double-conversion external dependency
- remove flatc (but flatbuffers is still required, arrow just shipped
with generated files and that's it)
CHANGED:
- remove pre-generated headers, it is shipped with the arrow
- remove flatc (see above)
NOTES (see tests changes):
- and snappy error is reported as unsupported compression.
2020-07-07 23:54:32 +03:00
Nikita Mikhaylov
270551e599
done
2020-07-07 14:45:20 +03:00
Nikita Mikhaylov
53522c728b
Merge pull request #11662 from Avogar/orc_output_format
...
Add ORCBlockOutputFormat
2020-07-06 18:51:35 +04:00
Nikita Mikhaylov
dac7a39256
fix ubsan final
2020-07-06 12:29:22 +03:00
alexey-milovidov
cf1967bb29
Merge pull request #12089 from ClickHouse/fix-11595
...
Fix result_rows and result_bytes metrics for selects.
2020-07-04 23:41:51 +03:00
Nikita Mikhaylov
fba02badcb
fix test with UBSan
2020-07-03 17:53:54 +03:00
Nikolai Kochetov
6d08b7b45a
Remove LC converting to Arrow.
2020-07-03 14:04:36 +03:00
Nikolai Kochetov
3d37a63247
Remove LC converting to Arrow.
2020-07-03 13:37:55 +03:00
Nikolai Kochetov
b3f1842035
Remove LC converting to Arrow.
2020-07-03 13:35:44 +03:00
Nikolai Kochetov
e7ab4df148
Fix result_rows and result_bytes metrics for selects.
2020-07-02 18:00:37 +03:00
Nikolai Kochetov
57d727d078
Fix result_rows and result_bytes metrics for selects.
2020-07-02 17:51:10 +03:00
hcz
e284bf83d0
Optimize and fix
2020-07-01 11:21:53 +08:00
Nikita Mikhaylov
c45a054f65
fix clang-tidy
2020-06-30 16:06:35 +03:00
Nikita Mikhaylov
b74b27854a
yet another try
2020-06-30 14:38:09 +03:00
Nikita Mikhaylov
7c828861da
fix arcadia
2020-06-30 13:18:38 +03:00
Nikita Mikhaylov
b2712aa242
disable orc in arcadia at all
2020-06-29 23:48:18 +03:00
Nikita Mikhaylov
33e627777b
Merge branch 'master' of github.com:ClickHouse/ClickHouse into Avogar_orc_output_format
2020-06-29 17:25:33 +03:00
hcz
0a8a29272b
Add TabSeparatedRawRowInputFormat
2020-06-29 15:06:43 +08:00
Andrew Onyshchuk
9054862dde
Avro: allow missing fields
2020-06-27 21:23:21 -05:00
Avogar
81d459a5a8
Small update ORCBlockOutputFormat and add ORC output format to performance test.
2020-06-26 23:17:28 +03:00
alexey-milovidov
751d36f7c7
Merge pull request #11954 from oandrew/avro-uuid
...
Avro UUID support
2020-06-26 14:09:54 +03:00
Andrew Onyshchuk
9364d42eb0
Add CANNOT_PARSE_UUID extern const
2020-06-25 14:50:07 -05:00
Andrew Onyshchuk
fc1ae85600
Avro UUID support
2020-06-25 14:25:45 -05:00
Nikita Mikhaylov
717f63923a
Merge branch 'master' into orc_output_format
2020-06-23 15:07:05 +04:00
Nikita Mikhaylov
ec3d101768
style fix + bump tests
2020-06-23 13:50:56 +03:00
Alexey Milovidov
199a6576a5
Remove dependency
2020-06-21 19:37:12 +03:00
Alexey Milovidov
3f96bdaa24
Minor changes after merge
2020-06-21 15:16:08 +03:00
alexey-milovidov
4ee623ccac
Merge pull request #10242 from MovElb/movelb-postgresql-wire-protocol-impl
...
PostgreSQL wire protocol implementation
2020-06-21 14:39:22 +03:00
Avogar
2a923f434f
Fix build errors
2020-06-20 01:32:57 +03:00
Avogar
f6ab431f2f
Change String type to Binary
2020-06-19 17:50:44 +03:00
Avogar
07ad947c1d
Add arrays support
2020-06-19 17:21:35 +03:00
Mikhail Filimonov
9609bd9dee
Kafka better states, formats based on PeekableReadBuffer, and other minor fixes.
...
Add formats tests, fixes for JSONCompactEachRowWithNamesAndTypes, TSVWithNamesAndTypes. Some CR fixes
Add sanitizing for kafka_max_block_size and kafka_poll_max_batch_size
2020-06-19 10:49:05 +02:00
Avogar
71b5d267ce
Set compression to None
2020-06-18 21:02:13 +03:00
Avogar
75a66fbba3
Fix errors 2
2020-06-18 15:52:46 +03:00
Avogar
4fd3bcd823
Fix build errors
2020-06-18 03:17:08 +03:00
Avogar
7780a74bd8
Update ORCBlockOutputFormat and add orc include directory in include path
2020-06-17 22:32:11 +03:00
Alexander Tokmakov
3938b051c2
fix type duduction in values format
2020-06-17 16:21:06 +03:00
Avogar
93aee32ae4
Add ORCBlockOutputFormat
2020-06-14 18:35:32 +03:00
Alexander Kuzmenkov
435f53e05a
Improve error messages
...
Also support fmt for exception messages, the same way we use it in
logging macros.
2020-06-04 20:50:29 +03:00
Alexey Milovidov
6fea843d88
Fix Mac OS build
2020-06-02 16:56:03 +03:00
Alexey Milovidov
933644b8e4
Fix clang-tidy
2020-06-02 16:53:34 +03:00
Alexey Milovidov
cd07d61ce7
Updated tests
2020-06-02 05:39:21 +03:00
Alexey Milovidov
d34b1a8791
Merge branch 'master' into improve-performance-of-client-interactive-mode
2020-06-02 04:24:17 +03:00
Alexey Milovidov
c901a657f8
Improvement
2020-06-01 01:17:40 +03:00
Alexey Milovidov
a76bd99e84
Added setting "output_format_pretty_max_value_width"
2020-06-01 01:12:13 +03:00
Alexey Milovidov
2895cfb480
Limit value width in Pretty formats
2020-05-31 22:22:59 +03:00
Alexey Milovidov
25f941020b
Remove namespace pollution
2020-05-31 00:57:37 +03:00
Alexey Milovidov
5aff138956
Preparation for structured logging
2020-05-31 00:35:52 +03:00
MovElb
b4b5c90343
squash
2020-05-30 20:05:05 +03:00
Alexander Kuzmenkov
ac263f921e
Merge remote-tracking branch 'origin/master' into HEAD
2020-05-27 15:51:03 +03:00
Alexander Kuzmenkov
ad93644254
make single if
2020-05-27 15:50:12 +03:00
Alexander Kuzmenkov
c1cfc68cd6
revert accidental changes
2020-05-27 15:46:49 +03:00
Alexander Kuzmenkov
f54c892b49
Update src/Processors/Formats/Impl/ArrowBlockInputFormat.h
2020-05-27 15:41:59 +03:00
Alexander Kuzmenkov
61f833efc1
Update src/Processors/Formats/Impl/ArrowBlockInputFormat.h
2020-05-27 15:41:52 +03:00
Alexander Kuzmenkov
81a5af1938
Update src/Processors/Formats/Impl/ArrowBlockInputFormat.h
2020-05-27 15:41:45 +03:00
hcz
f2e3bcdd15
Improve code style in src and test files of Arrow formats
2020-05-25 10:50:55 +08:00
Alexey Milovidov
7e1813825b
Return old names of macros
2020-05-24 01:24:01 +03:00
Alexey Milovidov
9d2a0d2dd7
Apply all transformations again
2020-05-23 21:59:49 +03:00
Alexey Milovidov
a2ad11897f
Remove duplicate whitespaces (preparation)
2020-05-23 21:53:58 +03:00
Alexey Milovidov
1f13515a65
Make all LOG in single line (preparation)
2020-05-23 21:31:37 +03:00
Alexey Milovidov
e391b77d81
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+ << "[^"]+"\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+) << "([^"]+)"\);/\1_FORMATTED(\2, "\3{}\5", \4);/'
2020-05-23 19:56:05 +03:00
Alexey Milovidov
ee4ffbc332
find {base,src,programs} -name '*.h' -or -name '*.cpp' | xargs grep -l -P 'LOG_\w+\([^,]+, "[^"]+" << [^<]+\);' | xargs sed -i -r -e 's/(LOG_\w+)\(([^,]+), "([^"]+)" << ([^<]+)\);/\1_FORMATTED(\2, "\3{}", \4);/'
2020-05-23 19:47:56 +03:00
Nikolai Kochetov
f7456f8320
Merge pull request #11004 from ClickHouse/remove-experimental-use-processors-flag-2
...
Remove allow_processors flag from executeQuery()
2020-05-22 10:47:15 +03:00
hcz
ab37791134
Fix arrow stream termination
2020-05-21 15:28:10 +08:00
hcz
e11fa03bdd
Add ArrowStream input and output format
2020-05-21 12:07:47 +08:00
Azat Khuzhin
d93b9a57f6
Forward declaration for Context as much as possible.
...
Now after changing Context.h 488 modules will be recompiled instead of 582.
2020-05-21 01:53:18 +03:00
Nikolai Kochetov
bcd38ac207
Fix build.
2020-05-20 22:08:20 +03:00
Nikolai Kochetov
6f50700f35
Add PullingOutputFormat and PullingPipelineExecutor.
2020-05-20 21:53:18 +03:00
alexey-milovidov
a3346e9331
Merge pull request #10912 from ClickHouse/static-mutate
...
Static mutate
2020-05-17 09:05:35 +03:00
Nikolai Kochetov
8c2d0aa7c0
Fix comment.
2020-05-15 22:30:41 +03:00
Nikolai Kochetov
681955661f
Add PullingPipelineExecutor.
2020-05-15 00:03:38 +03:00
Nikolai Kochetov
4d0503f021
Make IColumn::mutate() static.
2020-05-14 11:30:18 +03:00
Alexey Milovidov
fd26550a70
Remove commented code before merge #10580
2020-05-10 03:41:52 +03:00
Alexey Milovidov
d9257a8682
Merge branch 'arrow-io-format' of https://github.com/FawnD2/ClickHouse into FawnD2-arrow-io-format
2020-05-10 03:26:10 +03:00
Ivan
85d783c247
Poco contrib refactoring ( #10396 )
...
* Remove config_common.h
* Refactor libcpuid contrib
* Remove support for libcpuinfo
* Define USE_CPUID in Arcadia
* Refactor Poco libraries
2020-05-08 17:11:19 +03:00
FawnD2
19022cb670
Remove redundant includes. Style fix
2020-05-05 15:56:54 +03:00
FawnD2
aaf4f4ad61
Add missing Float16 for parquet format
2020-05-05 04:46:04 +03:00
FawnD2
2f8435813c
Remove unused stream
2020-05-04 17:49:13 +03:00
FawnD2
da0a908d27
Remove unused stream
2020-05-04 17:47:09 +03:00
FawnD2
b492e92d41
Remove extra check
2020-05-04 17:32:36 +03:00
FawnD2
182de47012
Use file reader for Arrow format
2020-05-04 17:30:42 +03:00
FawnD2
6d6d04903f
Remove check
2020-05-04 16:33:20 +03:00
FawnD2
a044766d58
Remove check
2020-05-04 16:26:20 +03:00
FawnD2
17450811d4
Move getHeader at the beginning of generate()
2020-05-04 16:19:25 +03:00
FawnD2
83148289d8
Rename arg
2020-05-04 16:01:37 +03:00
FawnD2
7b23844b32
Common order of precompiled checks
2020-05-04 03:58:10 +03:00
FawnD2
02e12215e7
Apply reducing memory usage optimization for seekable files to ORC format
2020-05-04 03:52:28 +03:00
FawnD2
a554177724
Simplify ORC format
2020-05-04 02:23:20 +03:00
FawnD2
112758b99d
Merge branch 'master' into arrow-io-format
2020-05-04 00:53:17 +03:00
FawnD2
2359ba3497
Correct stop condition
2020-05-03 22:59:19 +03:00
FawnD2
6b2e97fd56
Increate total read length while reading
2020-05-03 22:54:28 +03:00
FawnD2
b4898bdd19
Read single batch
2020-05-03 22:49:06 +03:00
FawnD2
a8d914a811
Read full table
2020-05-03 22:22:41 +03:00
FawnD2
b86562b8bc
Style fixes
2020-05-03 21:29:04 +03:00
FawnD2
7cc7a87f9f
Simplify interfaces
2020-05-03 21:12:14 +03:00
FawnD2
a590826fbb
Format settings for Arrow
2020-05-03 15:26:39 +03:00
FawnD2
e5b59616cd
Style fix
2020-05-03 15:25:53 +03:00
Alexey Milovidov
469cdc315b
Added suggestion from @azat #10607
2020-05-03 14:40:37 +03:00
FawnD2
f62f1a5924
Add missed include
2020-05-03 06:08:37 +03:00
FawnD2
6e0b31fb2a
Arrow output format
2020-05-03 05:46:36 +03:00
FawnD2
aeed62d5e7
Remove unused error codes
2020-05-03 04:20:03 +03:00
FawnD2
4d0225f5a9
std::string -> String
2020-05-03 04:15:09 +03:00
FawnD2
f4fe23e3fe
Add ptr wrapper. Fix style
2020-05-03 04:01:05 +03:00
FawnD2
fa1c35753b
Move CH -> Arrow routine converter into separated class
2020-05-03 03:54:39 +03:00
Alexey Milovidov
554e7a0dd2
Preparations
2020-05-02 22:59:07 +03:00
Alexey Milovidov
e6ab4d655b
Fix bad code
2020-05-02 22:54:29 +03:00
Alexey Milovidov
f0e80fec68
Whitespaces
2020-05-02 22:41:01 +03:00
alexey-milovidov
330627f597
Update ArrowBlockInputFormat.h
2020-05-02 22:35:58 +03:00
alexey-milovidov
47e3f83bc8
Update ArrowBlockInputFormat.h
2020-05-02 22:34:34 +03:00
alexey-milovidov
07f824c746
Update ArrowBlockInputFormat.cpp
2020-05-02 22:33:47 +03:00
alexey-milovidov
dfeb76038d
Update ArrowBlockInputFormat.h
2020-05-02 22:32:33 +03:00
alexey-milovidov
fbb280b5ac
Update ArrowBlockInputFormat.cpp
2020-05-02 22:32:21 +03:00
alexey-milovidov
250a44a47f
Merge pull request #10607 from Avogar/json_as_string_format
...
Add JSONAsStringRowInputFormat
2020-05-02 22:17:22 +03:00
Matwey V. Kornilov
d7627fa4fb
Fix capnproto version check for capnp::UnalignedFlatArrayMessageReader
...
capnp::UnalignedFlatArrayMessageReader was introduced in
3aa2b2aa023f0fee61c6
2020-05-01 22:32:06 +03:00
Avogar
697d5b07ce
Fix build and style errors. Update test.
2020-05-01 19:41:14 +03:00
Avogar
6464a1a902
Add JSONAsStringRowInputFormat
2020-04-30 23:16:09 +03:00
FawnD2
538e6c39da
Arrow input format
2020-04-28 22:52:22 +03:00
Nikolai Kochetov
de6d2d01b9
Try to fix tests.
2020-04-27 21:15:55 +03:00
Nikolai Kochetov
2f06180c5e
Revert changes for CSVRowOutputFormat.
2020-04-27 18:21:53 +03:00
Nikolai Kochetov
201a184d31
Enable auto flust for output formats if enable_streaming was set.
2020-04-27 18:00:10 +03:00
Nikolai Kochetov
32baa073b5
Merge remote-tracking branch 'upstream/master' into insert_into_watch_from_live_view
2020-04-27 17:49:50 +03:00
alexey-milovidov
436e4e3f5e
Merge pull request #10502 from oandrew/avro-nested2
...
Avro nested fields - support complex types
2020-04-27 13:44:58 +03:00
Andrew Onyshchuk
570be94a48
Use correct type for DateTime64 (ubsan)
2020-04-26 21:10:30 -05:00
alexey-milovidov
c6ae6a7ba8
Merge pull request #10418 from ClickHouse/fix_logical_error_in_diagnostic_info
...
Better diagnostic info in input formats
2020-04-26 20:26:59 +03:00
alexey-milovidov
d1464a441f
Merge pull request #10503 from oandrew/read-columns-perf-fix
...
IRowInputFormat performance fix
2020-04-26 17:36:54 +03:00
Vitaliy Zakaznikov
626468d255
Adding support for streaming mode to JSONEachRowRowOutputFormat
2020-04-26 15:56:45 +02:00
Vitaliy Zakaznikov
369b4d53ef
Adding support for output_format_enable_streaming format setting.
2020-04-26 15:44:11 +02:00
Andrew Onyshchuk
4a5497498e
IRowInputFormat performance fix
2020-04-25 19:33:36 -05:00
Alexey Milovidov
d01bd44f14
Merge branch 'master' into execute-query-flush-format-2
2020-04-26 03:31:46 +03:00
alexey-milovidov
8820b43606
Merge pull request #10479 from Avogar/markdown_format
...
Markdown format
2020-04-26 03:25:48 +03:00
Andrew Onyshchuk
e7ba75ce98
Avro nested fields - support complex types
2020-04-25 17:42:49 -05:00
Azat Khuzhin
40c65f7bee
IRowInputFormat: omit "Skipped 0 rows with errors while reading the input stream"
...
This will just pollute the output log.
And this is significant for kafka streaming, since it will print this
message for each message (due to how
ReadBufferFromKafkaConsumer/KafkaBlockInputStream works).
You can see this in the server log of the test_storage_kafka
(integration test), for the table with kafka_skip_broken_messages > 0,
it will have 52 lines of such message, 2 with "Skipped 1 rows" and 50
with "Skipped 0 rows" (since there are 52 messages in summary, 2
malformed and 50 correct).
2020-04-25 15:16:29 +03:00
Alexander Tokmakov
53dcce55e9
better diagnostic info in input formats
2020-04-24 15:03:48 +03:00
Avogar
778f1aa1e7
Merge branch 'markdown_format' of github.com:avogar/ClickHouse into markdown_format
2020-04-24 14:07:04 +03:00
Avogar
2458da802c
Update MarkdownRowOutputFormat
2020-04-24 14:06:41 +03:00
Alexey Milovidov
03e17405cd
Checkpoint
2020-04-22 10:03:43 +03:00
Alexey Milovidov
603d132d45
Checkpoint
2020-04-22 09:34:20 +03:00
Alexey Milovidov
0a7edce036
Checkpoint
2020-04-22 09:01:33 +03:00
alexey-milovidov
7690c34e5b
Merge pull request #10424 from ClickHouse/tsv-bom-extra-case
...
Allow to parse BOM in TSV if the first column cannot contain BOM in its value
2020-04-22 06:56:43 +03:00
Alexander Kuzmenkov
d1c5e38a21
Merge pull request #10385 from CurtizJ/tuple-literal
...
Parse tuples of tuples as literals
2020-04-22 05:49:14 +03:00
Alexey Milovidov
e8a9c7d518
Allow to parse BOM in TSV if the first column cannot contain BOM in its value
2020-04-22 02:46:42 +03:00
Anton Popov
3ccff833c1
fix ConstantExpressionTemplate with arrays
2020-04-21 15:54:36 +03:00
alexey-milovidov
3d4696d4ef
Update AvroRowInputFormat.cpp
2020-04-20 23:34:48 +03:00
Alexey Milovidov
5262512f0c
Flush output for interactive applications
2020-04-19 22:02:36 +03:00
Andrew Onyshchuk
68f2e93443
Fix style
2020-04-18 22:54:22 -05:00
Andrew Onyshchuk
14647ed03b
Add Avro nested fields insert support
2020-04-18 15:46:24 -05:00
alexey-milovidov
d1b72f10d5
Merge pull request #10325 from Avogar/msgpack_format
...
Improve MsgPackRowInputFormat
2020-04-18 14:27:11 +03:00
