Commit Graph

1081 Commits

Author SHA1 Message Date
Kruglov Pavel
09ab5832b1
Merge pull request #44382 from Avogar/fix-bson-object-id
Fix reading ObjectId in BSON schema inference
2022-12-21 10:48:50 +01:00
Kruglov Pavel
643a35bed1
Merge pull request #44019 from Avogar/refactor-schema-inference
Refactor and improve schema inference for text formats
2022-12-20 17:29:03 +01:00
Kruglov Pavel
c0b17ca0af
Merge branch 'master' into fix-bson-object-id 2022-12-20 17:18:10 +01:00
Kruglov Pavel
3f1e40aacd
Merge branch 'master' into fix-orc 2022-12-20 13:32:46 +01:00
avogar
0c406adce2 Fix reading Map type in ORC format 2022-12-19 18:23:07 +00:00
avogar
21cdf6e6ae Fix reading ObjectId in BSON schema inference 2022-12-19 14:13:42 +00:00
avogar
4a51bdce86 Fix comments 2022-12-16 13:58:54 +00:00
Kruglov Pavel
3fad5c7f1f
Merge branch 'master' into refactor-schema-inference 2022-12-16 14:24:51 +01:00
avogar
a94a0d9c85 Fix tests, fix bugs 2022-12-14 21:17:00 +00:00
avogar
739ad23b1f Make better, fix bugs, improve error messages 2022-12-12 22:00:45 +00:00
avogar
c224e397ac Check if delimiters are empty, add comments 2022-12-08 20:00:10 +00:00
avogar
1ec5f8451b Merge branch 'master' of github.com:ClickHouse/ClickHouse into csv-custom-delimiter 2022-12-08 19:17:42 +00:00
avogar
7375a7d429 Refactor and improve schema inference for text formats 2022-12-07 21:19:27 +00:00
Kruglov Pavel
c35b2a6495
Add a limit for string size in RowBinary format (#43842) 2022-12-02 13:57:11 +01:00
Alexander Tokmakov
431f6551cb
Merge branch 'master' into fix_assertion_in_thread_status 2022-11-30 23:05:15 +03:00
Anton Popov
fe5fff0347
Merge pull request #43329 from xiedeyantu/support_nested_column
s3 table function can support select nested column using {column_name}.{subcolumn_name}
2022-11-29 22:27:19 +01:00
Alexander Tokmakov
e45105bf44 detach threads from thread group 2022-11-28 21:31:55 +01:00
Kruglov Pavel
2818ecf7f0
Merge pull request #43297 from arthurpassos/fix_arrow_list_column_parsing
Flatten list type arrow chunks on parsing
2022-11-25 18:13:27 +01:00
xiedeyantu
304b6ebf3a s3 table function can support select nested column using {column_name}.{subcolumn_name} 2022-11-23 23:36:12 +08:00
Arthur Passos
414fd07bba add docs 2022-11-17 17:28:51 -03:00
Arthur Passos
dd37ca7767 add docs 2022-11-17 17:25:27 -03:00
Arthur Passos
12d3f799a5 small change 2022-11-17 17:18:54 -03:00
Arthur Passos
fcc032a31e handle both zero based and non zero based arrow offsets 2022-11-17 17:15:24 -03:00
avogar
fcfdd73d17 Improve reading CSV field in CustomSeparated/Template format 2022-11-17 15:36:56 +00:00
Arthur Passos
ed080b8ba5 fix style 2022-11-16 13:22:23 -03:00
Arthur Passos
e1236340b5 Flatten list type arrow chunks on parsing 2022-11-16 12:27:01 -03:00
avogar
2af60f34eb Restrict document size in parallel parsing, allow to read ObjectId/JS code into String column 2022-11-15 13:35:17 +00:00
avogar
842d25c358 Minor improvements, better docs 2022-11-14 20:05:01 +00:00
avogar
098dfcff56 Fix tests 2022-11-14 15:48:23 +00:00
avogar
564d83bbc7 Better handle uint64 2022-11-11 13:24:12 +00:00
avogar
94c6dc42eb Use better types 2022-11-11 13:17:48 +00:00
avogar
cd36caf013 Fix style 2022-11-10 20:37:24 +00:00
avogar
e0b3b9efae Remove old test, clean up a bit 2022-11-10 20:21:29 +00:00
avogar
4d787f3953 Remove unneded method 2022-11-10 20:18:52 +00:00
avogar
9e89af28c6 Refactor BSONEachRow format, fix bugs, support more data types, support parallel parsing and schema inference 2022-11-10 20:15:14 +00:00
avogar
7cc87679e4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into BSONEachRow 2022-11-02 19:47:42 +00:00
Vladimir C
512abfe511 Fix style, remove commented code 2022-11-02 19:42:57 +00:00
vdimir
ef3dbf8192 clang-format BSONUtils 2022-11-02 19:42:49 +00:00
vdimir
223614ee1d Fix typos 2022-11-02 19:42:32 +00:00
vdimir
ab61932223 Apply clang-format for BSONEachRow 2022-11-02 19:42:13 +00:00
Mark Polokhov
2fff4887ac Add BSON input/output format 2022-11-02 19:39:14 +00:00
Kruglov Pavel
38124b6533
Merge pull request #42780 from Avogar/parallel-parsing
Support parallel parsing for LineAsString input format
2022-11-02 13:21:53 +01:00
Anton Popov
2ae3cfa9e0
Merge branch 'master' into dynamic-columns-14 2022-10-31 16:15:19 +01:00
avogar
fe0aea2e3a Support parallel parsing for LineAsString input format 2022-10-28 21:56:09 +00:00
Azat Khuzhin
56bc85746f Merge remote-tracking branch 'upstream/master' into build/shorten-64-to-32
Conflicts:
- src/Interpreters/ProcessList.cpp
2022-10-22 16:49:08 +02:00
Azat Khuzhin
5094c0dd6d Fix clang-tidy performance-inefficient-vector-operation
By some reason it appears only after static_cast<> was added [1]:

    /build/src/Processors/Formats/Impl/AvroRowInputFormat.cpp
    Oct 18 01:03:56 /build/src/Processors/Formats/Impl/AvroRowInputFormat.cpp:351:21: error: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation,-warnings-as-errors]
    Oct 18 01:03:56                     symbols.push_back(root_node->nameAt(i));
    Oct 18 01:03:56                     ^
    Oct 18 01:03:56 /build/src/Processors/Formats/Impl/AvroRowInputFormat.cpp:511:17: error: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation,-warnings-as-errors]
    Oct 18 01:03:56                 union_skip_fns.push_back(createSkipFn(root_node->leafAt(i)));
    Oct 18 01:03:56                 ^
    Oct 18 01:03:56 /build/src/Processors/Formats/Impl/AvroRowInputFormat.cpp:552:17: error: 'push_back' is called inside a loop; consider pre-allocating the container capacity before the loop [performance-inefficient-vector-operation,-warnings-as-errors]
    Oct 18 01:03:56                 field_skip_fns.push_back(createSkipFn(root_node->leafAt(i)));
    Oct 18 01:03:56                 ^
    Oct 18 01:03:56 197965 warnings generated.

  [1]: https://s3.amazonaws.com/clickhouse-builds/42190/453d91fa3539882dcef1d5ecd5097747499572d8/clickhouse_special_build_check/report.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:43 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Alexey Milovidov
ff26251477 Merge branch 'master' into fix-race-condition-finish-cancel 2022-10-21 04:14:21 +02:00
Alexander Tokmakov
fffecbb9ad better error message for unsupported delimiters in custom formats 2022-10-17 18:08:52 +02:00
Alexey Milovidov
f88ed8195b Fix trash 2022-10-17 04:21:08 +02:00