thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
88,166 Commits 9,326 Branches 1,783 Tags 2.1 GiB
04fdd75c56
Commit Graph

2650 Commits

Author SHA1 Message Date
zvonand
2454f1dd13 Decimal components attempt 2022-02-14 00:04:09 +03:00
taiyang-li
e53719a86b remove comments 2022-02-13 17:13:23 +08:00
taiyang-li
aabf2aac69 finish all tests 2022-02-13 17:06:58 +08:00
taiyang-li
6559941972 support datetime64 when transform ch chunk to arrow table 2022-02-13 14:56:01 +08:00
alexey-milovidov
4a2c69c073
Merge pull request #34067 from Algunenano/mv_fixes 
Fix  `parallel_view_processing=0` and `view_duration_ms` in views log
 2022-02-12 22:36:41 +03:00
Anton Popov
357bdd69c4 fix aggregation in order with distributed_aggregation_memory_efficient=0 2022-02-11 18:09:13 +03:00
Vladimir C
a2b1900333
Merge pull request #34496 from Avogar/jsonl 
Support .jsonl extension for JSONEachRow format
 2022-02-11 15:44:31 +01:00
W
7cb0433fae
Update buildPushingToViewsChain.h 
typo
 2022-02-11 14:34:20 +08:00
avogar
9e58ae7577 Support jsonl extension for JSONEachRow format 2022-02-10 16:00:37 +03:00
Kruglov Pavel
a4f5610764
Merge pull request #34476 from CurtizJ/avoid-settings-copy 
Avoid unnecessary copying of `Settings`
 2022-02-10 14:13:46 +03:00
Anton Popov
298838f891 avoid unnecessary copying of Settings 2022-02-10 12:13:51 +03:00
Anton Popov
18940b8637 Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-09 23:38:38 +03:00
mergify[bot]
d78525bd10
Merge branch 'master' into fix-removing-order-in-CreatingSetsTransform 2022-02-09 13:55:52 +00:00
Azat Khuzhin
4fa2ae76bc Fix memory leak in AggregatingInOrderTransform 
Reproducer:

    # NOTE: we need clickhouse from 33957 since right now LSan is broken due to getauxval().
    $ url=https://s3.amazonaws.com/clickhouse-builds/33957/e04b862673644d313712607a0078f5d1c48b5377/package_asan/clickhouse
    $ wget $url -o clickhouse-asan
    $ chmod +x clickhouse-asan
    $ ./clickhouse-asan server &

    $ ./clickhouse-asan client
    :) create table data (key Int, value String) engine=MergeTree() order by key
    :) insert into data select number%5, toString(number) from numbers(10e6)

    # usually it is enough one query, benchmark is just for stability of the results
    # note, that if the exception was not happen from AggregatingInOrderTransform then add --continue_on_errors and wait
    $ ./clickhouse-asan benchmark --query 'select key, uniqCombined64(value), groupArray(value) from data group by key' --optimize_aggregation_in_order=1 --memory_tracker_fault_probability=0.01, max_untracked_memory='2Mi'

LSan report:

    ==24595==ERROR: LeakSanitizer: detected memory leaks

    Direct leak of 3932160 byte(s) in 6 object(s) allocated from:
        0 0xcadba93 in realloc ()
        1 0xcc108d9 in Allocator<false, false>::realloc() obj-x86_64-linux-gnu/../src/Common/Allocator.h:134:30
        2 0xde19eae in void DB::PODArrayBase<>::realloc<DB::Arena*&>(unsigned long, DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h:161:25
        3 0xde5f039 in void DB::PODArrayBase<>::reserveForNextSize<DB::Arena*&>(DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h
        4 0xde5f039 in void DB::PODArray<>::push_back<>(DB::GroupArrayNodeString*&, DB::Arena*&) obj-x86_64-linux-gnu/../src/Common/PODArray.h:432:19
        5 0xde5f039 in DB::GroupArrayGeneralImpl<>::add() const obj-x86_64-linux-gnu/../src/AggregateFunctions/AggregateFunctionGroupArray.h:465:31
        6 0xde5f039 in DB::IAggregateFunctionHelper<>::addBatchSinglePlaceFromInterval() const obj-x86_64-linux-gnu/../src/AggregateFunctions/IAggregateFunction.h:481:53
        7 0x299df134 in DB::Aggregator::executeOnIntervalWithoutKeyImpl() obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:869:31
        8 0x2ca75f7d in DB::AggregatingInOrderTransform::consume() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingInOrderTransform.cpp:124:13

    ...

    SUMMARY: AddressSanitizer: 4523184 byte(s) leaked in 12 allocation(s).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-09 09:23:56 +03:00
Maksim Kita
b8e350054c clang-tidy move fix build 2022-02-08 21:21:32 +00:00
Nikolai Kochetov
82a7d70a31
Merge branch 'master' into fix-removing-order-in-CreatingSetsTransform 2022-02-08 19:29:03 +03:00
Nikolai Kochetov
d2d47b9595 Fixing build. 2022-02-08 16:27:33 +00:00
Maksim Kita
4bb69bcb15
Merge pull request #34398 from DevTeamBK/input_format 
Method called on already moved
 2022-02-08 15:20:07 +01:00
Nikolai Kochetov
7e54dafdc1 Fix wrong destruction order in CreatingSetsTransform. 2022-02-08 10:41:07 +00:00
Kruglov Pavel
b4fec2af7c
Merge pull request #34065 from Avogar/msgpack 
Support UUID in MsgPack format
 2022-02-08 11:42:17 +03:00
Rajkumar
6b3adbb0de Method called on already moved 2022-02-07 19:50:34 -08:00
avogar
a4c7ecde87 Make better 2022-02-07 17:51:26 +03:00
avogar
c3d30fd502 Fix comments 2022-02-07 17:11:44 +03:00
Kruglov Pavel
34a17075d3 FIx error messages 2022-02-07 17:11:44 +03:00
avogar
77b42bb9ff Support UUID in MsgPack format 2022-02-07 17:11:44 +03:00
HeenaBansal2009
eeec2478ba Fix clang-tidy issue 2022-02-06 22:36:35 -08:00
Alexey Milovidov
f98010e374 Small improvements 2022-02-06 07:14:01 +03:00
Alexey Milovidov
4a83dbc514 Fix linkage 2022-02-04 00:26:44 +03:00
Alexey Milovidov
c426f11096 Maybe better 2022-02-04 00:20:16 +03:00
Alexey Milovidov
7c12f5f37a Fix terribly low performance of LineAsString format 2022-02-04 00:07:31 +03:00
Anton Popov
9b844c6b42
Merge pull request #32748 from CurtizJ/read-in-order-fixed-prefix 
Support `optimize_read_in_order` if prefix of sorting key is already sorted
 2022-02-03 18:17:08 +03:00
mergify[bot]
150d7ba8b5
Merge branch 'master' into mv_fixes 2022-02-03 00:41:52 +00:00
Azat Khuzhin
1d19851590 Disable data skipping indexes by default for queries with FINAL 
This patch adds use_skip_indexes_if_final setting, that is OFF by
default. Since skipping data for queries with FINAL may produce
incorrect result.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-02-02 13:31:50 +03:00
Anton Popov
836a348a9c Merge remote-tracking branch 'upstream/master' into HEAD 2022-02-01 15:23:07 +03:00
mergify[bot]
4f0e011958
Merge branch 'master' into mv_fixes 2022-01-31 23:53:35 +00:00
Alexey Milovidov
e4e7169277 Remove some strange code 2022-02-01 02:52:36 +03:00
Alexey Milovidov
83136f3515 Allow \r in the middle of the line in format Regexp 2022-02-01 02:49:26 +03:00
Alexey Milovidov
872d0a0fbe Improve performance of format Regexp 2022-02-01 02:07:48 +03:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines 
Fix benign race condition for storage HDFS, S3, URL
 2022-01-31 22:41:54 +03:00
alesapin
93c0700c4c Fix typo 2022-01-31 16:46:58 +03:00
alesapin
056b9e335f Fix comment 2022-01-31 16:39:42 +03:00
alesapin
31753afb7e Fix cancel logic in parallel parsing 2022-01-31 16:38:15 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Anton Popov
78b9f15abb Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-30 03:24:37 +03:00
Anton Popov
b950a12cb3
Merge pull request #34068 from CurtizJ/fix-async-insert-native 
Fix asynchronous inserts with `Native` format
 2022-01-29 01:24:53 +03:00
Azat Khuzhin
1519985c98 Fix possible "Can't attach query to the thread, it is already attached" 
After detachQueryIfNotDetached() had been removed it is not enough to
use attachTo() for ThreadPool (scheduleOrThrowOnError()) since the query
may be already attached, if the thread doing multiple jobs, so
CurrentThread::attachToIfDetached() should be used instead.

This should fix all the places from the failures on CI [1]:

    $ fgrep DB::CurrentThread::attachTo -A1 ~/Downloads/47.txt  | fgrep -v attachTo | cut -d' ' -f5,6 | sort | uniq -c
         92 --
          2 /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/deque:1393: DB::ParallelParsingInputFormat::parserThreadFunction(std::__1::shared_ptr<DB::ThreadGroupStatus>,
          4 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeData.cpp:1595: void
         87 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:993: void

  [1]: https://github.com/ClickHouse/ClickHouse/runs/4954466034?check_suite_focus=true

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Azat Khuzhin
b0c862c297 Fix memory accounting for queries that uses < max_untracker_memory 
MemoryTracker starts accounting memory directly only after per-thread
allocation exceeded max_untracker_memory (or memory_profiler_step).

But even memory under this limit should be accounted too, and there is
code to do this in ThreadStatus dtor, however due to
PullingAsyncPipelineExecutor detached the query from thread group that
memory was not accounted.

So remove CurrentThread::detachQueryIfNotDetached() from threads that
uses ThreadFromGlobalPool since it has ThreadStatus, and the query will
be detached using CurrentThread::defaultThreadDeleter.

Note, that before this patch memory accounting works for HTTP queries
due to it had been accounted from ParallelFormattingOutputFormat, but
not for TCP.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-28 16:25:33 +03:00
Anton Popov
6c0959b907 fix asynchronous inserts with Native format 2022-01-28 03:25:15 +03:00
Raúl Marín
5f7ac9fbb1 MV: Limit the chain threads based on parallel_view_processing 2022-01-27 18:13:30 +01:00
Raúl Marín
3df56f46b3 Fix MV view_duration_ms 2022-01-27 18:09:47 +01:00
tavplubix
9e91a9dfd1
Merge pull request #33429 from zzsmdfj/MaterializeMySQL_support_set_and_other_dataType 
MaterializeMySQL support SET YEAR TIME GEOMETRY dataType
 2022-01-26 11:29:46 +03:00
Kruglov Pavel
9f12f4af13
Merge pull request #33302 from Avogar/formats-with-suffixes 
Allow to create new files on insert for File/S3/HDFS engines
 2022-01-25 10:56:15 +03:00
avogar
1f49acc164 Better naming 2022-01-24 16:28:36 +03:00
Anton Popov
6861adadcf
Merge pull request #33637 from azat/fix-optimize_aggregation_in_order 
RFC: Fix "Chunk should have AggregatedChunkInfo in GroupingAggregatedTransform"
 2022-01-22 19:05:39 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
zzsmdfj
411f43ec4d to MaterializeMySQL_support_set_and_other_dataType 2022-01-21 12:24:12 +08:00
avogar
67e396f8f4 Fix schema inference for JSONEachRow and JSONCompactEachRow 2022-01-20 16:31:24 +03:00
mergify[bot]
cbb87e762c
Merge branch 'master' into formats-with-suffixes 2022-01-19 17:21:18 +00:00
Amos Bird
f65d77cc74
Hotfix of missing header 2022-01-19 14:11:42 +08:00
mergify[bot]
b318f9b5db
Merge branch 'master' into formats-with-suffixes 2022-01-18 12:17:07 +00:00
Anton Popov
a25f2518e3
Merge pull request #33141 from 1over/feature_default_keyword 
Add support of DEFAULT keyword for INSERT
 2022-01-18 02:04:37 +03:00
Kruglov Pavel
a7df9cd53a
Merge branch 'master' into formats-with-suffixes 2022-01-14 21:03:49 +03:00
avogar
253035a5df Fix 2022-01-14 19:17:06 +03:00
Kruglov Pavel
d2e9f37bee
Merge branch 'master' into format-by-extention 2022-01-14 18:36:23 +03:00
avogar
89a181bd19 Make better 2022-01-14 18:16:18 +03:00
Azat Khuzhin
a4c2f23b07 Fix "Chunk should have AggregatedChunkInfo in GroupingAggregatedTransform" 
In case of optimize_aggregation_in_order there will be
ChunkInfoWithAllocatedBytes.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 17:44:09 +03:00
Azat Khuzhin
06402386eb Do not generate empty chunks in AggregatingInOrderTransform 
This is just a micro optimization and it should not affect anything,
real fixes are in separate patches (previous and next).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 17:44:09 +03:00
Azat Khuzhin
d9a64d1f86 Fix optimize_aggregation_in_order in case of empty result set 
Note, that this is not complete fix, see the next two patches.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-01-14 17:43:52 +03:00
Kruglov Pavel
5a908e8edd
Merge branch 'master' into formats-with-suffixes 2022-01-14 16:45:20 +03:00
Kruglov Pavel
d54a430d9c
Merge pull request #33566 from Avogar/fix-avro 
Fix segfault in Avro
 2022-01-14 16:01:56 +03:00
Kseniia Sumarokova
5da673c3a5
Merge pull request #31104 from bigo-sg/hive_table 
Implement hive table engine
 2022-01-14 09:39:17 +03:00
Kruglov Pavel
305d58a762
Merge pull request #33524 from Avogar/stacktrace-in-client 
Don't print exception twice in client in case of exception in parallel parsing
 2022-01-13 15:50:42 +03:00
taiyang-li
82a3024712 Merge remote-tracking branch 'ck/master' into bigo_hive_table 2022-01-13 18:11:23 +08:00
Nikolai Kochetov
872ee5dc09
Update src/Processors/Formats/Impl/AvroRowOutputFormat.h 
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
 2022-01-13 12:55:14 +03:00
tavplubix
bacdaa7b4d
Merge pull request #33564 from ClickHouse/fix_minor_bug_in_ttl 
Fix minor bug in TTL and flaky test
 2022-01-13 11:51:57 +03:00
avogar
c5ea4b1bc0 Fix segfault in Avro 2022-01-12 18:34:28 +03:00
avogar
8390e9ad60 Detect format by file name in file/hdfs/s3/url table functions 2022-01-12 18:29:31 +03:00
Alexander Tokmakov
72b67f37fc fix log message and flaky test 2022-01-12 18:20:22 +03:00
lgbo-ustc
5c71d3687a fixed some bugs 
1. interagtion test for test_hive_query failed
2. nullptr reference in arrowSchemaToCHHeader
 2022-01-12 17:01:05 +08:00
taiyang-li
66813a3aa9 merge master 2022-01-12 16:56:29 +08:00
avogar
9915ce7ded Fix segfault in arrowSchemaToCHHeader 2022-01-11 20:30:35 +03:00
avogar
0ae0aa712b Don't print exception twice in client in case of exception in parallel parsing 2022-01-11 18:37:07 +03:00
msaf1980
780a1b2abe graphite: split tagged/plain rollup rules (for merges perfomance) 2022-01-10 16:34:16 +05:00
Alexey Milovidov
a75f1ccb21 Fix build 2022-01-09 12:26:36 +03:00
Azat Khuzhin
c05dac89ee ReadFromRemote: Remove redundant ParallelReplicasReadingCoordinator 2022-01-07 20:04:57 +03:00
李扬
2df2442ad0
Merge branch 'master' into hive_table 2022-01-04 01:26:16 -06:00
taiyang-li
8730dda895 fix hivte text 2022-01-01 09:16:30 +08:00
taiyang-li
1e102bc1b2 merge master 2022-01-01 09:01:06 +08:00
alexey-milovidov
34b934a1e0
Merge pull request #33331 from ClickHouse/serxa/line-as-string-output-format 
Add LineAsString output format
 2021-12-31 14:38:36 +03:00
Sergei Trifonov
f1d398ae4b Add LineAsString output format 2021-12-30 20:38:54 +03:00
alesapin
16c36d72b1
Merge pull request #33296 from ClickHouse/fix_clang_tidy_3 
Fix clang tidy 3
 2021-12-29 22:43:42 +03:00
avogar
97788b9c21 Allow to create new files on insert for File/S3/HDFS engines 2021-12-29 21:19:13 +03:00
Kruglov Pavel
489a30859f
Merge pull request #32455 from Avogar/schema-inference 
Automatic schema inference for input formats
 2021-12-29 21:03:48 +03:00
alesapin
67fc9d7129 Fix clang-tidy 2021-12-29 18:35:54 +03:00
Nikolai Kochetov
c715204e18
Merge pull request #32271 from Algunenano/kill_scalar_github 
Be able to KILL scalar queries
 2021-12-29 16:00:59 +03:00
Kruglov Pavel
dd7f61b337
Merge branch 'master' into schema-inference 2021-12-29 12:59:05 +03:00
avogar
364b4f5d36 Fix special build 2021-12-29 12:21:01 +03:00
Kruglov Pavel
cb0ed7fcb7 Fix typo 2021-12-29 12:21:01 +03:00
avogar
26abf7aa62 Remove code duplication, use simdjson and rapidjson instead of Poco 2021-12-29 12:21:01 +03:00
avogar
74f09d6476 Fix tests 2021-12-29 12:18:56 +03:00
avogar
aaf9f85c67 Add more tests and fixes 2021-12-29 12:18:56 +03:00
avogar
dd994aa761 Add some tests and some fixes 2021-12-29 12:18:56 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
Maksim Kita
b5698ea134 Updated test_executable_table_function tests 2021-12-28 22:55:31 +03:00
Maksim Kita
d1db3c9f42 Updated ShellCommandSource 2021-12-28 22:55:31 +03:00
Maksim Kita
3386378050 Updated test_executable_table_function tests 2021-12-28 22:55:31 +03:00
Maksim Kita
61b36f22b8 Fixed style check 2021-12-28 22:55:31 +03:00
Maksim Kita
fbe65658a1 Fixed tests 2021-12-28 22:55:31 +03:00
Maksim Kita
c2977dbf58 Updated test_executable_table_function integration tests 2021-12-28 22:55:30 +03:00
Maksim Kita
de49a25185 Fixed build 2021-12-28 22:55:30 +03:00
Maksim Kita
cda8bd7f44 Added test_executable_dictionary source tests 2021-12-28 22:55:30 +03:00
Maksim Kita
5590cfa3aa Updated executable function integration tests 2021-12-28 22:55:30 +03:00
Maksim Kita
d3e507b9ef Added executable function integrations tests 2021-12-28 22:55:30 +03:00
Maksim Kita
2ffd83ee83 Fix shell command source start 2021-12-28 22:55:30 +03:00
Maksim Kita
eb4e400c54 Executable refactoring 2021-12-28 22:55:30 +03:00
alexey-milovidov
934c1b3e83
Merge pull request #33238 from kssenii/some-killing 
Allow to cancel some sources which failed to cancel
 2021-12-28 22:00:44 +03:00
taiyang-li
aa97634d9b merge master 2021-12-28 18:38:50 +08:00
kssenii
1f6ca619b7 Allow some killing 2021-12-27 22:42:56 +03:00
kssenii
1bd4936961 Add retry for postgres query 2021-12-27 11:03:04 +03:00
taiyang-li
9036b18c2f merge master 2021-12-27 15:12:48 +08:00
alexey-milovidov
0a55fa3dc2
Revert "Grouping sets dev" 2021-12-25 20:30:31 +03:00
alexey-milovidov
c583ea7e6b
Merge pull request #32484 from Algunenano/libcxx13_take2 
libc++ 13 compatibility
 2021-12-25 10:14:12 +03:00
Andrii Buriachevskyi
e8cc6df7bb Add support of DEFAULT keyword for INSERT 2021-12-24 13:10:19 +01:00
Alexey Milovidov
29d28c531f Move code around to avoid dlsym on Musl 2021-12-24 12:25:27 +03:00
Anton Popov
1f67177d4d fix optimize_read_in_order with preliminary merge and add some tests 2021-12-23 18:32:31 +03:00
Raúl Marín
cb22091b33 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-23 13:59:33 +01:00
Raúl Marín
77db850c0b Merge remote-tracking branch 'blessed/master' into libcxx13_take2 2021-12-23 12:42:39 +01:00
Nikolai Kochetov
b84591d17a
Merge pull request #32862 from ClickHouse/fix-mv-result-with-many-chunks 
Fix MV query with multiple chunk result.
 2021-12-23 14:37:17 +03:00
Raúl Marín
88b8fd8b60 Merge remote-tracking branch 'blessed/master' into libcxx13_take2 2021-12-23 09:16:19 +01:00
Alexey Milovidov
f37ff32c37 Whitespaces 2021-12-23 01:33:47 +03:00
mreddy017
3e50217501 Remove the additional white space as per the pipeline build error. 2021-12-23 01:30:56 +03:00
mreddy017
10eb2dbdb7 Addressing review comments 2021-12-23 01:30:56 +03:00
Harry-Lee
846c46ac4b Fix issue #80: union index out of boundary 2021-12-23 01:30:56 +03:00
Raúl Marín
dbb13036bc Hook progress callbacks with subqueries too 2021-12-22 23:13:45 +01:00
Raúl Marín
1d9916eb25 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-22 16:42:10 +01:00
tavplubix
d8fb293d57
Merge pull request #32900 from zzsmdfj/issue/#15182_MaterializeMySQL_support_bit_type 
Add support for BIT data type in MaterializedMySQL
 2021-12-22 15:58:29 +03:00
zzsmdfj
039afc5cfe #15182_MaterializeMySQL_support_bit_type-improve bit2Uint64 function 2021-12-22 16:17:18 +08:00
Raúl Marín
eddccb0065 Merge remote-tracking branch 'blessed/master' into kill_scalar_github 2021-12-21 10:00:17 +01:00
taiyang-li
2597925724 merge master 2021-12-21 15:55:39 +08:00
Maksim Kita
dd0d3de050
Merge pull request #32970 from kitaisreal/loops-remove-postfix-increment 
Loops remove postfix increment
 2021-12-20 19:51:07 +03:00
Raúl Marín
b553e51969 Merge remote-tracking branch 'blessed/master' into libcxx13_take2 2021-12-20 17:47:57 +01:00
Maksim Kita
51477adf1b Updated additional cases 2021-12-20 15:55:07 +03:00
kreuzerkrieg
f06c37d206 Stop reading incomplete stripes and skip rows. 2021-12-19 18:41:32 +02:00
alexey-milovidov
6b97af4c63
Merge pull request #26869 from taylor12805/grouping-sets-dev 
Grouping sets dev
 2021-12-17 20:50:15 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
alesapin
6bd7e425c6
Merge pull request #22535 from CurtizJ/sparse-serialization 
Sparse serialization and ColumnSparse
 2021-12-17 15:26:17 +03:00
mergify[bot]
785d16d733
Merge branch 'master' into kill_scalar_github 2021-12-17 10:21:54 +00:00
taiyang-li
d033fc4c24 merge master and fix conflict 2021-12-17 15:11:21 +08:00
zzsmdfj
e9eed1f927 to #15182_MaterializeMySQL_support_bit_type 2021-12-17 15:03:24 +08:00
Nikolai Kochetov
96384fe4f3
Update WindowTransform.cpp 2021-12-16 21:20:39 +03:00
Nikolai Kochetov
82f94a35f9
Update WindowTransform.cpp 2021-12-16 21:20:01 +03:00
Nikolai Kochetov
e28bcbfcea Fix MV query with multiple chunk result. 2021-12-16 21:17:05 +03:00
Dmitrii Mokhnatkin
2147658432
Proper handler for apache arrow column duplication 2021-12-15 18:30:32 +03:00
Raúl Marín
3de002c7c9 Merge remote-tracking branch 'blessed/master' into libcxx13_take2 2021-12-15 12:57:57 +01:00
mergify[bot]
bff0424139
Merge branch 'master' into kill_scalar_github 2021-12-15 09:22:58 +00:00
Dmitry Novik
56a3f4a000 Cleanup code 2021-12-14 22:15:14 +03:00
alesapin
d7663b2179 Merge branch 'master' into fix_special_build_check 2021-12-14 19:08:28 +03:00
alesapin
884801e1bd Fixing 2021-12-14 19:08:08 +03:00
Anton Popov
16312e7e4a Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-14 18:58:17 +03:00
Raúl Marín
44f3b1c9d2 Merge remote-tracking branch 'blessed/master' into libcxx13_take2 2021-12-14 13:05:01 +01:00
Anton Popov
bda0cc2f76
Merge pull request #32530 from Avogar/fix-async-inserts 
Fix async inserts for some input formats
 2021-12-14 14:07:05 +03:00
taiyang-li
ca3f7425a4 fix code 2021-12-14 17:37:31 +08:00
taiyang-li
8234d1176f merge master 2021-12-14 10:39:21 +08:00
alexey-milovidov
fd9d40925a
Merge pull request #32640 from ClickHouse/fix-function-file 
Fix surprisingly bad code in function `file`
 2021-12-13 19:34:36 +03:00
Raúl Marín
61d959df8f Fix arrow build 2021-12-13 16:49:22 +01:00
Raúl Marín
d9e4544239 Adapt to arrow 6 2021-12-13 16:49:21 +01:00
Kruglov Pavel
52328f6abc
Merge pull request #32490 from ClickHouse/quote-more-identifiers 
Fix issue with ambiguous query formatting in distributed queries.
 2021-12-13 16:41:01 +03:00
mergify[bot]
fa8f9671f4
Merge branch 'master' into fix-async-inserts 2021-12-13 11:13:20 +00:00
Alexey Milovidov
71926a3a97 Fix surprisingly bad code in function "file" 2021-12-13 07:57:54 +03:00
李扬
8675086104
Merge branch 'master' into hive_table 2021-12-12 09:01:46 -06:00
alexey-milovidov
71df622b1f
Merge pull request #31888 from ClickHouse/fix-31114 
Remove LowCardinality from window function args.
 2021-12-12 06:37:55 +03:00
Alexey Milovidov
58ffe89643 Merge branch 'master' into quote-more-identifiers 2021-12-12 03:33:20 +03:00
taiyang-li
5ef68fc479 fix building 2021-12-11 15:50:59 +08:00
Kruglov Pavel
764e205d36
Fix resetParser in MsgPack format 2021-12-10 21:37:08 +03:00
avogar
1be84d80d4 Fix async inserts for some input formats 2021-12-10 20:54:08 +03:00
Alexey Milovidov
5c90ed2ed9 Unambiguous formatting of distributed queries 2021-12-10 00:55:14 +03:00
Raúl Marín
3a7e524f9e Style 2021-12-09 18:24:28 +01:00
Raúl Marín
3fc4167c54 Rework how progress is reported in views 2021-12-09 17:08:29 +01:00
kssenii
f5a77fca39 Done 2021-12-09 14:44:25 +00:00
Nikolai Kochetov
f4f06ca368 Fix assert. 2021-12-09 16:59:21 +03:00
Nikita Mikhaylov
dbf5091016
Parallel reading from replicas (#29279) 2021-12-09 13:39:28 +03:00
Raúl Marín
0158133b62 SourceWithProgress: Don't use process_list_elem without checking it 2021-12-09 10:02:32 +01:00
Raúl Marín
edf4095139 Add QueryStatus to BuildQueryPipelineSettings and use it 2021-12-09 10:02:32 +01:00
Raúl Marín
9fe648ef8e Force PipeLineExecutor creators to pass a QueryStatus 
This is to avoid future issues
 2021-12-09 10:02:32 +01:00
Anton Popov
61a5f8a61a add comments 2021-12-08 18:56:30 +03:00
Anton Popov
d8367334a3 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-08 18:26:19 +03:00
Mikhail f. Shiryaev
03927f5fc6
Revert "graphite: split tagged/plain rollup rules (for merges perfomance)" 
This reverts commit 303552f515.
 2021-12-08 15:49:21 +01:00
Nikolai Kochetov
dd906b2a3e Remove comments. 2021-12-08 14:25:00 +03:00
Nikolai Kochetov
07b2d69ea2 Another try. 2021-12-08 14:22:43 +03:00
Kruglov Pavel
3498e13551
Merge pull request #32270 from ClickHouse/fix-strict-resize 
Fix possible Pipeline stuck in case of StrictResize processor.
 2021-12-08 11:03:53 +03:00
Nikolai Kochetov
2c6cb902c4 Revert "Better fix." 
This reverts commit f07fc08a49.
 2021-12-07 20:54:49 +03:00
Dmitry Novik
4afaa8697c remove ya.make 2021-12-07 16:47:42 +03:00
Mikhail f. Shiryaev
5fc20b3e6a
Merge pull request #25122 from msaf1980/rollup_rules_type 
GraphiteMergeTree Rollup rules type
 2021-12-07 10:18:08 +01:00
Nikolai Kochetov
f07fc08a49 Better fix. 2021-12-06 22:08:32 +03:00
Nikolai Kochetov
ce568bff90 Merge branch 'master' into fix-31114 2021-12-06 19:47:13 +03:00
msaf1980
303552f515
graphite: split tagged/plain rollup rules (for merges perfomance) 2021-12-06 16:49:08 +01:00
Nikolai Kochetov
1d75d8fd45 Fix assert. 2021-12-06 15:52:08 +03:00
Nikolai Kochetov
32b4875273 Fix possible Pipeline stuck in case of StrictResize processor. 2021-12-06 14:53:39 +03:00
Kruglov Pavel
cc71c537bc
Merge pull request #32204 from Avogar/skip-quoted-values 
Improve skiping unknown fields with Quoted escaping rule in Template/CustomSeparated formats
 2021-12-06 12:28:14 +03:00
Nikolai Kochetov
91c4c89c66
Merge pull request #31636 from Algunenano/pull_timeout 
Check max_execution_time in the pipeline and pulling executors
 2021-12-06 11:33:39 +03:00
Dmitriy Dorofeev
31648d95e2
use application/x-ndjson for streaming JSON (#32223) 2021-12-06 10:49:14 +03:00
taiyang-li
c678c8101e fix some bugs 2021-12-04 16:41:35 +08:00
Maksim Kita
0372403ebd
Merge pull request #32112 from FrankChen021/content-type 
Returns Content-Type as application/json for JSONEachRow if output_format_json_array_of_rows is enabled
 2021-12-03 18:59:54 +03:00
avogar
7549619b25 Improve skiping unknown fields with Quoted escaping rule in Template/CustomSeparated formats 2021-12-03 16:25:35 +03:00
Maksim Kita
6ec559f103
Update JSONEachRowRowOutputFormat.h 2021-12-03 12:48:28 +03:00
Raúl Marín
755ba5dc09 Don't forget to check the output of checkTimeLimitSoft 2021-12-02 15:09:53 +01:00
Raúl Marín
5662d0aa59 Use softer checks 2021-12-02 14:53:55 +01:00
taiyang-li
01cac01527 modify permission of RowInputFormatWithNamesAndTypes methods 2021-12-02 20:54:49 +08:00
taiyang-li
e2d1ed1568 fix error 2021-12-02 20:51:19 +08:00
taiyang-li
2f4e7e1d4e merge master 2021-12-02 19:48:21 +08:00
taiyang-li
9ec8272186 refactor hive text input format 2021-12-02 16:14:25 +08:00
kssenii
8edf146687 Merge branch 'master' of github.com:ClickHouse/ClickHouse into versioning 2021-12-02 07:57:14 +00:00
mergify[bot]
e568b16e02
Merge branch 'master' into content-type 2021-12-02 07:40:17 +00:00
frank chen
c49a7251ed returns content-type as json if possible 
Signed-off-by: frank chen <frank.chen021@outlook.com>
 2021-12-02 13:25:17 +08:00
Anton Popov
6f4d9a53b2 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-12-01 15:54:33 +03:00
Anton Popov
54f51444c0 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-01 15:49:02 +03:00
tavplubix
b623a387af
Merge pull request #31887 from ClickHouse/fix_cannot_create_empty_part 
Parse partition key value from `partition_id` when need to create part in empty partition
 2021-12-01 15:38:46 +03:00
taiyang-li
4aeadf3967 fix build error 2021-12-01 14:13:48 +08:00
Nikita Mikhaylov
6c366feed7
Fix race in ParallelFormattingOutputFormat constructor (#32004) 2021-12-01 02:10:33 +03:00
Kseniia Sumarokova
3c348a2998
Merge pull request #8331 from Vxider/window-view 
Support for Stream Processing
 2021-11-30 18:20:12 +03:00
taiyang-li
d213500a3e remove blank at end of line 2021-11-30 18:23:24 +08:00
taiyang-li
c6abe60bcc add new input format HiveTextRowInputFormat 2021-11-30 18:06:26 +08:00
kssenii
f658a8c181 Small simplification, some comments 2021-11-30 11:16:24 +03:00
taiyang-li
440fa9b69c implement getMissingValues for ORC/Parquet/Arrow 2021-11-30 15:44:59 +08:00
taiyang-li
cacf516e3e calculate column value by default expression & apply defaults_for_omitted_fields_ in ArrowColumnToCHColumn 2021-11-30 14:52:26 +08:00
taiyang-li
ad6ba24efd fix ArrowColumnToCHColumn 2021-11-30 10:49:57 +08:00
Raúl Marín
a7ae715950 Merge remote-tracking branch 'blessed/master' into pull_timeout 2021-11-29 19:51:18 +01:00
fanzhou
43db4594ba some changes 2021-11-29 19:35:33 +03:00
fanzhou
07a986fbf1 fix bug when no data in grouping sets transformer 2021-11-29 18:59:15 +03:00
fanzhou
534160b0d4 grouing sets transformer instead of fork processor in aggregating transformer 2021-11-29 18:59:15 +03:00
taiyang-li
6922f09ea3 reuse seekable read buffer with size 2021-11-29 20:19:36 +08:00
Nikolai Kochetov
b78d2709d6
Update WindowTransform.cpp 
add comment
 2021-11-29 12:04:00 +03:00
Alexander Tokmakov
2fb00172a9 try parse partition key value from partition_id 2021-11-27 15:07:08 +03:00
kssenii
be3b4ca8fe Merge branch 'master' of github.com:ClickHouse/ClickHouse into versioning 2021-11-27 09:44:31 +00:00
MaxTheHuman
e32d58a36d grouping sets dev: fix errors, something works 2021-11-26 22:46:15 +03:00
MaxTheHuman
fff23945fa grouping sets: fix 'Port already connected' error 2021-11-26 22:33:14 +03:00
MaxTheHuman
e7711d953a grouping sets cleanup 2021-11-26 22:33:14 +03:00
MaxTheHuman
ddd1799743 grouping sets development 2021-11-26 22:11:34 +03:00
Raúl Marín
f0ee0724ac Reduce dependencies on ASTSelectQuery.h 
243 -> 152
 2021-11-26 18:35:24 +01:00
MaxTheHuman
2bd07ef338 feat grouping-sets: initial changes 2021-11-26 20:24:35 +03:00
Raúl Marín
b2cfa70541 Reduce dependencies on ASTFunction.h 
481 -> 230
 2021-11-26 18:21:54 +01:00
Nikolai Kochetov
05c6f065c1 remove LowCardinality from window function args. 2021-11-26 17:57:48 +03:00
Kruglov Pavel
af998af710
Merge pull request #31489 from Avogar/parallel-formatting 
Support parallel formatting almost for all text formats
 2021-11-26 15:21:22 +03:00
Raúl Marín
eb0435c51d Reduce header exposure to ProcessList.h 2021-11-26 12:52:51 +01:00
Raúl Marín
c498b7ba59 Move limits check to ProcessList 2021-11-26 12:44:39 +01:00
Raúl Marín
be60759e68 Merge remote-tracking branch 'blessed/master' into pull_timeout 2021-11-26 11:37:19 +01:00
Kruglov Pavel
efe26ddefa
Merge pull request #31804 from Avogar/fix-tskv 
Fix possible Assertion '!hasPendingData()' failed in TSKV format
 2021-11-26 12:30:45 +03:00
Vladimir Chebotarev
fae71279e7
Windowed time decay functions (#29799) 2021-11-26 11:29:26 +03:00
taiyang-li
d35e2a1c83
Merge branch 'master' into hive_table 2021-11-26 11:44:50 +08:00
kssenii
223048d395 Merge branch 'master' of github.com:ClickHouse/ClickHouse into vxider-window-view 2021-11-26 01:06:06 +03:00
avogar
aa2da98844 Add test 2021-11-25 18:06:46 +03:00
avogar
37abab7fdb Better naming 2021-11-25 15:09:13 +03:00
Kruglov Pavel
5d1520be72
Merge pull request #31736 from Avogar/fix-json-with-progress 
Fix race in JSONEachRowWithProgress output format
 2021-11-25 13:58:41 +03:00
taiyang-li
72f60cceb9
Merge branch 'master' into hive_table 2021-11-25 17:33:26 +08:00
alesapin
fe7f21acf9
Merge pull request #31697 from ClickHouse/fix_31686 
Fix parsing of domain data types
 2021-11-25 11:31:41 +03:00
Kseniia Sumarokova
93cf66df12
Merge pull request #30936 from kssenii/seekable-read-buffers 
Reduce memory usage for some formats when reading with s3/url/hdfs
 2021-11-25 11:19:24 +03:00
freedomDR
c8c827fc78 fix code comment 2021-11-25 11:11:25 +08:00
avogar
e4ba685d15 Fix race in JSONEachRowWithProgressRowOutputFormat 2021-11-24 22:29:43 +03:00
Kruglov Pavel
758c0e1c5e
Fix build 2021-11-24 18:35:18 +03:00
avogar
f5447a5e74 Fix tests, support parallel formatting for Template format 2021-11-24 16:42:07 +03:00
Alexander Tokmakov
e5972e6f71 fix 2021-11-24 15:44:04 +03:00
Raúl Marín
34d0f403dc Make clang-tidy happy 2021-11-24 12:32:08 +01:00
kssenii
29579962cb Update test, tiny changes 2021-11-24 12:55:36 +03:00
Raúl Marín
cb6f99fe6f
Use getContext 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2021-11-24 09:58:03 +01:00
taiyang-li
89dcef69d5 merge master 2021-11-24 14:38:04 +08:00
avogar
4470365fb3 Fix 2021-11-23 19:56:44 +03:00
avogar
b81d8426d3 Clean up 2021-11-23 19:56:43 +03:00
avogar
a900a26691 Support parallel formatting for all text output formats 2021-11-23 19:56:43 +03:00
MaxTheHuman
ff0bc94055 fix typos 2021-11-23 17:58:16 +03:00
MaxTheHuman
391a6019d1 fixes 2021-11-23 17:58:16 +03:00
MaxTheHuman
b5bca809e1 erase blank line to restore initial state 2021-11-23 17:58:16 +03:00
MaxTheHuman
b693652f22 revert changes made to cube transform 2021-11-23 17:58:16 +03:00
MaxTheHuman
8b95b7d271 init commit with parsing and BAD realisation 2021-11-23 17:58:16 +03:00
Raúl Marín
15dc86bd37 Fix 00613_shard_distributed_max_execution_time flakyness 2021-11-23 14:50:23 +01:00
Raúl Marín
cbe3a47f2f PipelineExecutor: Avoid throwing in constructor after saving the querystatus 
Otherwise the query status would keep a pointer to the executor which is dying at
that very moment
 2021-11-23 13:24:51 +01:00
Raúl Marín
f39648dafb Style 2021-11-23 09:23:22 +01:00
Raúl Marín
c6d3065885 Check max_execution_time in the pipeline and pulling executors 2021-11-23 09:23:22 +01:00
lgbo
996d7125c0
Merge branch 'master' into hive_table 2021-11-23 10:19:02 +08:00
kssenii
7fae3b1cc9 Update tests 2021-11-22 20:57:31 +03:00
Anton Popov
ccd78e3838 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-22 17:19:35 +03:00
mergify[bot]
a9c1fd1647
Merge branch 'master' into fix-write-buffers 2021-11-22 13:59:31 +00:00
Anton Popov
43cfa4467b
Merge pull request #31476 from CurtizJ/merging-constraints 
Merging #18787 (Constraints)
 2021-11-22 15:57:59 +03:00
mergify[bot]
a7ba3e23a0
Merge branch 'master' into fix-write-buffers 2021-11-22 11:24:27 +00:00
Kruglov Pavel
814a36ba69
Merge pull request #31434 from Avogar/custom-with-names-and-types 
Add formats CustomSeparatedWithNames/WithNamesAndTypes
 2021-11-22 13:24:00 +03:00
kssenii
05c548035f Some fixes 2021-11-22 11:04:04 +03:00
alexey-milovidov
faae69f631
Merge pull request #31534 from aiven/kmichel-fix-json-colum-name-encoding 
Fix invalid JSON in column names
 2021-11-21 11:34:33 +03:00
Kruglov Pavel
d9c1a0c8ec
Merge branch 'master' into fix-write-buffers 2021-11-20 17:48:24 +03:00
Azat Khuzhin
6aa94ae032 Fix MySQLWire format (in case of multiple writes) 
In case of multiple writes File() engine will set doNotWritePrefix(),
and this will avoid serializations initilization, move this to do this
always.

Fixes: #31004
 2021-11-20 15:26:21 +03:00
kssenii
ff969b4605 Merge branch 'master' of github.com:ClickHouse/ClickHouse into seekable-read-buffers 2021-11-20 15:03:13 +03:00
Kevin Michel
edbeeaf6ec
Fix invalid JSON in column names 
If the column name contains invalid UTF-8 sequences
and the output data types are all considered safe,
then the output will not be sanitized and the generated
JSON will be invalid.

A minimal reproduction case is :
`SELECT length('\x80') FORMAT JSONCompact`
where we auto-generate a non-UTF-8 column name with only
integer outputs, whereas :
`SELECT '\x80' FORMAT JSONCompact`
would be sanitized because the column type is String and
will trigger UTF-8 sanitization over the entire document.
 2021-11-20 12:35:41 +01:00
kssenii
29978f9707 Merge branch 'master' of github.com:ClickHouse/ClickHouse into vxider-window-view 2021-11-20 12:13:39 +03:00
Kruglov Pavel
fdd1f53d3a
Update CustomSeparatedRowOutputFormat.h 2021-11-19 16:52:48 +03:00
Kruglov Pavel
3070bf1e4d
Update CustomSeparatedRowOutputFormat.cpp 2021-11-19 16:52:31 +03:00
Kruglov Pavel
eb1cc834dc
Merge branch 'master' into custom-with-names-and-types 2021-11-19 16:51:25 +03:00
kssenii
9bdad8ab64 Rewrite from DataStreams to Processors 2021-11-19 16:09:12 +03:00
taiyang-li
e8644807fe merge master and solve conflict 2021-11-19 15:01:58 +08:00
Vitaly Baranov
6634fcbac7 Rename Quota::ResourceType -> QuotaType and move it to Access/Common. 2021-11-19 00:14:23 +03:00
Kruglov Pavel
f27fcf8372
Merge pull request #31004 from Avogar/output-formats 
Small refactoring in formats
 2021-11-17 20:44:46 +03:00
avogar
1ebcbf4748 Fix style 2021-11-16 17:10:30 +03:00
Kruglov Pavel
be7a358f17 Fix 2021-11-16 17:10:30 +03:00