Commit Graph

815 Commits

Author SHA1 Message Date
Alexey Milovidov
e4e7169277 Remove some strange code 2022-02-01 02:52:36 +03:00
Alexey Milovidov
83136f3515 Allow \r in the middle of the line in format Regexp 2022-02-01 02:49:26 +03:00
Alexey Milovidov
872d0a0fbe Improve performance of format Regexp 2022-02-01 02:07:48 +03:00
alesapin
dd61d1c2de
Merge pull request #34172 from ClickHouse/fix_race_in_some_engines
Fix benign race condition for storage HDFS, S3, URL
2022-01-31 22:41:54 +03:00
alesapin
93c0700c4c Fix typo 2022-01-31 16:46:58 +03:00
alesapin
056b9e335f Fix comment 2022-01-31 16:39:42 +03:00
alesapin
31753afb7e Fix cancel logic in parallel parsing 2022-01-31 16:38:15 +03:00
Maksim Kita
5ef83deaa6 Update sort to pdqsort 2022-01-30 19:49:48 +00:00
Anton Popov
b950a12cb3
Merge pull request #34068 from CurtizJ/fix-async-insert-native
Fix asynchronous inserts with `Native` format
2022-01-29 01:24:53 +03:00
Azat Khuzhin
1519985c98 Fix possible "Can't attach query to the thread, it is already attached"
After detachQueryIfNotDetached() had been removed it is not enough to
use attachTo() for ThreadPool (scheduleOrThrowOnError()) since the query
may be already attached, if the thread doing multiple jobs, so
CurrentThread::attachToIfDetached() should be used instead.

This should fix all the places from the failures on CI [1]:

    $ fgrep DB::CurrentThread::attachTo -A1 ~/Downloads/47.txt  | fgrep -v attachTo | cut -d' ' -f5,6 | sort | uniq -c
         92 --
          2 /fasttest-workspace/build/../../ClickHouse/contrib/libcxx/include/deque:1393: DB::ParallelParsingInputFormat::parserThreadFunction(std::__1::shared_ptr<DB::ThreadGroupStatus>,
          4 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeData.cpp:1595: void
         87 /fasttest-workspace/build/../../ClickHouse/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp:993: void

  [1]: https://github.com/ClickHouse/ClickHouse/runs/4954466034?check_suite_focus=true

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 16:25:33 +03:00
Azat Khuzhin
b0c862c297 Fix memory accounting for queries that uses < max_untracker_memory
MemoryTracker starts accounting memory directly only after per-thread
allocation exceeded max_untracker_memory (or memory_profiler_step).

But even memory under this limit should be accounted too, and there is
code to do this in ThreadStatus dtor, however due to
PullingAsyncPipelineExecutor detached the query from thread group that
memory was not accounted.

So remove CurrentThread::detachQueryIfNotDetached() from threads that
uses ThreadFromGlobalPool since it has ThreadStatus, and the query will
be detached using CurrentThread::defaultThreadDeleter.

Note, that before this patch memory accounting works for HTTP queries
due to it had been accounted from ParallelFormattingOutputFormat, but
not for TCP.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-01-28 16:25:33 +03:00
Anton Popov
6c0959b907 fix asynchronous inserts with Native format 2022-01-28 03:25:15 +03:00
Kruglov Pavel
9f12f4af13
Merge pull request #33302 from Avogar/formats-with-suffixes
Allow to create new files on insert for File/S3/HDFS engines
2022-01-25 10:56:15 +03:00
avogar
1f49acc164 Better naming 2022-01-24 16:28:36 +03:00
avogar
67e396f8f4 Fix schema inference for JSONEachRow and JSONCompactEachRow 2022-01-20 16:31:24 +03:00
mergify[bot]
b318f9b5db
Merge branch 'master' into formats-with-suffixes 2022-01-18 12:17:07 +00:00
Anton Popov
a25f2518e3
Merge pull request #33141 from 1over/feature_default_keyword
Add support of DEFAULT keyword for INSERT
2022-01-18 02:04:37 +03:00
Kruglov Pavel
a7df9cd53a
Merge branch 'master' into formats-with-suffixes 2022-01-14 21:03:49 +03:00
avogar
253035a5df Fix 2022-01-14 19:17:06 +03:00
Kruglov Pavel
d2e9f37bee
Merge branch 'master' into format-by-extention 2022-01-14 18:36:23 +03:00
avogar
89a181bd19 Make better 2022-01-14 18:16:18 +03:00
Kruglov Pavel
5a908e8edd
Merge branch 'master' into formats-with-suffixes 2022-01-14 16:45:20 +03:00
Kruglov Pavel
d54a430d9c
Merge pull request #33566 from Avogar/fix-avro
Fix segfault in Avro
2022-01-14 16:01:56 +03:00
Kseniia Sumarokova
5da673c3a5
Merge pull request #31104 from bigo-sg/hive_table
Implement hive table engine
2022-01-14 09:39:17 +03:00
Kruglov Pavel
305d58a762
Merge pull request #33524 from Avogar/stacktrace-in-client
Don't print exception twice in client in case of exception in parallel parsing
2022-01-13 15:50:42 +03:00
Nikolai Kochetov
872ee5dc09
Update src/Processors/Formats/Impl/AvroRowOutputFormat.h
Co-authored-by: Bharat Nallan <bharatnc@gmail.com>
2022-01-13 12:55:14 +03:00
avogar
c5ea4b1bc0 Fix segfault in Avro 2022-01-12 18:34:28 +03:00
avogar
8390e9ad60 Detect format by file name in file/hdfs/s3/url table functions 2022-01-12 18:29:31 +03:00
lgbo-ustc
5c71d3687a fixed some bugs
1. interagtion test for test_hive_query failed
2. nullptr reference in arrowSchemaToCHHeader
2022-01-12 17:01:05 +08:00
taiyang-li
66813a3aa9 merge master 2022-01-12 16:56:29 +08:00
avogar
9915ce7ded Fix segfault in arrowSchemaToCHHeader 2022-01-11 20:30:35 +03:00
avogar
0ae0aa712b Don't print exception twice in client in case of exception in parallel parsing 2022-01-11 18:37:07 +03:00
李扬
2df2442ad0
Merge branch 'master' into hive_table 2022-01-04 01:26:16 -06:00
taiyang-li
8730dda895 fix hivte text 2022-01-01 09:16:30 +08:00
taiyang-li
1e102bc1b2 merge master 2022-01-01 09:01:06 +08:00
alexey-milovidov
34b934a1e0
Merge pull request #33331 from ClickHouse/serxa/line-as-string-output-format
Add LineAsString output format
2021-12-31 14:38:36 +03:00
Sergei Trifonov
f1d398ae4b Add LineAsString output format 2021-12-30 20:38:54 +03:00
avogar
97788b9c21 Allow to create new files on insert for File/S3/HDFS engines 2021-12-29 21:19:13 +03:00
avogar
364b4f5d36 Fix special build 2021-12-29 12:21:01 +03:00
Kruglov Pavel
cb0ed7fcb7 Fix typo 2021-12-29 12:21:01 +03:00
avogar
26abf7aa62 Remove code duplication, use simdjson and rapidjson instead of Poco 2021-12-29 12:21:01 +03:00
avogar
74f09d6476 Fix tests 2021-12-29 12:18:56 +03:00
avogar
aaf9f85c67 Add more tests and fixes 2021-12-29 12:18:56 +03:00
avogar
dd994aa761 Add some tests and some fixes 2021-12-29 12:18:56 +03:00
avogar
8112a71233 Implement schema inference for most input formats 2021-12-29 12:18:56 +03:00
kssenii
1f6ca619b7 Allow some killing 2021-12-27 22:42:56 +03:00
taiyang-li
9036b18c2f merge master 2021-12-27 15:12:48 +08:00
alexey-milovidov
c583ea7e6b
Merge pull request #32484 from Algunenano/libcxx13_take2
libc++ 13 compatibility
2021-12-25 10:14:12 +03:00
Andrii Buriachevskyi
e8cc6df7bb Add support of DEFAULT keyword for INSERT 2021-12-24 13:10:19 +01:00
Alexey Milovidov
29d28c531f Move code around to avoid dlsym on Musl 2021-12-24 12:25:27 +03:00