Commit Graph

2851 Commits

Author SHA1 Message Date
Alexey Milovidov
48b4d98b21 Amend 2021-01-27 04:48:41 +03:00
Alexey Milovidov
a6f5f40a65 Merge branch 'master' into disable-checksums-on-read 2021-01-27 04:09:00 +03:00
vdimir
fc8b8d23b2
Merge pull request #18812 from excitoon-favorites/betters3endpoints
Added prefix-based S3 endpoint settings
2021-01-26 20:47:57 +03:00
Vladimir Chebotarev
25037edfa6 Removed space in winking face smiley. 2021-01-26 11:28:47 +03:00
tavplubix
2d6a71fced
Merge pull request #19443 from ClickHouse/fix_virtual_parts_in_parts_to_do
Addition to #15537
2021-01-26 11:27:14 +03:00
Alexey Milovidov
8dfa933028 Amend 2021-01-25 23:48:10 +03:00
Alexey Milovidov
9ee5c1535e Allow to disable checksums on read 2021-01-25 23:29:04 +03:00
Anton Popov
658f24dcff
Merge pull request #19358 from CurtizJ/fix-subcolumns
Fix several cases, while reading subcolumns
2021-01-25 20:26:07 +03:00
tavplubix
5f07bfb9f8
Update ReplicatedMergeTreeQueue.cpp 2021-01-25 16:15:47 +03:00
tavplubix
a88a564aae
Update ReplicatedMergeTreeQueue.cpp 2021-01-25 12:51:06 +03:00
alesapin
fe133580ee
Merge pull request #19346 from azat/system.parts-_state-fix
Fix system.parts _state column
2021-01-25 10:11:19 +03:00
alexey-milovidov
ba3e064a63
Merge pull request #19371 from kssenii/test-coverage-with-factories
test coverage with factories
2021-01-25 07:11:09 +03:00
Vladimir Chebotarev
8ba8e8efd5 Added prefix-based S3 endpoint settings. 2021-01-25 06:40:57 +03:00
alexey-milovidov
73501102f3
Merge pull request #19528 from azat/merge_tree_min_for_concurrent_read-SIGSEGV
Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX
2021-01-25 00:18:42 +03:00
Azat Khuzhin
1c364b6ee3 Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX
In case of 0 or too huge value it will try to read not existing marks
and got:

    Logical error: 'Trying to get non existing mark 11936128518282651045, while size is 2'.
2021-01-24 14:39:57 +03:00
alexey-milovidov
7a10fbc9c3
Merge pull request #19479 from ClickHouse/columns-description-idiotic
Fix wrong serde of ColumnsDescription
2021-01-23 12:08:15 +03:00
alexey-milovidov
7f059e3765
Merge pull request #19449 from azat/dist-send-check-block
Mark distributed batch as broken in case of empty data block in one of files
2021-01-23 12:04:31 +03:00
Alexey Milovidov
7005e3a826 Fix wrong serde of ColumnsDescription 2021-01-23 04:11:18 +03:00
Alexey Milovidov
7d2108d4e9 Fix double whitespace 2021-01-23 02:57:35 +03:00
Azat Khuzhin
109dbe5df4 Check the stream before sending while hanlding async INSERTs into Distributed
It is possible to get corruption (even though it is very unlikely, and
initially it wasn't corruption) just before the data block goes in the
file on disk, and in case of batching, it will break the packets, since
it will write the packet type but will not write any data after.
2021-01-22 21:29:58 +03:00
Alexander Tokmakov
963dadae54 fix wrong parts in parts_to_do 2021-01-22 19:07:19 +03:00
tavplubix
9801bfc0ee
Merge branch 'master' into fix_18063 2021-01-22 17:28:05 +03:00
alexey-milovidov
6e1d660e36
Merge pull request #19390 from ClickHouse/http_referer
Add http_referer to client info
2021-01-22 17:25:09 +03:00
alexey-milovidov
9818bd319a
Merge pull request #19381 from azat/parts-types-metrics
Add metrics for MergeTree parts (Wide/Compact/InMemory) types
2021-01-22 17:24:50 +03:00
Alexander Kuzmenkov
b58a696b24
Merge pull request #19379 from azat/Buffer-less-lock-contention
Reduce lock contention for multiple layers of the Buffer engine
2021-01-22 13:17:34 +03:00
alexey-milovidov
4e11e7cfa9
Merge pull request #18772 from azat/optimize-memory-tracking-fix
[RFC] Fix memory tracking for OPTIMIZE TABLE/merges
2021-01-22 03:48:06 +03:00
Alexey Milovidov
afc6f2fd8e Add http_referer to client info 2021-01-22 01:55:45 +03:00
kssenii
daab2c91bb Better 2021-01-21 21:15:11 +00:00
Azat Khuzhin
b0a80af888 Reduce lock contention for multiple layers of the Buffer engine
Otherwise you can see something like this for the following query:

    ```sql
    WITH
        arrayMap(x -> demangle(addressToSymbol(x)), s.trace) AS trace_array,
        arrayStringConcat(trace_array, '\n') AS trace_string
    SELECT
        p.thread_id,
        p.query_id,
        p.query,
        trace_string
    FROM
    (
        SELECT
            query_id,
            query,
            arrayJoin(thread_ids) AS thread_id
        FROM system.processes
    ) AS p
    INNER JOIN system.stack_trace AS s ON p.thread_id = s.thread_id
    ORDER BY p.query_id ASC
    SETTINGS enable_global_with_statement = 0, allow_introspection_functions = 1
    FORMAT PrettyCompactNoEscapes
    ```

Lots of the following:

    ```sql
    INSERT INTO buffer (...) VALUES

    __lll_lock_wait
    pthread_mutex_lock
    std::__1::mutex::lock()
    DB::StorageBuffer::reschedule()
    DB::BufferBlockOutputStream::write(DB::Block const&)
    ```

That will wait one of this:

    ```
    INSERT INTO buffer (...) VALUES

    ...
    DB::PushingToViewsBlockOutputStream::write(DB::Block const&)
    DB::AddingDefaultBlockOutputStream::write(DB::Block const&)
    DB::SquashingBlockOutputStream::finalize()
    DB::SquashingBlockOutputStream::writeSuffix()
    DB::PushingToViewsBlockOutputStream::writeSuffix()
    DB::StorageBuffer::writeBlockToDestination(DB::Block const&, std::__1::shared_ptr<DB::IStorage>)
    DB::StorageBuffer::flushBuffer(DB::StorageBuffer::Buffer&, bool, bool, bool)
    ```

P.S. we cannot simply unlock the buffer during flushing, see comments in
the code
2021-01-21 22:09:24 +03:00
Azat Khuzhin
cb951c2116 Add metrics for MergeTree parts types
- PartsWide
- PartsCompact
- PartsInMemory
2021-01-21 21:17:00 +03:00
Azat Khuzhin
c68f7cd5b1 Measure time that spend during flush of the Buffer to the underlying 2021-01-21 21:11:39 +03:00
kssenii
c1702f34ee Add factories info into system.query_log 2021-01-21 15:46:37 +00:00
alexey-milovidov
062f00aa5d
Merge pull request #19290 from azat/dist-broken-on-EOF-fix
Do not mark file for distributed send as broken on EOF
2021-01-21 17:00:36 +03:00
tavplubix
ac50b1b231
Update StorageMaterializedView.cpp 2021-01-21 15:57:18 +03:00
Anton Popov
ac3de63a71 fix several cases, while reading subcolumns 2021-01-21 15:34:11 +03:00
alesapin
fe6b964b32 Revert "Revert "Auto version update to [21.2.1.1] [54446]""
This reverts commit 42f63e14b5.
2021-01-21 12:39:46 +03:00
Azat Khuzhin
5f3059555a Fix system.parts _state column
There was LOGICAL_ERROR when querying this column, due to incorrect order:

    SELECT
        *,
        _state
    FROM system.parts

    2021.01.21 10:22:57.731556 [ 22851 ] {02a07c6d-467d-4681-9203-4dc11cc6fbee} <Fatal> : Logical error: 'Invalid Field get from type String to type UInt64'.
2021-01-21 10:41:23 +03:00
Alexander Tokmakov
6560ec3ed5 fix segfault on aggregation when MV has unexpected structure 2021-01-20 19:36:18 +03:00
Azat Khuzhin
8a00816396 Do not mark file for distributed send as broken on EOF
- the sender will got ATTEMPT_TO_READ_AFTER_EOF (added in
  946c275dfb) when the client just go
  away, i.e. server had been restarted, and this is incorrect to mark the
  file as broken in this case.

- since #18853 the file will be checked on the sender locally, and
  in case the file was truncated CANNOT_READ_ALL_DATA will be thrown.
  But before #18853 the sender will not receive
  ATTEMPT_TO_READ_AFTER_EOF from the client in case of file was truncated
  on the sender, since the client will just wait for more data, IOW just hang.

- and I don't see how ATTEMPT_TO_READ_AFTER_EOF can be received while
  reading local file.
2021-01-20 01:10:17 +03:00
alesapin
70d9c7ea03
Merge pull request #19237 from ClickHouse/fix_wait_for_log_entries
Don't wait forever for log update after table was dropped
2021-01-19 10:22:52 +03:00
alesapin
4ee96869a2 Don't wait forever for log update after table was dropped 2021-01-18 15:15:07 +03:00
Alexander Kuzmenkov
d22c04568d
Merge pull request #13405 from excitoon-favorites/s3keepalive
Connection pools for S3
2021-01-18 12:52:37 +03:00
alexey-milovidov
1444ccb85e
Merge pull request #19206 from azat/dist-pull
Tiny changes in DistributedBlockOutputStream
2021-01-18 01:04:50 +03:00
alexey-milovidov
15f4ae26c2
Merge pull request #17310 from CurtizJ/multiple-nested
Allow nested with multiple nesting and subcolumns of complex types
2021-01-17 15:00:26 +03:00
Azat Khuzhin
a6631287a7 DistributedBlockOutputStream: add more comments 2021-01-17 12:50:37 +03:00
Azat Khuzhin
b725e1d131 DistributedBlockOutputStream: Remove superfluous brackets for string construction 2021-01-17 12:48:51 +03:00
Azat Khuzhin
2955e25e83 Fix inserted blocks accounting for insert_distributed_one_random_shard=1
It is tricky due to block splitting

Refs: https://github.com/ClickHouse/ClickHouse/pull/18294
2021-01-17 12:45:42 +03:00
Azat Khuzhin
858f07c796 Update comment for query AST cloning during inesrt into multiple local shards
Refs: https://github.com/ClickHouse/ClickHouse/pull/18264#discussion_r558839456
2021-01-17 12:40:41 +03:00
alexey-milovidov
7de745ce77
Merge pull request #18554 from kssenii/pg2ch
Add PostgreSQL table function, dictionary source, database engine
2021-01-16 23:55:05 +03:00
Alexey Milovidov
029302d766 Merge with master 2021-01-16 17:09:44 +03:00