thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
56,415 Commits 9,326 Branches 1,783 Tags 2.1 GiB
bedc472bb0
Commit Graph

2883 Commits

Author SHA1 Message Date
Azat Khuzhin
109dbe5df4 Check the stream before sending while hanlding async INSERTs into Distributed 
It is possible to get corruption (even though it is very unlikely, and
initially it wasn't corruption) just before the data block goes in the
file on disk, and in case of batching, it will break the packets, since
it will write the packet type but will not write any data after.
 2021-01-22 21:29:58 +03:00
Alexander Tokmakov
963dadae54 fix wrong parts in parts_to_do 2021-01-22 19:07:19 +03:00
tavplubix
9801bfc0ee
Merge branch 'master' into fix_18063 2021-01-22 17:28:05 +03:00
alexey-milovidov
6e1d660e36
Merge pull request #19390 from ClickHouse/http_referer 
Add http_referer to client info
 2021-01-22 17:25:09 +03:00
alexey-milovidov
9818bd319a
Merge pull request #19381 from azat/parts-types-metrics 
Add metrics for MergeTree parts (Wide/Compact/InMemory) types
 2021-01-22 17:24:50 +03:00
Alexander Kuzmenkov
b58a696b24
Merge pull request #19379 from azat/Buffer-less-lock-contention 
Reduce lock contention for multiple layers of the Buffer engine
 2021-01-22 13:17:34 +03:00
alexey-milovidov
4e11e7cfa9
Merge pull request #18772 from azat/optimize-memory-tracking-fix 
[RFC] Fix memory tracking for OPTIMIZE TABLE/merges
 2021-01-22 03:48:06 +03:00
Alexey Milovidov
afc6f2fd8e Add http_referer to client info 2021-01-22 01:55:45 +03:00
kssenii
daab2c91bb Better 2021-01-21 21:15:11 +00:00
Azat Khuzhin
b0a80af888 Reduce lock contention for multiple layers of the Buffer engine 
Otherwise you can see something like this for the following query:

    ```sql
    WITH
        arrayMap(x -> demangle(addressToSymbol(x)), s.trace) AS trace_array,
        arrayStringConcat(trace_array, '\n') AS trace_string
    SELECT
        p.thread_id,
        p.query_id,
        p.query,
        trace_string
    FROM
    (
        SELECT
            query_id,
            query,
            arrayJoin(thread_ids) AS thread_id
        FROM system.processes
    ) AS p
    INNER JOIN system.stack_trace AS s ON p.thread_id = s.thread_id
    ORDER BY p.query_id ASC
    SETTINGS enable_global_with_statement = 0, allow_introspection_functions = 1
    FORMAT PrettyCompactNoEscapes
    ```

Lots of the following:

    ```sql
    INSERT INTO buffer (...) VALUES

    __lll_lock_wait
    pthread_mutex_lock
    std::__1::mutex::lock()
    DB::StorageBuffer::reschedule()
    DB::BufferBlockOutputStream::write(DB::Block const&)
    ```

That will wait one of this:

    ```
    INSERT INTO buffer (...) VALUES

    ...
    DB::PushingToViewsBlockOutputStream::write(DB::Block const&)
    DB::AddingDefaultBlockOutputStream::write(DB::Block const&)
    DB::SquashingBlockOutputStream::finalize()
    DB::SquashingBlockOutputStream::writeSuffix()
    DB::PushingToViewsBlockOutputStream::writeSuffix()
    DB::StorageBuffer::writeBlockToDestination(DB::Block const&, std::__1::shared_ptr<DB::IStorage>)
    DB::StorageBuffer::flushBuffer(DB::StorageBuffer::Buffer&, bool, bool, bool)
    ```

P.S. we cannot simply unlock the buffer during flushing, see comments in
the code
 2021-01-21 22:09:24 +03:00
Azat Khuzhin
cb951c2116 Add metrics for MergeTree parts types 
- PartsWide
- PartsCompact
- PartsInMemory
 2021-01-21 21:17:00 +03:00
Azat Khuzhin
c68f7cd5b1 Measure time that spend during flush of the Buffer to the underlying 2021-01-21 21:11:39 +03:00
kssenii
c1702f34ee Add factories info into system.query_log 2021-01-21 15:46:37 +00:00
alexey-milovidov
062f00aa5d
Merge pull request #19290 from azat/dist-broken-on-EOF-fix 
Do not mark file for distributed send as broken on EOF
 2021-01-21 17:00:36 +03:00
tavplubix
ac50b1b231
Update StorageMaterializedView.cpp 2021-01-21 15:57:18 +03:00
Anton Popov
ac3de63a71 fix several cases, while reading subcolumns 2021-01-21 15:34:11 +03:00
alesapin
fe6b964b32 Revert "Revert "Auto version update to [21.2.1.1] [54446]"" 
This reverts commit 42f63e14b5.
 2021-01-21 12:39:46 +03:00
Azat Khuzhin
5f3059555a Fix system.parts _state column 
There was LOGICAL_ERROR when querying this column, due to incorrect order:

    SELECT
        *,
        _state
    FROM system.parts

    2021.01.21 10:22:57.731556 [ 22851 ] {02a07c6d-467d-4681-9203-4dc11cc6fbee} <Fatal> : Logical error: 'Invalid Field get from type String to type UInt64'.
 2021-01-21 10:41:23 +03:00
Alexander Tokmakov
6560ec3ed5 fix segfault on aggregation when MV has unexpected structure 2021-01-20 19:36:18 +03:00
Azat Khuzhin
8a00816396 Do not mark file for distributed send as broken on EOF 
- the sender will got ATTEMPT_TO_READ_AFTER_EOF (added in
  946c275dfb) when the client just go
  away, i.e. server had been restarted, and this is incorrect to mark the
  file as broken in this case.

- since #18853 the file will be checked on the sender locally, and
  in case the file was truncated CANNOT_READ_ALL_DATA will be thrown.
  But before #18853 the sender will not receive
  ATTEMPT_TO_READ_AFTER_EOF from the client in case of file was truncated
  on the sender, since the client will just wait for more data, IOW just hang.

- and I don't see how ATTEMPT_TO_READ_AFTER_EOF can be received while
  reading local file.
 2021-01-20 01:10:17 +03:00
alesapin
70d9c7ea03
Merge pull request #19237 from ClickHouse/fix_wait_for_log_entries 
Don't wait forever for log update after table was dropped
 2021-01-19 10:22:52 +03:00
alesapin
4ee96869a2 Don't wait forever for log update after table was dropped 2021-01-18 15:15:07 +03:00
Alexander Kuzmenkov
d22c04568d
Merge pull request #13405 from excitoon-favorites/s3keepalive 
Connection pools for S3
 2021-01-18 12:52:37 +03:00
alexey-milovidov
1444ccb85e
Merge pull request #19206 from azat/dist-pull 
Tiny changes in DistributedBlockOutputStream
 2021-01-18 01:04:50 +03:00
alexey-milovidov
15f4ae26c2
Merge pull request #17310 from CurtizJ/multiple-nested 
Allow nested with multiple nesting and subcolumns of complex types
 2021-01-17 15:00:26 +03:00
Azat Khuzhin
a6631287a7 DistributedBlockOutputStream: add more comments 2021-01-17 12:50:37 +03:00
Azat Khuzhin
b725e1d131 DistributedBlockOutputStream: Remove superfluous brackets for string construction 2021-01-17 12:48:51 +03:00
Azat Khuzhin
2955e25e83 Fix inserted blocks accounting for insert_distributed_one_random_shard=1 
It is tricky due to block splitting

Refs: https://github.com/ClickHouse/ClickHouse/pull/18294
 2021-01-17 12:45:42 +03:00
Azat Khuzhin
858f07c796 Update comment for query AST cloning during inesrt into multiple local shards 
Refs: https://github.com/ClickHouse/ClickHouse/pull/18264#discussion_r558839456
 2021-01-17 12:40:41 +03:00
alexey-milovidov
7de745ce77
Merge pull request #18554 from kssenii/pg2ch 
Add PostgreSQL table function, dictionary source, database engine
 2021-01-16 23:55:05 +03:00
Alexey Milovidov
029302d766 Merge with master 2021-01-16 17:09:44 +03:00
Alexey Milovidov
24c8e53440 Merge branch 'master' into multiple-nested 2021-01-16 16:28:40 +03:00
alexey-milovidov
e67e4588e7
Update PostgreSQLConnection.h 2021-01-16 15:26:21 +03:00
alexey-milovidov
a15092eeb7
Merge pull request #18264 from ucasFL/insert-cluster 
Support insert into table function cluster
 2021-01-16 13:22:49 +03:00
alexey-milovidov
a5a19de878
Update DistributedBlockOutputStream.cpp 2021-01-16 13:22:25 +03:00
alexey-milovidov
4efc7a7dc3
Update MergeTreeIOSettings.h 2021-01-16 13:07:58 +03:00
alexey-milovidov
e32b1e3fe6
Update StorageTinyLog.cpp 2021-01-16 11:53:28 +03:00
Alexey Milovidov
4ee9c57190 Add TODO 2021-01-16 11:43:59 +03:00
Alexey Milovidov
b25e334313 Add assert 2021-01-16 11:30:35 +03:00
alexey-milovidov
2e2988e5d8
Merge pull request #19146 from azat/server-memory-limit-blocking 
MemoryTracker: Do not ignore server memory limits during blocking by default
 2021-01-16 11:09:19 +03:00
alesapin
67fd381034
Merge pull request #19123 from ClickHouse/additional_check_in_writer 
Fix max granules size in MergeTreeDataWriter
 2021-01-16 10:17:44 +03:00
alexey-milovidov
5f189c5756
Merge pull request #19122 from ClickHouse/data-part-better-code 
Add metrics for part number in MergeTree in ClickHouse
 2021-01-16 00:20:15 +03:00
alexey-milovidov
e09ec9fed5
Merge pull request #18373 from amosbird/fix-18364 
Fix 2-arg functions with constant in PK analysis
 2021-01-16 00:20:03 +03:00
Alexey Milovidov
67f207d8ca More safe connection string 2021-01-16 00:18:51 +03:00
Alexey Milovidov
52cfc1d110 Minor changes 2021-01-15 23:09:31 +03:00
Alexey Milovidov
ba1921f380 Minor changes 2021-01-15 22:59:49 +03:00
Azat Khuzhin
61b2d0ce42 MemoryTracker: Do not ignore server memory limits during blocking by default 2021-01-15 22:46:58 +03:00
Alexey Milovidov
b293d56a63 Merge branch 'master' into kssenii-pg2ch 2021-01-15 22:32:03 +03:00
alesapin
184dbedb06 Fix stupid error 2021-01-15 21:50:30 +03:00
alexey-milovidov
b97beea22a
Merge pull request #19101 from ClickHouse/check_compression_codec_read 
Fix compression codec read for empty files
 2021-01-15 20:55:58 +03:00
alexey-milovidov
971ff2ee0a
Merge pull request #19086 from ClickHouse/faster-parts-removal 
Faster parts removal, more safe and efficient interface of IDisk
 2021-01-15 20:37:35 +03:00
feng lv
9829c09720 fix 2021-01-15 15:54:35 +00:00
Alexey Milovidov
a19e7edd14 Merge branch 'master' into kssenii-pg2ch 2021-01-15 17:33:19 +03:00
alexey-milovidov
ca825f36f1
Merge pull request #18853 from azat/dist-send-checksums 
Improve checksum checks for async INSERT into Distributed on the sender
 2021-01-15 17:12:54 +03:00
alesapin
8ccaa6ede9 Additional check for huge granules in MergeTreeDataWriter 2021-01-15 15:40:37 +03:00
Alexey Milovidov
e238fd64ac Add part metrics 2021-01-15 15:28:53 +03:00
Alexey Milovidov
6a2a5e53ed Slightly better code of IMergeTreeDataPart #18955 2021-01-15 15:15:13 +03:00
alexey-milovidov
78fff6bc39
Merge branch 'master' into multiple-nested 2021-01-15 14:54:27 +03:00
Alexey Milovidov
4bae04d500 Merge branch 'master' into amosbird/fix-18364 2021-01-15 14:37:35 +03:00
alexey-milovidov
4a71971b43
Update KeyCondition.cpp 2021-01-15 14:36:07 +03:00
alexey-milovidov
8d58ce532a
Merge pull request #19064 from CurtizJ/restrict-modify-ttl 
Restrict MODIFY TTL for tables created in old syntax
 2021-01-15 14:09:47 +03:00
Alexey Milovidov
d553e46a06 Merge branch 'master' into faster-parts-removal 2021-01-15 13:25:20 +03:00
alexey-milovidov
461d370e8d
Merge pull request #19020 from ClickHouse/fix_18879 
Fix duplicate UUIDs of LiveView on server startup
 2021-01-15 13:04:08 +03:00
alesapin
d601faa669
Merge pull request #18935 from fastio/bugfix_attach_partition_does_not_reset_mutation 
Bugfix: attach partition should reset the mutation
 2021-01-15 12:21:16 +03:00
alesapin
e106df2ad0 Fix comment 2021-01-15 12:10:03 +03:00
alesapin
0662d6bd7d Fix compression codec read for empty files 2021-01-15 12:04:23 +03:00
Azat Khuzhin
ecae6c1c60 Avoid reading the distributed batch just to read the block header 
Before this patch batched mode of the DirectoryMonitor is 2x slower then
non-batched, after it should be more or less the same as non-batched.
 2021-01-14 22:38:46 +03:00
Alexey Milovidov
8276a1c8d2 Faster parts removal, more safe and efficient interface of IDisk 2021-01-14 19:24:13 +03:00
alesapin
09e825e28c
Merge pull request #19040 from ClickHouse/fix_redundant_exception_while_dropping_part 
Avoid redundant exception while dropping part
 2021-01-14 18:59:59 +03:00
Anton Popov
ac426c3da6 restrict MODIFY TTL for tables created in old syntax 2021-01-14 15:32:20 +03:00
Alexander Tokmakov
64975c08e7 fix 2021-01-14 13:33:39 +03:00
alexey-milovidov
2ca185c9f9
Merge pull request #16995 from sundy-li/alias_column_partition_prune 
Add optimize_respect_aliases
 2021-01-14 11:30:45 +03:00
Nikolai Kochetov
7b33ad5e44
Merge pull request #19010 from ClickHouse/fix-insert-lc-to-tiny-log 
Fix insert LC to TinyLog
 2021-01-14 11:25:38 +03:00
alesapin
bfc27254b2 Avoid redundant exception while dropping part 2021-01-14 11:07:13 +03:00
alexey-milovidov
3b02a3ef02
Merge pull request #18976 from ClickHouse/fix_terminate_when_not_enough_memory 
Finally get rid of allocations in ThreadPool::worker(...)
 2021-01-13 23:48:46 +03:00
Alexander Tokmakov
5f0f8ae3cd fix duplicate UUIDs of LiveView on server startup 2021-01-13 21:13:46 +03:00
Nikolai Kochetov
da0cb8d47f Added comment. 2021-01-13 16:32:27 +03:00
ygrek
8f2a830d83
add zstd long range option (#17184) 
* add zstd long compression option

* tests: add zstd long read-write test

Co-authored-by: Joris Giovannangeli <joris.giovannangeli@ahrefs.com>
Co-authored-by: ip <igor@ahrefs.com>
 2021-01-13 16:22:59 +03:00
Nikolai Kochetov
12f3b22623 Do not skip streams after serializeBinaryBulkStatePrefix. 2021-01-13 15:20:10 +03:00
Alexander Tokmakov
a6510cc4b7 Merge branch 'master' into fix_terminate_when_not_enough_memory 2021-01-13 14:05:54 +03:00
alesapin
c97469773d
Merge pull request #18969 from ClickHouse/fix_drop_part_deduplication 
Fix DROP PART query break deduplication
 2021-01-13 11:50:26 +03:00
alesapin
73e536a074
Merge pull request #18928 from ClickHouse/more_checks_in_writer_wide 
More checks in merge tree writer wide
 2021-01-13 09:59:45 +03:00
Anton Popov
d7200ee2ed minor changes 2021-01-13 02:20:32 +03:00
Alexander Tokmakov
ff6c3c75c2 add protection from unsafe allocations 2021-01-12 18:41:24 +03:00
alesapin
dead1016d6 Fix deduplication block names parsing 2021-01-12 13:55:02 +03:00
sundy-li
ad01534bd5 Merge branch 'master' into alias_column_partition_prune 2021-01-12 18:28:30 +08:00
alexey-milovidov
2c71b997de
Merge pull request #18464 from hexiaoting/dev_fp_as_pk 
Disallow floating point as partition key
 2021-01-12 13:08:41 +03:00
alesapin
f3e55183ad Better test and check 2021-01-12 11:46:31 +03:00
tavplubix
1d6bfe8ad7
Merge pull request #18944 from ClickHouse/fix_mutation_commands_escaping 
Fix mutations text serialization
 2021-01-12 11:29:24 +03:00
alesapin
5ae31b8068 Slightly relax check 2021-01-12 10:59:14 +03:00
Azat Khuzhin
05608687d6 Get back memory tracking blocker in calculateAndSerializePrimaryIndex() 
But reduce scope, to avoid leaking too much memory, since there are old
values in last_block_index_columns.

The scope of the MemoryTracker::BlockerInThread has been increased in #8290
 2021-01-12 01:00:37 +03:00
Azat Khuzhin
82edbfb581 Account query memory limits and sampling for OPTIMIZE TABLE/merges 2021-01-12 00:49:41 +03:00
Azat Khuzhin
bd05d9db2f Fix memory tracking for OPTIMIZE TABLE queries 
Because of BlockerInThread in
MergeTreeDataPartWriterOnDisk::calculateAndSerializePrimaryIndex memory
was accounted incorrectly and grows constantly.

And IIUC there is no need in that blocker, since INSERT SELECT shares
the same thread group.
 2021-01-12 00:49:40 +03:00
alesapin
e3a50d13a5 Add missed header 2021-01-11 17:51:23 +03:00
alesapin
b1c4e5b9f2 Fix mutations text escaping 2021-01-11 17:43:05 +03:00
fastio
8dde70b937 Bugfix: attach partition should reset the mutation 2021-01-11 21:26:43 +08:00
alesapin
c5df8f324c More checks in writer wide 2021-01-11 15:03:00 +03:00
tavplubix
213c653ef8
Merge pull request #17656 from bharatnc/ncb/ddl-worker-queue-table 
add  system.distributed_ddl_queue table
 2021-01-11 14:59:26 +03:00
Anton Popov
91dc347ff3 add some comments 2021-01-11 14:36:31 +03:00
kssenii
38a9cba850 Fix 2021-01-11 10:55:38 +00:00