thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-10-05 08:00:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
47,227 Commits 8,978 Branches 1,768 Tags 1.9 GiB
f9dd4cc98d
Commit Graph

1389 Commits

Author SHA1 Message Date
Ivan Babrou
8784994d65 Allow conditions outside of PK with exact range 
Conditions that are outside of PK are marked as `unknown` in `KeyCondition`,
so it's safe to allow them, as long as they are always combined by `AND`.
 2020-07-11 18:59:26 -07:00
Azat Khuzhin
3bee98c6f0 Fix lifetime_bytes/lifetime_rows for Buffer direct block write 2020-07-12 01:16:05 +03:00
Ivan Babrou
d9d8d0242e Optimize PK lookup for queries that match exact PK range 
Existing code that looks up marks that match the query has a pathological
case, when most of the part does in fact match the query.

The code works by recursively splitting a part into ranges and then discarding
the ranges that definitely do not match the query, based on primary key.

The problem is that it requires visiting every mark that matches the query,
making the complexity of this sort of look up O(n).

For queries that match exact range on the primary key, we can find
both left and right parts of the range with O(log 2) complexity.

This change implements exactly that.

To engage this optimization, the query must:

* Have a prefix list of the primary key.
* Have only range or single set element constraints for columns.
* Have only AND as a boolean operator.

Consider a table with `(service, timestamp)` as the primary key.

The following conditions will be optimized:

* `service = 'foo'`
* `service = 'foo' and timestamp >= now() - 3600`
* `service in ('foo')`
* `service in ('foo') and timestamp >= now() - 3600 and timestamp <= now`

The following will fall back to previous lookup algorithm:

* `timestamp >= now() - 3600`
* `service in ('foo', 'bar') and timestamp >= now() - 3600`
* `service = 'foo'`

Note that the optimization won't engage when PK has a range expression
followed by a point expression, since in that case the range is not continuous.

Trace query logging provides the following messages types of messages,
each representing a different kind of PK usage for a part:

```
Used optimized inclusion search over index for part 20200711_5710108_5710108_0 with 9 steps
Used generic exclusion search over index for part 20200711_5710118_5710228_5 with 1495 steps
Not using index on part 20200710_5710473_5710473_0
```

Number of steps translates to computational complexity.

Here's a comparison for before and after for a query over 24h of data:

```
Read 4562944 rows, 148.05 MiB in 45.19249672 sec.,   100966 rows/sec.,   3.28 MiB/sec.
Read 4183040 rows, 135.78 MiB in 0.196279627 sec., 21311636 rows/sec., 691.75 MiB/sec.
```

This is especially useful for queries that read data in order
and terminate early to return "last X things" matching a query.

See #11564 for more thoughts on this.
 2020-07-11 12:26:54 -07:00
Denis Glazachev
edb6ef8c09 Merge commit 'ceac649c01b0158090cd271776f3219f5e7ff57c' into ldap-per-user-authentication 
* commit 'ceac649c01b0158090cd271776f3219f5e7ff57c': (75 commits)
  [docs] split misc statements (#12403)
  Update 00405_pretty_formats.reference
  Update PrettyCompactBlockOutputFormat.cpp
  Update PrettyBlockOutputFormat.cpp
  Update DataTypeNullable.cpp
  Update 01383_remote_ambiguous_column_shard.sql
  add output_format_pretty_grid_charset setting in docs
  add setting output_format_pretty_grid_charset
  Added a test for #11135
  Update index.md
  RIGHT and FULL JOIN for MergeJoin (#12118)
  Update MergeTreeIndexFullText.cpp
  restart the tests
  [docs] add syntax highlight (#12398)
  query fuzzer
  Fix std::bad_typeid when JSON functions called with argument of wrong type.
  Allow typeid_cast() to cast nullptr to nullptr.
  fix another context-related segfault
  [security docs] actually, only admins can create advisories
  query fuzzer
  ...
 2020-07-11 21:32:36 +04:00
Azat Khuzhin
32a45d0dee Implement lifetime_rows/lifetime_bytes for Buffer engine 
Buffer engine is usually used on INSERTs, but right now there is no way
to track number of INSERTed rows per-table, since only summary metrics
exists:
- StorageBufferRows
- StorageBufferBytes

But it can be pretty useful to track INSERTed rows rate (and it can be
exposed via http_handlers for i.e. prometheus)
 2020-07-11 16:06:11 +03:00
Azat Khuzhin
433fdffc19 Add lifetime_rows/lifetime_bytes interface (exported via system.tables) 2020-07-11 15:33:11 +03:00
Azat Khuzhin
84c93a6b02 Tiny IStorage refactoring 2020-07-11 15:17:06 +03:00
alexey-milovidov
e22547c29d
Merge pull request #12388 from ClickHouse/bloom-filter-arg-check 
Check arguments of bloom filter index
 2020-07-10 20:54:16 +03:00
alexey-milovidov
caef1d8e24
Update MergeTreeIndexFullText.cpp 2020-07-10 20:53:58 +03:00
alexey-milovidov
d819624d7c
Merge pull request #12378 from ClickHouse/allow-clear-column-with-dependencies 
Allow to CLEAR column even if there are depending DEFAULT expressions
 2020-07-10 20:18:14 +03:00
alexey-milovidov
031c773260
Merge pull request #12384 from ClickHouse/support-negative-float-constants-in-key-condition 
Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables
 2020-07-10 20:16:35 +03:00
Azat Khuzhin
610382b693 kafka: fix SIGSEGV if there is an message with error in the middle of the batch 
ReadBufferFromKafkaConsumer does not handle the case when there is
message with an error on non first position in the current batch, since
it goes through messages in the batch after poll and stop on first valid
message.

But later it can try to use message as valid:
- while storing offset
- get topic name
- ...

And besides the message itself is also invalid (you can find this in the
gdb traces below).

So just filter out messages win an error error after poll.

SIGSEGV was with the following stacktrace:
    (gdb) bt
    3  0x0000000010f05b4d in rd_kafka_offset_store (app_rkt=0x0, partition=0, offset=0) at ../contrib/librdkafka/src/rdkafka_offset.c:656
    4  0x0000000010e69657 in cppkafka::Consumer::store_offset (this=0x7f2015210820, msg=...) at ../contrib/cppkafka/include/cppkafka/message.h:225
    5  0x000000000e68f208 in DB::ReadBufferFromKafkaConsumer::storeLastReadMessageOffset (this=0x7f206a136618) at ../contrib/libcxx/include/iterator:1508
    6  0x000000000e68b207 in DB::KafkaBlockInputStream::readImpl (this=0x7f202c689020) at ../src/Storages/Kafka/KafkaBlockInputStream.cpp:150
    7  0x000000000dd1178d in DB::IBlockInputStream::read (this=this@entry=0x7f202c689020) at ../src/DataStreams/IBlockInputStream.cpp:60
    8  0x000000000dd34c0a in DB::copyDataImpl<> () at ../src/DataStreams/copyData.cpp:21
    9  DB::copyData () at ../src/DataStreams/copyData.cpp:62
    10 0x000000000e67c8f2 in DB::StorageKafka::streamToViews () at ../contrib/libcxx/include/memory:3823
    11 0x000000000e67d218 in DB::StorageKafka::threadFunc () at ../src/Storages/Kafka/StorageKafka.cpp:488

And some information from it:

    (gdb) p this.current.__i
    $14 = (std::__1::__wrap_iter<cppkafka::Message const*>::iterator_type) 0x7f1ca8f58660

    # current-1
    (gdb) p $14-1
    $15 = (const cppkafka::Message *) 0x7f1ca8f58600
    (gdb) p $16.handle_
    $17 = {__ptr_ = {<std::__1::__compressed_pair_elem<rd_kafka_message_s*, 0, false>> = { __value_ = 0x7f203577f938}, ...}
    (gdb) p *(rd_kafka_message_s*)0x7f203577f938
    $24 = {err = RD_KAFKA_RESP_ERR__TRANSPORT, rkt = 0x0, partition = 0, payload = 0x7f202f0339c0, len = 63, key = 0x0, key_len = 0, offset = 0, _private = 0x7f203577f8c0}

    # current
    (gdb) p $14-0
    $28 = (const cppkafka::Message *) 0x7f1ca8f58660
    (gdb) p $28.handle_.__ptr_
    $29 = {<std::__1::__compressed_pair_elem<rd_kafka_message_s*, 0, false>> = { __value_ = 0x7f184f129bf0}, ...}
    (gdb) p *(rd_kafka_message_s*)0x7f184f129bf0
    $30 = {err = RD_KAFKA_RESP_ERR_NO_ERROR, rkt = 0x7f1ed44fe000, partition = 1, payload = 0x7f1fc9bc6036, len = 242, key = 0x0, key_len = 0, offset = 2394853582209,

    # current+1
    (gdb) p (*($14+1)).handle_.__ptr_
    $44 = {<std::__1::__compressed_pair_elem<rd_kafka_message_s*, 0, false>> = { __value_ = 0x7f184f129d30}, ...}
    (gdb) p *(rd_kafka_message_s*)0x7f184f129d30
    $45 = {err = RD_KAFKA_RESP_ERR_NO_ERROR, rkt = 0x7f1ed44fe000, partition = 1, payload = 0x7f1fc9bc612f, len = 31, key = 0x0, key_len = 0, offset = 2394853582210,
      _private = 0x7f184f129cc0}

    # distance from the beginning
    (gdb) p messages.__end_-messages.__begin_
    $34 = 65536
    (gdb) p ($14-0)-messages.__begin_
    $37 = 8965
    (gdb) p ($14-1)-messages.__begin_
    $38 = 8964

    # parsing info
    (gdb) p allowed
    $39 = false
    (gdb) p new_rows
    $40 = 1
    (gdb) p total_rows
    $41 = 8964

    # current buffer is invalid
    (gdb) p *buffer.__ptr_
    $50 = {<DB::ReadBuffer> = {<DB::BufferBase> = {pos = 0x7f202f0339c0 "FindCoordinator response error: Local: Broker transport failure", bytes = 47904863385, working_buffer = {
            begin_pos = 0x7f202f0339c0 "FindCoordinator response error: Local: Broker transport failure",
            end_pos = 0x7f202f0339c0 "FindCoordinator response error: Local: Broker transport failure"}, internal_buffer = {

v0: check message errors in ReadBufferFromKafkaConsumer::nextImpl() (but
this may lead to using of that messages after and SIGSEGV again, doh).
v2: skip messages with an error after poll.
 2020-07-10 11:41:44 +03:00
Alexey Milovidov
47eaffbe63 Additional checks 2020-07-10 11:21:40 +03:00
Alexey Milovidov
4b86f36d37 Check arguments of bloom filter index 2020-07-10 11:13:21 +03:00
alesapin
5cae87e664
Merge pull request #12335 from ClickHouse/fix_alter_exit_codes 
Fix alter rename error messages
 2020-07-10 11:05:20 +03:00
Peng Jian
958bb1f3aa COS (Tencent Cloud Object Storage) integration 
To allow import and export to COS.
 2020-07-10 15:26:55 +08:00
Alexey Milovidov
276b3a0215 Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables #11905 2020-07-10 09:30:49 +03:00
Alexey Milovidov
a4b35a8a6f Allow to CLEAR column even if there are depending DEFAULT expressions #12333 2020-07-10 08:54:35 +03:00
alexey-milovidov
c16d8e094b
Merge pull request #12308 from ClickHouse/fix-codec-bad-exception-code 
Fix wrong exception code in codecs Delta, DoubleDelta #12110
 2020-07-10 08:40:46 +03:00
Alexey Milovidov
afc00fa0b8 Merge branch 'master' into fix-codec-bad-exception-code 2020-07-10 04:12:24 +03:00
Alexey Milovidov
7fc90aa070 Fix error 2020-07-10 02:45:29 +03:00
Alexey Milovidov
ad6fcd57b2 Merge branch 'master' into fix-race-condition-replicated-merge-tree-queue 2020-07-10 02:21:24 +03:00
alesapin
0156f43ed3 Human readable errors in alter rename queries 2020-07-09 17:30:38 +03:00
alesapin
9dea4ab323 Initial version 2020-07-09 17:14:44 +03:00
Vladimir Chebotarev
faf6be6576
Implemented single part uploads for DiskS3 (#12026) 
* Implemented single part uploads for DiskS3.
* Added `min_multi_part_upload_size` to disk configuration.
 2020-07-09 17:09:17 +03:00
Alexander Tokmakov
445acd0294 Merge branch 'master' into database_atomic_improvements 2020-07-09 16:02:54 +03:00
Denis Glazachev
5a4762bb8a
Merge branch 'master' into ldap-per-user-authentication 2020-07-09 14:50:02 +04:00
alesapin
47f05dcadd
Merge pull request #12304 from CurtizJ/fix-ttl-rename 
Fix TTL after renaming column.
 2020-07-09 13:06:27 +03:00
Alexey Milovidov
39e79cfe3e Merge branch 'master' into fix-codec-bad-exception-code 2020-07-09 07:50:18 +03:00
Alexey Milovidov
6c458a6287 Fix race condition in ReplicatedMergeTreeQueue 2020-07-09 07:43:28 +03:00
alexey-milovidov
827990d681
Merge pull request #12305 from ClickHouse/fix-virtual-columns-filter 
Fix virtual columns filter
 2020-07-09 07:06:07 +03:00
Alexey Milovidov
ea970fd57c Remove bad ugliness 2020-07-09 04:00:16 +03:00
Alexey Milovidov
b78e1145e8 Fix filtering by virtual columns #12166 2020-07-09 02:52:57 +03:00
Anton Popov
0e4871eec2 fix TTL after renaming column 2020-07-09 02:25:31 +03:00
Alexey Milovidov
eaa13d8176 Fix strange code CC @Enmk. Prove: g++ -xc++ -include vector - <<< 'int main() { return std::vector<const char>{1, 2, 3}.size(); }' 2020-07-09 02:04:42 +03:00
Alexander Tokmakov
9c4efa1f36 fix, add test 2020-07-09 01:50:15 +03:00
tavplubix
33c3545125
Merge pull request #12197 from nikitamikhaylov/csv-file-bugfix 
Headers for CSVWithNames in StorageFile
 2020-07-08 20:42:49 +03:00
alesapin
84f8bf1e14
Merge pull request #10697 from CurtizJ/polymorphic-parts 
Polymorphic parts (in-memory format)
 2020-07-08 13:25:45 +03:00
alexey-milovidov
ffdd6e1af6
Merge pull request #12173 from nikitamikhaylov/summing-bug-fix 
Exclude partition key columns from SummingMergeTree
 2020-07-08 01:46:12 +03:00
Denis Glazachev
9effacfbc1 Merge branch 'master' into ldap-per-user-authentication 
* master: (1102 commits)
  Update README.md
  Update README.md
  Update README.md
  Update index.md
  [docs] add intrdocution for statements page (#12189)
  Revert "Run perf tests with memory sampling (for allocations >1M)"
  Sanitize LINK_LIBRARIES property for the directories (#12160)
  [docs] refactor Domains overview (#12186)
  DOCS-647: toStartOfSecond (#12190)
  [docs] add intrdocution for commercial page (#12187)
  DOCSUP-1348 Russian translation for new functions (#133) (#12194)
  changelog fixes
  Update index.md (#12191)
  Update zh kafka.md title (#12192)
  Added test for #3767
  style fix for #12152
  Tests for fixed issues #10846 and #7347
  changelog fixes
  [docs] introduction for special table engines (#12170)
  [docs] introduction for third-party interfaces (#12175)
  ...

# Conflicts:
#	src/Access/ya.make
#	src/Common/ErrorCodes.cpp
 2020-07-08 00:42:09 +04:00
alexey-milovidov
546fcc9ed6
Merge pull request #12151 from ClickHouse/fix-transform-query-aliases 
Fix transform query for external databases in presense of aliases
 2020-07-07 21:46:35 +03:00
Nikita Mikhaylov
270551e599 done 2020-07-07 14:45:20 +03:00
Anton Popov
416d9648c2 remove questionable functionality 2020-07-07 12:58:14 +03:00
tavplubix
488a8cbc9f
Merge pull request #12120 from filimonov/query_context_for_system_logs 
Add query context for system logs and to Buffer
 2020-07-07 09:55:20 +03:00
Mikhail Filimonov
93517b4e82 Same change for Kafka - just in case, and to make it conform. 2020-07-06 20:48:33 +02:00
alesapin
3f90f65678 Merge remote-tracking branch 'origin' into CurtizJ-polymorphic-parts 2020-07-06 21:34:17 +03:00
Nikita Mikhaylov
2b539e6ace fix style 2020-07-06 20:26:15 +03:00
Mikhail Filimonov
8038383f06 Fix #10437, CR fixes 2020-07-06 19:24:33 +02:00
Nikita Mikhaylov
d31ed58f01 done 2020-07-06 17:33:31 +03:00
Alexander Tokmakov
d1be5ec641 fix symlinks, add short syntax of attach database 2020-07-06 16:41:43 +03:00
Alexander Kuzmenkov
d648628a4b
Merge pull request #12075 from azat/StorageKafka-SIGSEGV-fix 
Fix SIGSEGV in StorageKafka on DROP TABLE
 2020-07-06 13:56:06 +03:00
Alexey Milovidov
95a7a09c37 Fix transform query for external databases in presense of aliases #12032 2020-07-06 04:50:45 +03:00
alexey-milovidov
ea0fb005fb
Merge pull request #12116 from vitlibar/fix-table-dictionary-dependency 
Fix handling dictionary-table dependency
 2020-07-06 04:19:43 +03:00
alexey-milovidov
eecf7ae3ee
Merge pull request #12140 from ClickHouse/fix-division 
Fix potential overflow in integer division
 2020-07-05 19:00:41 +03:00
myrrc
8c3417fbf7
ILIKE operator (#12125) 
* Integrated CachingAllocator into MarkCache

* fixed build errors

* reset func hotfix

* upd: Fixing build

* updated submodules links

* fix 2

* updating grabber allocator proto

* updating lost work

* updating CMake to use concepts

* some other changes to get it building (integration into MarkCache)

* further integration into caches

* updated Async metrics, fixed some build errors

* and some other errors revealing

* added perfect forwarding to some functions

* fix: forward template

* fix: constexpr modifier

* fix: FakePODAllocator missing member func

* updated PODArray constructor taking alloc params

* fix: PODArray overload with n restored

* fix: FakePODAlloc duplicating alloc() func

* added constexpr variable for alloc_tag_t

* split cache values by allocators, provided updates

* fix: memcpy

* fix: constexpr modifier

* fix: noexcept modifier

* fix: alloc_tag_t for PODArray constructor

* fix: PODArray copy ctor with different alloc

* fix: resize() signature

* updating to lastest working master

* syncing with 273267

* first draft version

* fix: update Searcher to case-insensitive

* added ILIKE test

* fixed style errors, updated test, split like and ilike,  added notILike

* replaced inconsistent comments

* fixed show tables ilike

* updated missing test cases

* regenerated ya.make

* Update 01355_ilike.sql

Co-authored-by: myrrc <me-clickhouse@myrrec.space>
Co-authored-by: alexey-milovidov <milovidov@yandex-team.ru>
 2020-07-05 18:57:59 +03:00
alexey-milovidov
cb126a23ef
Update StorageDictionary.h 2020-07-05 18:45:05 +03:00
alexey-milovidov
1a760e18a5
Update StorageDictionary.h 2020-07-05 18:44:28 +03:00
Vitaly Baranov
99e9b15e00 Make code clearer: use enum instead of bool internal. 2020-07-05 17:14:29 +03:00
Alexey Milovidov
73a5c38398 Fix potential overflow in integer division #12119 2020-07-05 03:29:03 +03:00
alexey-milovidov
c696354dd4
Merge pull request #12115 from ianton-ru/disk-type-in-system-disks 
Add type column in system.disks
 2020-07-05 01:04:17 +03:00
alexey-milovidov
e05283d5d3
Merge pull request #12060 from ClickHouse/fix-12053 
Check wrong type for filter.
 2020-07-04 23:46:34 +03:00
Anton Popov
73676f5022
Improve performace of reading in order of sorting key. (#11696) 
* simplify reading in order of sorting key

* add perf test for reading many parts

* Revert "simplify reading in order of sorting key"

This reverts commit 7267d7c46e.

* add threshold for preliminary merge for reading in order

* better threshold

* limit threads in test
 2020-07-04 15:48:51 +03:00
alesapin
a2b6d58053
Merge pull request #11069 from kssenii/add-storage-rabbitmq-read-only 
Add storage RabbitMQ
 2020-07-04 10:38:50 +03:00
alesapin
bfa8cdc7a4
Move skip lists for tests to skip_list.json file (#12107) 
* Add unbundled mode flag

* Move skip lists to clickhouse-test

* More verbose message about skip

* Make skip-list optional

* Update clickhouse-test
 2020-07-04 10:35:45 +03:00
alesapin
8dc204350f
Fix version column in replicated version collapsing merge tree (#12121) 2020-07-04 10:35:17 +03:00
alexey-milovidov
6677e0a503
Merge pull request #12073 from zhang2014/feature/atfer_column_modify_column 
ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN
 2020-07-03 22:54:05 +03:00
Vitaly Baranov
ab343132de Fix handling dependency of table with ENGINE=Dictionary on dictionary. 2020-07-03 22:05:53 +03:00
Anton Ivashkin
23b44ca6fe Add type column in system.disks 2020-07-03 15:45:54 +03:00
alesapin
0a6f3ca9fd Add unbundled mode flag 2020-07-03 13:11:16 +03:00
alesapin
c57edd2018 Tiny fixes 2020-07-03 11:02:35 +03:00
Azat Khuzhin
bd5e5e9462 kafka: remove outdated comment 
As stated by @filimonov it is not relevant (after #11599)
 2020-07-03 09:23:39 +03:00
Azat Khuzhin
de8bc99d48 kafka: avoid superior polling after DROP/DETACH TABLE 
Before this patch isStalled() was checked before polledDataUnusable(),
and after DROP TABLE isStalled() == true (although this looks tricky).
 2020-07-03 09:23:39 +03:00
Azat Khuzhin
8508200020 kafka: improve logging during engine shutdown 
This will help with tracking possible issues, when you need to know was
buffer released or not.
 2020-07-03 09:23:39 +03:00
Azat Khuzhin
e7179c402c kafka: check that the data is still usable after parsing 2020-07-03 09:23:38 +03:00
Azat Khuzhin
6104872cae kafka: fix SIGSEGV on DROP TABLE 
After #11599 it is possible that messages of the
ReadBufferFromKafkaConsumer will be cleaned-up right in
read_kafka_message callback (from KafkaBlockInputStream) if the stop
flag isset (i.e. DROP TABLE is waiting the consumer), and if
read_kafka_message already processed some rows it will not return 0 and
the loop after will try to get current topic from the buffer, which uses
messages in the underlying and this will got SIGSEGV:

    12:14:56.173262 [ 55421 ] {f7930856-d478-4e41-af56-24ce7b693e95} <Debug> executeQuery: (from 0.0.0.0:0, user: ) DROP TABLE IF EXISTS data.queue
    12:14:56.173285 [ 55421 ] {f7930856-d478-4e41-af56-24ce7b693e95} <Trace> StorageKafka (newly_queue): Waiting for cleanup
    12:14:56.180016 [ 55390 ] {} <Trace> BaseDaemon: Received signal 11
    12:14:56.180267 [ 4914 ] {} <Fatal> BaseDaemon: ########################################
    12:14:56.181879 [ 4914 ] {} <Fatal> BaseDaemon: (version 20.6.1.1, build id: 4CE0298F08583658) (from thread 55468) (no query) Received signal Segmentation fault (11)
    12:14:56.181900 [ 4914 ] {} <Fatal> BaseDaemon: Address: 0x8 Access: read. Address not mapped to object.
    12:14:56.181909 [ 4914 ] {} <Fatal> BaseDaemon: Stack trace:
    12:14:56.184676 [ 4914 ] {} <Fatal> BaseDaemon: 3. /ch/contrib/cppkafka/include/cppkafka/message.h:111: DB::KafkaBlockInputStream::readImpl() @ 0xe343f1c in /usr/lib/debug/usr/bin/clickhouse
    12:14:56.185553 [ 4914 ] {} <Fatal> BaseDaemon: 4. /ch/contrib/libcxx/include/vector:1003: DB::IBlockInputStream::read() @ 0xd9d95bd in /usr/lib/debug/usr/bin/clickhouse
    12:14:56.188238 [ 4914 ] {} <Fatal> BaseDaemon: 5. /ch/src/DataStreams/copyData.cpp:26: DB::copyData() @ 0xd9f712a in /usr/lib/debug/usr/bin/clickhouse
    12:14:56.188780 [ 4914 ] {} <Fatal> BaseDaemon: 6. /ch/contrib/libcxx/include/vector:1532: DB::StorageKafka::streamToViews() @ 0xe335e73 in /usr/lib/debug/usr/bin/clickhouse
    12:14:56.189331 [ 4914 ] {} <Fatal> BaseDaemon: 7. /ch/src/Storages/Kafka/StorageKafka.cpp:491: DB::StorageKafka::threadFunc() @ 0xe336738 in /usr/lib/debug/usr/bin/clickhouse

55421 thread (shows that it still waiting for deactivation):

    5  std::__1::lock_guard<>::lock_guard () at ../contrib/libcxx/include/__mutex_base:90
    6  DB::BackgroundSchedulePoolTaskInfo::deactivate (this=0x7fc7e4465f20) at ../src/Core/BackgroundSchedulePool.cpp:59
    7  DB::StorageKafka::shutdown (this=0x7fc7e45e4600) at ../contrib/libcxx/include/memory:3821

And just in case thread where read_kafka_message is called:

    0  DB::ReadBufferFromKafkaConsumer::nextImpl (this=0x7fd4901d4118) at ../contrib/libcxx/include/atomic:1491
    1  DB::ReadBuffer::next (this=0x7fd4901d4118) at ../src/IO/ReadBuffer.h:59
    2  DB::ReadBuffer::eof (this=0x7fd4901d4118) at ../src/IO/ReadBuffer.h:81
    3  DB::skipWhitespaceIfAny (buf=...) at ../src/IO/ReadHelpers.h:945
    4  DB::JSONEachRowRowInputFormat::readRow (ext=..., columns=..., this=0x7fd499a7a020) at ../src/Processors/Formats/Impl/JSONEachRowRowInputFormat.cpp:222
    5  DB::JSONEachRowRowInputFormat::readRow (this=0x7fd499a7a020, columns=..., ext=...) at ../src/Processors/Formats/Impl/JSONEachRowRowInputFormat.cpp:218
    6  DB::IRowInputFormat::generate (this=0x7fd499a7a020) at ../src/Processors/Formats/IRowInputFormat.cpp:64
    7  DB::ISource::work (this=0x7fd499a7a020) at ../src/Processors/ISource.cpp:48
    8  DB::KafkaBlockInputStream::<lambda()>::operator()(void) const () at ../contrib/libcxx/include/memory:3826
    9  DB::KafkaBlockInputStream::readImpl (this=0x7fd46e718820) at ../contrib/libcxx/include/new:340

Cc: @filimonov
 2020-07-03 09:22:44 +03:00
Alexander Tokmakov
383c583bdd add uuid to atomic databases 2020-07-02 23:39:31 +03:00
Anton Popov
4422df2e37 Merge remote-tracking branch 'upstream/master' into HEAD 2020-07-02 20:18:21 +03:00
alesapin
c6c7ee30d9 Less race conditions 2020-07-02 19:44:04 +03:00
alesapin
96df2e6b71 Better shutdown and conversion 2020-07-02 17:35:10 +03:00
alexey-milovidov
82061da77c
Merge pull request #12040 from azat/dist-fixes 
Tiny fixes for distributed queries
 2020-07-02 17:02:56 +03:00
alesapin
64583ceb22 Merge branch 'master' into add-storage-rabbitmq-read-only 2020-07-02 16:08:29 +03:00
Anton Popov
71059e4dff
Merge pull request #12062 from nvartolomei/nv/set-index-tuple-types 
Try fix pk in tuple performance
 2020-07-02 15:51:03 +03:00
Nikolai Kochetov
79cd33a567 Fix tests. 2020-07-02 10:44:47 +03:00
Alexander Kazakov
cc2cadb7f5
Merge pull request #11893 from excitoon-favorites/autostartmoves 
In *MergeTree: Parts moving task shall be started if new storage policy needs them
 2020-07-02 10:44:38 +03:00
zhang2014
9ed5c6d925 ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN 2020-07-01 22:58:52 +08:00
Nicolae Vartolomei
3854ce6d84 Rewrite Set lookup to make it more readable 2020-07-01 15:05:54 +01:00
Anton Popov
68c9896fa2 fix arcadia build 2020-07-01 00:15:39 +03:00
Anton Popov
d45697a1b5 in-memory-parts: lazy creation of WAL 2020-06-30 21:47:12 +03:00
Nikolai Kochetov
3585700506 Fix tests. 2020-06-30 19:41:43 +03:00
Vitaly Baranov
c39eb8f71b Fix partial revokes (complex cases). 2020-06-30 18:47:02 +03:00
Nicolae Vartolomei
8f1845185e Try fix pk in tuple performance 
Possible approach for fixing #10574

The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).

However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.

The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.
 2020-06-30 16:33:38 +01:00
Nikolai Kochetov
66da0733ab Check type of filter for prewhere. 2020-06-30 17:20:27 +03:00
Vladimir Chebotarev
bb8da71eff Moves task shall be started if new storage policy needs them. 2020-06-30 15:36:44 +03:00
kssenii
f797efb04c Fix build 2020-06-30 01:48:11 +00:00
Anton Popov
53e955c6dd several fixes 2020-06-29 23:36:18 +03:00
Azat Khuzhin
9c94993f14 Fix "Sharding key is not deterministic" message 2020-06-29 23:00:14 +03:00
kssenii
fd9b416871 Fix and simplify code 2020-06-29 15:47:25 +00:00
Anton Popov
a43cb93be5 remove questionable functionality 2020-06-29 18:46:50 +03:00
alesapin
718262e7ce
Merge pull request #11973 from ClickHouse/alter-enum-partition-key 
Allow to perform "metadata-only" ALTER of partition key.
 2020-06-29 16:01:07 +03:00
kssenii
88ece429c9 Move writing to RabbitMQ into background task 2020-06-29 12:33:01 +00:00
kssenii
3d2cc9d4b2 Remove libevent library 2020-06-29 12:11:17 +00:00
kssenii
5fc0b93400 Use ConcurentBoundedQueue instead of vector 2020-06-29 09:50:33 +00:00
alesapin
e72bc39654
Merge pull request #11983 from ClickHouse/storage_mongodb 
Merging #10931 (Storage MongoDB)
 2020-06-29 11:49:18 +03:00
Alexey Milovidov
cc597bb30b Implement for ReplicatedMergeTree 2020-06-28 22:39:31 +03:00
Alexey Milovidov
6aa89d9073 Typos 2020-06-28 22:39:31 +03:00
Alexey Milovidov
e3410ca617 Write partition key changes to ZooKeeper 2020-06-28 22:39:31 +03:00
Alexey Milovidov
e5f4a4b4b0 Whitespace 2020-06-28 22:39:31 +03:00
Alexey Milovidov
1f9c27de34 Fix clang-tidy 2020-06-28 22:39:31 +03:00
Alexey Milovidov
b6e31f0a5e Only allow Enum extension within the same width 2020-06-28 22:39:31 +03:00
Alexey Milovidov
e4d8cc2b44 Slightly more safe 2020-06-28 22:39:31 +03:00
Alexey Milovidov
97ad23b905 Allow to ALTER partition key in some cases 2020-06-28 22:39:31 +03:00
Alexey Milovidov
678d127503 Whitespace 2020-06-28 22:39:31 +03:00
Alexey Milovidov
c4d5ca485e Remove the usage of obsolete setting 2020-06-28 22:39:31 +03:00
kssenii
649eb8e348 Move reading from RabbitMQ into background task 2020-06-28 18:20:18 +00:00
Alexey Milovidov
d1f4a0e473 Fix array size overflow in generateRandom 2020-06-28 17:33:02 +03:00
alexey-milovidov
935d2be826
Merge pull request #12003 from ClickHouse/fix-typos 
Fix typos in code
 2020-06-28 15:49:17 +03:00
alexey-milovidov
5a99cdb0d9
Merge pull request #11996 from BohuTANG/mysql_engine_enum 
ISSUES-3985 Support MySQL engine reading Enums type
 2020-06-27 22:18:25 +03:00
Alexey Milovidov
1462a66d1e Fix typos 2020-06-27 22:05:00 +03:00
Alexey Milovidov
46caa211d5 Better diagnostics of "Replica {} appears to be already active" message 2020-06-27 16:55:00 +03:00
BohuTANG
f609fd34f7 Support Enums type for MySQL engine #3985 2020-06-27 09:10:07 +08:00
alesapin
6b3092290d Trying to fix ya.make 2020-06-26 19:33:51 +03:00
alesapin
b5f79bc28c Remove strange header 2020-06-26 19:32:15 +03:00
alesapin
4969da85d9 Fixes 2020-06-26 17:28:00 +03:00
alesapin
11f88340a5 Merge branch 'mongo' of https://github.com/ageraab/ClickHouse into storage_mongodb 2020-06-26 16:03:06 +03:00
alesapin
9710a67704 Fix style check 2020-06-26 15:27:12 +03:00
alesapin
6caf2e7a3f Fix protocol check 2020-06-26 14:38:37 +03:00
alesapin
6f1824f0ea Correct merge with master 2020-06-26 14:30:23 +03:00
alesapin
e9c47dc89c Merge branch 'master' into CurtizJ-polymorphic-parts 2020-06-26 14:27:19 +03:00
alesapin
b3c31370c2
Merge pull request #11970 from ClickHouse/fix-estimation-of-the-number-of-marks 
Fix estimation of the number of marks while reading from MergeTree
 2020-06-26 11:28:36 +03:00
alexey-milovidov
2163716e45
Merge pull request #11955 from ClickHouse/revive-mmap-2 
Allow to use direct_io and mmap_io for secondary indices.
 2020-06-26 03:16:56 +03:00
alexey-milovidov
178a5217bc
Merge pull request #11953 from ClickHouse/revive-mmap 
Revive mmap IO + add a test
 2020-06-26 03:15:07 +03:00
alexey-milovidov
8f429f6027
Merge pull request #11935 from zlobober/allocated_bytes 
Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.
 2020-06-26 00:57:47 +03:00
Alexey Milovidov
8eed47857b Fix estimation of the number of marks for various thresholds 2020-06-25 23:20:22 +03:00
Alexey Milovidov
8872417d00 Respect direct_io/mmap settings while reading secondary indices 2020-06-25 22:31:54 +03:00
Alexey Milovidov
5608f15749 Revive mmap IO 2020-06-25 22:15:41 +03:00
alexey-milovidov
a34032cace
Merge pull request #11300 from blinkov/sentry 
Opt-in support for sending crash reports
 2020-06-25 17:20:50 +03:00
alesapin
36eb2c3028 Simplify code around locks 2020-06-25 13:25:22 +03:00
alesapin
d5847d29d6 Avoid memory leaks 2020-06-25 12:44:39 +03:00
Maxim Akhmedov
29d2928f93 Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}. 2020-06-25 00:27:53 +03:00
alesapin
addee61bcb Trying to do everything on top of libuv, add heartbeats 2020-06-25 00:14:49 +03:00
alesapin
cb30dbfe28 Correct merge with master 2020-06-24 20:32:57 +03:00
alesapin
3fc65b3269 Merge branch 'master' into kssenii-rabbit-mq 2020-06-24 20:14:28 +03:00
alesapin
63f8f89ca0
bump CI 2020-06-24 19:06:01 +03:00
alesapin
08f5ed6da3 Fix unitialized memory 2020-06-24 17:12:44 +03:00
Nikolai Kochetov
f7f16fdbf7
Merge pull request #11895 from ClickHouse/fix-nullable-prewhere-type-3 
Fix nullable prewhere type 3
 2020-06-24 14:32:08 +03:00
alexey-milovidov
18eb141ea1
Merge pull request #11715 from azat/dist-optimize_skip_unused_shards-fixes 
Control nesting level for shards skipping and disallow non-deterministic functions
 2020-06-24 12:54:58 +03:00
tavplubix
2401bc9143
Merge pull request #11896 from ClickHouse/merging_drop_replica 
Merging #10679
 2020-06-24 11:42:39 +03:00
Anton Popov
8c3f496826 add test with alters and in-memory parts 2020-06-24 01:16:29 +03:00
Ivan Blinkov
45d1ca2567 merge master 2020-06-24 00:18:16 +03:00
alexey-milovidov
33dfbdcbe9
Update StorageSystemTables.cpp 2020-06-23 22:28:26 +03:00
Alexander Tokmakov
7b5de16da9 minor improvements 2020-06-23 22:15:01 +03:00
Nikolai Kochetov
658a2d0dc0 Fix anotehr one nullable prewhere column. 2020-06-23 21:52:32 +03:00
amudong
2a51286527 Optimize code 2020-06-23 16:22:53 +08:00
alesapin
4185fe9975 Merge branch 'master' into atomic_metadata5 2020-06-23 11:13:05 +03:00
alesapin
292016e045 Add missed lock 2020-06-23 11:04:43 +03:00
amudong
cca8df6b78 format comment 2020-06-23 12:24:05 +08:00
amudong
0101947403 rewrite static StorageReplicatedMergeTree::dropReplica 2020-06-23 12:12:30 +08:00
amudong
7723dc4935 add drop relica database and the whole replica 
fix removeReplicaByZKPath

fix bug: add zkpath empty judge

fix: rewrite code

delete useless code.

fix:ast fromat

fix bug

add test_drop_replica

add drop_replica doc

add drop databse checkAccess

refactor dropReplica

update tests

add static method StorageReplicatedMergeTree::dropReplicaByZkPath

update doc and delete useless code

fix conflict

fix doc

fix doc

fix StorageReplicatedMergeTree::dropReplica

fix bug

delete useless code
 2020-06-23 12:12:30 +08:00
amudong
2e6a3eff8e fix: can not drop local table and replace ALTER with SYSTEM 
fix test_drop_replica

fix drop replica '/path/to/zk/' ending in '/' and update doc
 2020-06-23 12:12:30 +08:00
sundy-li
15ad830290 only drop inactive replica 
update doc

Increase timeout to release the zookeeper Ephemeral nodes

Fix code comment

use PartitionManager

make integrations test passed
 2020-06-23 12:12:30 +08:00
sundy-li
34df59baf8 Add tests and docs 
Fix integration-tests
 2020-06-23 12:12:30 +08:00
sundy-li
d4aa3cea4b fix merge state 
Fix style && build
 2020-06-23 12:12:30 +08:00
sundy-li
906a43e4a8 Add drop replica alter support 2020-06-23 12:12:30 +08:00
alexey-milovidov
a7b224a578
Merge pull request #11869 from ClickHouse/fix-nullable-prewhere-type-2 
Fix nullable prewhere type 2
 2020-06-23 02:48:59 +03:00
Anton Popov
78d28be8cf check in-memory parts, comments and style-fixes 2020-06-22 21:57:50 +03:00
robot-clickhouse
4bd47b8adc Auto version update to [20.6.1.1] [54436] 2020-06-22 20:40:26 +03:00
alesapin
ef89a6f728
Merge pull request #11786 from ClickHouse/fix_alter_with_materialized 
Fix alter key when materialized column passed
 2020-06-22 19:06:25 +03:00
Nikolai Kochetov
ae21aca3a3 Fix header for nullable prewhere column. 2020-06-22 19:01:59 +03:00
alesapin
c76b4b3481 Clang-tidy fixes 2020-06-22 18:51:11 +03:00
alesapin
f262c4f652 Better comments 2020-06-22 12:49:21 +03:00
alesapin
b1e8976df4 Merge with master 2020-06-22 12:04:27 +03:00
alesapin
b9e74f4e82 Merge branch 'master' into atomic_metadata5 2020-06-22 12:03:53 +03:00
Ivan Blinkov
2c0ff29c48 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-22 10:06:36 +03:00
Denis Glazachev
6c1b2a4831 Merge branch 'master' into ldap-per-user-authentication 
* master: (114 commits)
  Update alter.md (#11823)
  Update PODArray.h
  Added a test for #6549
  Update create.md
  Update create.md
  replacingmergetree translate into "替换合并树“ better (#11814)
  Whitespace
  Fix style
  Added a test
  Fix another 10% of bad code #11529
  Update join.md (#11798)
  Update memory.md (#11800)
  Whitespace
  Check for tests with "fail" in their names
  Fix 10% of bad code #11756
  Update PODArray.h
  avoid duplicate short system name
  adapt recent result as well
  [website] tune benchmark pages styling + add extra checks
  Added MSan suppression for Hyperscan
  ...

# Conflicts:
#	src/Common/ErrorCodes.cpp
 2020-06-20 22:01:12 +04:00
alexey-milovidov
9bfefc99ab
Merge pull request #11599 from filimonov/better_kafka_states 
Kafka work with formats based on PeekableReadBuffer and other improvements
 2020-06-20 20:36:54 +03:00
alexey-milovidov
076fe8e8ba
Update KafkaBlockInputStream.cpp 2020-06-20 20:34:42 +03:00
Alexey Milovidov
ad672f7ce2 Whitespace 2020-06-20 10:26:33 +03:00
Alexey Milovidov
b8769e1ada Fix style 2020-06-20 10:24:51 +03:00
alesapin
4c0879ae30 Better logging in storages 2020-06-19 20:17:13 +03:00
alesapin
c9fa5d2ec3 Better naming 2020-06-19 18:39:41 +03:00
alesapin
a2e7e9f232 Remove unused constructor and rename method 2020-06-19 18:21:48 +03:00
alesapin
3c47faa9da Move partition key initialization into registerMergeTree 2020-06-19 18:14:08 +03:00
Alexey Milovidov
468eaa8d93 Remove unused method 2020-06-19 17:21:26 +03:00
Alexey Milovidov
5211a42c04 Remove leader election, step 3 2020-06-19 17:18:58 +03:00
alesapin
016ee6316f Add missed check 2020-06-19 15:05:29 +03:00
alesapin
ce0c5faf9e Merge branch 'fix_alter_with_materialized' into atomic_metadata5 2020-06-19 14:00:32 +03:00
alesapin
4b6db63fff Fix alter key when materialized column passed 2020-06-19 13:53:20 +03:00
Mikhail Filimonov
9609bd9dee Kafka better states, formats based on PeekableReadBuffer, and other minor fixes. 
Add formats tests, fixes for JSONCompactEachRowWithNamesAndTypes, TSVWithNamesAndTypes. Some CR fixes
Add sanitizing for kafka_max_block_size and kafka_poll_max_batch_size
 2020-06-19 10:49:05 +02:00
Denis Glazachev
5db60202b6 Merge branch 'master' into ldap-per-user-authentication 2020-06-19 00:11:08 +04:00
alesapin
009977c20c Fix locks 2020-06-18 22:45:37 +03:00
Azat Khuzhin
d34e6217bc Add logging of adjusting conditional settings for distributed queries 2020-06-18 21:49:29 +03:00
Azat Khuzhin
041533eae2 Disable optimize_skip_unused_shards if sharding_key has non-deterministic func 
Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.
 2020-06-18 21:49:29 +03:00
alesapin
8b131e2079 Remove int contention 2020-06-18 20:19:11 +03:00
alesapin
c666763cc0 Remove unused method better 2020-06-18 20:09:06 +03:00
alesapin
e888dafdc2 Remove unused method 2020-06-18 20:03:42 +03:00
alesapin
b3ee8967dc Fix style 2020-06-18 19:28:20 +03:00
alesapin
54e5fe7dbc Less locks 2020-06-18 19:19:40 +03:00
alesapin
d79982f497 Better locks in Storages 2020-06-18 19:10:47 +03:00
alesapin
c8a58299ac Fix storage buffer metadata 2020-06-18 14:42:48 +03:00
alesapin
1a69c3234a Fix style 2020-06-18 14:09:55 +03:00
alesapin
d4c49816ab Really atomic metadata 2020-06-18 14:02:31 +03:00
alesapin
4de5331b0d Fix SystemTables 2020-06-18 13:39:33 +03:00
alesapin
aab4ce6394 Truncate with metadata 2020-06-18 13:29:13 +03:00
alesapin
35ce47951d Fix storage merge 2020-06-18 12:22:54 +03:00
alesapin
760e9a8488 Fix crash 2020-06-18 12:08:24 +03:00
alesapin
f4037b8f54 Fix build 2020-06-18 12:00:43 +03:00
alesapin
ebb36bec8a Merge branch 'master' into atomic_metadata5 2020-06-18 11:57:16 +03:00
alesapin
d0beeaf7c4
Merge pull request #11744 from ClickHouse/stop_fetches_before_metadata_alter 
Cancel fetches before metadata alter
 2020-06-18 11:32:16 +03:00
alesapin
870d891257 Cancel fetches before metadata alter 2020-06-17 23:25:43 +03:00
Azat Khuzhin
caa88777be Parse metadata stored in zookeeper before checking for equality 
After #11325 trailing whitespaces has been removed for data skipping
indicies, and it may be different, if you have multiple skip indices,
and in this case new server will not load such tables, because metadata
will be different.

Fix this by re-parse metadata in zookeeper.
 2020-06-17 21:03:22 +03:00
alesapin
dffdece350 getColumns in StorageInMemoryMetadta (only compilable) 2020-06-17 19:39:58 +03:00
alesapin
ef8781cce7 Better getVirtuals method 2020-06-17 17:37:21 +03:00
alesapin
33c27de54d Check methods in metadata 2020-06-17 17:32:25 +03:00
alesapin
31abbe5dbd Select query in metadata 2020-06-17 17:06:22 +03:00
alesapin
eaaef83742 Settings changes in StorageInMemoryMetadata 2020-06-17 16:46:01 +03:00
alesapin
ed8f3b2fc4 TTL in storage in memory metadata 2020-06-17 16:39:26 +03:00
Vitaly Baranov
7011401cd4
Merge pull request #11670 from vitlibar/rbac-improve-parser-multiple-entities 
RBAC: improve syntax
 2020-06-17 16:26:02 +03:00
alesapin
7064a366e2 Missed change for primary key 2020-06-17 15:40:27 +03:00
alesapin
1afdebeebd Primary key in storage metadata 2020-06-17 15:39:20 +03:00
alesapin
1da393b218 Sampling key in StorageInMemoryMetadata 2020-06-17 15:07:09 +03:00
alesapin
eca6e9087e Fix race condition 2020-06-17 14:52:19 +03:00
alesapin
ba04d02f1e Compilable sorting key in metadata 2020-06-17 14:05:11 +03:00
Nikita Mikhaylov
d2be6c036b
Merge pull request #11602 from ClickHouse/fix-buffer-sample 
Fix buffer sample
 2020-06-17 14:44:48 +04:00
Nikita Mikhaylov
ff0262626a
Merge pull request #11645 from azat/load-balancing-round-robin 
Add round_robin load_balancing
 2020-06-17 14:34:59 +04:00
alesapin
ab61abccc1 Partition key in StorageInMemoryMetadata 2020-06-17 13:34:23 +03:00
alesapin
62f2c17a66 Secondary indices in StorageInMemoryMetadata 2020-06-17 12:38:47 +03:00
alesapin
2fac3290f0 Merge branch 'master' into atomic_metadata5 2020-06-17 11:14:03 +03:00
alexey-milovidov
e07d02015c
Merge pull request #11661 from ClickHouse/return-not-nullable-from-count-distinct-2 
Return non-Nullable results from COUNT(DISTINCT), more complete.
 2020-06-17 09:25:45 +03:00
alexey-milovidov
598fd75caf
Merge pull request #11639 from ClickHouse/remove-leader-election-2 
Remove leader election, step 2: allow multiple leaders
 2020-06-17 09:25:23 +03:00
alexey-milovidov
6e61f708d7
Merge pull request #11690 from ClickHouse/odbc-bridge-timeouts 
Pass proper timeouts when communicating with XDBC bridge.
 2020-06-17 03:18:29 +03:00
Alexey Milovidov
a655765040 Merge branch 'master' into return-not-nullable-from-count-distinct-2 2020-06-17 01:15:57 +03:00
alexey-milovidov
f387752cf1
Merge pull request #11702 from ClickHouse/fix-mv-header 
Fix mv header
 2020-06-16 22:43:38 +03:00
Alexey Milovidov
0a6054eece Merge branch 'master' into return-not-nullable-from-count-distinct-2 2020-06-16 22:31:33 +03:00
Alexey Milovidov
b7ec5f94f2 Merge branch 'master' into odbc-bridge-timeouts 2020-06-16 22:14:43 +03:00
alesapin
fa60903620 Fix race condition 2020-06-16 21:41:11 +03:00
alesapin
d214a504d7
Merge pull request #11681 from ClickHouse/fix_mutations_premature_finish 
Fix mutations premature "finish"
 2020-06-16 21:26:14 +03:00
alesapin
ccc2bda666 getConstraints() in StorageInMemoryMetadata (suspicious commit, but pretend to work) 2020-06-16 19:55:04 +03:00
alesapin
1ddeb3d149 Buildable getSampleBlock in StorageInMemoryMetadata 2020-06-16 18:51:29 +03:00
alesapin
71f99a274d Compileable getSampleBlockWithColumns in StorageInMemoryMetadata 2020-06-16 17:25:08 +03:00
Ivan Blinkov
7e2bb1fd83 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-16 16:02:55 +03:00
alesapin
08b9aa6b2e getSampleBlockWithVirtuals in StorageInMemoryMetadata 2020-06-16 15:58:05 +03:00
alesapin
53cb5210de Move getSampleBlockNonMaterialized to StorageInMemoryMetadata 2020-06-16 15:48:10 +03:00
alesapin
0bcd22008a Get column dependencies in StorageInMemoryMetadata 2020-06-16 15:19:21 +03:00
alesapin
0f286ac133 Copy some methods to metdata 2020-06-16 15:03:27 +03:00
alesapin
eae901f58d Merge branch 'master' into atomic_metadata5 2020-06-16 14:44:45 +03:00
alesapin
0a33ccb77b Fix remove method 2020-06-16 14:12:52 +03:00
alesapin
d1052f2dfe Better logic 2020-06-16 13:50:47 +03:00
alesapin
d5b76b2fe1 Fix parts count in mutations 2020-06-16 13:34:59 +03:00
alesapin
0cb14c836c
Merge pull request #11691 from ClickHouse/fix-race-condition-system-sync-replicas 
Fix race condition in SYSTEM SYNC REPLICA
 2020-06-16 13:29:18 +03:00
alesapin
362aaf1385
Merge pull request #11516 from ClickHouse/consistent_metadata4 
Storage metadata as a single struct.
 2020-06-16 13:28:22 +03:00
alesapin
605c904fc6 Add method for compatibility 2020-06-16 11:39:12 +03:00
Nikolai Kochetov
66c0ef2610 Do not materialize header for merge tables. 2020-06-16 11:08:32 +03:00
Anton Popov
b19d48a11c Merge remote-tracking branch 'upstream/master' into HEAD 2020-06-16 06:37:55 +03:00
Alexey Milovidov
125b007360 Merge branch 'master' into remove-leader-election-2 2020-06-16 05:56:29 +03:00
Alexey Milovidov
22a92faab6 Avoid connection to replica when fetches are cancelled 2020-06-16 05:14:53 +03:00
Alexey Milovidov
53d985909e Fix race condition in SYSTEM SYNC REPLICA 2020-06-16 01:54:19 +03:00
Alexey Milovidov
186d336e31 Use proper timeouts when communicating with xdbc-bridge 2020-06-16 01:35:15 +03:00
Anton Popov
a3ac224ae4 in-memory parts: sync insert to replicated 2020-06-15 22:57:12 +03:00
Alexey Milovidov
d10109dc38 Remove excessive statement #11131 2020-06-15 22:18:33 +03:00
alesapin
36ba0192df Metadata in read and write methods of IStorage 2020-06-15 22:08:58 +03:00
Vitaly Baranov
15132d47c9 Slightly improve syntax of CREATE POLICY. 
Also rename column source=>storage in table system.row_policies.
 2020-06-15 22:07:42 +03:00
alesapin
b47a7327fd All set methods in metadata 2020-06-15 21:08:05 +03:00
alesapin
33a74a3ea0 TTL methods in StorageInMemoryMetadata 2020-06-15 20:50:53 +03:00
alesapin
e667eb57b2 Working set*Keys methods 2020-06-15 20:24:49 +03:00
alesapin
5fc41c7ecc Move set*Key methods to StorageInMemoryMetadata 2020-06-15 20:17:06 +03:00
Vitaly Baranov
4bd00b02e2 Improve syntax of CREATE QUOTA. Now resource types and key types could be written with underscores. 
Also rename columns key_type=>keys and source=>storage in table system.quotas.
 2020-06-15 20:10:34 +03:00
Vitaly Baranov
92b9f4a88d Rename ExtendedRoleSet => RolesOrUsersSet. 2020-06-15 20:10:34 +03:00
alesapin
aa30649ce5 Working setColumns, setConstraints, setIndices 2020-06-15 20:10:14 +03:00
alesapin
af2fe2ba55 Compilable setColumns, setConstraints, setIndices 2020-06-15 19:55:33 +03:00
alesapin
8f66c1b22c Merge with master 2020-06-15 17:31:37 +03:00
alesapin
8945d0073a Fix misunderstanding bug in mutations finalization 2020-06-15 16:37:40 +03:00
Artem Zuikov
24059efad5
Change push down logic in VIEW (#11513) 2020-06-15 15:36:10 +03:00
alesapin
2c9ce0f3fa
Bump CI 2020-06-15 13:14:36 +03:00
alesapin
bc9a3c0458 Merge branch 'master' into consistent_metadata4 2020-06-15 11:20:30 +03:00
alesapin
c20ce687cf
bump ci 2020-06-15 11:12:01 +03:00
alexey-milovidov
fd6115cccf
Merge pull request #11648 from ClickHouse/simplify-code-functions-comparison-3 
Allow comparison with String in index analysis; simplify code.
 2020-06-15 05:47:03 +03:00
Alexey Milovidov
d2c66f9688 Added comments 2020-06-15 05:14:59 +03:00
Alexey Milovidov
bbe5f4c909 Revert "Remove "current_password" because it is harmful" 
This reverts commit 66ccb2f6b1.
 2020-06-15 05:13:41 +03:00
Alexey Milovidov
5866401f60 Less noise in cleanup thread 2020-06-15 05:12:06 +03:00
Alexey Milovidov
66ccb2f6b1 Remove "current_password" because it is harmful 2020-06-15 04:12:01 +03:00
Alexey Milovidov
689b6901f8 Fix typo 2020-06-15 04:04:42 +03:00
Alexey Milovidov
1c438a133e Leader election both backward and forward compatible 2020-06-15 03:59:12 +03:00
Alexey Milovidov
b596831342 Merge branch 'master' into remove-leader-election-2 2020-06-15 03:16:41 +03:00
Alexey Milovidov
8dac30ae95 Split file for better build times 2020-06-14 21:42:10 +03:00
kssenii
9e1b8b2872 Better exchanges, fix build, better comments, better tests 2020-06-14 18:36:56 +00:00
Alexey Milovidov
f6c52fe1c2 Allow comparison with String in index analysis; simplify code #11630 2020-06-14 21:31:42 +03:00
alexey-milovidov
561d4b5b01
Merge pull request #11655 from ClickHouse/position-in-system-columns 
Added column "position" to system.columns and "column_position" to system.parts_columns
 2020-06-14 21:01:41 +03:00
Alexey Milovidov
394fb64a9c Better way of implementation 2020-06-14 20:42:11 +03:00
alexey-milovidov
87223b2cb7
Merge pull request #11652 from ClickHouse/allow-drop-and-rename-without-zookeeper 
Allow to drop table if there is no metadata in ZooKeeper; allow to rename.
 2020-06-14 09:00:38 +03:00
Alexey Milovidov
d6cf62e587 Added column "position" to system.columns and "column_position" to system.parts_columns 2020-06-14 04:56:22 +03:00
Alexey Milovidov
400e9fb64f Allow to drop table if there is no metadata in ZooKeeper; allow to rename 2020-06-14 04:23:53 +03:00
Azat Khuzhin
c139a05370 Forward declaration in StorageDistributed 2020-06-14 01:09:21 +03:00
kssenii
dcd7b7351c Support headers-exchange type 2020-06-13 21:47:22 +00:00
Alexey Milovidov
2cd82a25f5 Remove trivial count query optimization if row-level security is set #11352 2020-06-13 23:59:20 +03:00
kssenii
b8a4c7708a Make local exchanges unique for each table 2020-06-13 18:15:59 +00:00
alexey-milovidov
f53da4d36b
Merge pull request #11608 from ClickHouse/fix-nullable-prewhere-type 
Fix nullable prewhere type
 2020-06-13 20:29:12 +03:00
alesapin
3427da1c43 Fix incorrect usage of rows TTL 2020-06-13 13:39:25 +03:00
alesapin
18b58e8483 Revert too strict lock 2020-06-13 12:12:45 +03:00
alesapin
d636cdf4b0 Remove outdated comment 2020-06-13 11:55:03 +03:00
alesapin
31b852c46d Remove redundant locks 2020-06-13 11:53:40 +03:00
alesapin
fcfb6d3bc2 Merge with master 2020-06-13 11:51:07 +03:00
alesapin
5d3de747cd Merge remote-tracking branch 'origin/master' into consistent_metadata4 2020-06-13 11:50:27 +03:00
Alexey Milovidov
6f0db5ef10 Fix error 2020-06-13 04:35:49 +03:00
Alexey Milovidov
21897f2abd Instrument 2020-06-13 04:35:49 +03:00
Alexey Milovidov
6ff671b092 Improvement 2020-06-13 04:35:49 +03:00
Alexey Milovidov
ab00e34305 Miscellaneous 2020-06-13 04:35:49 +03:00
Alexey Milovidov
85c0706901 Step 2: allow multiple leaders 2020-06-13 04:35:49 +03:00
Alexey Milovidov
b8b55a5b99 More LeaderElection to Storage/MergeTree 2020-06-13 04:35:49 +03:00
Alexey Milovidov
52ac009754 Update versions on merge and mutation 2020-06-13 04:34:05 +03:00
Alexey Milovidov
833c8178ae Whitespaces 2020-06-13 04:34:05 +03:00
Alexey Milovidov
6869c122a8 Step 1 towards removing leader election: check and modify version of the "log" node 2020-06-13 04:34:05 +03:00
alexey-milovidov
54b77ff967
Merge pull request #11633 from ClickHouse/avoid-errors-using-zk-api 
Avoid errors due to implicit int<->bool conversions when using ZK API
 2020-06-13 04:32:32 +03:00
kssenii
462e8bcdc9 Support transactions for publishing 2020-06-12 18:10:13 +00:00
kssenii
3b0a3e00c0 Some fixes 2020-06-12 18:09:49 +00:00
alexey-milovidov
fc8da2abd3
Merge pull request #11613 from azat/system.zookeeper-path-prepend-slash 
Prepend slash for path while querying system.zookeeper
 2020-06-12 18:19:10 +03:00
Alexey Milovidov
72257061d5 Avoid errors due to implicit int<->bool conversions when using ZK API 2020-06-12 18:09:12 +03:00
Nikolai Kochetov
83155e139c Try fix tests. 2020-06-12 17:59:14 +03:00
alesapin
0edf5ff7a2 Fix race condition 2020-06-12 17:32:47 +03:00
Ivan Blinkov
e98a91e9f7 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-12 16:53:00 +03:00
alesapin
465c4b65b7 Slightly better interfaces and comments 2020-06-12 12:37:52 +03:00
alesapin
4eb8a1cfef Fix gcc build 2020-06-12 00:03:25 +03:00
Alexey Milovidov
540f8532cb Fix error 2020-06-11 22:59:15 +03:00
Azat Khuzhin
176ebb26e0 Prepend slash to path for system.zookeeper queries 
path should starts with '/', otherwise ZBADARGUMENTS will be thrown in
ZooKeeper::sendThread and the session will fail.
 2020-06-11 20:58:53 +03:00
Nikolai Kochetov
7b8c1657a5 Fix nullable prewhere column type. 2020-06-11 19:34:36 +03:00
alesapin
a39d010f94 Merge branch 'master' into consistent_metadata4 2020-06-11 18:47:41 +03:00
alexey-milovidov
5f073c3ef4
Update StorageReplicatedMergeTree.cpp 2020-06-11 17:11:13 +03:00
Nikolai Kochetov
ed0826efa7 Remove extra columns from sample in StorageBuffer. 2020-06-11 15:18:45 +03:00
kssenii
626eb53baa Fix multiple bindings for single queue & rm hardcoded strings 2020-06-11 11:13:13 +00:00
kssenii
9350472ee4 Support multiple bindings 2020-06-11 10:29:08 +00:00
Alexey Milovidov
790bc4e8e0 Better code 2020-06-11 06:35:59 +03:00
Alexey Milovidov
b72a095e99 Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree 2020-06-11 06:24:52 +03:00
kssenii
18820814f5 Merge 2020-06-10 23:25:33 +00:00
kssenii
d7b2694806 Support all exchange types 2020-06-10 23:06:23 +00:00
Denis Glazachev
9e3a28a6b8 Merge branch 'master' into ldap-per-user-authentication 
* master: (414 commits)
  Update file.md
  Update merge.md
  Update dictionary.md
  Update external-data.md
  Update distributed.md
  Update null.md
  Update set.md
  Update join.md
  Update url.md
  Update view.md
  Update materializedview.md
  Update memory.md
  Update buffer.md
  Update generate.md
  removed a sentence about global lock during rename (#11577)
  greatCircleAngle en translation (#11584)
  Update configuration-files.md
  try fix flacky test
  Update why.html
  Update rich.html
  ...

# Conflicts:
#	src/Common/ErrorCodes.cpp
#	utils/ci/jobs/quick-build/run.sh
 2020-06-11 03:06:17 +04:00
kssenii
472b04b69c Better producer 2020-06-10 19:59:37 +00:00
alesapin
dd3cf0fe0a Don't miss columns TTLs update 2020-06-10 21:02:45 +03:00
kssenii
897a592ee6 Move subscription 2020-06-10 15:41:49 +00:00
alesapin
24db834046 Style 2020-06-10 14:27:16 +03:00
alesapin
8be957ecb5 Better checks around metadata 2020-06-10 14:16:31 +03:00
Ivan Blinkov
90ee4d52f4 merge master 2020-06-10 13:18:41 +03:00
alesapin
ec933d9d03 Better naming 2020-06-10 12:09:51 +03:00
alesapin
75977bd939 Gcc fixed 2020-06-10 00:22:01 +03:00
alesapin
4f398b5dea Merge remote-tracking branch 'origin/master' into consistent_metadata4 2020-06-09 23:53:38 +03:00
alesapin
4c77ba664a Better assignment 2020-06-09 20:42:04 +03:00
alesapin
d2fcf5aea5 Fixes for gcc 2020-06-09 20:28:29 +03:00
alesapin
936cc9d573 Update key 2020-06-09 20:22:03 +03:00
alesapin
4c2e0561f5 Fix primary key update 2020-06-09 20:21:38 +03:00
alesapin
6f673b0981 Fix comments bug 2020-06-09 19:33:28 +03:00
Nikolai Kochetov
5cc2f318c9
Merge pull request #11495 from vzakaznikov/live_view_support_for_prewhere 
Adding support for PREWHERE in live view tables.
 2020-06-09 19:09:17 +03:00
Ivan Blinkov
6f7064928f Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-09 19:07:57 +03:00
alesapin
f7a397242f Merge branch 'master' into consistent_metadata4 2020-06-09 18:59:59 +03:00
Alexander Kuzmenkov
dd41f6a574
Merge pull request #11527 from azat/dist-insert-local-fix 
Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication
 2020-06-09 15:21:05 +03:00
Ivan
47ad338cb2
Refactor CMake build files (#11390) 
* Get rid of lib_name.cmake
* Refactor Boost and HyperScan libraries
* Refactor lz4
* Fix build with xxHash
 2020-06-09 13:54:49 +03:00
Ivan Blinkov
5b3fef526e Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-09 11:51:29 +03:00
alesapin
223f45685f Review fixes 2020-06-09 05:03:22 +03:00
alesapin
de59629b38 Fix benign race condition during shutdown 2020-06-09 04:48:30 +03:00
Azat Khuzhin
d2383f0f5d Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication 2020-06-08 21:58:56 +03:00
alesapin
0d375e4522 Fix constraints 2020-06-08 21:49:54 +03:00
alesapin
a5a9048109 At least fixed gcc-9 build 2020-06-08 21:23:26 +03:00
alesapin
7b63a88120 Remove redundant error code 2020-06-08 18:19:41 +03:00
alesapin
d625162fce Better comment 2020-06-08 17:23:32 +03:00
alesapin
37a1d128c9 Better 2020-06-08 17:22:11 +03:00
alesapin
a30b72a128 Less debug 2020-06-08 17:21:12 +03:00
alesapin
2354939994 Remove log debug 2020-06-08 17:19:20 +03:00
alesapin
638cd50ce4 Fix several bugs 2020-06-08 17:18:38 +03:00
alesapin
2226f79f16 Fix some bugs 2020-06-08 15:57:25 +03:00
alesapin
ac67d0485c Merge branch 'master' into consistent_metadata4 2020-06-08 15:06:51 +03:00
Vitaliy Zakaznikov
5fe67c4292 * Removing supportsPrewhere() from StorageLiveView.h as it is not valid. 
* Updating test to check using PREWHERE in query against live view table.
* Updating test to check using PREWHERE in the stored query against the table that does
  not support PREWHWERE.
 2020-06-08 13:12:40 +02:00
kssenii
17e7cc03c0 Add consumer failure handler 2020-06-08 01:11:48 +00:00
Vitaliy Zakaznikov
75c97ca82a Adding support for PREWHERE in live view tables. 2020-06-08 00:19:23 +02:00
Alexey Milovidov
f2cfb7d132 Whitespaces 2020-06-08 00:05:36 +03:00
kssenii
ce448d9291 Better event handler 2020-06-07 11:24:10 +00:00
alexey-milovidov
e3c528e661
Merge pull request #11174 from excitoon-favorites/s3tests 
Added tests for improved S3 URL parsing
 2020-06-07 02:44:40 +03:00
Ivan
d2178cbacd
Merge pull request #11388 from filimonov/kafka_better_settings 
Better settings for Kafka
 2020-06-07 01:26:21 +03:00
alexey-milovidov
5ff63d9ad5
Merge pull request #11486 from ClickHouse/speed-up-build 
Speed up build by removing old example programs
 2020-06-07 00:22:03 +03:00
Ivan
657ce426d3
Fix build 
Checked locally - no PR
 2020-06-06 22:27:48 +03:00
BayoNet
e9b7919e12
DOCS-271: Updated the MergeTree() ORDER BY description (#11433) 
* CLICKHOUSEDOCS-271: Updated the MergeTree() ORDER BY description.

* CLICKHOUSEDOCS-271: Fixes grammar.

* CLICKHOUSEDOCS-271: Updated by comments.

Co-authored-by: Sergei Shtykov <bayonet@yandex-team.ru>
 2020-06-06 20:44:48 +03:00
Ivan
058821fa02
Merge branch 'master' into kafka_better_settings 2020-06-06 19:50:38 +03:00
Alexey Milovidov
d3d02a67f4 Speed up build by removing old example programs 2020-06-06 19:19:16 +03:00
alexey-milovidov
3e84b17cf8
Merge pull request #11394 from azat/system.distribution_queue 
Add system.distribution_queue table
 2020-06-06 18:59:59 +03:00
alexey-milovidov
82e849e6a1
Update StorageDistributed.cpp 2020-06-06 18:57:52 +03:00
Anton Popov
5c42408add
Merge pull request #9113 from dimarub2000/group_by_in_order_optimization 
[WIP] Optimization of GROUP BY with respect to table sorting key.
 2020-06-06 14:25:59 +03:00
Artem Zuikov
c58f9bd43d refactoring: remove TableWithColumnNames 2020-06-06 00:17:00 +03:00
Anton Popov
66e31d4311 in-memory parts: several fixes 2020-06-05 23:49:24 +03:00
Ivan Blinkov
653a9afda0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into sentry 2020-06-05 22:51:28 +03:00
alesapin
abaf47f0cd Make metadata single structure 2020-06-05 20:29:40 +03:00
kssenii
cb618a32b8 Fix style 2020-06-05 14:27:56 +00:00
kssenii
68b94c5c20 Fixes 2020-06-05 13:46:20 +00:00
alesapin
2b23d1aa33 Fix reference 2020-06-05 15:13:24 +03:00
alesapin
5bc7f67e61 Better metadata for select query and renames 2020-06-05 14:54:54 +03:00
Alexander Kuzmenkov
eaaf005a03
Merge pull request #11410 from ClickHouse/aku/logical-error-synonyms 
Remove assorted synonyms of LOGICAL_ERROR
 2020-06-05 13:10:40 +03:00
Azat Khuzhin
ff85125326 Fix readability-qualified-auto 2020-06-04 20:23:46 +03:00
Alexander Kuzmenkov
029005be7c clang-tidy 2020-06-04 17:26:16 +03:00
alesapin
22235c1951 Tiny fixes for storage s3 2020-06-04 16:48:20 +03:00
alesapin
bba558d68e Merge branch 'custom-auth' of https://github.com/GrigoryPervakov/ClickHouse into merging_s3_auth 2020-06-04 16:19:04 +03:00
Vladimir Chebotarev
d62e85f875 Improved logging of globs in S3. 2020-06-04 16:12:08 +03:00
Nikita Mikhaylov
4e4218e10d
Merge pull request #11385 from ClickHouse/akz/bg-processing-pool-update1 
Extended BackgroundProcessingPool::startTask() to control task execution concurrency
 2020-06-04 16:19:29 +04:00
Mikhail Filimonov
26d93fdfba Fix the test, adjust default for queued.min.messages 2020-06-04 14:14:24 +02:00
Alexander Kuzmenkov
30d7d41ebc
Merge pull request #11381 from ClickHouse/aku/fuzz-related 
Fuzzing-related changes.
 2020-06-04 15:06:11 +03:00
Alexander Kuzmenkov
0943ef3e4b Do not set global context twice in unit tests. 2020-06-04 15:04:05 +03:00