Commit Graph

1687 Commits

Author SHA1 Message Date
alesapin
1ea00d5e7a Fix test and better messages 2020-08-28 14:12:51 +03:00
alesapin
b599d0a1ff Simplify function 2020-08-28 13:47:29 +03:00
alesapin
0896b49533 Small fixes 2020-08-28 13:18:56 +03:00
kssenii
4e0c619721 Global refactoring 2020-08-28 10:03:54 +00:00
alesapin
77faf9587f Better interface 2020-08-28 12:07:20 +03:00
Dongdong Yang
7e48edf42c fix issue #14202 2020-08-28 13:29:15 +08:00
Alexey Milovidov
5763737d97 Fixups 2020-08-28 03:53:22 +03:00
Alexey Milovidov
ed1d120de0 Added another test; fixup 2020-08-28 03:28:37 +03:00
Alexey Milovidov
c4e8aaac16 Fixups 2020-08-28 03:07:51 +03:00
Alexey Milovidov
6082697c4d Support for quorum 2020-08-28 02:39:12 +03:00
Alexey Milovidov
cdba5e727c Fix mistake 2020-08-28 02:30:07 +03:00
Alexey Milovidov
88db4938f5 Fix error; refinements 2020-08-28 02:22:00 +03:00
Alexey Milovidov
c04dc4e722 Merge branch 'master' into manual-write-duplicate-parts-to-replicas 2020-08-28 01:56:11 +03:00
Amos Bird
1476a9e236
Add replicated_merge_tree_settings table 2020-08-27 22:41:59 +08:00
Amos Bird
05a5a13e08
specific ReplicatedMergeTree settings 2020-08-27 22:37:41 +08:00
Amos Bird
078b14610d
ALTER MODIFY SAMPLE BY 2020-08-27 22:31:30 +08:00
Amos Bird
0c1cf22c00
Allow FETCH PARTITION from other zookeepers 2020-08-27 22:19:18 +08:00
alesapin
32db38b4d2 Fix compression codec in WAL 2020-08-27 17:06:14 +03:00
alesapin
d426751e26 Default codec if part is not stored on disk 2020-08-27 16:32:23 +03:00
alesapin
0ff5e5412e Merge branch 'master' into parts_default_compression 2020-08-27 16:01:37 +03:00
alexey-milovidov
ea0122de7e
Merge pull request #14120 from ClickHouse/parallel-insert-more-storages
Support parallel INSERT for more table engines
2020-08-27 15:08:56 +03:00
alexey-milovidov
9baa0fbf81
Merge pull request #14129 from CurtizJ/fix-performance
Slightly optimize very short queries with LowCardinality
2020-08-27 15:07:19 +03:00
alesapin
1225c09a02 Fix default codec for in-memory parts 2020-08-27 11:35:55 +03:00
alesapin
886a7b485e Fixed codec in clone part 2020-08-27 09:49:41 +03:00
bharatnc
ffb73adb11 fix style check 2020-08-26 15:31:48 -07:00
bharatnc
593f054b10 minor fix exception message 2020-08-26 15:31:48 -07:00
bharatnc
3a35aedb56 add comments explaining index_granularity_bytes safe guard 2020-08-26 15:31:48 -07:00
bharatnc
4ac9b59cec format file 2020-08-26 15:31:48 -07:00
bharatnc
37416045ba setting min_index_granularity_bytes
This PR adds a new setting called `min_index_granularity_bytes`.

If the `index_granularity_bytes` is > 0 and is lesser than the
`min_index_granularity_bytes`, throw an exception inside the
`registerStorageMergeTree.cpp` file.
2020-08-26 15:31:48 -07:00
Azat Khuzhin
a588947fe2 Fix DistributedFilesToInsert metric (zeroed when it should not)
CurrentMetrics::Increment add amount for specified metric only for the
lifetime of the object, but this is not the intention, since
DistributedFilesToInsert is a gauge and after #10263 it can exit from
the callback (and enter again later, for example after SYSTEM STOP
DISTRIBUTED SEND it will always exit from it, until SYSTEM START
DISTRIBUTED SEND).

So make Increment member of a class (this will also fix possible issues
with substructing value on DROP TABLE).
2020-08-27 00:43:00 +03:00
alesapin
44eb702fbe Remove redundant code 2020-08-26 22:41:57 +03:00
alesapin
3a680dc8fb Fix removal bug 2020-08-26 22:40:04 +03:00
kssenii
4fecfdbe2f Better & cleaner 2020-08-26 18:52:45 +00:00
Anton Popov
eeb78bf291 slightly optimize very short queries with LowCardinality 2020-08-26 21:46:18 +03:00
Alexey Milovidov
2a09aa53cc Support parallel INSERT for more table engines 2020-08-26 19:41:30 +03:00
alesapin
1fb93a8bf2 Remove read helpers 2020-08-26 18:57:06 +03:00
alesapin
18eb0dbcaa Better remove 2020-08-26 18:39:11 +03:00
alesapin
2fc80189af Add default compression codec to merge tree data part 2020-08-26 18:29:46 +03:00
alexey-milovidov
c157b7a685
Merge pull request #14056 from ClickHouse/alternative-14043
Alternative implementation of #14043
2020-08-26 13:15:30 +03:00
alesapin
21d78f8513 Better codec description representation 2020-08-26 11:45:13 +03:00
alesapin
7428c46737 Merge branch 'master' into recompressing_in_background 2020-08-26 10:28:03 +03:00
alexey-milovidov
d3a9389465
Merge branch 'master' into bharatnc-ncb/timezones-table 2020-08-26 04:26:37 +03:00
Alexey Milovidov
0e021c6151 Merge branch 'master' into manual-write-duplicate-parts-to-replicas 2020-08-26 03:24:35 +03:00
alexey-milovidov
c1c5474730
Merge pull request #14051 from javisantana/fix/info_in_read_only_table
adds some info to the exception about zookeeper path
2020-08-26 02:27:01 +03:00
Alexey Milovidov
0101fa0af9 Fix error 2020-08-26 02:12:16 +03:00
Alexey Milovidov
63feb75306 Some tweaks 2020-08-25 23:49:22 +03:00
Alexey Milovidov
64af0db94a Fix error 2020-08-25 23:40:32 +03:00
Alexey Milovidov
7aadd3803c Fix error 2020-08-25 22:46:47 +03:00
Alexey Milovidov
b3845b10d4 More simple 2020-08-25 20:54:44 +03:00
Alexey Milovidov
39730bfc30 Alternative implementation of #14043 2020-08-25 20:42:35 +03:00
alexey-milovidov
ea4aa34697
Little bit more pretty 2020-08-25 20:09:35 +03:00
Javi Santana
4942d85090 adds some info to the exception about zookeeper path 2020-08-25 18:09:22 +02:00
alesapin
1180bad3a7 Add ability to specify DefaultCompression codec which correspond to settings specified in config.xml 2020-08-25 18:02:32 +03:00
kssenii
31b1b6fe9f Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into rabbitmq-improvements 2020-08-25 14:47:03 +00:00
Alexey Milovidov
31240aa243 Minor modifications 2020-08-24 23:35:38 +03:00
bharatnc
70d00abe4f try and fix gcc-9 build.
This tries to fix the failing gcc-9 unbundled build:

https://clickhouse-builds.s3.yandex.net/13880/e437d85b927a47da69b8bdc4dde95917c2261cbe/clickhouse_build_check/build_log_753740668_1598037713.txt
2020-08-24 23:31:21 +03:00
bharatnc
a44a90fe99 remove older script that generated tz data 2020-08-24 23:30:41 +03:00
Alexey Milovidov
b504b9a917 Fix conflict + minor modification. 2020-08-24 23:29:56 +03:00
bharatnc
b85bcc5b8b add time_zones tables
This table adds a new table called the `time_zones` table:

```
:) select * from system.time_zones limit 10;

SELECT *
FROM system.time_zones
LIMIT 10

┌─time_zone──────────┐
│ Africa/Abidjan     │
│ Africa/Accra       │
│ Africa/Addis_Ababa │
│ Africa/Algiers     │
│ Africa/Asmara      │
│ Africa/Asmera      │
│ Africa/Bamako      │
│ Africa/Bangui      │
│ Africa/Banjul      │
│ Africa/Bissau      │
└────────────────────┘

10 rows in set. Elapsed: 0.012 sec.
```

The available timezones are parsed from under
`ClickHouse/contrib/cctz/testdata/zoneinfo` through a script
and the table content is constructed from the
StorageSystemTimeZones.generated.cpp`.

I was able to test this locally and was able to get a list of time
zones as shown above.
2020-08-24 23:26:00 +03:00
alexey-milovidov
7ea5364299
Merge pull request #13648 from filimonov/tzdata_version2
tzdata improvements
2020-08-24 23:07:37 +03:00
Anton Popov
4331158d30 merge with master 2020-08-24 16:09:23 +03:00
Anton Popov
4df5c72878 Merge remote-tracking branch 'upstream/master' into HEAD 2020-08-24 12:57:27 +03:00
Alexander Tokmakov
dd4b8b9663 fix lock order inversion when renaming distributed table 2020-08-20 16:36:22 +03:00
tavplubix
de3a6d78fc
Merge pull request #13822 from zhang2014/fix/materialize_integration_test
ISSUES-4006 try fix materialize mysql database integration test
2020-08-19 17:32:38 +03:00
Artem Zuikov
becc186c91
Add support for extended precision integers and decimals (#13097) 2020-08-19 14:52:17 +03:00
alexey-milovidov
23ccb0b6be
Merge pull request #13677 from hagen1778/merge-tree-fail-fast-on-rows-limit
[mergeTree]: fail fast if max_rows_to_read limit exceeded on parts scan
2020-08-18 22:24:39 +03:00
Alexey Milovidov
9615b1f06a Generate ya.make 2020-08-18 12:46:02 +03:00
Alexander Tokmakov
f5c14a0766 Merge branch 'master' into fix/materialize_integration_test 2020-08-17 21:47:53 +03:00
zhang2014
ead3d34323 ISSUES-4006 try fix build 2020-08-17 18:11:50 +08:00
roman
35e28b4c6b [mergeTree]: make exception message more clear 2020-08-17 09:52:04 +01:00
zhang2014
2cbc1df3ce ISSUES-4006 try fix bad integration test 2020-08-17 13:14:02 +08:00
Vitaly Baranov
29a6558d33 Add system table system.user_directories 2020-08-16 19:15:39 +03:00
Vitaly Baranov
2909ed1bc0 Better initialization of access storages. Make list of access storages dynamic. 2020-08-16 19:15:34 +03:00
Alexey Milovidov
5059d1f441 Merge branch 'master' into manual-write-duplicate-parts-to-replicas 2020-08-16 15:58:57 +03:00
alexey-milovidov
f3755fecb7
Merge pull request #13793 from amosbird/fnc
Fix set index with const column pred
2020-08-16 15:18:31 +03:00
Alexey Milovidov
44fb1ec107 Fix typo 2020-08-16 13:15:35 +03:00
Amos Bird
ae2c50e9e8
Fix set index with const column pred 2020-08-16 15:16:37 +08:00
alexey-milovidov
cd417bb066
Merge pull request #13757 from ClickHouse/fix-flaky-test-3
Fix flaky test 01305_replica_create_drop_zookeeper
2020-08-16 10:15:30 +03:00
kssenii
1e7eb49481 Add one more mes property, support format_schema 2020-08-15 15:05:53 +00:00
alexey-milovidov
0e3b9d0da6
Merge pull request #13747 from ClickHouse/fix-race-condition-in-test
Fix race condition in test
2020-08-15 15:55:47 +03:00
Alexey Milovidov
8c5e5ec1cf Fix flaky test 01305_replica_create_drop_zookeeper 2020-08-15 15:30:17 +03:00
kssenii
c6fdeb6c02 Better 2020-08-15 08:00:56 +00:00
Alexey Milovidov
9e6d4802ac Do not drop cache if DROP/RENAME in Atomic database 2020-08-15 09:56:47 +03:00
kssenii
9a409db5c0 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into rabbitmq-improvements 2020-08-15 06:51:22 +00:00
Alexey Milovidov
ee084f0ec9 Fix race condition in DETACH and background merge 2020-08-15 08:21:02 +03:00
alexey-milovidov
7e72437b9d
Merge pull request #13680 from ClickHouse/use_atomic_engine_for_system_db
Use Atomic engine for system database
2020-08-15 01:15:09 +03:00
Alexander Tokmakov
03eb8f5f21 fix sync 2 2020-08-14 16:51:32 +03:00
Nikolai Kochetov
76df13c31c Fix tests. 2020-08-14 16:16:36 +03:00
roman
b637699ccd [mergeTree]: fail fast if max_rows_to_read limit exceeded on parts scan
The motivation behind this change is to skip ranges scan for all selected parts
if it is clear that `max_rows_to_read` is already exceeded. The change is quite
noticeable for queries over big number of parts.
2020-08-14 13:53:48 +01:00
tavplubix
e6df1f4868
Merge pull request #13676 from amosbird/bd
Make drop table a little bit more robust
2020-08-14 14:14:58 +03:00
Alexander Tokmakov
d8d50f20fd fix sync 1 2020-08-14 14:04:47 +03:00
Alexander Tokmakov
128b19fdba Merge branch 'master' into use_atomic_engine_for_system_db 2020-08-14 13:47:44 +03:00
Artem Zuikov
89e9673336
Hotfix for pushdown with StorageMerge (#13679) 2020-08-14 12:38:18 +03:00
Alexey Milovidov
7a6e0deea5 Fix build after merge 2020-08-14 08:28:26 +03:00
Amos Bird
de4a070d6a
log error message 2020-08-14 09:54:51 +08:00
alexey-milovidov
c06cfda3cb
Merge pull request #13678 from ClickHouse/fix-typo-2
Fix typo due to copy-paste
2020-08-14 04:15:23 +03:00
alexey-milovidov
eeefcd6493
Merge pull request #13337 from ClickHouse/refactor-pipes-3
Refactor Pipe
2020-08-14 04:12:22 +03:00
Alexander Tokmakov
cd3155ff19 fix 2020-08-13 21:05:31 +03:00
Alexander Tokmakov
0f1c4bc9c4 attach all system tables 2020-08-13 20:09:40 +03:00
Alexey Milovidov
bcd29dc9f8 Fix typo due to copy-paste 2020-08-13 18:41:26 +03:00
Amos Bird
3b92692458
Make drop table a little bit more robust 2020-08-13 22:22:04 +08:00
alexey-milovidov
b243a0c37a
Merge pull request #13626 from ClickHouse/fix_deadlock_in_replication_queue
Fix deadlock and logical error in replication queue
2020-08-13 00:53:09 +03:00
Alexander Tokmakov
a6ff049eec use Atomic for system database 2020-08-12 23:40:13 +03:00
tavplubix
4aff48dc3f
Merge pull request #10851 from zhang2014/feature/materialize_mysql_database
ISSUES-4006 support MaterializeMySQL database engine
2020-08-12 15:59:28 +03:00
Mikhail Filimonov
a0ae1b2b42 tzdata improvements
Expose version of embedded tzdata via TZDATA_VERSION in system.build_options.
Autogenerate list of shipped timezones. Script to check tzdb updates.
2020-08-12 14:21:40 +02:00
Alexander Tokmakov
e4da1d8e51 fix deadlock and logical error in replication queue 2020-08-12 00:36:02 +03:00
zhang2014
93d73c78c9 ISSUES-4006 resolve review comment & add test case 2020-08-11 17:03:43 +08:00
Vladimir Chebotarev
7371cf6bd1
Proper remote host checking in S3 redirects (#13404)
* Proper remote host checking in S3 redirects.

Co-authored-by: Alexander Kazakov <Akazz@users.noreply.github.com>
2020-08-11 09:09:08 +03:00
zhang2014
2311cda334 ISSUES-4006 convert _sign & _version to materialized column 2020-08-10 22:32:05 +08:00
Nikolai Kochetov
9b67cd9faf Merge branch 'master' into refactor-pipes-3 2020-08-10 10:50:17 +03:00
alexey-milovidov
fa1c4d1eef
Merge pull request #13538 from CurtizJ/fix-skip-indices
Fix secondary indices corruption in compact parts
2020-08-09 20:13:13 +03:00
Anton Popov
2280619ad0 fix secondary indices corruption in compact parts 2020-08-09 04:45:10 +03:00
alexey-milovidov
aefa9b297b
Merge pull request #13449 from ClickHouse/mixed_granularity_parts_by_default
Enable mixed granularity parts by default
2020-08-09 01:36:59 +03:00
kssenii
139a19d583 Merge 2020-08-08 17:20:50 +00:00
kssenii
2ea32a710a More tests, better reconnect 2020-08-08 17:03:55 +00:00
alexey-milovidov
c0fef71507
Merge pull request #13511 from ClickHouse/codespell
Fix typos in code with codespell.
2020-08-08 17:09:16 +03:00
alexey-milovidov
92fb97c76b
Merge pull request #13416 from ClickHouse/storage-memory-race
Possibly fix race in StorageMemory.
2020-08-08 17:08:13 +03:00
Alexey Milovidov
306aefd33e Change error code #13416 2020-08-08 17:05:46 +03:00
Alexey Milovidov
12f66fa82c Fix 99% of typos 2020-08-08 04:01:47 +03:00
Alexey Milovidov
edd89a8610 Fix half of typos 2020-08-08 03:47:03 +03:00
Alexey Milovidov
bbbc52362b Remove strange file 2020-08-08 03:13:51 +03:00
Alexander Kuzmenkov
6a596d5c40
Merge pull request #13076 from Jokser/s3-file-cache
Ability to cache mark and index files for S3 disk
2020-08-08 01:04:53 +03:00
alexey-milovidov
7786fd4119
Merge pull request #13228 from ClickHouse/decimal-too-large-negative-exponent
Fix assert when decimal has too large negative exponent
2020-08-07 21:57:20 +03:00
robot-clickhouse
a82c0f5371 Auto version update to [20.8.1.1] [54438] 2020-08-07 21:45:49 +03:00
zhang2014
6da8633741 ISSUES-4006 try fix build failure & test failure 2020-08-08 00:14:10 +08:00
Nikolai Kochetov
c9f81125bc Review fixes. 2020-08-07 16:02:28 +03:00
alesapin
5de2491eda Enable mixed granularity parts by default 2020-08-07 11:55:51 +03:00
zhang2014
bd21f8e6b9 ISSUES-4006 fix conflicts commit when merge 2020-08-07 12:55:45 +08:00
zhang2014
e496225641 ISSUES-4006 try fix build failure & review comment 2020-08-07 12:53:33 +08:00
Alexey Milovidov
0ac3f0481f Probably fix error 2020-08-07 04:27:29 +03:00
kssenii
1213161cf4 Add some message properties 2020-08-06 20:34:13 +00:00
Nikolai Kochetov
d83c7cc0e6 Merge branch 'master' into refactor-pipes-3 2020-08-06 17:32:50 +03:00
Nikolai Kochetov
ac52726f7d Renamed iterators. Added comment. 2020-08-06 17:16:52 +03:00
kssenii
24b032b378 Allow multiple consumers for same queues 2020-08-06 13:50:44 +00:00
Nikolai Kochetov
285f323653 Fix build. 2020-08-06 16:22:17 +03:00
Nikolai Kochetov
caef062135 Possibly fix race in StorageMemory. 2020-08-06 15:34:16 +03:00
Nikolai Kochetov
20e63d2271 Refactor Pipe [part 6] 2020-08-06 15:24:05 +03:00
Anton Popov
33699d8d4a
Merge pull request #13282 from amosbird/mha
Report broken parts in compact MergeTree
2020-08-05 14:27:32 +03:00
kssenii
add698a86b Merge 2020-08-04 21:44:33 +00:00
kssenii
053f31cb77 Better confirmListener 2020-08-04 20:50:41 +00:00
Alexey Milovidov
d91471ea00 Merge branch 'master' into stack-frams-size-limit 2020-08-04 15:28:47 +03:00
zhang2014
c99da34039 ISSUES-4006 try fix unit test failure 2020-08-04 19:09:30 +08:00
alesapin
943eb2f990
bump CI 2020-08-04 12:42:51 +03:00
Amos Bird
019f6d3a76
Report broken parts in compact MergeTree 2020-08-04 16:45:52 +08:00
Nikolai Kochetov
09fbce1b1e Merge branch 'master' into refactor-pipes-3 2020-08-04 11:32:34 +03:00
zhang2014
1eddb3fbc8 ISSUES-4006 fix build failure after merge master 2020-08-04 10:31:27 +08:00
zhang2014
54c3db0567 ISSUES-4006 some refactor for query convert 2020-08-04 10:31:26 +08:00
zhang2014
a9e1794bd3 ISSUES-4006 filter sign = -1 rows 2020-08-04 10:31:26 +08:00
zhang2014
9324c0ee49 ISSUES-4006 fix crash when dump data 2020-08-04 10:31:26 +08:00
zhang2014
0336a4ad58 ISSUES-4006 suport synchronous MySQL incremental data[part 2] 2020-08-04 10:31:23 +08:00
Vitaly Baranov
a804f9499d Use references while iterating through settings. 2020-08-04 04:00:38 +03:00
alexey-milovidov
c43a27782e
Merge pull request #13153 from ClickHouse/merge-tree-settings-sanity-check
Sanity checks for MergeTreeSettings
2020-08-04 01:27:19 +03:00
Alexey Milovidov
1d091738dd Merge branch 'master' into stack-frams-size-limit 2020-08-03 20:41:47 +03:00
kssenii
d5b1332b67 Stop publish untill batch is confirmed 2020-08-03 14:40:51 +00:00
Nikolai Kochetov
2cca4d5fcf Refactor Pipe [part 2]. 2020-08-03 16:54:14 +03:00
Alexander Tokmakov
5b92f43f12 Merge branch 'master' into database_atomic_improvements 2020-08-03 14:35:40 +03:00
Nikolai Kochetov
e411916bde Refactor Pipe [part 1]. 2020-08-03 14:33:11 +03:00
kssenii
60124b769c Merge 2020-08-03 09:17:23 +00:00
kssenii
62293f80c0 Small fixes 2020-08-03 09:14:25 +00:00
Nikolai Kochetov
cf49a83980
Merge pull request #13201 from azat/fix-logging-in-MergeTreeDataSelectExecutor
Fix logging in MergeTreeDataSelectExecutor for multiple threads (attach to thread group)
2020-08-03 10:08:58 +03:00
Alexey Milovidov
77516b48fa Merge branch 'master' into merge-tree-settings-sanity-check 2020-08-03 00:30:56 +03:00
Alexey Milovidov
0fd7af8a63 Merge branch 'master' into stack-frams-size-limit 2020-08-03 00:06:00 +03:00
Alexey Milovidov
4ed0bf3af1 Better code 2020-08-03 00:01:39 +03:00
Alexey Milovidov
3c489ce159 Fix assertion in KeyCondition 2020-08-02 23:55:20 +03:00
Alexey Milovidov
5f808aa503 Fix bad code 2020-08-02 23:41:52 +03:00
alexey-milovidov
3b2954c09a
Merge pull request #13209 from ClickHouse/minor-changes-in-debian-scripts
Minor changes in Debian scripts
2020-08-02 22:44:38 +03:00
alesapin
75a9053511
Merge pull request #13150 from nvartolomei/nv/freeze-verbose-absolute-path
Use absolute paths to backup locations (shadow)
2020-08-02 19:25:24 +03:00
Alexey Milovidov
41111d1e57 Merge branch 'master' into stack-frams-size-limit 2020-08-02 19:18:33 +03:00
Alexey Milovidov
778abb346f Merge branch 'master' into merge-tree-settings-sanity-check 2020-08-02 17:13:17 +03:00
Alexey Milovidov
7bd5505902 Merge branch 'master' into minor-changes-in-debian-scripts 2020-08-02 17:07:48 +03:00
Alexey Milovidov
6f690b7c0d Normalize ya.make files, fix "Arcadia" build 2020-08-02 16:57:38 +03:00
Azat Khuzhin
e37c42c56c Fix logging in MergeTreeDataSelectExecutor for multiple threads (attach to thread group) 2020-08-02 13:40:01 +03:00
Azat Khuzhin
101217470e Use "Not using primary index on part" over "Not using index on part" (add "primary") 2020-08-02 13:40:01 +03:00
Vitaly Baranov
18b21511a9
Merge pull request #13013 from vitlibar/implement-custom-settings
Implement custom settings
2020-08-02 05:01:14 +03:00
Alexey Milovidov
d3c0635fc0 Merge branch 'master' into stack-frams-size-limit 2020-08-02 04:35:09 +03:00
Alexey Milovidov
53450b1b09 Merge branch 'master' into merge-tree-settings-sanity-check 2020-08-02 02:47:57 +03:00
Alexey Milovidov
084e261687 Trigger CI 2020-08-02 01:06:09 +03:00
alexey-milovidov
566044f9c8
Merge pull request #12991 from ClickHouse/kafka-fix-msan
Fix MSan error in "rdkafka" library #12990
2020-08-02 00:22:44 +03:00
Alexey Milovidov
8fe201f1b9 Merge branch 'master' into merge-tree-settings-sanity-check 2020-08-01 19:20:11 +03:00
kssenii
c2bed351ae Add consumer connection track and restore 2020-08-01 13:05:06 +00:00
Nikolai Kochetov
c786f05388
Merge pull request #13189 from ClickHouse/refactor-pipes-2
Remove TreeExecutorBIS
2020-08-01 11:56:50 +03:00
Vitaly Baranov
56665a15f7 Rework and rename the template class SettingsCollection => BaseSettings. 2020-07-31 20:54:18 +03:00
Vitaly Baranov
e40a8ac176 Use names of the settings instead of their indices in SettingsConstraints and settings profiles. 2020-07-31 19:11:27 +03:00
Vitaly Baranov
300727afa3 Rework the StringField* classes and make conversion String => StringField* explicit. 2020-07-31 19:11:27 +03:00
Vitaly Baranov
90602b869a Make SettingsChanges a class. 2020-07-31 19:11:27 +03:00
Vitaly Baranov
4ad99f1472 Make type names of settings shorter: SettingUInt64 => UInt64. 2020-07-31 19:11:27 +03:00
Vitaly Baranov
30b34e6a15 Use SettingSeconds only in Settings, use std::chrono::seconds in other places. 2020-07-31 19:11:27 +03:00
Nikolai Kochetov
39530f837e Remove TreeExecutorBlockInputStream. 2020-07-31 16:23:19 +03:00
alesapin
45894fcf99 Better comments 2020-07-31 15:22:32 +03:00
alexey-milovidov
bbc8473388
Update StorageMergeTree.cpp 2020-07-31 15:00:07 +03:00
alexey-milovidov
9c9ef0db9b
Merge branch 'master' into database_atomic_improvements 2020-07-31 14:56:49 +03:00
alesapin
14928874a1 Fix mutations status check 2020-07-31 14:37:16 +03:00
alesapin
2eb01fd655
Merge pull request #13104 from ClickHouse/volumes-refactorings
Merging #12670
2020-07-31 11:21:30 +03:00
kssenii
5a934c079e Add connection restore in insert, better confirms 2020-07-31 04:59:56 +00:00
kssenii
763c337be9 Remove redundant, move subscription 2020-07-31 04:59:18 +00:00
alexey-milovidov
74edcfb87e
Merge pull request #13073 from nikitamikhaylov/rewrite-tests
BayesAB test
2020-07-31 02:30:07 +03:00
Alexey Milovidov
2c40539df6 Check ALTERs; update test 2020-07-31 00:42:55 +03:00
Alexey Milovidov
c491b2c153 Lower stack frame size 2020-07-30 23:58:08 +03:00
Alexey Milovidov
362655fa43 Fix some bad tests 2020-07-30 22:59:49 +03:00
Alexey Milovidov
a9b02e8184 Fix trivial bug in code 2020-07-30 22:33:20 +03:00
Alexey Milovidov
b9f49d31df Sanity checks for MergeTreeSettings 2020-07-30 22:08:13 +03:00
alexey-milovidov
fc35ce69fa
Merge pull request #12666 from ClickHouse/forward_alter_error_to_client
Forward exceptions to client if exception happened during ALTER
2020-07-30 21:36:06 +03:00
Nicolae Vartolomei
85174d64a0 Use absolute paths to backup locations (shadow)
Tests are frankenstain. Had to hide absolute path `/tmp/clickhouse.test..IP69I/data/shadow/test_01417_single_part_7/data/default/table_for_freeze/7_8_8_0` to make them deterministic and independend of the method they are run (local vs remote target)
2020-07-30 18:03:26 +01:00
alexey-milovidov
4c3241808d
Merge pull request #12728 from ClickHouse/prevent-inconsistent-merge-tree-settings
Safe logic for incorrect MergeTreeSettings
2020-07-30 17:55:39 +03:00
Pavel Kovalenko
00181d34fd DiskS3 caching improvements. 2020-07-30 16:42:05 +03:00
alexey-milovidov
e6265bf299
Merge pull request #13100 from ClickHouse/keep-less-amount-of-logs-in-zookeeper
Keep less amount of logs in ZooKeeper
2020-07-30 15:41:55 +03:00
alexey-milovidov
a22bdb1989
Merge pull request #13081 from CurtizJ/fix-index_analysis
Fix wrong index analysis with functions
2020-07-30 15:30:55 +03:00
alesapin
d3ea1d914b Tiny fixes and avoid dynamic casts 2020-07-30 13:04:49 +03:00
alesapin
724b7be555 Merge branch 'volumes-related-refactorings' of https://github.com/excitoon-favorites/ClickHouse into volumes-refactorings 2020-07-30 12:20:48 +03:00
Alexey Milovidov
6517ddcbd4 Keep less amount of logs in ZooKeeper 2020-07-30 05:47:05 +03:00
Vladimir Chebotarev
bcc926fbb2 Returned max_data_part_size to system.storage_policies. 2020-07-30 04:19:32 +03:00
Alexander Kuzmenkov
881220eec2
Merge pull request #12755 from bobrik/ivan/obvious-skip-overall
Show marks before applying skipping indices
2020-07-29 20:59:08 +03:00
Nikita Mikhaylov
1d6c2ba689 rewrite other test + remove useless includes 2020-07-29 20:37:57 +03:00
Anton Popov
4c266d1e5d fix wrong index analysis with functions 2020-07-29 19:09:38 +03:00
alesapin
7a806eb90f
Merge pull request #13017 from ClickHouse/freeze_partition_with_name
Alter table freeze with verbose output
2020-07-29 18:42:48 +03:00
alesapin
ba8a29a3a4 Better logging on mutation skip 2020-07-29 16:12:55 +03:00
Alexander Kuzmenkov
80fcbfb1eb
Merge pull request #13034 from ClickHouse/aku/block-structure-mismatch
Block structure mismatch is a program error
2020-07-29 13:01:11 +03:00
alesapin
7fc1e45c3b Review fixe 2020-07-29 11:32:52 +03:00
Alexander Kuzmenkov
f6f2812f73 Block structure mismatch is a program error 2020-07-28 21:35:18 +03:00
Alexander Kuzmenkov
1b9269ae0c fixup 2020-07-28 19:58:19 +03:00
Alexander Kuzmenkov
297cf65f1f Merge remote-tracking branch 'origin/master' into HEAD 2020-07-28 19:56:35 +03:00
Vladimir Chebotarev
faedb04722 Minor fixes. 2020-07-28 19:45:46 +03:00
alesapin
b02951fd3a Remove new lines 2020-07-28 18:48:35 +03:00
alesapin
c75830e878 Alter table freeze with verbose output 2020-07-28 18:10:36 +03:00
alesapin
585d9247ec
Merge pull request #12959 from ClickHouse/test_for_backup
Check backup restore in integration test.
2020-07-28 17:59:52 +03:00
Alexander Kazakov
2bde393499
Merging #12548 - Correction to merge_with_ttl_timeout logic by @excitoon (#12982)
* Fixed `merge_with_ttl_timeout` logic.
* Separate TTL-merge timers for each partition

Co-authored-by: Vladimir Chebotarev <vladimir.chebotarev@gmail.com>
2020-07-28 17:38:34 +03:00
Alexander Kuzmenkov
ba7c33f806
Merge pull request #12754 from bobrik/ivan/obvious-skip
Show total granules examined by skipping indices
2020-07-28 17:14:25 +03:00
kssenii
469e46a539 Fix build 2020-07-28 12:46:44 +00:00
alexey-milovidov
7a22717180
Merge pull request #12760 from ClickHouse/sticky_mutations_bug
Sticking mutations bug
2020-07-28 14:42:54 +03:00
kssenii
0ee54c8a4a Fix build, async acks -> sync acks, fix tests 2020-07-28 10:43:25 +00:00
alesapin
1bece3de0a Remove strange code and test 2020-07-28 12:36:08 +03:00
alexey-milovidov
4d76f41449
Merge pull request #12658 from azat/kafka-unavail-broker-SIGSEGV
Fix SIGSEGV in StorageKafka when broker is unavailable
2020-07-28 00:03:54 +03:00
Alexey Milovidov
798ef1b9f9 Adjustments; added test result 2020-07-27 23:18:23 +03:00
Alexey Milovidov
ec84ed30cb Merge branch 'master' into prevent-inconsistent-merge-tree-settings 2020-07-27 21:39:25 +03:00
Nikolai Kochetov
abdd160db1
Merge pull request #12589 from bobrik/ivan/parallel-ranges
Parallelize PK range and skipping index stages
2020-07-27 19:23:00 +03:00
alexey-milovidov
7dac65ce0f
Merge pull request #12659 from zhang2014/fix/ISSUES-10572
ISSUES-10572 fix bloom filter with const column
2020-07-27 14:38:56 +03:00
Anton Popov
c23bc8aace
Merge pull request #12492 from CurtizJ/polymorphic-parts-4
Make read buffer size lower, while reading from compact parts
2020-07-27 14:22:54 +03:00
alesapin
1707f84a44 Less strict check and rare rename bug 2020-07-27 12:42:37 +03:00
Vladimir Chebotarev
1b3f5c99f5 Real fix of test. 2020-07-26 21:27:36 +03:00
Vladimir Chebotarev
f5af64514f Test fix (removed redundant code). 2020-07-26 21:27:36 +03:00
Vladimir Chebotarev
8039d45910 Minor fix in StorageDistributed. 2020-07-26 21:27:36 +03:00
Gleb Novikov
ee07c74868 Fixed integration test_multiple_disks 2020-07-26 21:27:36 +03:00
Gleb Novikov
7f5b6fba78 Generic volume is coming...
1. SingleDiskVolume for temporary volumes
2. Generic VolumePtr in StoragePolicies
3. Removed max_data_part_size in system.storage_policies, added volume_type
2020-07-26 21:27:36 +03:00
alesapin
e0bf5913e7 Fix sticking mutations 2020-07-26 17:21:57 +03:00
Alexander Tokmakov
6bfe88e83f Merge branch 'master' into database_atomic_improvements 2020-07-26 13:40:05 +03:00
alesapin
c66dc23a47 Fix error style 2020-07-25 23:56:29 +03:00
alesapin
eae7dc852e Fix build 2020-07-25 23:54:33 +03:00
kssenii
22b16060f6 More reliable publishing 2020-07-25 16:56:29 +00:00
kssenii
2b57857afc Add dl-exchange, commits 2020-07-25 16:56:12 +00:00
kssenii
f9a4bf9e61 Add queue resume read 2020-07-25 16:55:58 +00:00
kssenii
f0f6111655 Move exchange init, add bridge-exchange 2020-07-25 16:55:44 +00:00
kssenii
ac448db491 Add virtuals 2020-07-25 16:55:32 +00:00
kssenii
a88e391bd2 Fix bug with insert, simplify exchanges logic 2020-07-25 16:55:18 +00:00
alesapin
393a9195ca Better exception message 2020-07-25 17:45:46 +03:00
alesapin
78d357f0d2 Add a test for sticking mutations bug 2020-07-25 17:42:20 +03:00
Ivan Babrou
e835ec0b56 Show marks before applying skipping indices
This change makes skipping index efficiency more obvious, changing this:

```
Selected 30 parts by date, 30 parts by key, 592 marks to read from 541 ranges
```

Into this:

```
Selected 30 parts by date, 30 parts by key, 48324 marks by primary key, 592 marks to read from 541 ranges
```
2020-07-24 15:45:38 -07:00
Ivan Babrou
67d4529783 Show total granules examined by skipping indices
This change makes skipping index efficiency more obvious, changing this:

```
Index `idx_duration` has dropped 59 granules.
```

Into this:

```
Index `idx_duration` has dropped 59 / 61 granules.
```
2020-07-24 14:50:32 -07:00
Azat Khuzhin
d3b95d3fb0 Link clickhouse_storages_system with Poco::JSON (fixes unbundled+shared build) 2020-07-24 00:57:45 +03:00
Anton Popov
d3fbeb296e write compressed blocks for every column in granule 2020-07-23 19:56:12 +03:00
Anton Popov
61018c275d Merge remote-tracking branch 'upstream/master' into polymorphic-parts-4 2020-07-23 18:38:29 +03:00
Anton Popov
1d0493cfcc better code in ReaderCompact and perf test added 2020-07-23 18:37:44 +03:00
Nikolai Kochetov
dad9d369a1 Merge branch 'master' into bobrik-parallel-randes 2020-07-23 16:21:32 +03:00
Alexey Milovidov
6b4be2b9fe Safe logic for incorrect MergeTreeSettings #10897 2020-07-23 15:31:09 +03:00
Azat Khuzhin
c97fb4b0e5
Remove unused bits in MergeTreeRangeReader (#12671)
Refs: #7769
Cc: @amosbird
2020-07-23 11:20:09 +03:00
alesapin
978f286625 Trying to fix FreeBSD build 2020-07-23 11:00:18 +03:00
alesapin
6fdcb59df7 Merge branch 'master' into forward_alter_error_to_client 2020-07-23 10:48:58 +03:00
Alexander Tokmakov
237accd154 Merge branch 'master' into database_atomic_improvements 2020-07-23 06:33:20 +03:00
alesapin
355150afce Fix deadlock and better code 2020-07-22 22:29:54 +03:00
Artem Zuikov
2afd123eda
Refactoring: extract TreeOptimizer from SyntaxAnalyzer (#12645) 2020-07-22 20:13:05 +03:00
alesapin
8c8bdd5070 Fix tests and work with multiple mutations 2020-07-22 18:19:54 +03:00
Nikolai Kochetov
76d784be9c
Merge pull request #12613 from ClickHouse/remove-sort-description-from-streams
Remove sort description from streams
2020-07-22 17:35:05 +03:00
alesapin
7585fc4b52 Fix style 2020-07-22 16:22:05 +03:00
alesapin
de846e5e3c Forward exception to client during alter 2020-07-22 15:36:19 +03:00
Nikolai Kochetov
b27066389a Do not create ThreadPool for single thread. 2020-07-22 14:51:35 +03:00
zhang2014
580889f1e4 ISSUES-10572 fix bloom filter with const column 2020-07-22 17:42:12 +08:00
Azat Khuzhin
e2d9176617 Fix SIGSEGV in StorageKafka when broker is unavailable 2020-07-22 11:28:07 +03:00
Alexander Tokmakov
9bcaaea3e0 Merge branch 'master' into database_atomic_improvements 2020-07-22 05:24:48 +03:00
alesapin
be106b662e Comment and style 2020-07-21 18:19:41 +03:00
Nikolai Kochetov
486a4932c3 Fix tests. 2020-07-21 17:08:18 +03:00
alesapin
daaa5cc382 Add setting allow_non_metadata_alters 2020-07-21 17:05:30 +03:00
Nikolai Kochetov
755f15def3 Make MergeTreeSetIndex::checkInRange const. 2020-07-21 14:22:45 +03:00
alesapin
379a5a750a Fix error message about granularity 2020-07-21 14:10:23 +03:00
Nikolai Kochetov
12c5e376c6 Remove mutable from RPNElement. 2020-07-21 14:02:58 +03:00
Nikolai Kochetov
cab9146ff3 Remove SortDescription from IBlockInputStream. 2020-07-21 11:05:52 +03:00
Nikolai Kochetov
d6583698a9 Merge branch 'master' into bobrik-parallel-randes 2020-07-20 19:05:28 +03:00
Nikolai Kochetov
0cc55781d8 Try fix tests. 2020-07-20 18:09:00 +03:00
Alexander Kuzmenkov
40ee6eba19
Merge pull request #12545 from ClickHouse/fix_load_part_without_checksums
Fix loading parts without checksums
2020-07-20 14:40:53 +03:00
Ivan Babrou
72622a9b00 Parallelize PK range and skipping index stages
This runs PK lookup and skipping index stages on parts
in parallel, as described in #11564.

While #12277 sped up PK lookups, skipping index stage
may still be a bottleneck in a select query. Here we
parallelize both stages between parts.

On a query that uses a bloom filter skipping index to pick
2,688 rows out of 8,273,114,994 on a two day time span,
this change reduces latency from 10.5s to 1.5s.
2020-07-19 21:49:41 -07:00
tavplubix
5634482393
Merge pull request #12556 from azat/distributed-remove-files-on-drop
Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE
2020-07-17 19:29:57 +03:00
Alexander Tokmakov
c27de9b424 fixes 2020-07-17 16:11:44 +03:00
alexey-milovidov
9fb40a3f7b
Merge pull request #12386 from fastio/master
COS (Tencent Cloud Object Storage) integration
2020-07-17 14:09:16 +03:00
alesapin
ac802a8e96
Merge pull request #12543 from ClickHouse/fix_marks_type_on_mutation
Fix incorrect adaptive granularity by default
2020-07-17 11:09:54 +03:00
Azat Khuzhin
6ea1b19476 Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE 2020-07-17 08:59:57 +03:00
Alexander Tokmakov
a612080df1 Merge branch 'master' into database_atomic_improvements 2020-07-17 00:43:59 +03:00
alexey-milovidov
fde8c87a1f
Merge pull request #12426 from ClickHouse/log-engine-rollback-on-insert-error
Rollback insertion error in Log engines
2020-07-16 22:50:48 +03:00
Anton Popov
97e8a88b30
Merge pull request #12277 from bobrik/ivan/exact-range-speedup
WIP: Optimize PK lookup for queries that match exact PK range
2020-07-16 19:17:50 +03:00
alesapin
b3ec3b1ed1 Fix loading parts without checksums 2020-07-16 13:54:49 +03:00
Vitaly Baranov
000b197ad1
Merge pull request #11234 from traceon/ldap-per-user-authentication
Add LDAP authentication support
2020-07-16 13:17:21 +03:00
alesapin
42983c5416 Fix incorrect adaptive granularity by default 2020-07-16 12:10:15 +03:00
Alexey Milovidov
6df282e813 Fixup 2020-07-16 11:33:51 +03:00
alexey-milovidov
8966c09ed6
Merge pull request #12519 from vzakaznikov/fix_data_duplication_and_tests_for_live_view
Fixing race condition in live view tables which could cause data duplication and live view tests
2020-07-16 11:03:28 +03:00
Alexey Milovidov
68f9fd3767 Debug tests 2020-07-16 06:02:20 +03:00
Alexey Milovidov
82ea884d01 Fix incorrect unit test 2020-07-16 05:37:12 +03:00
Alexey Milovidov
3408b7e259 Merge branch 'master' into log-engine-rollback-on-insert-error 2020-07-16 05:34:02 +03:00
Denis Glazachev
59cb758cf7 Merge branch 'master' into ldap-per-user-authentication 2020-07-16 02:29:24 +04:00
Alexander Tokmakov
f6949b2f47 use dictionary ids 2020-07-15 22:25:31 +03:00
Alexey Milovidov
e1e2204279 Whitespace 2020-07-15 19:37:52 +03:00
Vitaliy Zakaznikov
370dd3396b Fixing clang build. 2020-07-15 16:18:53 +02:00
Anton Popov
c6423b2b28 lower read buffer size, while reading from compact parts 2020-07-15 14:53:23 +03:00
Vitaliy Zakaznikov
560151f6cd * Fix bug in StorageLiveView.cpp
* Fixing synchronization of the first insert in live view tests
2020-07-15 13:24:33 +02:00
Alexander Tokmakov
2fe2833a0d fix build 2020-07-15 13:18:30 +03:00
Alexander Tokmakov
09abe71b60 Merge branch 'master' into database_atomic_improvements 2020-07-15 13:07:08 +03:00
alesapin
614540eddf
Merge pull request #12382 from ClickHouse/clear-all-columns
Better errors for CLEAR/DROP columns (possibly in partitions)
2020-07-15 12:52:06 +03:00
alexey-milovidov
9c68124110
Merge pull request #12302 from azat/kafka-error-in-the-batch-SIGSEGV
kafka: fix SIGSEGV if there is a message with error in the middle of the batch
2020-07-15 05:20:26 +03:00
Alexander Tokmakov
eea0cdcf25 add uuid to generated config 2020-07-14 23:32:13 +03:00
alesapin
9e41fbca55 Remove check for drop detached partition 2020-07-14 16:56:30 +03:00
Alexander Kuzmenkov
b515dd5b83 Merge remote-tracking branch 'origin/master' into HEAD 2020-07-14 15:40:27 +03:00
Alexander Kuzmenkov
b24f727aea typo 2020-07-14 15:40:18 +03:00
alesapin
014bb070ec Fix tests 2020-07-14 11:19:39 +03:00
alexey-milovidov
fd4adf27d6
Merge pull request #12456 from CurtizJ/fix-12437
Fix #12437
2020-07-14 09:28:31 +03:00
alexey-milovidov
1893d89ce3
Merge pull request #12448 from ClickHouse/fix-trash-rabbitmq
Fix trash from RabbitMQ
2020-07-14 01:11:37 +03:00
alesapin
1f576ee039 Some intermediate solution 2020-07-13 20:27:52 +03:00
Alexey Milovidov
cb46bca157 Merge branch 'master' into fix-trash-rabbitmq 2020-07-13 19:51:17 +03:00
alesapin
4a53264a86 Remove redundant and duplicated code 2020-07-13 19:19:08 +03:00
robot-clickhouse
0f23642a3d Auto version update to [20.7.1.1] [54437] 2020-07-13 18:26:03 +03:00
Alexander Tokmakov
fdb02edc7c Merge branch 'master' into database_atomic_improvements 2020-07-13 17:28:48 +03:00
Peng Jian
7579d65ebc Registering StorageS3 and TableFunctionS3 under two different names (S3 and COSN). 2020-07-13 22:13:30 +08:00
Alexander Kuzmenkov
d6e7ab5988 Fuzzing-related fixes 2020-07-13 16:58:48 +03:00
Alexander Tokmakov
04c3e7cab6 add implicit uuid macro 2020-07-13 16:18:38 +03:00
Denis Glazachev
f787702922 Merge branch 'master' into ldap-per-user-authentication
* master: (27 commits)
  Whitespaces
  Fix typo
  Fix UBSan report in base64
  Correct default secure port for clickhouse-benchmark #11044
  Remove test with bug #10697
  Update in-functions.md (#12430)
  Allow nullable key in MergeTree
  Update arithmetic-functions.md
  [docs] add rabbitmq docs (#12326)
  Lower block sizes and look what will happen #9248
  Fix lifetime_bytes/lifetime_rows for Buffer direct block write
  Retrigger CI
  Fix up  test_mysql_protocol failed
  Implement lifetime_rows/lifetime_bytes for Buffer engine
  Add comment regarding proxy tunnel usage in PocoHTTPClient.cpp
  Add lifetime_rows/lifetime_bytes interface (exported via system.tables)
  Tiny IStorage refactoring
  Trigger integration-test-runner image rebuild.
  Delete log.txt
  Fix test_mysql_client/test_python_client error
  ...
2020-07-13 15:46:27 +04:00
Anton Popov
a9530d2883 in-memory parts: fix reading from nested 2020-07-13 12:10:55 +03:00
alexey-milovidov
ae7eff98ed
Merge pull request #12433 from amosbird/np
Allow nullable key in MergeTree
2020-07-13 04:36:00 +03:00
Alexey Milovidov
8f2055b0a0 Fix trash from RabbitMQ 2020-07-13 04:11:48 +03:00
Amos Bird
cac5a89169
Allow nullable key in MergeTree 2020-07-12 22:21:51 +08:00
Alexey Milovidov
49f60ef3a4 Fix build 2020-07-12 08:26:33 +03:00
Alexey Milovidov
204a4af394 Rollback insertion error in Log engines #12402 2020-07-12 05:32:18 +03:00
Ivan Babrou
8784994d65 Allow conditions outside of PK with exact range
Conditions that are outside of PK are marked as `unknown` in `KeyCondition`,
so it's safe to allow them, as long as they are always combined by `AND`.
2020-07-11 18:59:26 -07:00
Azat Khuzhin
3bee98c6f0 Fix lifetime_bytes/lifetime_rows for Buffer direct block write 2020-07-12 01:16:05 +03:00
Ivan Babrou
d9d8d0242e Optimize PK lookup for queries that match exact PK range
Existing code that looks up marks that match the query has a pathological
case, when most of the part does in fact match the query.

The code works by recursively splitting a part into ranges and then discarding
the ranges that definitely do not match the query, based on primary key.

The problem is that it requires visiting every mark that matches the query,
making the complexity of this sort of look up O(n).

For queries that match exact range on the primary key, we can find
both left and right parts of the range with O(log 2) complexity.

This change implements exactly that.

To engage this optimization, the query must:

* Have a prefix list of the primary key.
* Have only range or single set element constraints for columns.
* Have only AND as a boolean operator.

Consider a table with `(service, timestamp)` as the primary key.

The following conditions will be optimized:

* `service = 'foo'`
* `service = 'foo' and timestamp >= now() - 3600`
* `service in ('foo')`
* `service in ('foo') and timestamp >= now() - 3600 and timestamp <= now`

The following will fall back to previous lookup algorithm:

* `timestamp >= now() - 3600`
* `service in ('foo', 'bar') and timestamp >= now() - 3600`
* `service = 'foo'`

Note that the optimization won't engage when PK has a range expression
followed by a point expression, since in that case the range is not continuous.

Trace query logging provides the following messages types of messages,
each representing a different kind of PK usage for a part:

```
Used optimized inclusion search over index for part 20200711_5710108_5710108_0 with 9 steps
Used generic exclusion search over index for part 20200711_5710118_5710228_5 with 1495 steps
Not using index on part 20200710_5710473_5710473_0
```

Number of steps translates to computational complexity.

Here's a comparison for before and after for a query over 24h of data:

```
Read 4562944 rows, 148.05 MiB in 45.19249672 sec.,   100966 rows/sec.,   3.28 MiB/sec.
Read 4183040 rows, 135.78 MiB in 0.196279627 sec., 21311636 rows/sec., 691.75 MiB/sec.
```

This is especially useful for queries that read data in order
and terminate early to return "last X things" matching a query.

See #11564 for more thoughts on this.
2020-07-11 12:26:54 -07:00
Denis Glazachev
edb6ef8c09 Merge commit 'ceac649c01b0158090cd271776f3219f5e7ff57c' into ldap-per-user-authentication
* commit 'ceac649c01b0158090cd271776f3219f5e7ff57c': (75 commits)
  [docs] split misc statements (#12403)
  Update 00405_pretty_formats.reference
  Update PrettyCompactBlockOutputFormat.cpp
  Update PrettyBlockOutputFormat.cpp
  Update DataTypeNullable.cpp
  Update 01383_remote_ambiguous_column_shard.sql
  add output_format_pretty_grid_charset setting in docs
  add setting output_format_pretty_grid_charset
  Added a test for #11135
  Update index.md
  RIGHT and FULL JOIN for MergeJoin (#12118)
  Update MergeTreeIndexFullText.cpp
  restart the tests
  [docs] add syntax highlight (#12398)
  query fuzzer
  Fix std::bad_typeid when JSON functions called with argument of wrong type.
  Allow typeid_cast() to cast nullptr to nullptr.
  fix another context-related segfault
  [security docs] actually, only admins can create advisories
  query fuzzer
  ...
2020-07-11 21:32:36 +04:00
Azat Khuzhin
32a45d0dee Implement lifetime_rows/lifetime_bytes for Buffer engine
Buffer engine is usually used on INSERTs, but right now there is no way
to track number of INSERTed rows per-table, since only summary metrics
exists:
- StorageBufferRows
- StorageBufferBytes

But it can be pretty useful to track INSERTed rows rate (and it can be
exposed via http_handlers for i.e. prometheus)
2020-07-11 16:06:11 +03:00
Azat Khuzhin
433fdffc19 Add lifetime_rows/lifetime_bytes interface (exported via system.tables) 2020-07-11 15:33:11 +03:00
Azat Khuzhin
84c93a6b02 Tiny IStorage refactoring 2020-07-11 15:17:06 +03:00
Anton Popov
fbec940e0f fix reading and check query for compact parts with different codecs 2020-07-11 02:33:36 +03:00
alexey-milovidov
e22547c29d
Merge pull request #12388 from ClickHouse/bloom-filter-arg-check
Check arguments of bloom filter index
2020-07-10 20:54:16 +03:00
alexey-milovidov
caef1d8e24
Update MergeTreeIndexFullText.cpp 2020-07-10 20:53:58 +03:00
alexey-milovidov
d819624d7c
Merge pull request #12378 from ClickHouse/allow-clear-column-with-dependencies
Allow to CLEAR column even if there are depending DEFAULT expressions
2020-07-10 20:18:14 +03:00
alexey-milovidov
031c773260
Merge pull request #12384 from ClickHouse/support-negative-float-constants-in-key-condition
Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables
2020-07-10 20:16:35 +03:00
Anton Popov
24f627e52c fix reading from compact parts with different codecs 2020-07-10 18:57:10 +03:00
Azat Khuzhin
610382b693 kafka: fix SIGSEGV if there is an message with error in the middle of the batch
ReadBufferFromKafkaConsumer does not handle the case when there is
message with an error on non first position in the current batch, since
it goes through messages in the batch after poll and stop on first valid
message.

But later it can try to use message as valid:
- while storing offset
- get topic name
- ...

And besides the message itself is also invalid (you can find this in the
gdb traces below).

So just filter out messages win an error error after poll.

SIGSEGV was with the following stacktrace:
    (gdb) bt
    3  0x0000000010f05b4d in rd_kafka_offset_store (app_rkt=0x0, partition=0, offset=0) at ../contrib/librdkafka/src/rdkafka_offset.c:656
    4  0x0000000010e69657 in cppkafka::Consumer::store_offset (this=0x7f2015210820, msg=...) at ../contrib/cppkafka/include/cppkafka/message.h:225
    5  0x000000000e68f208 in DB::ReadBufferFromKafkaConsumer::storeLastReadMessageOffset (this=0x7f206a136618) at ../contrib/libcxx/include/iterator:1508
    6  0x000000000e68b207 in DB::KafkaBlockInputStream::readImpl (this=0x7f202c689020) at ../src/Storages/Kafka/KafkaBlockInputStream.cpp:150
    7  0x000000000dd1178d in DB::IBlockInputStream::read (this=this@entry=0x7f202c689020) at ../src/DataStreams/IBlockInputStream.cpp:60
    8  0x000000000dd34c0a in DB::copyDataImpl<> () at ../src/DataStreams/copyData.cpp:21
    9  DB::copyData () at ../src/DataStreams/copyData.cpp:62
    10 0x000000000e67c8f2 in DB::StorageKafka::streamToViews () at ../contrib/libcxx/include/memory:3823
    11 0x000000000e67d218 in DB::StorageKafka::threadFunc () at ../src/Storages/Kafka/StorageKafka.cpp:488

And some information from it:

    (gdb) p this.current.__i
    $14 = (std::__1::__wrap_iter<cppkafka::Message const*>::iterator_type) 0x7f1ca8f58660

    # current-1
    (gdb) p $14-1
    $15 = (const cppkafka::Message *) 0x7f1ca8f58600
    (gdb) p $16.handle_
    $17 = {__ptr_ = {<std::__1::__compressed_pair_elem<rd_kafka_message_s*, 0, false>> = { __value_ = 0x7f203577f938}, ...}
    (gdb) p *(rd_kafka_message_s*)0x7f203577f938
    $24 = {err = RD_KAFKA_RESP_ERR__TRANSPORT, rkt = 0x0, partition = 0, payload = 0x7f202f0339c0, len = 63, key = 0x0, key_len = 0, offset = 0, _private = 0x7f203577f8c0}

    # current
    (gdb) p $14-0
    $28 = (const cppkafka::Message *) 0x7f1ca8f58660
    (gdb) p $28.handle_.__ptr_
    $29 = {<std::__1::__compressed_pair_elem<rd_kafka_message_s*, 0, false>> = { __value_ = 0x7f184f129bf0}, ...}
    (gdb) p *(rd_kafka_message_s*)0x7f184f129bf0
    $30 = {err = RD_KAFKA_RESP_ERR_NO_ERROR, rkt = 0x7f1ed44fe000, partition = 1, payload = 0x7f1fc9bc6036, len = 242, key = 0x0, key_len = 0, offset = 2394853582209,

    # current+1
    (gdb) p (*($14+1)).handle_.__ptr_
    $44 = {<std::__1::__compressed_pair_elem<rd_kafka_message_s*, 0, false>> = { __value_ = 0x7f184f129d30}, ...}
    (gdb) p *(rd_kafka_message_s*)0x7f184f129d30
    $45 = {err = RD_KAFKA_RESP_ERR_NO_ERROR, rkt = 0x7f1ed44fe000, partition = 1, payload = 0x7f1fc9bc612f, len = 31, key = 0x0, key_len = 0, offset = 2394853582210,
      _private = 0x7f184f129cc0}

    # distance from the beginning
    (gdb) p messages.__end_-messages.__begin_
    $34 = 65536
    (gdb) p ($14-0)-messages.__begin_
    $37 = 8965
    (gdb) p ($14-1)-messages.__begin_
    $38 = 8964

    # parsing info
    (gdb) p allowed
    $39 = false
    (gdb) p new_rows
    $40 = 1
    (gdb) p total_rows
    $41 = 8964

    # current buffer is invalid
    (gdb) p *buffer.__ptr_
    $50 = {<DB::ReadBuffer> = {<DB::BufferBase> = {pos = 0x7f202f0339c0 "FindCoordinator response error: Local: Broker transport failure", bytes = 47904863385, working_buffer = {
            begin_pos = 0x7f202f0339c0 "FindCoordinator response error: Local: Broker transport failure",
            end_pos = 0x7f202f0339c0 "FindCoordinator response error: Local: Broker transport failure"}, internal_buffer = {

v0: check message errors in ReadBufferFromKafkaConsumer::nextImpl() (but
this may lead to using of that messages after and SIGSEGV again, doh).
v2: skip messages with an error after poll.
2020-07-10 11:41:44 +03:00