Commit Graph

1523 Commits

Author SHA1 Message Date
Nikolai Kochetov
d328bfa41f Review fixes. Add setting max_optimizations_to_apply. 2021-02-26 19:29:56 +03:00
tavplubix
09f62447af
Merge pull request #21060 from mxzlxy/fix-bug-for-replicated-engine-parameter
fix default_replica_path and default_replica_name values are useless …
2021-02-26 17:18:56 +03:00
Anton Ivashkin
5b267b7eec Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into s3_zero_copy_replication 2021-02-26 13:15:18 +03:00
Anton Ivashkin
c891cf4557 Fixes by review response 2021-02-26 12:48:57 +03:00
alesapin
653d962cdd Fix style and add low-level check 2021-02-25 17:43:58 +03:00
alesapin
5c6c318737 Restrict mutations for engines which doesn't support them 2021-02-25 13:07:48 +03:00
alesapin
427e5f295c
Merge pull request #21007 from CurtizJ/alter-bad-conversions
Do not start mutation for alters with wrong type conversion
2021-02-24 10:08:15 +03:00
tavplubix
4661073750
Update src/Storages/MergeTree/registerStorageMergeTree.cpp 2021-02-23 00:09:20 +03:00
jennyma
6433ab3da6 fix default_replica_path and default_replica_name values are useless on Replicated(*)MergeTree engine when the engine needs specify other parameters 2021-02-22 18:34:01 +08:00
Denis Glazachev
c1158b50f4 Merge branch 'master' into row-policy-with-prewhere
* master: (155 commits)
  Update version_date.tsv after release 20.8.13.15
  Update version_date.tsv after release 20.12.7.3
  Update version_date.tsv after release 21.1.5.4
  Update version_date.tsv after release 21.2.4.6
  fix
  Add test to skip list
  Fix WriteBufferFromHTTPServerResponse usage in other places (add missing finalize())
  Fix WriteBufferFromHTTPServerResponse usage in odbc-bridge
  Update config.xml
  Suppress signed overflow in AggregateFunctionGroupArrayMoving 2
  Update BaseDaemon.cpp
  review suggestions
  Fix bash syntax in 01731_async_task_queue_wait
  Do not use view() in 01731_async_task_queue_wait to fix ANTLR parser
  Increase buffer for uncaught exception / std::terminate
  Even more better
  Fix uncaught exception when HTTP client goes away
  test for decimal ( p , s) in dictionaries
  Just little better
  Fixed style check
  ...
2021-02-21 13:25:47 +04:00
Denis Glazachev
dfde9de0e9 Remove commented-out code 2021-02-20 23:30:21 +04:00
tavplubix
a697b578bc
Merge pull request #16193 from ClickHouse/database_replicated
DatabaseReplicated: continuation of #10485
2021-02-20 19:39:34 +03:00
Anton Popov
48e188681c do not start mutation for alters with wrong type conversion 2021-02-20 19:11:08 +03:00
Nikolai Kochetov
673e24d7ef Refactor 2021-02-20 14:00:16 +03:00
Denis Glazachev
85ea5cafc4 Always apply row_level_filter immediately 2021-02-20 05:23:48 +04:00
Denis Glazachev
8cc72b8fe0 Merge branch 'master' into row-policy-with-prewhere
* master: (160 commits)
  Make Poco HTTP Server zero-copy again (#19516)
  Fixed documentation
  ccache 4.2+ does not requires any quirks for SOURCE_DATE_EPOCH
  Add a function `htmlOrXmlCoarseParse` to extract content from html or xml format string. (#19600)
  Reinterpret function added Decimal, DateTim64 support
  Add test
  Update InterpreterSelectQuery.cpp
  Improved serialization for data types combined of Arrays and Tuples. Improved matching enum data types to protobuf enum type. Fixed serialization of the Map data type. Omitted values are now set by default.
  Log stdout and stderr when failed to start docker in integration tests.
  Added comment
  Don't backport base commit of branch in the same branch (#20628)
  Fix fasttest retry for failed tests
  Dictionary create source with functions crash fix
  Added error reinterpretation tests
  Update run.sh
  Updated documentation
  fix subquery with limit
  Rename untyped function reinterpretAs into reinterpret
  ignore data store files
  Support vhost
  ...
2021-02-19 19:49:37 +04:00
Denis Glazachev
5e5b3b80ce Remove debug printouts 2021-02-19 19:46:52 +04:00
Ivan
414f470c79
Make Poco HTTP Server zero-copy again (#19516)
* Refactoring: part 1

* Refactoring: part 2

* Handle request using ReadBuffer interface

* Struggles with ReadBuffer's

* Fix URI parsing

* Implement parsing of multipart/form-data

* Check HTTP_LENGTH_REQUIRED before eof() or will hang

* Fix HTTPChunkedReadBuffer

* Fix build and style

* Fix test

* Resist double-eof

* Fix arcadian build
2021-02-19 15:51:26 +03:00
Alexander Tokmakov
bf6f64a3fb Merge branch 'master' into database_replicated 2021-02-16 01:28:19 +03:00
Nikolai Kochetov
85277d6a41 Comment debug output. 2021-02-15 22:54:47 +03:00
Nikolai Kochetov
ae73600fb0 Refactor row level security actions. 2021-02-15 22:48:06 +03:00
Denis Glazachev
45e90961f7 Store and process alias_actions in FilterInfo 2021-02-14 21:16:40 +04:00
Denis Glazachev
99a04b6c64 Merge branch 'master' into row-policy-with-prewhere
* master: (759 commits)
  Suppress UBSan report in Decimal comparison
  Suppress UBSan report in Decimal comparison
  Fix UBSan report in arrayDifference
  Update README.md
  Non significant change in AggregationCommon
  Print stack trace on SIGTRAP
  Fix dependent test
  Fix tests for better parallel run
  Add test for already working code
  Revert "Fix access control manager destruction order"
  Update index.md
  Update index.md
  Update index.md
  Bit more complicated example for isIPv4String - ru
  Bit more complicated example for isIPv4String
  cleanup
  Replace database with ordinary
  Added comments
  Split tests to make them stable
  Fixes
  ...

# Conflicts:
#	src/Storages/MergeTree/MergeTreeRangeReader.cpp
2021-02-14 02:24:00 +04:00
Denis Glazachev
79592b73f8 Store filter info in prewhere info instead of multiple prewheres
Some cleanups
2021-02-14 02:07:13 +04:00
Azat Khuzhin
68f23b7087 Improve logging during MergeTree reading
- Remove "Not using primary index on part {}" message (too noisy)
- Add number of total marks before filtering by primary key into the
  common message
- Make "Index {} has dropped {} / {} granules." not per-part, but
  per-query
2021-02-13 18:08:55 +03:00
Pavel Kruglov
f0163c2acf Don't create empty parts on INSERT 2021-02-12 17:02:04 +03:00
alesapin
a2bb586170
Merge pull request #20335 from ClickHouse/fix_scheduling_of_non_successful_background_tasks
Fix backoff for failed background tasks in replicated merge tree
2021-02-12 10:06:04 +03:00
alexey-milovidov
7b55eb8c58
Merge pull request #20178 from amosbird/delayinactive
Delay or throw insertion when too many inactive parts
2021-02-11 20:52:04 +03:00
alesapin
3253638969 Fix backoff for failed background tasks in replicated merge tree 2021-02-11 14:46:18 +03:00
alesapin
56453256b7
Merge pull request #20300 from ClickHouse/restrict_to_drop_version_column
Don't allow to drop or rename version column
2021-02-11 11:04:00 +03:00
alesapin
121c5c9605
Merge pull request #20279 from ClickHouse/relax_check_in_data_writer
Relax check in data writer for non adaptive tables
2021-02-11 10:57:07 +03:00
alesapin
c2bb2c2902
Merge pull request #20097 from ClickHouse/remove-adding-missed-step
Build actions dag to evaluate missing defaults.
2021-02-11 10:51:21 +03:00
alexey-milovidov
e33477a16e
Merge pull request #20282 from ClickHouse/dont_allow_ttl_for_old_syntax
Don't allow TTL in old syntax
2021-02-11 00:46:30 +03:00
Anton Popov
365e2fb746
Merge pull request #20231 from fuwhu/refine-loadDataParts-code
refine code in MergeTreeData::loadDataParts to avoid parsing WAL file as data part
2021-02-10 22:42:31 +03:00
alesapin
891fce3275 Don't allow to drop or rename version column 2021-02-10 22:23:48 +03:00
alesapin
a63d73db58 Don't allow TTL in old syntax 2021-02-10 15:09:12 +03:00
alesapin
420f2958e2
Merge pull request #19991 from ClickHouse/constant-propagation-fix
Do not use inputs which values are known constants in ActionsDAG.
2021-02-10 14:13:52 +03:00
alesapin
21faea1f68 Relax check in data writer for non adaptive tables 2021-02-10 13:43:31 +03:00
Amos Bird
afb5846a24
refactor 2021-02-10 11:22:24 +08:00
HuFuwang
4ac8860cea refine code in MergeTreeData::loadDataParts to avoid parsing WAL file as data part. 2021-02-10 10:13:43 +08:00
tavplubix
86530e603c
Merge pull request #20027 from bharatnc/ncb/event_time_micro_part_log
add event_time_microseconds field to system.part_log table
2021-02-09 14:46:40 +03:00
Nikolai Kochetov
229acbad5d
Merge pull request #20077 from ClickHouse/remove-adding-const-column-transform
Remove AddingConstColumn step and transform.
2021-02-09 11:28:44 +03:00
Alexander Tokmakov
5e0cbfe972 Merge branch 'master' into database_replicated 2021-02-08 23:01:12 +03:00
Alexander Tokmakov
78c1d69b8c better code 2021-02-08 22:36:17 +03:00
Nicolae Vartolomei
c336cac306 Remove dead code 2021-02-08 17:36:58 +00:00
Nicolae Vartolomei
a8f37e7704 Remove dead code 2021-02-08 17:19:35 +00:00
bharatnc
60c8dfa22c PartLog - populate event_time_microseconds for MergeParts 2021-02-08 07:25:44 -08:00
Nikolai Kochetov
d7fa440356
Merge branch 'master' into remove-adding-const-column-transform 2021-02-08 15:49:23 +03:00
Anton Ivashkin
4d44d75bc7 Fix build after merge one more time 2021-02-08 14:45:10 +03:00
Amos Bird
109a392e0c
Fix ubsan 2021-02-08 19:41:16 +08:00
Anton Ivashkin
1b6ce056a5 Merge master 2021-02-08 12:34:20 +03:00
Amos Bird
ab5c7b75a4
Delay or throw insertion when too many inactive parts 2021-02-07 19:03:55 +08:00
Maxim Akhmedov
0cd3601928 Allow using MergeTreeWhereOptimizer not only with MergeTree-based storages 2021-02-06 18:08:42 +03:00
alesapin
011109c82a
Merge pull request #17348 from xjewer/alex/CLICKHOUSE-606_deduplication_UUID
CLICKHOUSE-606: query deduplication based on parts' UUID
2021-02-05 22:47:34 +03:00
Anton Ivashkin
e64c63c611 Merge master 2021-02-05 20:10:06 +03:00
Anton Ivashkin
df6c882aab Fix build after merge 2021-02-05 18:52:40 +03:00
Nikolai Kochetov
85c175883e Rename functions. 2021-02-05 18:11:26 +03:00
alesapin
7cbc135e72 More isolated code 2021-02-05 12:54:34 +03:00
Nikolai Kochetov
d9aa1096cf Build actions dag to evaluate missing defaults. 2021-02-04 23:36:50 +03:00
vdimir
30f98795bc
Merge remote-tracking branch 'upstream/master' into disk-s3-backup-restore-metadata 2021-02-04 18:38:28 +03:00
Nikolai Kochetov
e3d6ffd9dc Remove AddingConstColumn step and transform. 2021-02-04 17:25:11 +03:00
Alexey Milovidov
474f151707 More docs for MergeTreeSettings 2021-02-04 08:26:22 +03:00
Alexey Milovidov
dc2afc4795 Fix double whitespace #18383 2021-02-03 21:12:14 +03:00
Alexander Tokmakov
d010f97db0 Merge branch 'master' into database_replicated 2021-02-03 20:13:25 +03:00
Pavel Kovalenko
b7151aa754 Merge remote-tracking branch 'origin/master' into disk-s3-backup-restore-metadata
# Conflicts:
#	src/Disks/DiskDecorator.h
#	src/Disks/IDisk.h
#	src/Disks/S3/DiskS3.cpp
2021-02-03 14:22:18 +03:00
Alexander Tokmakov
6456ccf0da better test 2021-02-02 22:39:04 +03:00
Alexey Milovidov
12485eee6b Fix some of the issues found by Coverity 2021-02-02 22:07:23 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
2021-02-02 16:53:39 +00:00
Nikolai Kochetov
b2d7790794 Fix test. 2021-02-02 17:30:54 +03:00
Maksim Kita
d0151de4bb
Merge pull request #19608 from kreuzerkrieg/Add_IStoragePolicy_interface
Add IStoragePolicy interface
2021-02-02 11:03:20 +03:00
Pavel Kruglov
a3f1b825cc Fix build 2021-02-01 21:17:12 +03:00
alexey-milovidov
6b2f0435c8
Merge pull request #19375 from Avogar/select-final
Improve do_not_merge_across_partitions_select_final optimization.
2021-02-01 20:31:08 +03:00
alesapin
2aa8a6304b
Merge pull request #15450 from CurtizJ/fix-ttl-group-by
Fix some cases of TTL expressions
2021-02-01 16:48:07 +03:00
Nikolai Kochetov
19e4a33f9d
Merge pull request #19544 from amosbird/limitconcurrency
Per MergeTree table query limit
2021-02-01 16:09:12 +03:00
alesapin
d9598c47b4
Merge pull request #19702 from ClickHouse/fix_rare_bug_after_part_corruption
Fix incorrect virtual_parts after part corruption
2021-02-01 10:09:48 +03:00
alexey-milovidov
d19feb724b
Merge pull request #19799 from CurtizJ/fix-uint8-filtering
Fix filtering by Uint8 greater than 127
2021-01-29 21:34:42 +03:00
Denis Glazachev
31706e69bc Merge branch 'master' into row-policy-with-prewhere
* master: (75 commits)
  Temporary retrun dicttoxml to integration tests runner
  Fix option
  Fix dependency on ODBC for Yandex internal build
  Remove useless headers
  Add test-connect tool
  Revert "Remove old non-automated test"
  Fix trivial bug in arrayEnumerateUniq
  Fix UBSan report in "round"
  Translate comment in test
  Added test to ANTLR skip_list.json
  Update Client.cpp
  Fixed race in tests
  Fixed test_kafka_flush_by_time test.
  LowCardinality UUID fix
  Update mergetree.md
  Clickhouse client query param CTE added test
  correct style zh doc typo
  Fixed data race in function DictGetNoType
  add empty line after error messages in client
  Fixed style issues
  ...

# Conflicts:
#	src/Storages/MergeTree/MergeTreeReadPool.h
2021-01-29 22:26:11 +04:00
Pavel Kruglov
78371e15dc Update test, reduce num_threads_for_lonely_parts if data is small 2021-01-29 21:00:08 +03:00
alesapin
f4236fd765 Fix style 2021-01-29 20:12:53 +03:00
Nikolai Kochetov
47a7273fe3 Fix build 2021-01-29 19:21:52 +03:00
Nikolai Kochetov
afdc9635cb Update MergeTreeRangeReader 2021-01-29 19:12:53 +03:00
alesapin
c373d92a80 Less strict check 2021-01-29 18:50:08 +03:00
Nikolai Kochetov
4929fe2063 Update MergeTreeRangeReader 2021-01-29 18:13:09 +03:00
Pavel Kruglov
71f4acd48b Use one pool for lonely parts, update tests 2021-01-29 17:30:14 +03:00
Pavel Kruglov
a437ee4e31 Merge branch 'master' of github.com:ClickHouse/ClickHouse into select-final 2021-01-29 14:25:47 +03:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
Anton Popov
031132038b fix filtering by uint8 greater than 127 2021-01-29 10:39:18 +03:00
alexey-milovidov
fa7a795c3a
Merge pull request #19588 from ClickHouse/disable-checksums-on-read
Allow to disable checksums on read
2021-01-29 02:37:26 +03:00
alesapin
2881c830e3 Merge branch 'master' into fix_rare_bug_after_part_corruption 2021-01-28 23:16:52 +03:00
Denis Glazachev
009d1e3fc5 Merge branch 'master' into row-policy-with-prewhere
* master: (247 commits)
  Update 01014_lazy_database_basic.sh
  Update 00459_group_array_insert_at.sql
  Same for ALTER
  Syntax and links
  Added test.
  Check where and prewhere identifiers exist.
  Remove redundant lsof
  Add lsof to fasttest
  Add new line to status
  Make integration odbc tests idempotent
  Minor code improvement in JOIN
  Arcadia does not have bitmaps
  Make Fuzzer more reliable
  Add missing lsof for fasttest docker image
  Update test
  The test most likely would not work in Arcadia
  Less parser depth
  Fix stack overflow in coroutine
  Fuzzer: better messages.
  Fixed build.
  ...
2021-01-28 21:48:12 +04:00
alesapin
be1104d6c1 Fix typos 2021-01-28 19:07:47 +03:00
alesapin
c1d18fc8a2 Fix very rare case 2021-01-28 18:10:53 +03:00
alesapin
a7ca26fec4
Merge pull request #19708 from ClickHouse/fix_flaky_test_multiple_ttl
Fix flaky test test_concurrent_ttl_merges
2021-01-28 15:26:56 +03:00
alesapin
d14f4a525b Merge branch 'master' into fix_rare_bug_after_part_corruption 2021-01-28 11:09:41 +03:00
alesapin
402e031a22 Throw exception only in debug mode 2021-01-28 11:07:18 +03:00
Alexey Milovidov
065d22cf2b Merge branch 'master' into disable-checksums-on-read 2021-01-28 03:07:31 +03:00
Anton Popov
e5125b8c73 add comments and test for compatibility 2021-01-28 02:39:15 +03:00
alesapin
c74631c650 Fix test and add logical error 2021-01-27 21:54:05 +03:00
Anton Popov
a8f3078ce9 Merge remote-tracking branch 'upstream/master' into HEAD 2021-01-27 19:48:55 +03:00
alesapin
5622e6daa6 Fix rare max_number_of_merges_with_ttl_in_pool limit overrun for non-replicated MergeTree 2021-01-27 14:56:12 +03:00