thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
111,211 Commits 9,326 Branches 1,783 Tags 2.1 GiB
13f29a7242
Commit Graph

12060 Commits

Author SHA1 Message Date
Kruglov Pavel
a06596bdba
Fix special build 2023-02-28 14:33:41 +01:00
alesapin
712a8309c5
Merge pull request #46976 from ClickHouse/decrease-log-level 
Decrease log level in "disks"
 2023-02-28 12:28:30 +01:00
kssenii
d4e6fc4546 Fix test 2023-02-27 21:43:45 +01:00
Nikolai Kochetov
738551dbf1
Merge pull request #46624 from CheSema/detached_mt 
add multitreading for StorageSystemDetachedParts
 2023-02-27 20:54:28 +01:00
Alexey Milovidov
29da7fc965 Decrease log level in "disks" 2023-02-27 17:48:38 +01:00
kssenii
ad88251ee7 Fix tests 2023-02-27 17:42:04 +01:00
Anton Popov
7b85b35534 fix skip indexes 2023-02-27 15:11:25 +00:00
Anton Popov
caa1d22759
Merge branch 'master' into fix-issues-with-sparse 2023-02-27 15:25:19 +01:00
Robert Schulze
6f0709032d
Merge pull request #46921 from ClickHouse/rs/inv-idx-checksum 
A better alternative to #46344
 2023-02-27 14:44:44 +01:00
Nikolai Kochetov
3392b96cc3
Merge branch 'master' into aggregate-projections-analysis-query-plan 2023-02-27 14:07:04 +01:00
kssenii
b19264cf9f Remove redundant 2023-02-27 12:32:13 +01:00
Robert Schulze
4c9b9b362c
Replace dynamic_cast by typeid_cast 2023-02-27 10:01:21 +00:00
Robert Schulze
cc0c0c6133
A better alternative to #46344 
The (experimental) inverted index writes/reads files different from the
standard files written by the other skip indexes. The original problem
was that with database engine "ordinary", DROP TABLE of a table with
inverted index finds unknown files in persistence and complains. The
same will happen with engine "atomic" but deferred. As a hotfix, the
error was silenced by explicitly adding the four files created in a
specific test to the deletion code.

This PR tries a cleaner solution where all needed files are provided via
the normal checksum structure. One drawback remains which is that the
affected files were written earlier and we don't have their checksums
available. Therefore, the inverted index is currently excluded from
CHECK TABLE.

Minimal repro:
  SET allow_experimental_inverted_index = 1;
  DROP TABLE IF EXISTS tab;
  CREATE TABLE tab(s String, INDEX af(s) TYPE inverted(2)) ENGINE = MergeTree() ORDER BY s;
  INSERT INTO tab VALUES ('Alick a01');
  CHECK TABLE tab;
  DROP TABLE IF EXISTS tab;

  run ./clickhouse-test with --db-engine Ordinary
 2023-02-26 21:06:38 +00:00
Alexey Milovidov
73aec86f5a Change exception message 2023-02-26 21:36:26 +01:00
Sema Checherinda
1666527dae do listing combined with getting size in one thread 2023-02-26 12:09:05 +01:00
Alexey Milovidov
d448a48f16
Merge pull request #46831 from azat/merge-default-prewhere 
Allow PREWHERE for Merge with different DEFAULT expression for column
 2023-02-26 10:02:34 +03:00
Robert Schulze
5e68d50b39
Merge pull request #46752 from ClickHouse/rs/periods-in-kafka-topics 
Allow configuration of Kafka topics with periods
 2023-02-26 07:49:28 +01:00
Alexey Milovidov
8f40bbd5b4
Revert "Fix several RENAME COLUMN bugs." 2023-02-26 05:47:05 +03:00
Alexey Milovidov
eb89ab471f
Merge pull request #45911 from ClickHouse/mutations_rename_hang 
Fix several `RENAME COLUMN` bugs.
 2023-02-26 01:56:40 +03:00
Alexey Milovidov
656af55831
Merge pull request #46384 from ClickHouse/fixes_for_993 
Fixes for 993
 2023-02-26 01:55:20 +03:00
Robert Schulze
fbba9e319d
Merge branch 'master' into rs/periods-in-kafka-topics 2023-02-25 21:45:20 +01:00
Robert Schulze
3618e37dd2
Reduce indentation by early out 2023-02-25 15:36:59 +00:00
Sema Checherinda
0c9643f1ed make listing in parrallel as well 2023-02-25 16:27:27 +01:00
Nikolai Kochetov
57b5f9e7d7 Fix case with no detached parts for table. 2023-02-25 13:33:20 +01:00
Sema Checherinda
e2bfa83429 cancel workers when exception 2023-02-25 13:33:20 +01:00
Sema Checherinda
08dc874a37 do not run support threads if no tasks left 2023-02-25 13:33:20 +01:00
Sema Checherinda
e5cbe4311e create limites count of support threads for a block 2023-02-25 13:33:20 +01:00
Sema Checherinda
ef0c1841af work with comments on review 2023-02-25 13:33:20 +01:00
Sema Checherinda
3b58249320 pack several files in one thread 2023-02-25 13:33:20 +01:00
Sema Checherinda
8abc1f0d5d fix build and style 2023-02-25 13:33:20 +01:00
Sema Checherinda
dce319e1c5 add multitreading for StorageSystemDetachedParts 2023-02-25 13:33:20 +01:00
Alexander Tokmakov
cad1e0b768 fix 2023-02-25 01:18:34 +01:00
kssenii
c2bcc4119f Better 2023-02-24 20:57:37 +01:00
kssenii
96b57c8a65 Better support for equal keys 2023-02-24 20:38:40 +01:00
Anton Popov
3a947ecddc do not allow const and non-deterministic secondary indexes 2023-02-24 19:17:44 +00:00
Alexander Tokmakov
ca1c793cac
Merge pull request #46820 from Algunenano/avoid_KeeperMultiException_system_errors 
Don't report ZK node exists to system.errors when a block was created concurrently by a different replica
 2023-02-24 19:42:32 +03:00
kssenii
6224ca9cd6 Fix 2023-02-24 17:08:05 +01:00
Robert Schulze
6402050125
Fixup comments 2023-02-24 16:59:38 +01:00
Azat Khuzhin
86e9b131c7 Allow PREWHERE for Merge with different DEFAULT expression for column 
It can be not that special to have Merge engine w/o default expression,
while MergeTree will have it. So improve PREWHERE for such kinds, and
also allow PREWHERE for DEFAULT and MATERIALIZED (since both are
physical columns on disk and this make sense).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-24 16:33:49 +01:00
Antonio Andelic
56a126f7af Fix progress bar with URL 2023-02-24 14:49:14 +00:00
kssenii
f2a9109fe1 Merge remote-tracking branch 'upstream/master' into named-collections-granular-access 2023-02-24 15:20:46 +01:00
Alexander Gololobov
b43b98f7c0
Merge pull request #46785 from ClickHouse/fix_combined_prewhere_column 
Fix combined PREWHERE column accumulated from multiple steps
 2023-02-24 14:00:17 +01:00
Kruglov Pavel
2150b836b1
Merge branch 'master' into fix-kafka-protobuf 2023-02-24 12:48:07 +01:00
Raúl Marín
7ff8d2d3d5 Don't report ZK node exists to system.errors 2023-02-24 12:36:29 +01:00
pufit
2fbe75957d Add default column for system.settings and system.server_settings 2023-02-23 18:24:57 -05:00
Anton Popov
d926713cf5 Merge remote-tracking branch 'upstream/master' into HEAD 2023-02-23 23:04:22 +00:00
Anton Popov
5e5e802348 return back optimization for ALTER MODIFY Nullable 2023-02-23 22:55:47 +00:00
Mikhail f. Shiryaev
18a20268d8
Merge pull request #46792 from ClickHouse/23.3-prepare 
Update version after release
 2023-02-23 23:51:28 +01:00
Alexander Tokmakov
fe92fd8a61 Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-02-23 22:35:31 +01:00
Robert Schulze
6906cafa99
Change to S3 cfg syntax 2023-02-23 20:04:41 +00:00
Antonio Andelic
1efba8995d Update version to 23.3.1.2537 2023-02-23 19:35:57 +00:00
Alexander Gololobov
47f51fe187
Merge pull request #46771 from ClickHouse/fix_prewhere_logging 
More concise logging at trace level for PREWHERE steps
 2023-02-23 20:20:30 +01:00
Alexander Gololobov
8b22373488 Apply filter at last step if it has been accumulated from multiple steps 2023-02-23 16:58:28 +01:00
Alexander Gololobov
cbcb1a7852 More concise logging at trace level 2023-02-23 13:31:56 +01:00
Kseniia Sumarokova
7e064ccbd4
Merge pull request #46667 from kssenii/hide-named-collections-under-setting 
Allow to hide only values from system.named_collections while making collection visible
 2023-02-23 11:44:50 +01:00
Kseniia Sumarokova
30a8eb0c2f
Merge pull request #46670 from kssenii/hide-disk-setting 
Hide arguments of custom disk merge tree setting
 2023-02-23 11:42:53 +01:00
Alexey Milovidov
c0bc549e77
Merge pull request #46709 from azat/bg-pool-no-query 
BackgroundSchedulePool should not have any query context
 2023-02-23 10:24:46 +03:00
Alexey Milovidov
e3f24b2b68
Merge pull request #46744 from ClickHouse/remove_unused_field 
Remove unused MergeTreeReadTask::remove_prewhere_column
 2023-02-23 08:02:54 +03:00
Robert Schulze
81bf43157f
Allow configuration of Kafka topics with periods 
The Kafka table engine allows global configuration and per-Kafka-topic
configuration. The latter uses syntax <kafka_TOPIC>, e.g. for topic
"football":

  <kafka_football>
      <retry_backoff_ms>250</retry_backoff_ms>
      <fetch_min_bytes>100000</fetch_min_bytes>
  </kafka_football>

Some users had to find out the hard way that such configuration doesn't
take effect if the topic name contains a period, e.g. "sports.football".
The reason is that ClickHouse configuration framework already uses
periods as level separators to descend the configuration hierarchy.
(Besides that, per-topic configuration at the same level as global
configuration could be considered ugly.)

Note that Kafka topics may contain characters "a-zA-Z0-9._-" (*) and
a tree-like topic organization using periods is quite common in
practice.

This PR deprecates the existing per-topic configuration syntax (but
continues to support it for backward compat) and introduces a new
per-topic configuration syntax below the global Kafka configuration of
the form:

<kafka>
   <topic name="football">
       <retry_backoff_ms>250</retry_backoff_ms>
       <fetch_min_bytes>100000</fetch_min_bytes>
   </topic>
</kafka>

The period restriction doesn't apply to XML attributes, so <topic
name="sports.football"> will work. Also, everything Kafka-related is
below <kafka>.

Considered but rejected alternatives:
- Extending Poco ConfigurationView with custom separators (e.g."/"
  instead of "."). Won't work easily because ConfigurationView only
  builds a path but defers descending the configuration tree to the
  normal configuration classes.
- Reloading the configuration file in StorageKafka (instead of reading
  the loaded file) but with a custom separator. This mode is supported
  by XML configuration. Too ugly and error-prone since the true
  configuration is composed from multiple configuration files.

(*) https://stackoverflow.com/a/37067544
 2023-02-22 20:35:09 +00:00
Alexander Tokmakov
e660c0838c fix multipart requests 2023-02-22 17:54:35 +01:00
Alexander Gololobov
67dcd9694c Remove unused MergeTreeReadTask::remove_prewhere_column 2023-02-22 17:49:22 +01:00
pufit
8e7533fa57
Merge pull request #46564 from AVMusorin/update-time-distribution-queue 
Added `last_exception_time` column into distribution_queue table
 2023-02-22 10:43:35 -05:00
kssenii
c4761d6cc6 Fix checks 2023-02-22 14:27:43 +01:00
kssenii
f0ca3ef207 Merge remote-tracking branch 'upstream/master' into named-collections-granular-access 2023-02-22 14:02:29 +01:00
kssenii
f518ddd96d Merge remote-tracking branch 'upstream/master' into named-collections-finish 2023-02-22 13:43:18 +01:00
Kruglov Pavel
3ba3fdbfa3
Merge pull request #46607 from kssenii/delay-loading-of-named-collections 
Do not load named collections on server startup (on first access instead)
 2023-02-22 13:22:34 +01:00
Azat Khuzhin
2ca47a6eb6 BackgroundSchedulePool should not have any query context 
BackgroundSchedulePool is used for some peridic jobs, not from the query
context, i.e. flush of Buffer table.

And for such jobs there cannot be any query context, and more
importantly it will not work correctly since that query_context will
eventually expires.

And this is the reason of this failures [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/46668/015991bc5e20c787851050c2eaa13f0fef3aac00/stateless_tests_flaky_check__asan_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-22 10:50:51 +01:00
Alexander Tokmakov
592af6d652 fix incomplete interst through http 2023-02-22 02:34:03 +01:00
AVMusorin
0bf0fe488e
added last_exception_time column into distribution_queue table 2023-02-21 23:46:57 +01:00
Alexander Tokmakov
f5c232707d cancel insert queries correctly 2023-02-21 22:04:39 +01:00
Alexander Gololobov
b61bb56a5a
Merge pull request #46611 from ClickHouse/prewhere_progress 
Fix reporting progress when rows are filtered in PREWHERE
 2023-02-21 21:27:56 +01:00
kssenii
b15be14792 Fix table function remote 2023-02-21 18:55:43 +01:00
kssenii
9a7c71b78e Allow to hide only values from system.named_collections 2023-02-21 18:07:57 +01:00
Alexey Milovidov
848286eca7
Merge pull request #40945 from arthurpassos/add_select_final_mt_setting 
Add user setting to force select final at query level
 2023-02-21 19:58:45 +03:00
Nikolai Kochetov
c5f93eb108 Fix more tests. 2023-02-21 15:44:50 +00:00
kssenii
d84215d005 Fix style check 2023-02-21 16:20:57 +01:00
Kseniia Sumarokova
655e7f29f7
Merge pull request #46595 from kssenii/update-rabbit-producer 
Fix rabbitmq test
 2023-02-21 16:18:55 +01:00
Anton Popov
b66a5bdcf0 fix mutations with sparse columns 2023-02-21 15:13:29 +00:00
kssenii
b3ca976f5f Finish for streaming engines 2023-02-21 14:50:55 +01:00
Alexander Gololobov
5354b0d069
Merge pull request #46140 from ClickHouse/split_prewhere_dag 
Split PREWHERE DAG into multiple read steps
 2023-02-21 14:37:09 +01:00
kssenii
68e06ecb99 Replace for table function remote, and external storage 2023-02-21 14:33:37 +01:00
Kseniia Sumarokova
09a979f106
Merge branch 'master' into update-rabbit-producer 2023-02-21 11:50:30 +01:00
Vladimir C
a791250a4a
Merge pull request #38614 from bharatnc/ncb/profile-events-partlog 2023-02-21 11:06:44 +01:00
Kseniia Sumarokova
24d0490ef3
Merge branch 'master' into delay-loading-of-named-collections 2023-02-21 00:24:18 +01:00
Kruglov Pavel
828a06b511
Merge branch 'master' into fix-kafka-protobuf 2023-02-20 21:45:43 +01:00
Kruglov Pavel
b0424c1021
Merge pull request #46171 from Avogar/insert-null-as-default 
Use default of column type in `insert_null_as_default` if column DEFAULT values is not specified
 2023-02-20 21:45:02 +01:00
kssenii
a54b011670 Finish for mysql 2023-02-20 21:37:38 +01:00
Nikolai Kochetov
ee597b939f Merge branch 'master' into aggregate-projections-analysis-query-plan 2023-02-20 19:46:52 +00:00
Arthur Passos
4f2fcf763b
Merge branch 'master' into add_select_final_mt_setting 2023-02-20 16:22:14 -03:00
Kseniia Sumarokova
47604ad77e
Merge pull request #46106 from kssenii/return-materialized-postrgesql-tests-back 
Enable disabled MaterializedPostgreSQL tests back in CI
 2023-02-20 19:04:58 +01:00
Alexander Gololobov
c84a64d22f Return chunks with 0 rows from MergeTreeSource to report progress when rows are filtered in PREWHERE 2023-02-20 17:35:01 +01:00
kssenii
9d16205c8a Load named collections on first access 2023-02-20 16:33:45 +01:00
Nikolai Kochetov
84e442e620 Fixing build and style. 2023-02-20 14:21:20 +00:00
kssenii
80a0aa1327 Merge remote-tracking branch 'upstream/master' into return-materialized-postrgesql-tests-back 2023-02-20 14:37:39 +01:00
kssenii
361678ad73 rabbitmq-test-fix 2023-02-20 14:16:40 +01:00
Nikolai Kochetov
2a8460d962 Merge branch 'master' into aggregate-projections-analysis-query-plan 2023-02-20 13:54:05 +01:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
Arthur Passos
3d9b23e5f3
Merge branch 'master' into add_select_final_mt_setting 2023-02-19 15:01:41 -03:00
Alexey Milovidov
5359dcc014
Merge pull request #46554 from ucasfl/iceberg-fix 
fix build without avro
 2023-02-19 14:35:29 +03:00
Alexander Gololobov
520b381675 Fixes for executing actions on block with unknown row count 2023-02-19 12:14:23 +01:00
Maksim Kita
6dffe52e68
Merge pull request #46471 from kitaisreal/analyzer-planner-fixes-before-enable-by-default 
Analyzer planner fixes before enable by default
 2023-02-19 13:49:18 +03:00
flynn
7b4fbf33b3 fix 2023-02-19 03:47:14 +00:00
Alexander Gololobov
0ed0b191f8 Fix for Nullable step filter column 2023-02-18 19:12:32 +01:00
flynn
177261a5ac fix typo 2023-02-18 17:34:37 +00:00
flynn
830398f0bd fix build without avro 
fix
 2023-02-18 17:32:35 +00:00
Alexander Gololobov
88fb6c3c25 Fix for incorrect combined PREWHERE column 2023-02-18 18:29:16 +01:00
Maksim Kita
cbd961de98 Fixed code review issues 2023-02-18 17:06:00 +01:00
Nikolay Degterinsky
f93aae80b8
Merge pull request #46407 from evillique/parameters-rename-query 
Add parameterized RENAME queries
 2023-02-18 16:04:55 +01:00
Kseniia Sumarokova
b0ea047e76
Merge pull request #46533 from kssenii/fix-disk-setting 
Allow `disk` merge tree setting in query to override config merge tree setting `storage_policy`
 2023-02-18 12:36:54 +01:00
pufit
4496db66b2 Implement system.server_settings 2023-02-17 23:46:09 -05:00
Kseniia Sumarokova
bda4c94f66
Merge pull request #45384 from ucasfl/iceberg 2023-02-17 23:38:24 +01:00
Kseniia Sumarokova
a0b28edd46
Merge pull request #46521 from kssenii/fix-stress-test-logical-errors 
Fix stress test
 2023-02-17 21:01:20 +01:00
Nikolay Degterinsky
5104b2e8bf
Merge branch 'master' into parameters-rename-query 2023-02-17 17:30:37 +01:00
kssenii
afb36dd120 Allow disk setting to override storage_policy from config merge tree settings insteaf of exception 2023-02-17 17:11:29 +01:00
Maksim Kita
6c1436f78e Fixed tests 2023-02-17 16:50:46 +01:00
Antonio Andelic
ab51c1d975
Merge branch 'master' into custom-key-parallel-replicas 2023-02-17 15:06:41 +01:00
flynn
ebd88aab41
Merge branch 'master' into iceberg 2023-02-17 21:53:29 +08:00
Vladimir C
0f182f6164
Merge pull request #46454 from azat/merge-alias-prewhere 2023-02-17 14:10:52 +01:00
flynn
b3a9468661 fix 2023-02-17 12:42:24 +00:00
flynn
4b1d997b82 fix 2023-02-17 12:27:53 +00:00
kssenii
bae85c50e9 Fix stress test 2023-02-17 12:40:17 +01:00
Alexander Tokmakov
9a664efeb6
Merge branch 'master' into fixes_for_993 2023-02-17 14:17:08 +03:00
Sema Checherinda
b551b9fb3a
Merge pull request #46452 from CheSema/resolve-race-get-csn 
resolve race in getCSNAndAssert
 2023-02-17 11:31:58 +01:00
flynn
1d5b7ebc73 fix 2023-02-17 09:01:13 +00:00
flynn
561b575111 fix style 2023-02-17 08:35:38 +00:00
flynn
a39f6f419b refactor 2023-02-17 08:27:52 +00:00
Arthur Passos
ba6b7b1610 fix test_alias_column 2023-02-16 16:01:25 -03:00
kssenii
417052e4b2 Fix remaining tests 2023-02-16 18:56:19 +01:00
Azat Khuzhin
beff5ac795 Fix PREWHERE for Merge with different default types 
In case of underlying table has an ALIAS for this column, while in Merge
table it is not marked as an alias, there will NOT_FOUND_COLUMN_IN_BLOCK
error.

Further more, when underlying tables has different default type for the
column, i.e. one has ALIAS and another has real column, then you will
also get NOT_FOUND_COLUMN_IN_BLOCK, because Merge engine should take
care of this.

Also this patch reworks how PREWHERE is handled for Merge table, and now
if you use PREWHERE on the column that has the same type and default
type (ALIAS, ...) then it will be possible, and only if the type
differs, it will be prohibited and throw ILLEGAL_PREWHERE error.

And last, but not least, also respect this restrictions for
optimize_move_to_prewhere.

v2: introduce IStorage::supportedPrewhereColumns()
v3: Remove excessive condition for PREWHERE in StorageMerge::read()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-16 17:21:53 +01:00
kssenii
75d62ee24a Remove support of feature which does not work reliably 2023-02-16 16:41:31 +01:00
Alexander Gololobov
a6061e022e Merge branch 'master' into split_prewhere_dag 2023-02-16 14:43:58 +01:00
flynn
eae73a1511 fix 2023-02-16 13:42:19 +00:00
youennL-cs
6526c2a8ab
[RFC] Replacing merge tree new engine (#41005) 
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add keyword 'CLEANUP' when OPTIMIZE

* Cleanup uniquely when it's a replacingMergeTree

* Propagate CLEANUP information and change from 'with_cleanup' to 'cleanup'

* Cleanup data flagged as 'is_deleted'

* Fix merge when optimize and add a test

* Fix OPTIMIZE and INSERT + add tests

* New fix for cleanup at the merge

* Cleanup debug logs

* Add the SETTINGS option 'clean_deleted_rows' that can be 'never' or 'always'

* Fix regression bug; Now REplicatedMergeTree can be called as before without 'is_deleted'

* Add Replicated tests

* Disable tag 'long' for our test and cleanup some white spaces

* Update tests

* Fix tests and remove additional useless whitespace

* Fix replica test

* Style clean && add condition check for is_deleted values

* clean_deleted_rows settings is nom an enum

* Add valid default value to the clean_deleted_rows settings

* Update cleanup checkers to use the enum and fix typos in the test

* Fix submodule contrib/AMQP-CPP pointer

* Add missing messages in test reference and remove a print with non derterministic order

* fix replica test reference

* Fix edge case

* Fix a typo for the spell checker

* Fix reference

* Fix a condition to raise an error if is_deleted differ from 0/1 and cleanup

* Change tests file name and update number

* This should fix the ReplacingMergeTree parameter set

* Fix replicated parameters

* Disable allow_deprecated_syntax_for_merge_tree for our new column

* Fix a test

* Remove non deterministic order print in the test

* Test on replicas

* Remove a condition, when checking optional parameters, that should not be sueful since we disabled the deprected_syntaxe

* Revert "Remove a condition, when checking optional parameters, that should not be useful since we disabled the deprected_syntaxe"

This reverts commit b65d64c05e.

* Fix replica management and limit the number of argument to two maximum, due to the possiblity of deprecated table create/attach failing otherwise

* Test a fix for replicated log information error

* Try to add sync to have consistent results

* Change path of replicas that should cause one issue and add few prints in case it's not that

* Get cleanup info on replicas only if information found

* Fix style issues

* Try to avoid replication error 'cannot select parts...' and and replica read/write field order

* Cleanup according to PR reviews
 and add tests on error raised.

* Update src/Storages/MergeTree/registerStorageMergeTree.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* Select ... FINAL don't show rows with is_deleted = true

* Update and fix SELECT ... FINAL merge parameter

* Remove is_deleted rows only on the version inserted when merge

* Fix (master) updates issues

* Revert changes that should not be commited

* Add changes according to review

* Revert changes that should not be commited - part 2

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
 2023-02-16 16:03:16 +03:00
flynn
7f4c23ec8a fix 2023-02-16 12:48:22 +00:00
Maksim Kita
05baf271f0 Analyzer fix table functions with invalid arguments analysis 2023-02-16 12:17:04 +01:00
Maksim Kita
57b5d9852f Fixed build 2023-02-16 12:17:03 +01:00
Maksim Kita
b72ea982b0 Remove unnecessary includes of InterpreterSelectQuery header 2023-02-16 12:17:03 +01:00
Maksim Kita
6b2adc1ec2 Analyzer storage Merge fixes 2023-02-16 12:17:03 +01:00
Maksim Kita
b1ab2af7ad Analyzer support storage Merge 2023-02-16 12:17:03 +01:00
Maksim Kita
84065fb13f Analyzer added distributed table functions support 2023-02-16 12:17:03 +01:00
Maksim Kita
a090a8449d Analyzer distributed read fix 2023-02-16 12:17:03 +01:00
Maksim Kita
f73c49b082 Fixed style check 2023-02-16 12:17:03 +01:00
Maksim Kita
f8442b2a8d Analyzer support LiveView 2023-02-16 12:17:03 +01:00
Maksim Kita
a12aea45f6 Analyzer matcher resolve fix 2023-02-16 12:17:02 +01:00
Maksim Kita
25da9dcef7 StorageDistributed Planner initialization fix 2023-02-16 12:17:02 +01:00
Maksim Kita
0fe3ba36c4 Analyzer added ParitionPruner support 2023-02-16 12:17:02 +01:00
Maksim Kita
bb56af0e2b MergeTreeIndexGin crash fix 2023-02-16 12:17:02 +01:00
Maksim Kita
77fee97705 MergeTreeIndexFullText fix prepared set index analysis 2023-02-16 12:17:02 +01:00
Alexey Milovidov
a9b4f2ef3a
Merge pull request #46435 from ClickHouse/rs/consistent-db-metric 
Exclude internal databases from async metric "NumberOfDatabases"
 2023-02-16 14:09:59 +03:00
Kseniia Sumarokova
187f7482e1
Merge pull request #46044 from ucasfl/log-engine 
Storage Log family support settings storage_policy
 2023-02-16 12:09:47 +01:00
Nikolay Degterinsky
d8c5c7c503
Merge branch 'master' into parameters-rename-query 2023-02-16 11:30:20 +01:00
Kseniia Sumarokova
4303ee9114
Merge pull request #46404 from kssenii/fix-rabbit-test-4 
Fix test test_rabbitmq_overloaded_insert
 2023-02-16 11:28:18 +01:00
Kseniia Sumarokova
947753bd4c
Merge pull request #46409 from kssenii/follow-up-to-46168 
Follow-up to #46168
 2023-02-16 11:26:54 +01:00
flynn
2968cdc8f6 fix 2023-02-16 10:18:22 +00:00
Nikolay Degterinsky
ebec5b8050
Merge pull request #46422 from evillique/move-sources 
Move MongoDB and PostgreSQL sources to Sources folder
 2023-02-16 10:51:45 +01:00
Sergei Trifonov
3ad41b29ec
Merge pull request #46247 from ClickHouse/round-robin-merge-scheduler 2023-02-16 08:03:22 +01:00
flynn
ecc39978d7 fix conflict 2023-02-16 02:23:55 +00:00
Alexander Gololobov
cdf3828575
Removed unneeded default value for enable_multiple_prewhere_read_steps 2023-02-15 22:41:55 +01:00
Alexander Gololobov
6936471150
fix include 
Co-authored-by: Nikita Taranov <nikita.taranov@clickhouse.com>
 2023-02-15 22:37:40 +01:00
Alexander Gololobov
8b528739c1
fix typo 
Co-authored-by: Nikita Taranov <nikita.taranov@clickhouse.com>
 2023-02-15 22:37:18 +01:00
Alexander Gololobov
3ff0c7edcc
fix typo 
Co-authored-by: Nikita Taranov <nikita.taranov@clickhouse.com>
 2023-02-15 22:37:04 +01:00
Antonio Andelic
17494d79c2
Merge pull request #46070 from ClickHouse/s3-client-polish 
Polish S3 client
 2023-02-15 21:22:47 +01:00
Nikolai Kochetov
f524dae924 Fix more tests. 2023-02-15 19:17:24 +00:00
Sema Checherinda
dd1b509637 resolve race in getCSNAndAssert 2023-02-15 18:16:39 +01:00
Han Fei
5458d4216f
Merge branch 'master' into dev-prewhere 2023-02-15 17:59:44 +01:00
Alexander Tokmakov
885f14311c fix 2023-02-15 16:37:44 +01:00
Kruglov Pavel
4f380370a9
Fix s3Cluster schema inference in parallel distributed insert select (#46381) 
* Fix s3Cluster schema inference in parallel distributed insert select
* Try fix flaky test
* Try SYSTEM SYNC REPLICA to avoid test flakiness
 2023-02-15 15:30:43 +01:00
Kruglov Pavel
e56dead5aa
Merge branch 'master' into fix-kafka-protobuf 2023-02-15 15:16:25 +01:00
flynn
de11a05d89 remove redundant header 2023-02-15 13:27:25 +00:00
Robert Schulze
44815240f5
Exclude internal databases from async metric "NumberOfDatabases" 2023-02-15 12:58:00 +00:00
Alexey Milovidov
77f60311da
Merge branch 'master' into add_select_final_mt_setting 2023-02-15 15:35:37 +03:00
flynn
289c5c60d3 fix 2023-02-15 11:24:06 +00:00
Vladimir C
645fee3d94
Merge pull request #46009 from ClickHouse/vdimir/fix_read_in_order_final_desc 2023-02-15 11:23:07 +01:00
kssenii
f75d69a954 Fix 2023-02-15 11:14:50 +01:00
Antonio Andelic
4dd4aac125
Merge branch 'master' into custom-key-parallel-replicas 2023-02-15 11:02:37 +01:00
Nikolay Degterinsky
24bbf885db
Merge branch 'master' into parameters-rename-query 2023-02-15 10:53:44 +01:00
Robert Schulze
ec33204c05
Merge pull request #46344 from ClickHouse/rs/stabilize-full_text_seaerch_test 
Try to stabilize test 02346_full_text_search.sql
 2023-02-15 10:47:00 +01:00
Antonio Andelic
f76ef72ac0
Merge branch 'master' into s3-client-polish 2023-02-15 09:20:06 +01:00
Nikolay Degterinsky
bcc4adf73d
Merge branch 'master' into move-sources 2023-02-15 08:46:09 +01:00
flynn
18cf72147e remove more redundant header files 2023-02-15 07:02:44 +00:00
flynn
50c0693afd fix 2023-02-15 06:49:53 +00:00
flynn
1468e9b9d9 fix 2023-02-15 06:37:05 +00:00
flynn
e4e3adf160 fix 2023-02-15 05:05:46 +00:00
flynn
22a3439eec Merge branch 'master' of github.com:ClickHouse/ClickHouse into log-engine 2023-02-15 03:57:11 +00:00
flynn
f31451822e fix 2023-02-15 03:56:01 +00:00
Nikolay Degterinsky
6e4b660033 Move MongoDB and PostgreSQL sources to Sources folder 2023-02-14 22:35:10 +00:00
Alexander Tokmakov
e96340e183
Merge pull request #46410 from ClickHouse/fix_noisy_log_messages 
Fix noisy log messages
 2023-02-15 01:22:42 +03:00
Nikolay Degterinsky
1fc28c883f Fix tests 2023-02-14 22:19:46 +00:00
Alexander Tokmakov
8ddbc85919
Merge branch 'master' into followup_revert_39737 2023-02-15 01:15:17 +03:00
Sergei Trifonov
b7a4fa8bd9
Merge branch 'master' into round-robin-merge-scheduler 2023-02-14 20:39:02 +01:00
serxa
a7bbf02bac fix possible deadlock 2023-02-14 19:37:09 +00:00
pufit
e914e7c98d
Merge pull request #45148 from mateng0915/replicas_status_api_optimize 
Feature: Optimize the replica delay api logic
 2023-02-14 13:44:52 -05:00
Han Fei
937fade982 clean up 2023-02-14 18:54:19 +01:00
Han Fei
86fda9bd22 some clean up 2023-02-14 18:52:09 +01:00
Han Fei
32050acde5 Merge branch 'master' into dev-prewhere 2023-02-14 18:39:13 +01:00
Kseniia Sumarokova
fe155bb210
Merge branch 'master' into follow-up-to-46168 2023-02-14 18:28:46 +01:00
Anton Popov
ecc6ff707b
Merge pull request #46387 from CurtizJ/fix-update-from-empty-nested 
Fix `ALTER UPDATE` of empty `Nested` columns
 2023-02-14 16:28:32 +01:00
kssenii
5bda358fb7 Follow-up to #46168 2023-02-14 15:45:38 +01:00
Alexander Tokmakov
1bf3735ca9 fix noisy log messages 2023-02-14 15:33:27 +01:00
vdimir
abd9119583
Remove unnecessary ProfileEventsScope 2023-02-14 14:11:12 +00:00
vdimir
4c1aeb7972
Use overload of PartLog::addNewPart with profile_events 2023-02-14 14:11:12 +00:00
vdimir
07d7478bc7
wip 2023-02-14 14:11:10 +00:00
vdimir
994dcb93f9
Rename ScopedProfileEvents -> ProfileEventsScope 2023-02-14 14:11:10 +00:00
vdimir
a228f7f419
[wip] ProfileCounters for each part 2023-02-14 14:11:09 +00:00
Alexey Milovidov
f554ff9d99
Remove trash 2023-02-14 14:11:08 +00:00
Nicolae Vartolomei
96e4411694
Fix style 2023-02-14 14:11:07 +00:00
John Skopis
f7604cc686
[feat] Add ProfileEvents map to PartLog 
closes #10316
 2023-02-14 14:11:02 +00:00
flynn
e05b4e9745 fix conflict 2023-02-14 13:49:28 +00:00
Antonio Andelic
f703b4c25b Merge branch 'master' into s3-client-polish 2023-02-14 13:33:45 +00:00
Antonio Andelic
15df1fcd72
Merge pull request #46339 from ClickHouse/refactor-s3 
Extract common logic for S3
 2023-02-14 14:30:43 +01:00
Antonio Andelic
adde580756
Merge branch 'master' into custom-key-parallel-replicas 2023-02-14 14:09:12 +01:00
Antonio Andelic
1826c29c82 Merge branch 'master' into s3-client-polish 2023-02-14 13:08:27 +00:00
Nikita Taranov
cbd10c4493
Fix mbm with aio on top of query plan (#45892) 2023-02-14 14:02:54 +01:00
Nikolay Degterinsky
b9d3160107 Add parameterized RENAME queries 2023-02-14 13:01:06 +00:00
Robert Schulze
95c50b8653
Address the actual problem (not an optimal approach yet ...) 2023-02-14 12:32:09 +00:00
kssenii
bead024ecb Fix test 2023-02-14 12:45:57 +01:00
Kseniia Sumarokova
0b55021c68
Merge pull request #46337 from kssenii/rabbitmq-tests-fix-2 
Fix flaky test_storage_rabbitmq::test_rabbitmq_address
 2023-02-14 11:44:34 +01:00
Sema Checherinda
fba7ee6332
Merge pull request #46347 from CheSema/fix-select-candite-to-check 
fix candidate selection
 2023-02-14 11:18:42 +01:00
mateng915
1faf09ec21
Merge branch 'master' into replicas_status_api_optimize 2023-02-14 17:55:23 +08:00
Robert Schulze
61e315f30d
Merge pull request #45520 from ClibMouse/z-build-0120 
s390x build support
 2023-02-14 10:01:38 +01:00
Antonio Andelic
5ab24285fc Fix arg parsing 2023-02-14 08:33:59 +00:00
Antonio Andelic
005888b0a4
Merge pull request #46330 from ClickHouse/delete-update-keepermap 
Add support for DELETE/UPDATE for KeeperMap
 2023-02-14 09:10:41 +01:00
Anton Popov
1ab37ad5ed fix update of empty Nested columns 2023-02-14 02:31:42 +00:00
mateng915
d28792528a
Merge branch 'master' into replicas_status_api_optimize 2023-02-14 10:03:07 +08:00
Alexander Tokmakov
e712fbecb2 fix race between drop and create 2023-02-14 00:45:28 +01:00
Alexander Gololobov
d76e4f60a8 Fix for combined node for condition group not found 2023-02-13 23:14:18 +01:00
Nikolai Kochetov
1c0c662fe1
Merge pull request #46275 from ClickHouse/less-mem-usage-for-vertical-merge 
Less memory usage for vertical merge (non-remote disk)
 2023-02-13 21:10:43 +01:00
Alexander Tokmakov
a4d9688775 fix 'Directory not empty after drop' with zero copy replication 2023-02-13 19:35:07 +01:00
Alexander Gololobov
0fe080cc6f Logger instead of debug prints 2023-02-13 17:43:41 +01:00
Alexander Gololobov
18a29e81fc Better formatting in PrewhereExprInfo::dump() 2023-02-13 17:42:27 +01:00
flynn
68748d6aef fix 2023-02-13 16:06:00 +00:00
Antonio Andelic
cdf570a64f Better 2023-02-13 15:23:48 +00:00
kssenii
614d57d245 Fix race 2023-02-13 16:02:41 +01:00
Alexander Gololobov
f0011862a9 Move prewhere splitting logic into a separate file 2023-02-13 16:00:03 +01:00
Antonio Andelic
95db6fb1a6
Merge pull request #45659 from ClickHouse/add-support-for-settings-alias 
Add support for settings aliases/Rename `replication_alter_partitions_sync` to `alter_sync`
 2023-02-13 15:31:12 +01:00
Suzy Wang
e140bed3ea
Merge branch 'master' into z-build-0120 2023-02-13 09:06:48 -05:00
Sema Checherinda
a48b6ec612 fix candidate selection 2023-02-13 14:52:55 +01:00
Alexander Tokmakov
69f579b8ec remove an unused argument 2023-02-13 14:46:46 +01:00
Alexander Gololobov
39e4ca9850 Cleanup 2023-02-13 14:40:54 +01:00
Alexander Gololobov
c39cfee954 Properly set "remove" flag for step prewhere_columns when they are used in futher steps 2023-02-13 14:11:18 +01:00
flynn
c49a293a9f refactor and get rid of s3 2023-02-13 12:39:54 +00:00
Antonio Andelic
3a6ea861d8 Extract common argument parsing logic 2023-02-13 12:27:49 +00:00
Alexander Gololobov
3060bee18a Properly handle constants in multiple steps 2023-02-13 12:58:33 +01:00
Alexey Milovidov
8265db80ff
Merge pull request #46309 from ClickHouse/revert-39737 
[RFC] Revert #39737
 2023-02-13 14:41:02 +03:00
Kruglov Pavel
e139070043
Merge branch 'master' into fix-kafka-protobuf 2023-02-13 12:38:18 +01:00
Antonio Andelic
454705b261 Add support for DELETE/UPDATE for KeeperMap 2023-02-13 11:22:43 +00:00
Vladimir C
89cb55aa1f
whitespace 2023-02-13 12:19:31 +01:00
Alexey Milovidov
9be1bba92f
Merge pull request #46310 from ClickHouse/replicas_status_remove_mutex 
Remove mutex from replicas_status
 2023-02-13 13:36:11 +03:00
flynn
d3dd9421da refactor and get rid of s3 2023-02-13 08:29:22 +00:00
Alexey Milovidov
1b912c07cd
Merge pull request #42391 from ClickHouse/filimonov-kafka-Local-No-offset-stored 
Attempt to fix 'Local: No offset stored message' from Kafka
 2023-02-13 10:49:49 +03:00
Antonio Andelic
8d16fe5793
Merge branch 'master' into add-support-for-settings-alias 2023-02-13 08:46:00 +01:00
mateng915
c876ad901b
Merge branch 'master' into replicas_status_api_optimize 2023-02-13 11:38:55 +08:00
Alexander Gololobov
a89465d1cb Fix build after merge 2023-02-12 22:48:28 +01:00
Alexander Gololobov
3ad0683de9 Refactor DAG splitting into steps 2023-02-12 21:55:53 +01:00
Alexander Gololobov
12e576cddb Slight refactoring 2023-02-12 21:55:53 +01:00
Alexander Gololobov
8fc08c74f0 Pass action_settings to getReadTaskColumns() 2023-02-12 21:55:53 +01:00
Alexander Gololobov
e460d53619 Use different name when filter column is CASTed to UInt8 2023-02-12 21:55:41 +01:00
Alexander Gololobov
7634b9f56e Added settings to enable the new logic 2023-02-12 21:54:23 +01:00
Alexander Gololobov
120b112743 Cleanup 2023-02-12 21:54:23 +01:00
Alexander Gololobov
31f5dbe2a3 Cast intermediate step result to Bool if needed 2023-02-12 21:54:23 +01:00
Alexander Gololobov
1b207d9fa0 Fix for overwritten row level filter step 2023-02-12 21:54:23 +01:00
Alexander Gololobov
249e670c9f Properly preserve columns that are required by next steps 2023-02-12 21:54:23 +01:00
Alexander Gololobov
45ef2f6d60 Split prewhere actions into separate conjuctive steps 2023-02-12 21:54:23 +01:00
alesapin
1683cd5d6f
Merge branch 'master' into mutations_rename_hang 2023-02-12 13:06:48 +01:00
Alexey Milovidov
e6b44756dc Fixed build 2023-02-12 07:45:28 +01:00
Alexey Milovidov
b8cbc947ce Remove mutex from replicas_status 2023-02-12 07:39:16 +01:00
Alexey Milovidov
e2079b78a3 [RFC] Revert #39737 2023-02-12 07:24:42 +01:00
Alexey Milovidov
384ae944fe Merge branch 'master' into filimonov-kafka-Local-No-offset-stored 2023-02-12 04:01:07 +01:00
Alexey Milovidov
5f00bc49f8
Merge branch 'master' into addParts 2023-02-12 05:55:18 +03:00
Alexey Milovidov
a97f9b2caf
Update MergeTreeData.cpp 2023-02-12 05:55:07 +03:00
Alexey Milovidov
395d6b6bd5
Merge pull request #46282 from CurtizJ/return-back-vertical-merges 
Allow vertical merges from compact to wide parts [2]
 2023-02-12 05:51:12 +03:00
serxa
df6b4c4372 fix typo 2023-02-11 20:18:19 +00:00
serxa
9a7f70f595 fix comment 2023-02-11 16:54:57 +00:00
serxa
eb8cf6f816 Merge branch 'round-robin-merge-scheduler' of github.com:ClickHouse/ClickHouse into round-robin-merge-scheduler 2023-02-11 16:45:55 +00:00
serxa
d2f8377c47 fix check 2023-02-11 16:45:45 +00:00
Sergei Trifonov
297587531b
Merge branch 'master' into round-robin-merge-scheduler 2023-02-11 17:19:27 +01:00
serxa
c58b165b0f add config option to select scheduling policy 2023-02-11 16:18:42 +00:00
Kseniia Sumarokova
ed6f623f3a
Merge pull request #46168 from kssenii/resubmit-prefetches 
Allow to increase prefetching of data for reads
 2023-02-11 16:22:47 +01:00
flynn
50dace9571 fix 2023-02-11 13:58:19 +00:00
flynn
d3255fa420 Merge branch 'master' of github.com:ClickHouse/ClickHouse into log-engine 2023-02-11 12:26:16 +00:00
alesapin
5c83ddabf9 Merge branch 'master' into mutations_rename_hang 2023-02-11 12:18:58 +01:00
kssenii
3067c1d723 Merge remote-tracking branch 'upstream/master' into resubmit-prefetches 2023-02-11 11:36:23 +01:00
Kseniia Sumarokova
2c397551a0
Update RabbitMQProducer.cpp 2023-02-11 11:23:29 +01:00
Kseniia Sumarokova
0e48ddc4c7
Merge pull request #46283 from kssenii/add-logs-for-rabbit 
Add more logging to RabbitMQ (to help debug tests)
 2023-02-11 11:21:13 +01:00
Sergei Trifonov
fbedf4d520
Merge branch 'master' into round-robin-merge-scheduler 2023-02-11 10:30:26 +01:00
Sergei Trifonov
2931c3bbe1
Merge pull request #45711 from ClickHouse/io-scheduler-integration 
Integrate IO scheduler with buffers for remote reads and writes
 2023-02-11 10:26:58 +01:00
robot-ch-test-poll2
4e9d1275c7
Merge pull request #46278 from CheSema/race-check-table 
fix data race between check table request and background checker
 2023-02-11 09:50:48 +01:00
kssenii
3545568440 Fix build 2023-02-10 22:15:44 +01:00
Kseniia Sumarokova
3be36c7979
Merge pull request #41976 from kssenii/allow-single-disk-instead-of-storage-policy 
Allow configuring storage as `SETTINGS disk='<disk_name>'` (instead of `storage_policy`) and explicit disk creation `SETTINGS disk=disk(<disk_configuration>)`
 2023-02-10 21:38:50 +01:00
Vladimir C
ea670acb72
Add comment to requestReadingInOrder 2023-02-10 20:05:18 +01:00
serxa
8c3f85e58d more review fixes 2023-02-10 17:39:28 +00:00
Kruglov Pavel
8bbf62508f
Merge branch 'master' into fix-kafka-protobuf 2023-02-10 18:32:25 +01:00
kssenii
96e745be09 Add logging 2023-02-10 18:14:49 +01:00
Anton Popov
1205932a19 add setting to disable vertical merges from compact parts 2023-02-10 16:33:46 +00:00
kssenii
468d5216da Remove logging because of test 00002_log_and_exception_messages_formatting 2023-02-10 17:25:26 +01:00
Sema Checherinda
ef133ee30d fix data race between check table query and background part checker thread 2023-02-10 16:22:35 +01:00
Suzy Wang
1c2a52c57b
Update src/Storages/StorageS3Cluster.cpp 2023-02-10 10:09:24 -05:00
Anton Popov
a252d3a139 Revert "Merge pull request #46236 from ClickHouse/revert-45681-compact-parts-vertical-merge" 
This reverts commit 27bc317ef9, reversing
changes made to ccf7b6f4c1.
 2023-02-10 14:13:29 +00:00
Anton Popov
27bc317ef9
Merge pull request #46236 from ClickHouse/revert-45681-compact-parts-vertical-merge 
Revert "Allow vertical merges from compact to wide parts"
 2023-02-10 14:31:33 +01:00
alesapin
99dc7fbde8
Update src/Storages/MutationCommands.h 2023-02-10 14:27:59 +01:00