Commit Graph

8484 Commits

Author SHA1 Message Date
kssenii
5cd757dd9a Merge remote-tracking branch 'origin/master' into s3-queue-parallelize-ordered-mode 2024-01-26 16:33:22 +01:00
serxa
a40da92761 Merge branch 'master' into fix-async-loader-shutdown-deadlock 2024-01-25 11:31:05 +00:00
Kseniia Sumarokova
951c874aa9
Merge pull request #59080 from ClickHouse/data-lakes-survive-restart
Allow server to start with broken data lake table and fix itself lazily
2024-01-25 11:50:16 +01:00
Nikita Taranov
067af44cbd
Merge pull request #59077 from ClickHouse/nickitat-patch-17
Fix `test_parallel_replicas_invisible_parts`
2024-01-24 17:19:36 +01:00
Kseniia Sumarokova
50a0494df7
Update test.py 2024-01-24 16:53:10 +01:00
kssenii
7f8f379d7f Parallel & disrtibuted processing for ordered mode 2024-01-24 16:32:15 +01:00
Kruglov Pavel
89bafbcda6
Merge pull request #59133 from Avogar/allow-to-ignore-schema-evolution-in-iceberg
Allow to ignore schema evolution in Iceberg table engine under a setting
2024-01-24 13:19:09 +01:00
robot-clickhouse
41e6dd7a9d Automatic style fix 2024-01-24 12:15:19 +00:00
serxa
0badcffdc1 test: table metadata load failure should shutdowns the server properly 2024-01-24 12:01:32 +00:00
Kseniia Sumarokova
4b40b14bba
Update test.py 2024-01-24 11:23:13 +01:00
Yarik Briukhovetskyi
f0be1ac2c4
Merge pull request #58936 from wheatdog/level-int-to-word
Display word-descriptive log level while enabling structured log formatting in json
2024-01-24 00:18:25 +01:00
avogar
be3b5dc45f Allow to ignore schema evolution in Iceberg table engine under a setting 2024-01-23 17:45:41 +00:00
Tim Liou
7981d182bc Merge branch 'master' into level-int-to-word 2024-01-23 22:29:07 +08:00
Robert Schulze
a1c83e2f51
Revert "Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition" 2024-01-23 15:26:04 +01:00
kssenii
a4082f5b97 Add a test 2024-01-23 13:28:25 +01:00
Robert Schulze
9752079983
Cosmetics 2024-01-23 10:58:09 +00:00
Robert Schulze
e23e7e6b73
Merge pull request #53710 from rschu1ze/system-reload-asynchronous-metrics
Add statement `SYSTEM RELOAD ASYNCHRONOUS METRICS`
2024-01-23 11:56:25 +01:00
Robert Schulze
8fc918b272
Fix style, pt. III 2024-01-23 06:23:34 +00:00
Robert Schulze
e72b44a0bf
Fix style, pt. II 2024-01-23 05:58:24 +00:00
Nikita Taranov
09e24ed6c5
Merge pull request #58068 from zhongyuankai/limit_materialized_view_size
A limit on the number of materialized views attached to a table.
2024-01-22 22:50:56 +01:00
Arthur Passos
24b8bbe9fa
Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition (#39507)
* temp commit

* temp commit

* draft impl for feedback

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* aa

* aa

* Add integ tests and remove partition key restriction

* fix small incosistency in partition id

* style fix

* style fix

* style fix

* use existing DataPartStorageBuilder instead of new one

* Refactor part clone to make it more readable and maintainable

* Add MergeTreeDataPartCloner docs

* define ErrorCodes::BAD_ARGUMENTS

* Rebase

* camel case methods

* address some comments

* yet another rebase?

* Move from integ tests to stateless tests

* address more comments

* add finalize on min_max_idx files

* Add sync option to DistinctPartitionExpCloner

* just a temp test

* revert temp change

* Use echoOn to distinguish test queries

* remove comment

* fix build issue during rebase

* atempt to fix build after rebase

* finally fix build

* clear minmaxidx hyperrectangle before loading it

* Fix error on min_max files deletion where it was being assumed that partition expression contained all columns

* get it to the state it was previously

* add missing include

* getting functional?

* refactoring and renaming

* some more refactoring

* extern bad arguments

* try to fix style

* improvements and docs

* remove duplicate includes

* fix crash

* make tests more stable by ordering

* rebase once again..

* fix

* make ci happy?

* fix rebase issues

* docs

* rebase, but prolly needs to be improved

* refactor out from nasty inheritance to static methods

* fix style

* work around optional

* refactor & integrate some changes

* update column_type

* add tests by dencrane

* set utc

* fix ref file

* fix tests

* use MergeTree instead of SummingMergeTree

* mark MergeTreeDataPart::getBlock as const

* address a few comments

* compute module function name size at compile time

* simplify branching in getPartitionAstFieldsCount

* remove column_indexes argument

* merge getBlock with buildBlock

* add some const specifiers

* small adjustments

* remove no longer needed isNull check

* use std::min and max to update global min max idx

* add some assertions

* forward declare some symbols

* fix grammar

* forward decl

* try to fix build..

* remove IFunction forward decl

* Revert "use std::min and max to update global min max idx"

This reverts commit b2fe79dda7.

* Revert "remove no longer needed isNull check"

This reverts commit 129db2610f.

* Revert "Revert "remove no longer needed isNull check""

This reverts commit 9416087dd8.

* Revert "Revert "use std::min and max to update global min max idx""

This reverts commit 20246d4416.

* remove some comments

* partial use of MonotonicityCheckMatcher

* ranges

* remove KeyDescriptionMonotonicityChecker

* remove duplication of applyfunction

* move functions to anonymous namespace

* move functions to cpp

* Relax partition compatibility requirements by accepting subset, add tests from partitioned to unpartitioned

* updte reference file

* Support for partition by a, b, c to partition by a, b

* refactoring part 1

* refactoring part 2, use hyperrectangle, still not complete

* refactoring part 3, build hyperrectangle with intersection of source & destination min max columns

* Support attaching to table with partition expression of multiple expressions

* add tests

* rename method

* remove some code duplication

* draft impl for replicatedmergetree, need to dive deeper

* ship ref file

* fix impl for replicatedmergetree..

* forbid attach empty partition replicatedmergetree

* Add replicated merge tree integration tests

* add test missing files

* fix black

* do not check for monotonicity of empty partition

* add empty tests & fix replicated

* remove no longer needed buildBlockWithMinMaxINdexes

* remove column logic in buildHyperrectangle

* simplify implementation by using existing methods

* further simplify implementation

* move all MergeTreeDataPartClone private methods to .cpp file

* decrease decomposition

* use different namespaces

* reduce code duplication

* fix style

* address a few comments

* add chassert to assert arguments size on MonotonicityCheckVisitor

* remove deleteMinMaxFiles method

* remove useless checks from sanitycheck

* add tests for attach partition (not id)

* Remove sanityCheckASTPartition and bring back conditional getPartitionIDFromQuery

* remove empty block comment

* small fixes

* fix formatting

* add missing include

* remove duplicate iuncludes

* trigger ci

* reduce some code duplication

* use updated partition id on replicatedmergetree

* fix build

* fix build

* small refactor

* do not use insert increment on fetch part

* remove duplicate includes

* add one more integ test

* black

* black

* rely on partition exp instead of partition id on replicated part fetch to decide if it is a different partition exp

* add one more integ test

* add order by clause

* fix black

---------

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
2024-01-22 21:00:04 +01:00
robot-clickhouse
48f0580512 Automatic style fix 2024-01-22 17:39:44 +00:00
Robert Schulze
d739a7230a
Fix style 2024-01-22 17:37:16 +00:00
kssenii
1aa8e62bb7 Fix 2024-01-22 18:26:44 +01:00
Robert Schulze
90a0ea36b3
Add statement 'SYSTEM RELOAD ASYNCHRONOUS METRICS' 2024-01-22 17:14:58 +00:00
Nikita Taranov
704e5ac883
Fix test_parallel_replicas_invisible_parts 2024-01-22 17:52:13 +01:00
Kseniia Sumarokova
aec76d6653
Merge pull request #58877 from ClickHouse/multiple-read-file-log-storage-in-mv
Multiple read file log storage in mv
2024-01-22 12:02:41 +01:00
Vitaly Baranov
01aafda27b
Merge pull request #58804 from vitlibar/allow-kill-query-for-backups
Allow KILL QUERY for backups
2024-01-20 23:48:18 +01:00
Igor Nikonov
4be068c73b
Merge pull request #58993 from ClickHouse/revert-58992-revert_flaky
Revive: Parallel replicas custom key: skip unavailable replicas
2024-01-20 19:44:56 +01:00
Kseniia Sumarokova
5d67e26fc4
Merge pull request #58869 from ClickHouse/force-check-for-cached-custom-disk
Fix for custom cached disk creation
2024-01-20 14:47:36 +01:00
Alexey Milovidov
587295aca1
Merge pull request #58981 from azat/prom-handler
Fix timeout for prometheus exporter for HTTP/1.1 (due to keep-alive)
2024-01-20 09:42:05 +01:00
Igor Nikonov
3d8be9d588 Comment for disabled assert 2024-01-19 14:54:20 +00:00
robot-clickhouse
522cce991a Automatic style fix 2024-01-19 14:37:53 +00:00
Igor Nikonov
a4d0260fa7 Update tests
+ setting max_replica_delay_for_distributed_queries to avoid replica lag
  affecting load balancing
+ disable flaky assert for asan/tsan builds (will investigated
  separately)
2024-01-19 14:23:15 +00:00
zhongyuankai
cbb51659f6
Merge branch 'master' into limit_materialized_view_size 2024-01-19 22:00:12 +08:00
Igor Nikonov
17c7fdfed9 Disable check which is flaky atm 2024-01-19 12:58:21 +00:00
Igor Nikonov
131f71b50a
Revert "Revert flaky" 2024-01-19 13:51:30 +01:00
Raúl Marín
477489a05d Revert "Merge pull request #57235 from ClickHouse/pr-custom-key-failover"
This reverts commit 8ba9b4a7ef, reversing
changes made to 0f39245cb8.
2024-01-19 12:46:05 +00:00
Raúl Marín
44b621fb48 Revert "Merge pull request #58909 from ClickHouse/fix-test-custom-key-failover"
This reverts commit deeb09ff37, reversing
changes made to a4cab68a06.
2024-01-19 12:44:55 +00:00
Azat Khuzhin
38ad4ef493 Fix timeout for prometheus exporter for HTTP/1.1 (due to keep-alive)
Before:

    $ time curl -s --http1.1 127.1:9363/metrics > /dev/null
    real    0m10.018s # default keep_alive_timeout is 10 seconds
    user    0m0.005s
    sys     0m0.001s

After

    $ time curl -s --http1.1 127.1:9363/metrics > /dev/null
    real    0m0.008s
    user    0m0.006s
    sys     0m0.000s

And if you will look at the test_prometheus_endpoint, you will see that
it takes > 30 seconds (it obtains metrics 3 times), after this patch it
should be finished more or less instantly.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-01-19 13:29:17 +03:00
Kseniia Sumarokova
25be226c05
Merge pull request #58951 from ClickHouse/fix-materialized-pg-test
Fix flaky test_postgresql_replica_database_engine_2/test.py::test_dependent_loading
2024-01-19 10:27:18 +01:00
Nikita Taranov
9d36fe4a48
Merge pull request #58923 from nickitat/add_test
Add regression test for parallel replicas (follow up #58722, #58844)
2024-01-18 21:54:18 +01:00
Kseniia Sumarokova
7f92bf6f0b
Merge pull request #58921 from ClickHouse/add-a-test-for-azure
Fix and test for azure #58697
2024-01-18 21:41:54 +01:00
kssenii
10a47ef185 Fix 2024-01-18 16:56:17 +01:00
Tim Liou
483bbaf14d Run black locally to format the code 2024-01-18 23:42:10 +08:00
Tim Liou
61826cf21a Validate log level values 2024-01-18 21:40:23 +08:00
Vitaly Baranov
4c6d3e7efa Add server settings "shutdown_wait_backups_and_restores" (default=true)
to set whether shutdown should wait for running backups to finish or just cancel them.
2024-01-18 13:27:14 +01:00
Kseniia Sumarokova
7092948d53
Update test.py 2024-01-18 11:02:13 +01:00
Igor Nikonov
fa664a3656 Merge remote-tracking branch 'origin/master' into fix-test-custom-key-failover 2024-01-17 19:36:04 +00:00
Igor Nikonov
265bfa0af8 Using current database in cluster 2024-01-17 19:35:35 +00:00