thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
137,459 Commits 9,328 Branches 1,783 Tags 2.1 GiB
1264b5337e
Commit Graph

8852 Commits

Author SHA1 Message Date
kssenii
a4082f5b97 Add a test 2024-01-23 13:28:25 +01:00
Robert Schulze
9752079983
Cosmetics 2024-01-23 10:58:09 +00:00
Robert Schulze
e23e7e6b73
Merge pull request #53710 from rschu1ze/system-reload-asynchronous-metrics 
Add statement `SYSTEM RELOAD ASYNCHRONOUS METRICS`
 2024-01-23 11:56:25 +01:00
kssenii
b76630c955 Merge remote-tracking branch 'origin/master' into broken-projections-better-handling 2024-01-23 11:34:58 +01:00
Robert Schulze
8fc918b272
Fix style, pt. III 2024-01-23 06:23:34 +00:00
Robert Schulze
e72b44a0bf
Fix style, pt. II 2024-01-23 05:58:24 +00:00
Nikita Taranov
09e24ed6c5
Merge pull request #58068 from zhongyuankai/limit_materialized_view_size 
A limit on the number of materialized views attached to a table.
 2024-01-22 22:50:56 +01:00
Arthur Passos
24b8bbe9fa
Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition (#39507) 
* temp commit

* temp commit

* draft impl for feedback

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* aa

* aa

* Add integ tests and remove partition key restriction

* fix small incosistency in partition id

* style fix

* style fix

* style fix

* use existing DataPartStorageBuilder instead of new one

* Refactor part clone to make it more readable and maintainable

* Add MergeTreeDataPartCloner docs

* define ErrorCodes::BAD_ARGUMENTS

* Rebase

* camel case methods

* address some comments

* yet another rebase?

* Move from integ tests to stateless tests

* address more comments

* add finalize on min_max_idx files

* Add sync option to DistinctPartitionExpCloner

* just a temp test

* revert temp change

* Use echoOn to distinguish test queries

* remove comment

* fix build issue during rebase

* atempt to fix build after rebase

* finally fix build

* clear minmaxidx hyperrectangle before loading it

* Fix error on min_max files deletion where it was being assumed that partition expression contained all columns

* get it to the state it was previously

* add missing include

* getting functional?

* refactoring and renaming

* some more refactoring

* extern bad arguments

* try to fix style

* improvements and docs

* remove duplicate includes

* fix crash

* make tests more stable by ordering

* rebase once again..

* fix

* make ci happy?

* fix rebase issues

* docs

* rebase, but prolly needs to be improved

* refactor out from nasty inheritance to static methods

* fix style

* work around optional

* refactor & integrate some changes

* update column_type

* add tests by dencrane

* set utc

* fix ref file

* fix tests

* use MergeTree instead of SummingMergeTree

* mark MergeTreeDataPart::getBlock as const

* address a few comments

* compute module function name size at compile time

* simplify branching in getPartitionAstFieldsCount

* remove column_indexes argument

* merge getBlock with buildBlock

* add some const specifiers

* small adjustments

* remove no longer needed isNull check

* use std::min and max to update global min max idx

* add some assertions

* forward declare some symbols

* fix grammar

* forward decl

* try to fix build..

* remove IFunction forward decl

* Revert "use std::min and max to update global min max idx"

This reverts commit b2fe79dda7.

* Revert "remove no longer needed isNull check"

This reverts commit 129db2610f.

* Revert "Revert "remove no longer needed isNull check""

This reverts commit 9416087dd8.

* Revert "Revert "use std::min and max to update global min max idx""

This reverts commit 20246d4416.

* remove some comments

* partial use of MonotonicityCheckMatcher

* ranges

* remove KeyDescriptionMonotonicityChecker

* remove duplication of applyfunction

* move functions to anonymous namespace

* move functions to cpp

* Relax partition compatibility requirements by accepting subset, add tests from partitioned to unpartitioned

* updte reference file

* Support for partition by a, b, c to partition by a, b

* refactoring part 1

* refactoring part 2, use hyperrectangle, still not complete

* refactoring part 3, build hyperrectangle with intersection of source & destination min max columns

* Support attaching to table with partition expression of multiple expressions

* add tests

* rename method

* remove some code duplication

* draft impl for replicatedmergetree, need to dive deeper

* ship ref file

* fix impl for replicatedmergetree..

* forbid attach empty partition replicatedmergetree

* Add replicated merge tree integration tests

* add test missing files

* fix black

* do not check for monotonicity of empty partition

* add empty tests & fix replicated

* remove no longer needed buildBlockWithMinMaxINdexes

* remove column logic in buildHyperrectangle

* simplify implementation by using existing methods

* further simplify implementation

* move all MergeTreeDataPartClone private methods to .cpp file

* decrease decomposition

* use different namespaces

* reduce code duplication

* fix style

* address a few comments

* add chassert to assert arguments size on MonotonicityCheckVisitor

* remove deleteMinMaxFiles method

* remove useless checks from sanitycheck

* add tests for attach partition (not id)

* Remove sanityCheckASTPartition and bring back conditional getPartitionIDFromQuery

* remove empty block comment

* small fixes

* fix formatting

* add missing include

* remove duplicate iuncludes

* trigger ci

* reduce some code duplication

* use updated partition id on replicatedmergetree

* fix build

* fix build

* small refactor

* do not use insert increment on fetch part

* remove duplicate includes

* add one more integ test

* black

* black

* rely on partition exp instead of partition id on replicated part fetch to decide if it is a different partition exp

* add one more integ test

* add order by clause

* fix black

---------

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2024-01-22 21:00:04 +01:00
robot-clickhouse
48f0580512 Automatic style fix 2024-01-22 17:39:44 +00:00
Robert Schulze
d739a7230a
Fix style 2024-01-22 17:37:16 +00:00
kssenii
1aa8e62bb7 Fix 2024-01-22 18:26:44 +01:00
Robert Schulze
90a0ea36b3
Add statement 'SYSTEM RELOAD ASYNCHRONOUS METRICS' 2024-01-22 17:14:58 +00:00
Nikita Taranov
704e5ac883
Fix test_parallel_replicas_invisible_parts 2024-01-22 17:52:13 +01:00
kssenii
15e3a5b396 Try fix flaky test 2024-01-22 14:57:10 +01:00
Kseniia Sumarokova
aec76d6653
Merge pull request #58877 from ClickHouse/multiple-read-file-log-storage-in-mv 
Multiple read file log storage in mv
 2024-01-22 12:02:41 +01:00
Vitaly Baranov
01aafda27b
Merge pull request #58804 from vitlibar/allow-kill-query-for-backups 
Allow KILL QUERY for backups
 2024-01-20 23:48:18 +01:00
Igor Nikonov
4be068c73b
Merge pull request #58993 from ClickHouse/revert-58992-revert_flaky 
Revive: Parallel replicas custom key: skip unavailable replicas
 2024-01-20 19:44:56 +01:00
Kseniia Sumarokova
e5c3b67f37
Update test.py 2024-01-20 16:10:09 +01:00
Kseniia Sumarokova
caf9d8df6d
Update test.py 2024-01-20 14:59:13 +01:00
Kseniia Sumarokova
5d67e26fc4
Merge pull request #58869 from ClickHouse/force-check-for-cached-custom-disk 
Fix for custom cached disk creation
 2024-01-20 14:47:36 +01:00
Alexey Milovidov
587295aca1
Merge pull request #58981 from azat/prom-handler 
Fix timeout for prometheus exporter for HTTP/1.1 (due to keep-alive)
 2024-01-20 09:42:05 +01:00
Igor Nikonov
3d8be9d588 Comment for disabled assert 2024-01-19 14:54:20 +00:00
robot-clickhouse
522cce991a Automatic style fix 2024-01-19 14:37:53 +00:00
Igor Nikonov
a4d0260fa7 Update tests 
+ setting max_replica_delay_for_distributed_queries to avoid replica lag
  affecting load balancing
+ disable flaky assert for asan/tsan builds (will investigated
  separately)
 2024-01-19 14:23:15 +00:00
zhongyuankai
cbb51659f6
Merge branch 'master' into limit_materialized_view_size 2024-01-19 22:00:12 +08:00
Igor Nikonov
17c7fdfed9 Disable check which is flaky atm 2024-01-19 12:58:21 +00:00
Igor Nikonov
131f71b50a
Revert "Revert flaky" 2024-01-19 13:51:30 +01:00
Raúl Marín
477489a05d Revert "Merge pull request #57235 from ClickHouse/pr-custom-key-failover" 
This reverts commit 8ba9b4a7ef, reversing
changes made to 0f39245cb8.
 2024-01-19 12:46:05 +00:00
Raúl Marín
44b621fb48 Revert "Merge pull request #58909 from ClickHouse/fix-test-custom-key-failover" 
This reverts commit deeb09ff37, reversing
changes made to a4cab68a06.
 2024-01-19 12:44:55 +00:00
robot-clickhouse
7c66141e08 Automatic style fix 2024-01-19 11:11:13 +00:00
kssenii
a196d04a1c Update test 2024-01-19 12:04:16 +01:00
Azat Khuzhin
38ad4ef493 Fix timeout for prometheus exporter for HTTP/1.1 (due to keep-alive) 
Before:

    $ time curl -s --http1.1 127.1:9363/metrics > /dev/null
    real    0m10.018s # default keep_alive_timeout is 10 seconds
    user    0m0.005s
    sys     0m0.001s

After

    $ time curl -s --http1.1 127.1:9363/metrics > /dev/null
    real    0m0.008s
    user    0m0.006s
    sys     0m0.000s

And if you will look at the test_prometheus_endpoint, you will see that
it takes > 30 seconds (it obtains metrics 3 times), after this patch it
should be finished more or less instantly.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-19 13:29:17 +03:00
kssenii
48e489682b Merge remote-tracking branch 'origin/master' into broken-projections-better-handling 2024-01-19 10:49:27 +01:00
Kseniia Sumarokova
25be226c05
Merge pull request #58951 from ClickHouse/fix-materialized-pg-test 
Fix flaky test_postgresql_replica_database_engine_2/test.py::test_dependent_loading
 2024-01-19 10:27:18 +01:00
Nikita Taranov
9d36fe4a48
Merge pull request #58923 from nickitat/add_test 
Add regression test for parallel replicas (follow up #58722, #58844)
 2024-01-18 21:54:18 +01:00
Kseniia Sumarokova
7f92bf6f0b
Merge pull request #58921 from ClickHouse/add-a-test-for-azure 
Fix and test for azure #58697
 2024-01-18 21:41:54 +01:00
kssenii
10a47ef185 Fix 2024-01-18 16:56:17 +01:00
MikhailBurdukov
be58c1fe44 Fix style 2024-01-18 15:43:43 +00:00
MikhailBurdukov
f6fb20d6cf Backoff policy for failed mutation. 2024-01-18 15:43:43 +00:00
Tim Liou
483bbaf14d Run black locally to format the code 2024-01-18 23:42:10 +08:00
Tim Liou
61826cf21a Validate log level values 2024-01-18 21:40:23 +08:00
Vitaly Baranov
4c6d3e7efa Add server settings "shutdown_wait_backups_and_restores" (default=true) 
to set whether shutdown should wait for running backups to finish or just cancel them.
 2024-01-18 13:27:14 +01:00
Kseniia Sumarokova
7092948d53
Update test.py 2024-01-18 11:02:13 +01:00
Smita Kulkarni
e71977b724 Merge branch 'master' into Azure_backup 2024-01-18 08:29:01 +01:00
Igor Nikonov
fa664a3656 Merge remote-tracking branch 'origin/master' into fix-test-custom-key-failover 2024-01-17 19:36:04 +00:00
Igor Nikonov
265bfa0af8 Using current database in cluster 2024-01-17 19:35:35 +00:00
Igor Nikonov
3b8ce04392 Add test for load balancing 2024-01-17 18:57:08 +00:00
Nikita Taranov
06a3a81b24 fix 2024-01-17 19:42:23 +01:00
Nikita Taranov
4fddf60f05 impl 2024-01-17 19:41:12 +01:00
kssenii
ef48b90bca Fix and test 2024-01-17 18:44:09 +01:00
Alexey Milovidov
1161dbbd54
Merge pull request #57993 from mkmkme/mkmkme/reload-config-again 
Update `query_masking_rules` when reloading the config, attempt 2
 2024-01-17 17:09:36 +01:00
robot-clickhouse
c2ccfba8ef Automatic style fix 2024-01-17 15:02:37 +00:00
kssenii
dc5d09fc55 Fix tests 2024-01-17 15:52:07 +01:00
kssenii
1e9de73bf5 Fix style check 2024-01-17 15:40:46 +01:00
kssenii
4d9c173008 Merge remote-tracking branch 'origin/broken-projections-better-handling' into broken-projections-better-handling 2024-01-17 15:40:27 +01:00
kssenii
dea8b10ae9 Fix test 2024-01-17 15:39:57 +01:00
Igor Nikonov
9e17877da5 Fix: use cluster() instead of defining distributed table 2024-01-17 14:11:09 +00:00
Kruglov Pavel
6d064512e1
Merge pull request #58614 from Blargian/58363_disable_ansi_pretty_automatically 
58363 Automatically disable ANSI escape sequences in Pretty formats if the output is not a terminal
 2024-01-17 13:45:41 +01:00
Кирилл Гарбар
dfbc2b142c GetTableDataPath and new docs 2024-01-17 12:50:09 +03:00
kssenii
1f384dd8a8 Fix 2024-01-17 10:32:56 +01:00
Mikhail Koviazin
6e6197019c
Reintroduce commit 3067ca6 
Update `query_masking_rules` when reloading the config

This reverts commit 7e55eab1c3.
 2024-01-17 08:10:49 +02:00
Alexander Tokmakov
38b85e6409
Merge pull request #58649 from azat/tests/test_replicated_database 
Fix test_replicated_database::test_startup_without_zk flakiness
 2024-01-16 21:08:33 +01:00
robot-clickhouse
216769f43e Automatic style fix 2024-01-16 18:55:23 +00:00
kssenii
7bc6a858c7 Rewrite bash test to integration test 2024-01-16 19:44:55 +01:00
Blargian
1b81efc337 fix failing 02421 stateless and (hopefully) fix failing integration tests 2024-01-16 19:48:23 +02:00
robot-clickhouse
7262e93a27 Automatic style fix 2024-01-16 16:26:28 +00:00
János Benjamin Antal
5a178ce1ef Add proper cleanup to kafka test 2024-01-16 16:03:44 +00:00
Vitaly Baranov
085833aaae Use special statuses BACKUP_CANCELLED and RESTORE_CANCELLED in system.backups 2024-01-16 15:07:24 +01:00
Vitaly Baranov
185f6cc37c Add tests for cancelling backup/restore queries. 2024-01-16 15:07:20 +01:00
kssenii
8decf5039a Fix for custom cached disk 2024-01-16 14:41:12 +01:00
vdimir
54a79c1f3d
Merge pull request #58123 from ClickHouse/vdimir/cluster_discovery_password 
ClusterDiscovery supports setting username and password
 2024-01-16 11:05:54 +01:00
zhongyuankai
cca2fa6976 fix test 2024-01-16 15:39:33 +08:00
Alexander Tokmakov
8bdd6542ce
Merge pull request #58647 from ClickHouse/tavplubix-patch-10 
Update test_replicated_database/test.py
 2024-01-16 00:32:08 +01:00
János Benjamin Antal
3e1d7bf685
Merge pull request #58477 from ClickHouse/multiple-read-kafka-storage-in-mv 
Delay reading from StorageKafka to allow multiple reads in materialized views
 2024-01-15 12:31:59 +01:00
vdimir
1f0a29cf10
Fix test_cluster_discovery/test_password.py 2024-01-15 11:31:22 +00:00
vdimir
53e2a97d33
Add missing configs to test_cluster_discovery 2024-01-15 11:22:54 +00:00
vdimir
8dc1756532
Support <secret> in cluster discovery 2024-01-15 11:22:53 +00:00
vdimir
6ab507e722
Add test case for cluster discovery with wrong password 2024-01-15 11:22:52 +00:00
vdimir
012713fc31
ClusterDiscovery supports setting username and password 2024-01-15 11:22:48 +00:00
zhongyuankai
826b29ef4d Max materialized views size for table 2024-01-15 10:02:57 +08:00
Kseniia Sumarokova
77f4cd8dd3
Merge pull request #58721 from ClickHouse/support-point-datatype-for-mysql 
Support Point data type for MySQL
 2024-01-12 17:02:01 +01:00
kssenii
8b1d2805ae Add one more test case to the test 2024-01-12 10:41:04 +01:00
Dmitry Novik
e3aed6ba4e
Fix integration test 2024-01-12 09:40:15 +01:00
kssenii
fada92173b Support Point data type for MySQL 2024-01-11 19:19:08 +01:00
Yakov Olkhovskiy
84809b36c3
Merge pull request #58475 from ClickHouse/revert-58450-revert-56064-feature-server-iface-metrics 
Second attempt: Use CH Buffer for HTTP out stream, add metrics for interfaces
 2024-01-11 12:06:59 -05:00
Igor Nikonov
8ba9b4a7ef
Merge pull request #57235 from ClickHouse/pr-custom-key-failover 
Parallel replicas custom key: skip unavailable replicas
 2024-01-11 12:22:32 +01:00
Konstantin Bogdanov
7f4ccebea4
Merge pull request #58550 from danipozo/hide-credentials-in-base-backup-name 
Hide credentials in system.backup_log base_backup_name column
 2024-01-11 03:42:17 +03:00
Igor Nikonov
8d4a395afa Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2024-01-10 20:28:04 +00:00
Igor Nikonov
e9fc869254 Fix test flakiness with hedge connections enabled 2024-01-10 15:59:48 +00:00
Yakov Olkhovskiy
2e124b19b8
Merge branch 'master' into revert-58450-revert-56064-feature-server-iface-metrics 2024-01-10 09:05:18 -05:00
Azat Khuzhin
2cf74aea5c Fix test_replicated_database::test_startup_without_zk flakiness 
CI founds [1]

    E           assert 'ZooKeeper' in "Received exception from server (version 24.1.1):\nCode: 999. DB::Exception: Received from 172.16.1.6:9000. Coordination::Exception. Coordination::Exception: Coordination error: Connection loss, path /clickhouse/databases/startup. Stack trace:\n\n0. ? @ 0x000000000c734f1b in /usr/bin/clickhouse\n1. ? @ 0x000000000fbef3f4 in /usr/bin/clickhouse\n2. ? @ 0x000000000fbeebc8 in /usr/bin/clickhouse\n3. ? @ 0x0000000012ded0f9 in /usr/bin/clickhouse\n4. ? @ 0x0000000012deaa71 in /usr/bin/clickhouse\n5. ? @ 0x00000000107e51e8 in /usr/bin/clickhouse\n6. ? @ 0x00000000107eb87a in /usr/bin/clickhouse\n7. ? @ 0x0000000011213ec3 in /usr/bin/clickhouse\n8. ? @ 0x00000000111e9acf in /usr/bin/clickhouse\n9. ? @ 0x000000001120e85e in /usr/bin/clickhouse\n10. ? @ 0x000000001174f0a2 in /usr/bin/clickhouse\n11. ? @ 0x0000000011748c9a in /usr/bin/clickhouse\n12. ? @ 0x00000000126fb5a9 in /usr/bin/clickhouse\n13. ? @ 0x00000000127130f9 in /usr/bin/clickhouse\n14. ? @ 0x000000001517bf12 in /usr/bin/clickhouse\n15. ? @ 0x000000001517cd11 in /usr/bin/clickhouse\n16. ? @ 0x00000000152740c7 in /usr/bin/clickhouse\n17. ? @ 0x00000000152726bc in /usr/bin/clickhouse\n18. ? @ 0x00007f7f36485ac3\n19. ? @ 0x00007f7f36517660\n. (KEEPER_EXCEPTION)\n(query: CREATE DATABASE startup ENGINE = Replicated('/clickhouse/databases/startup', 'shard1', 'replica1');)\n"

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/58610/441a3fbb9b1c088f528841727f013b829f63fc8a/integration_tests__release__[3_4]/integration_run_test_replicated_database_test_py_0.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-10 14:44:29 +03:00
Alexander Tokmakov
b954667ae9
Update test.py 2024-01-10 11:07:13 +01:00
Robert Schulze
4cacfa6b43
Merge pull request #57509 from jinjunzh/qatzstd_main 
Add Intel-QAT-based `ZSTD_QAT` compression codec
 2024-01-10 10:31:39 +01:00
Kseniia Sumarokova
1b517105db
Merge pull request #57323 from valbok/mysql-binlog-client 
MaterializedMySQL: Introduce MySQL Binlog Client
 2024-01-10 09:08:13 +01:00
Robert Schulze
ced9e93ac6
Merge remote-tracking branch 'rschu1ze/master' into qatzstd_main 2024-01-09 20:36:02 +00:00
Sema Checherinda
7c7e72c4b7
Merge pull request #58573 from ClickHouse/chesema-s3-client-creation 
fix and test that S3Clients are reused
 2024-01-09 13:28:20 +01:00
Sema Checherinda
6dbf7c967c
Merge pull request #57663 from ClickHouse/CheSema-remote-blobs-naming-2 
generate object storage key by template
 2024-01-09 13:26:56 +01:00
Кирилл Гарбар
85163994dc Merge remote-tracking branch 'ClickHouse/master' 2024-01-09 04:51:51 +03:00
Кирилл Гарбар
bebb2c9dca Common test functions, check flags are removed 2024-01-09 04:22:40 +03:00
Igor Nikonov
7fd851b4e9 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2024-01-08 20:43:20 +00:00
Dani Pozo
d8f4b5fd13 Fix black formatting 2024-01-08 12:47:58 +01:00
Dani Pozo
b42eaf6c62 Add test for hiding credentials in system.backups table 2024-01-08 12:20:32 +01:00
Кирилл Гарбар
c7b14d190e Remove flags directory 2024-01-08 07:09:38 +03:00
robot-clickhouse
29173a602e Automatic style fix 2024-01-08 02:39:19 +00:00
Yakov Olkhovskiy
45ac4d2b67 Merge remote-tracking branch 'origin/master' into revert-58450-revert-56064-feature-server-iface-metrics 2024-01-07 15:23:46 +00:00
Robert Schulze
53965bb9f1
Merge remote-tracking branch 'rschu1ze/master' into qatzstd_main 2024-01-07 14:11:22 +00:00
Sema Checherinda
d5f86f671d fix and test that S3Clients reused 2024-01-07 02:19:06 +01:00
robot-clickhouse
4a473c37ec Automatic style fix 2024-01-06 15:55:20 +00:00
Alexey Milovidov
2751b351cb Fix test 2024-01-06 16:44:38 +01:00
Yakov Olkhovskiy
9ca4be5d91 Merge remote-tracking branch 'origin/master' into revert-58450-revert-56064-feature-server-iface-metrics 2024-01-06 00:24:31 +00:00
robot-clickhouse
630ee67262 Automatic style fix 2024-01-05 23:02:58 +00:00
Yakov Olkhovskiy
0964bf3638 add clickhouse-library-bridge logging for test_catboost_evaluate 2024-01-05 22:51:17 +00:00
Nikolay Degterinsky
56aa835954
Merge pull request #58444 from bharatnc/ncb/support-hints-for-db-engines 
support hints for database engines
 2024-01-05 17:12:44 +01:00
Sergei Trifonov
8d6b012223
Merge pull request #57877 from azat/merges-mutations-throttling 
Add ability to throttle merges/mutations
 2024-01-05 17:08:50 +01:00
Sema Checherinda
2861cc70ba fix style 2024-01-05 13:42:30 +00:00
Sema Checherinda
8ab4aa3423 enable randomization setting 2024-01-05 13:42:30 +00:00
Sema Checherinda
53f36f8b21 object storage key template 2024-01-05 13:42:30 +00:00
Alexander Tokmakov
d2752f4b66
Merge pull request #57603 from evillique/allow-not-resolving-hostnames 
Allow avoiding resolving hostnames in DDLWorker
 2024-01-04 15:46:28 +01:00
Alexander Tokmakov
972756486a
Merge pull request #58350 from ClickHouse/rdb_dont_wait_inactive 
An option to avoid waiting for inactive Replicated database replicas
 2024-01-04 14:50:11 +01:00
Alexander Tokmakov
10e3ae9e68 Merge branch 'master' into allow-not-resolving-hostnames 2024-01-04 13:22:15 +01:00
robot-clickhouse
224a58ddc5 Automatic style fix 2024-01-04 10:30:44 +00:00
János Benjamin Antal
1a1fa07128 Add missing asserts 2024-01-04 10:18:04 +00:00
János Benjamin Antal
2517d23e27 Add tests 2024-01-04 10:14:41 +00:00
Nikita Taranov
66d2db5283
New parallel replicas coordinator implementation (#57968) 2024-01-04 00:27:04 +01:00
Bharat Nallan Chakravarthy
d63765fb19 Merge upstream/master into ncb/support-hints-for-db-engines 2024-01-03 08:44:11 -08:00
Alexander Tokmakov
b1688b1953 Merge branch 'master' into rdb_dont_wait_inactive 2024-01-03 17:03:01 +01:00
Val Doroshchuk
5c221d123d MaterializedMySQL: Introduce MySQL Binlog Client 
One binlog connection for many databases.

Suggesting to disable this feature by default for now. It should be explicitly enabled by SETTINGS use_binlog_client=1.
But if you would permanently enable it in MaterializedMySQLSettings, it should keep old behavior and all tests should pass too.

1. Introduced `IBinlog` and its impl to read the binlog events from socket - `BinlogFromSocket`, or file - `BinlogFromFile`. Based on prev impl of `EventBase` and the same old binlog parsers. It fully keeps BC with old version. Fixed `./check-mysql-binlog` to test new impl.
2. Introduced `BinlogEventsDispatcher`, it reads the event from the source `IBinlog` and sends it to currently attached `IBinlog` instances.
3. Introduced `BinlogClient`, which is used to group a list of `BinlogEventsDispatcher` by MySQL binlog connection which is defined by `user:password@host:port`. All dispatchers with the same binlog position should be merged to one.
4. Introduced `BinlogClientFactory`, which is a singleton and it is used to track all binlogs created over the instance.
5. Introduced `use_binlog_client` setting to `MaterializedMySQL`, which forces to reuse a `BinlogClient` if it already exists in `BinlogClientCatalog` or create new one. By default, it is disabled.
6. Introduced `max_bytes_in_binlog_queue` setting to define the limit of bytes in binlog's queue of events. If bytes in the queue increases this limit, `BinlogEventsDispatcher` will stop reading new events from source `IBinlog` until the space for new events will be freed.
7. Introduced `max_milliseconds_to_wait_in_binlog_queue` setting to define max ms to wait when the max bytes exceeded.
7. Introduced `max_milliseconds_to_wait_in_binlog_queue` setting to define max ms to wait when the max bytes exceeded.
8. Introduced `max_bytes_in_binlog_dispatcher_buffer` setting to define max bytes in the binlog dispatcher's buffer before it is flushed to attached binlogs.
9. Introduced `max_flush_milliseconds_in_binlog_dispatcher` setting to define max milliseconds in the binlog dispatcher's buffer to wait before it is flushed to attached binlogs.
10. Introduced `system.mysql_binlogs` system table, which shows a list of active binlogs.
11. Introduced `UnparsedRowsEvent` and `MYSQL_UNPARSED_ROWS_EVENT`, which defines that an event is not parsed and should be explicitly parsed later.
12. Fixed bug when not possible to apply DDL since syntax error or unsupported SQL.

@larspars is the author of following:
`GTIDSets::contains()`
`ReplicationHelper`
`shouldReconnectOnException()`
 2024-01-03 15:26:09 +01:00
Kruglov Pavel
570d1c013b
Merge pull request #57053 from Avogar/better-parsing-exceptions 
Better exception messages in input formats
 2024-01-03 14:47:51 +01:00
Bharat Nallan Chakravarthy
2da0a30626 add a test 2024-01-02 13:52:52 -08:00
Smita Kulkarni
4122de9721 Updated tests and added settings 2024-01-02 20:19:01 +01:00
Nikolay Degterinsky
debbeb505d
Merge pull request #58409 from ClickHouse/evillique-patch-1 
Fix test_user_valid_until
 2024-01-02 17:06:11 +01:00
avogar
5f6a806d1c Fix style 2024-01-02 15:43:44 +00:00
avogar
0844528ec1 Fix tests 2024-01-02 15:22:42 +00:00
Kruglov Pavel
e69bda7f08
Merge pull request #57438 from Avogar/fix-resetting-buffers 
Fix working with read buffers in StreamingFormatExecutor
 2024-01-02 15:34:35 +01:00
Nikolay Degterinsky
004730fe87
Fix test_user_valid_until 2024-01-02 14:50:28 +01:00
Alexey Milovidov
78523639e0 Remove function 2023-12-30 14:52:54 +01:00
avogar
f20917048e Fix kafka tests 2023-12-29 17:02:46 +00:00
Alexander Tokmakov
bfc10bd234 an option to avoid waiting for inactive Replicated db replicas 2023-12-29 15:19:16 +01:00
Igor Nikonov
208a9193f6 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-28 21:28:36 +00:00
Kruglov Pavel
9389ea8745 Fix style 2023-12-28 18:47:57 +00:00
Kruglov Pavel
69f6147499 Fix style 2023-12-28 18:47:57 +00:00
avogar
c6085b3eb9 Fix working with read buffers in StreamingFormatExecutor 2023-12-28 18:47:57 +00:00
Michael Kolupaev
609b2c216c Fix some of the CI 2023-12-28 17:56:05 +00:00
koloshmet
49367186e3 fix fix fix 2023-12-28 17:56:04 +00:00
Kseniia Sumarokova
8e8fd84cb7
Merge pull request #58293 from ClickHouse/fix-s3-queue-test 
Fix test_storage_s3_queue/test.py::test_drop_table
 2023-12-28 17:18:11 +01:00
Alexander Tokmakov
38fe70c68a
Revert "Refreshable materialized views (takeover)" 2023-12-28 13:12:20 +01:00
kssenii
529a1d67cb Fix 2023-12-28 12:48:56 +01:00
Smita Kulkarni
bf60a06a11 Merge branch 'master' into Azure_backup 2023-12-28 11:51:49 +01:00
Alexey Milovidov
40a5dbdeba
Merge branch 'master' into mv 2023-12-28 03:16:27 +01:00
Кирилл Гарбар
764791809b Test for unusual zk path 2023-12-28 03:33:56 +03:00
Alexey Milovidov
30f6d03b37
Merge pull request #57977 from bianpengyuan/add-s3-read-only-setting 
Add a setting to specify s3 disk is read only.
 2023-12-28 01:24:50 +01:00
Кирилл Гарбар
697b4d5fa1 No need to expand uuid macro 2023-12-28 02:35:13 +03:00
Alexey Milovidov
5406c79c89
Merge pull request #58264 from ClickHouse/fs-cache-better-multi-cache-configuration 
Make fs caches with the same path use the same cache objects
 2023-12-28 00:10:15 +01:00
Кирилл Гарбар
6f79f98719 Throw on ordinary db engine 2023-12-28 00:54:10 +03:00
Michael Kolupaev
673743e2ac Fix some of the CI 2023-12-27 20:24:55 +00:00
koloshmet
808cb0fa05 fix fix fix 2023-12-27 20:24:54 +00:00
Sema Checherinda
617ee5abaf
Merge pull request #57551 from bharatnc/ncb/async-metrics-pri-key-size 
add asynchronous metrics for total primary key size in memory and allocated
 2023-12-27 20:24:20 +01:00
Pengyuan Bian
26b553c3b6 format. 2023-12-27 18:27:36 +00:00
kssenii
041c5d4219 Make caches with the same path use the same cache objects 2023-12-27 18:37:43 +01:00
Pengyuan Bian
3dcc1f5448 update. 2023-12-27 07:54:18 +00:00
Igor Nikonov
bee15325fc Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-26 21:56:46 +00:00
Pengyuan Bian
2765acfe45 wip test. 2023-12-26 07:50:46 +00:00
Azat Khuzhin
837f4ea676 Add ability to throttle merges/mutations 
Main motivation was to has an ability to throttle background tasks, to
avoid affecting queries.

To new server settings had been added for this:
- max_mutations_bandwidth_for_server
- max_merges_bandwidth_for_server

Note, that they limit only reading, since usually you will not write
more data then you read, but sometimes it is possible in case of ALTER
UPDATE.

But for now, to keep things simple, I decided to limit this with only
2 settings instead of 4.

Note, that if the write throttling will be needed, then they can use the
same settings, and just create new throttler for write.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:31:49 +01:00
Azat Khuzhin
6ed9b53d1f Refactor test_throttling slightly for upcoming tests 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:29:43 +01:00
Alexey Milovidov
817d0c6553
Merge pull request #58202 from vitlibar/explicit-finalize-in-zip-archive-writer 
Explicit finalize in ZipArchiveWriter
 2023-12-24 03:30:21 +01:00
Vitaly Baranov
e1a136b791 Explicit finalize() function in ZipArchiveWriter. 
Simplify too complicated code in ZipArchiveWriter.
 2023-12-24 00:33:59 +01:00
Smita Kulkarni
3e22f29b45 Fixed parameters 2023-12-23 11:40:58 +01:00
jinjunzh
aaf98f0ed7 refine setting with enable_zstd_qat_codec 2023-12-22 14:24:15 -05:00
Bharat Nallan Chakravarthy
d2f0fe28dc fix test query 2023-12-22 08:48:14 -08:00
Raúl Marín
24f952b2b8
Merge pull request #57986 from Algunenano/zk_retries_quorum 
Support keeper failures in quorum check
 2023-12-22 11:10:04 +01:00
Bharat Nallan Chakravarthy
62b28d365c Merge with upstream/master 2023-12-21 17:04:25 -08:00
Bharat Nallan Chakravarthy
929bc01281 better test with retries 2023-12-21 11:20:16 -08:00
Raúl Marín
6d9da8edd5 Merge remote-tracking branch 'blessed/master' into zk_retries_quorum 2023-12-21 17:03:29 +01:00
Mikhail f. Shiryaev
177f8f7d0d
Create a single point of true for integration tests images 2023-12-21 16:09:56 +01:00
Igor Nikonov
db87ea0b86 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-20 20:48:51 +00:00
Kruglov Pavel
ca7001d70f
Merge pull request #57850 from shanielh/feature/s3-session-token 
Feature/s3 session token
 2023-12-20 17:22:09 +01:00
jsc0218
0a9475abc7
Merge pull request #57874 from joelynch/joelynch/host-psql 
bugfix: addresses_expr ignored for psql named collections
 2023-12-19 15:24:23 -05:00
Кирилл Гарбар
64073f09b6 Support converting tables with explicit engine arguments 2023-12-19 20:57:25 +03:00
Bharat Nallan Chakravarthy
6273c5920a fix tests 2023-12-19 08:26:54 -08:00
Raúl Marín
b458c77340 Fix integration test 2023-12-19 14:36:03 +01:00
Shani Elharrar
5f38e1d944 S3 Session Tokens: Added tests 2023-12-19 10:50:19 +02:00
robot-clickhouse
2be806618e Automatic style fix 2023-12-18 22:07:00 +00:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
Nikita Mikhaylov
ddf5da225d Fix integration test 2023-12-18 21:39:59 +00:00
pufit
36d1905881
Merge pull request #57752 from chizhonkova/refactor_sql_udf_loader 
Refactoring for UserDefinedSQLObjectsLoader
 2023-12-18 16:08:46 -05:00
Bharat Nallan Chakravarthy
73f71cc9fc add a test 2023-12-18 12:39:43 -08:00
Raúl Marín
c4f2511596
Merge pull request #57895 from Algunenano/fix_azure 
Fix test_unset_skip_unavailable_shards
 2023-12-18 12:01:57 +01:00
Кирилл Гарбар
f2a1ab522e Merge remote-tracking branch 'ClickHouse/master' 2023-12-18 13:45:28 +03:00
joelynch
403d9c809d
fix test for postgresql addresses_expr 2023-12-18 10:41:12 +01:00
Igor Nikonov
ec1539728d Fix load balancing for hedged connections 2023-12-16 21:17:01 +00:00
Кирилл Гарбар
1ca059594f Format tests 2023-12-16 13:48:25 +03:00
Igor Nikonov
a735820400 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-15 21:14:43 +00:00
Igor Nikonov
da2dc30056 Fix test: use different log_comment for each test instance 2023-12-15 21:02:55 +00:00
Alexey Milovidov
7e55eab1c3 Revert "Merge pull request #56573 from mkmkme/mkmkme/reload-config" 
This reverts commit 3067ca64df, reversing
changes made to 1e46460910.
 2023-12-15 19:28:25 +01:00
Sema Checherinda
0dfe530a7f
Merge pull request #56744 from MikhailBurdukov/native_copy_for_s3_disks 
Enabled s3 `copyObject` for copy between s3 disks.
 2023-12-15 16:05:16 +01:00
Nikita Mikhaylov
1780671443
ClickHouse Cloud promotion (#57638) 
Co-authored-by: Alexander Sapin <alesapin@gmail.com>
 2023-12-15 15:23:23 +01:00
Yarik Briukhovetskyi
325374c68b
Merge pull request #57342 from bharatnc/ncb/better-hint-if-table-doesnt-exist 
provide a better hint if a table doesn't exist
 2023-12-15 12:35:13 +01:00
Raúl Marín
1e8397f437 Fix test_unset_skip_unavailable_shards 2023-12-15 12:18:40 +01:00
Кирилл Гарбар
8d1dfebb4f Refactor tests 2023-12-14 23:01:50 +03:00
Bharat Nallan Chakravarthy
daef2febf3 Merge with 'upstream/master' 2023-12-14 11:19:44 -08:00
robot-clickhouse
217570a165 Automatic style fix 2023-12-14 18:49:32 +00:00
Igor Nikonov
1370a8c6c4 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-14 18:29:38 +00:00
Igor Nikonov
f8dc6e127e Update tests 2023-12-14 18:28:25 +00:00
joelynch
b4bef78e63
bugfix: addresses_expr ignored for psql named collections 2023-12-14 17:17:14 +01:00
robot-clickhouse
b05c73b3ae Automatic style fix 2023-12-14 13:19:46 +00:00
kssenii
a4a264c785 Fix 2023-12-14 14:04:17 +01:00
Igor Nikonov
42d0afef1f
Merge pull request #57749 from ClickHouse/pr-perf-improve-hope 
Parallel replicas: announcement response handling improvement
 2023-12-14 11:48:51 +01:00
Konstantin Bogdanov
808b4a0213
Merge pull request #57628 from azat/config-nodes-override 
Allow nodes in config with from_env/from_zk and non empty element with replace=1
 2023-12-14 13:20:46 +03:00
jsc0218
78773157b6
Merge pull request #57436 from ryanmjacobs/fix_issue_43186_pg_quote_array_ndims 
Fix several issues regarding PostgreSQL `array_ndims` usage.
 2023-12-13 13:41:43 -05:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
alesapin
230e1fe6c7
Merge pull request #57809 from ClickHouse/fix-flaky-test_parallel_replicas_over_distributed 
Fix flaky test_parallel_replicas_over_distributed
 2023-12-13 17:11:41 +01:00
Alexey Milovidov
54698cd2de
Merge pull request #55876 from ClickHouse/keeper-http-readiness-endpoint 
Keeper HTTP readiness endpoint
 2023-12-13 17:07:57 +01:00
Igor Nikonov
72f1c415f1 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-13 15:03:05 +00:00
Igor Nikonov
c165be76ab
Parallel replicas: friendly settings (#57542) 2023-12-13 14:42:06 +01:00
Igor Nikonov
2a0a5f755c Comment to about possible query results 2023-12-13 12:29:57 +00:00
robot-clickhouse
7d9e9fd42e Automatic style fix 2023-12-13 11:38:41 +00:00
Igor Nikonov
7762beaf6c Fix: w/o replicas sync query result can vary 2023-12-13 11:23:53 +00:00
Kseniia Sumarokova
b1325450e9
Merge pull request #57754 from ClickHouse/fix-loading-dependent-table-materialized-postgresql 
Table engine MaterializedPostgreSQL fix dependency loading
 2023-12-13 12:17:01 +01:00
Igor Nikonov
5abeadf20f Fix: stop merges, otherwise test can be flaky 2023-12-13 08:54:44 +00:00
robot-clickhouse
ea86b33d3b Automatic style fix 2023-12-12 21:59:08 +00:00
Igor Nikonov
3333a7f219 Test cleanup 2023-12-12 21:39:08 +00:00
Igor Nikonov
616d054dee Merge remote-tracking branch 'origin/master' into pr-perf-improve-hope 2023-12-12 21:27:31 +00:00
Igor Nikonov
7f4a028196 Test and fix 2023-12-12 21:26:58 +00:00
Alexander Tokmakov
3482224f83 Merge branch 'master' into keeper-http-readiness-endpoint 2023-12-12 21:19:57 +01:00
Кирилл Гарбар
a6549c106a Tests for engine convertation 2023-12-12 21:45:00 +03:00
robot-clickhouse
f2336ff025 Automatic style fix 2023-12-12 14:43:44 +00:00
kssenii
e77cb18d18 Better test 2023-12-12 15:30:17 +01:00
Natalya Chizhonkova
3d51192bad Refactoring for UserDefinedSQLObjectsLoader 2023-12-12 17:03:49 +03:00
Kruglov Pavel
8a447bf57c
Merge pull request #55892 from Avogar/schema-inference-union 
Add 'union' mode for schema inference
 2023-12-12 15:02:06 +01:00
Kruglov Pavel
b33f124555
Merge pull request #57538 from MikhailBurdukov/ignore_on_cluster_for_grants 
Ignore ON CLUSTER clause in grant/revoke queries for management of replicated access entities.
 2023-12-12 14:59:58 +01:00
Konstantin Bogdanov
511cfb393d
Remove time.sleep from test 2023-12-12 12:29:00 +01:00
robot-clickhouse
b49452fb45 Automatic style fix 2023-12-12 10:44:55 +00:00
Konstantin Bogdanov
8fe2cd1a7e
Update tests 2023-12-12 11:35:17 +01:00
Igor Nikonov
fd92cb4c32
Merge pull request #57757 from ClickHouse/fix-flaky-test_parallel_replicas_distributed 
Fix flaky test_parallel_replicas_distributed_read_from_all
 2023-12-12 10:43:19 +01:00
Konstantin Bogdanov
1f9c7336a9
Fix test helper naming 2023-12-12 09:32:16 +01:00
robot-clickhouse
e95041a3af Automatic style fix 2023-12-11 21:40:23 +00:00
Igor Nikonov
da43d49ffc Fix: sync is failed with stopped merges 2023-12-11 21:30:25 +00:00
alesapin
cbae29d26a
Merge pull request #57699 from hanfei1991/hanfei/keeper_soft_limit_ratio 
support keeper memory soft limit ratio
 2023-12-11 21:43:24 +01:00
Igor Nikonov
c007025ee3 Fix flaky test_parallel_replicas_distributed_read_from_all 2023-12-11 20:05:26 +00:00
Ryan Jacobs
40da568ef9
Merge branch 'master' into fix_issue_43186_pg_quote_array_ndims 2023-12-11 11:17:38 -08:00
kssenii
05e56f76ed Merge remote-tracking branch 'origin/master' into fix-loading-dependent-table-materialized-postgresql 2023-12-11 20:17:26 +01:00
Kseniia Sumarokova
907d0a79eb
Merge pull request #57568 from ClickHouse/fix-materialized-pg-issue-with-generated-columns 
MaterializedPostgreSQL: Support "generated columns" and default values
 2023-12-11 19:57:41 +01:00
Kruglov Pavel
3b1ffbd77c
Merge branch 'master' into schema-inference-union 2023-12-11 18:59:15 +01:00
kssenii
48ce040621 Fix 2023-12-11 17:30:23 +01:00
kssenii
a06179da00 Merge remote-tracking branch 'origin/master' into fix-materialized-pg-issue-with-generated-columns 2023-12-11 11:15:00 +01:00
robot-clickhouse
3adb83e406
Automatic style fix 2023-12-11 10:19:49 +01:00
Konstantin Bogdanov
89ba4a8452
Add HTTP readiness endpoint 2023-12-11 10:19:42 +01:00
MikhailBurdukov
119e451967 Merge branch 'master' into native_copy_for_s3_disks 2023-12-11 07:25:20 +00:00
Vitaly Baranov
4aba3ec685 Add some comments. 2023-12-11 00:56:40 +01:00
Vitaly Baranov
e2a7f4004f Fix test test_create_or_drop_tables_during_backup. 2023-12-11 00:56:28 +01:00
Alexey Milovidov
644ef7b63f
Revert "Fix RWLock inconsistency after write lock timeout" 2023-12-10 23:51:59 +03:00
Igor Nikonov
f7e6a17052 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-10 20:21:47 +00:00
Alexey Milovidov
98f9d2a5fc
Merge pull request #57714 from azat/tests/fix-test_dictionaries_update_and_reload 
Fix test_dictionaries_update_and_reload/test.py::test_reload_while_loading flakiness
 2023-12-10 17:55:50 +01:00
Alexey Milovidov
d119052895
Merge pull request #51858 from ClickHouse/CurtizJ-fix-ttl-non-deterministic 
Merging #37286
 2023-12-10 17:50:36 +01:00
Vitaly Baranov
a058a26b7e
Merge pull request #57454 from vitlibar/fix-rwlock 
Fix RWLock inconsistency after write lock timeout
 2023-12-10 14:09:29 +01:00
Azat Khuzhin
ce9fd2c57a Fix test_dictionaries_update_and_reload/test.py::test_reload_while_loading flakiness 
On CI this test fails due to start_time == prev_start_time:

            start_time, duration = get_loading_start_time("slow"), get_loading_duration("slow")
    >       assert start_time > prev_start_time
    E       assert time.struct_time(tm_year=2023, tm_mon=12, tm_mday=9, tm_hour=23, tm_min=42, tm_sec=2, tm_wday=5, tm_yday=343, tm_isdst=-1) > time.struct_time(tm_year=2023, tm_mon=12, tm_mday=9, tm_hour=23, tm_min=42, tm_sec=2, tm_wday=5, tm_yday=343, tm_isdst=-1)

The reason I guess is that there is sleep(0.5), while the
loading_start_time is DateTime not DateTime64, so you cannot distinguish
values if the difference is less then one second

CI: https://s3.amazonaws.com/clickhouse-test-reports/57710/7af1c0885daaf1e41470c5fdd92abfc7b6b2befc/integration_tests__asan__[3_4].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 08:36:10 +01:00
Alexey Milovidov
33b0ec7d55 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-10 01:21:43 +01:00
Kseniia Sumarokova
69adfffa02
Merge pull request #57588 from ClickHouse/materialized-postgres-escape-publication 
Add a test for #55251
 2023-12-09 12:14:28 +01:00
Han Fei
881e58dfcb support soft limit ratio 2023-12-08 21:48:27 +01:00
avogar
ee7af95bc0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-12-08 20:29:28 +00:00
Nikolay Degterinsky
10bfd054d8 Fix style & review 2023-12-08 19:40:09 +00:00
Nikita Mikhaylov
5251548753 Fix tests and build 2023-12-08 17:28:26 +00:00
alesapin
dc7347e0ba
Merge pull request #57271 from hanfei1991/hanfei/keeper_soft_limit 
support memory soft limit for keeper
 2023-12-08 16:12:47 +01:00
Kseniia Sumarokova
928993f2bd
Merge branch 'master' into fix-materialized-pg-issue-with-generated-columns 2023-12-08 15:37:29 +01:00
Kseniia Sumarokova
d0c5dd3b87
Merge branch 'master' into materialized-postgres-escape-publication 2023-12-08 15:29:10 +01:00
zhongyuankai
7b0f8d44e8
Make DirectoryMonitor handle cluster node list change (#42826) 2023-12-08 14:41:51 +01:00
kssenii
0c40465fec Test 2023-12-08 12:46:59 +01:00
Kseniia Sumarokova
f41d3be4b2
Fix bad merge 2023-12-08 11:28:14 +01:00
Kseniia Sumarokova
de7abe2341
Merge branch 'master' into materialized-postgres-escape-publication 2023-12-08 10:53:39 +01:00
Kseniia Sumarokova
1ec2d29302
Merge pull request #57567 from ClickHouse/kssenii-patch-8 
Fix possible segfault in PostgreSQLSource
 2023-12-08 10:51:08 +01:00
vdimir
6df715f5d7
Merge pull request #57627 from ClickHouse/prometheus-all-errors 
Expose the total number of errors to Prometheus endpoint
 2023-12-08 10:42:01 +01:00
robot-clickhouse
129e0349b7 Automatic style fix 2023-12-08 02:33:04 +00:00
Alexey Milovidov
d810bea8e3 Fix test 2023-12-08 03:17:13 +01:00
Alexey Milovidov
01135211e9 Fix test 2023-12-08 03:12:54 +01:00
Alexey Milovidov
ce9e134043 Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-12-08 03:10:09 +01:00
Vitaly Baranov
b4cac92e28 Add some comments. 2023-12-07 22:18:49 +01:00
robot-clickhouse
f7c3304045 Automatic style fix 2023-12-07 19:42:33 +00:00
Nikita Mikhaylov
0b5b042153 Add test 2023-12-07 20:09:44 +01:00
Azat Khuzhin
2c6604ec24 Allow nodes in config with from_env/from_zk and non empty element with replace=1 
Such nodes in config are useful as a nodes with default values, that can
be overwritten from ZooKeeper/env.

So after this patch the following is valid, and is interpreted as
default value 86400, and can be overwritten via env:

```xml
<asynchronous_metrics_update_period_s replace="1" from_env="CH_ASYNCHRONOUS_METRICS_UPDATE_PERIOD_S">86400</asynchronous_metrics_update_period_s>
```

While the following is not:

```xml
<asynchronous_metrics_update_period_s             from_env="CH_ASYNCHRONOUS_METRICS_UPDATE_PERIOD_S">86400</asynchronous_metrics_update_period_s>
```

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-07 18:01:40 +01:00
Igor Nikonov
94fb23af0b Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-07 11:52:40 +00:00
kssenii
188d4d66c9 Merge with master 2023-12-07 12:45:36 +01:00
kssenii
0486631f4c Fxi 2023-12-07 12:44:00 +01:00
robot-clickhouse
6cf7f75fb4 Automatic style fix 2023-12-07 10:11:44 +00:00
kssenii
246eeee2df Merge remote-tracking branch 'origin/master' into materialized-postgres-escape-publication 2023-12-07 10:48:15 +01:00
kssenii
56b20b6d36 Merge remote-tracking branch 'origin/master' into add-a-test-for-issue-with-table-override 2023-12-07 10:46:08 +01:00
kssenii
bbddd330c6 Add forgotten change 2023-12-07 10:46:00 +01:00
Kseniia Sumarokova
b3f22449f6
Merge pull request #57515 from ClickHouse/fix-materialized-pg-issue-with-incorrect-connection-options 
MaterializedPostgreSQL: fix issue #41922, add test for #41923
 2023-12-07 10:42:14 +01:00
Nikolay Degterinsky
995b51ef73 Allow avoiding resolving hostnames in DDLWorker 2023-12-07 03:43:26 +00:00
Ryan Jacobs
ca790b6eec
apply python style check 2023-12-06 19:26:50 -08:00
Ryan Jacobs
7d41426935
test_storage_postgresql: Add test_postgres_array_ndim_error_messges() 
Tests:

1. View with array column cannot be empty --> error message
2. View cannot have empty array --> error message
3. View cannot have NULL array value --> error message
4. Ensures PG identifiers that require quoting do not crash ClickHouse.

These apply to views that contain arrays.
 2023-12-06 18:39:09 -08:00
kssenii
f461b5b2d0 Add a test 2023-12-06 23:39:34 +01:00
Alexey Milovidov
05bc8ef1e0
Merge pull request #57548 from ClickHouse/s3queue_production_ready 
`S3Queue` is production ready
 2023-12-06 22:21:08 +01:00
Ryan Jacobs
119c2864a0
test_storage_postgresql: mixed-case identifier on array column 2023-12-06 12:59:58 -08:00
robot-clickhouse
4dab2515dc Automatic style fix 2023-12-06 19:46:41 +00:00
kssenii
6321576ff8 Add a test 2023-12-06 20:27:41 +01:00
robot-clickhouse
ec5726c1e6 Automatic style fix 2023-12-06 18:58:03 +00:00
kssenii
6368d060b7 Add a test 2023-12-06 19:39:18 +01:00
vdimir
8f32293104
Merge pull request #57416 from ClickHouse/vdimir/fix_test_encrypted_disk 
Change type of s3_cache in test_encrypted_disk
 2023-12-06 14:37:40 +01:00
Alexey Milovidov
f8de7f6bc2 Merge branch 'master' into s3queue_production_ready 2023-12-06 12:25:29 +01:00
Kseniia Sumarokova
e02883cd4a
Merge branch 'master' into fix-materialized-pg-issue-with-incorrect-connection-options 2023-12-06 12:16:04 +01:00
Igor Nikonov
b4922eece4 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-06 10:44:30 +00:00
Alexey Milovidov
76bbeaa676 S3Queue is production ready 2023-12-05 21:10:19 +01:00
Kseniia Sumarokova
f56fe785e8
Update test.py 2023-12-05 18:40:56 +01:00
MikhailBurdukov
c41511e51c ignore_on_cluster_for_grants 2023-12-05 15:11:32 +00:00
Nikolay Degterinsky
a776a5cac5
Merge pull request #57483 from joelynch/stop-listen-tcp-secure 
bugfix: correctly parse SYSTEM STOP LISTEN TCP SECURE
 2023-12-05 15:25:34 +01:00
robot-clickhouse
fee17abf7a Automatic style fix 2023-12-05 12:40:11 +00:00
kssenii
0d313acd03 Fix 2023-12-05 13:22:35 +01:00
Igor Nikonov
71204c2a82 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-05 12:00:45 +00:00
Alexey Milovidov
1f752b79db
Merge pull request #55894 from bharatnc/ncb/hostname-system-log-tables 
add hostname column to system log tables
 2023-12-05 06:42:43 +01:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
Bharat Nallan Chakravarthy
942802f308 few more test cases 2023-12-04 14:01:43 -08:00
joelynch
b312a9f4ee
bugfix: parse system stop listen tcp secure 2023-12-04 19:56:35 +01:00
Alexey Milovidov
02439eee58
Merge pull request #49351 from ClickHouse/async-loader-integration 
Asynchronous loading of tables
 2023-12-04 17:16:12 +01:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data 
Background merges correctly use temporary data storage in the cache
 2023-12-04 14:52:20 +01:00
vdimir
27589f5b0e
Enable temporary_data_in_cache in test_encrypted_disk 2023-12-04 10:34:17 +00:00
vdimir
b2e29d1416
Change type of s3_cache in test_encrypted_disk 2023-12-04 10:34:13 +00:00
Bharat Nallan Chakravarthy
25d1d59c6d fix tests 2023-12-03 20:16:16 -08:00
Bharat Nallan Chakravarthy
440dc66a5c Merge remote-tracking branch 'upstream/master' into ncb/hostname-system-log-tables 2023-12-03 15:19:47 -08:00
Vitaly Baranov
366095affc Fix test test_create_or_drop_tables_during_backup. 2023-12-04 00:15:31 +01:00
Sema Checherinda
32e3b2657b
Merge pull request #57205 from CheSema/zero-locks-leak 
fix zero-copy locks leaking
 2023-12-02 15:54:12 +01:00
Han Fei
fc0f080cb4 fix black 2023-12-01 20:39:38 +01:00
JackyWoo
8a68d51261
Add index to table system.numbers (#50909) 2023-12-01 19:59:25 +01:00
Han Fei
e4da317f05 fix test 2023-12-01 18:07:45 +01:00
Nikolai Kochetov
0b4131546a
Merge pull request #56813 from jsc0218/SystemTablesFilterEngine 
Able to Filter Engine When Scanning System Tables
 2023-12-01 16:02:27 +01:00
alesapin
fb98b212c5
Merge pull request #57311 from ClickHouse/make_alter_mat_view_non_experimental 
Make alter materialized view non experimental
 2023-12-01 12:18:43 +01:00
robot-clickhouse
a1219d0efd Automatic style fix 2023-11-30 21:12:29 +00:00
Igor Nikonov
b67bccd374 Integration test: custom key with failover 2023-11-30 20:54:53 +00:00
Sema Checherinda
60bf17ce8a fix style 2023-11-30 21:13:56 +01:00
Sema Checherinda
42f648f0d2 relock block when sessin expired 2023-11-30 21:13:56 +01:00
alesapin
17ac9e3409 Fix test 2023-11-30 19:42:12 +01:00
Igor Nikonov
7ab093c5d4 Fix existing intergation test 2023-11-30 18:42:03 +00:00
vdimir
b9860648c1
Remove temporary_data_in_cache from test_merge_tree_s3 2023-11-30 13:27:32 +00:00
Kruglov Pavel
575ee1e665
Merge pull request #56868 from zvonand/zvonand-fileCluster 
Introduce `fileCluster` table function
 2023-11-30 13:49:53 +01:00
Han Fei
a316b70917 Merge branch 'master' into hanfei/keeper_soft_limit 2023-11-30 12:29:40 +01:00
Alexander Tokmakov
a2dc2d9a53
Mark a setting obsolete (#57336) 
* Update Settings.h

* Update InterpreterUndropQuery.cpp

* Update 02888_obsolete_settings.reference

* Update test.py

* Update 02681_undrop_query.sql

* Update 02681_undrop_query_uuid.sh
 2023-11-29 19:24:11 +01:00
Alexander Tokmakov
97b63bbbca
Merge pull request #57339 from ClickHouse/fix_ddlworker_hostname_change 
Ignore finished ON CLUSTER tasks if hostname changed
 2023-11-29 17:46:17 +01:00
Nikolai Kochetov
823ba2db46
Merge pull request #57075 from yariks5s/s3_links_fix 
S3-style links bug fix
 2023-11-29 17:41:08 +01:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
zvonand
c329e382ab resolve conflicts 2023-11-29 16:03:07 +01:00
Han Fei
4c0efb0262
Merge pull request #53240 from hanfei1991/hanfei/statistic 
use statistic to order prewhere conditions better
 2023-11-29 12:26:44 +01:00
Nikolai Kochetov
731a1bfcf8 Cleanup 2023-11-29 11:11:58 +00:00
Alexander Tokmakov
84abbafcc5 ignore finished ddl tasks if hostname changed 2023-11-29 02:06:52 +01:00
Nikolai Kochetov
4d906d6e3c Cleanup 2023-11-28 18:35:19 +00:00
Yarik Briukhovetskyi
ad9583d6e9
style fix 2023-11-28 19:13:37 +01:00
Yarik Briukhovetskyi
a83a046d54 added tests 2023-11-28 17:33:46 +00:00
Nikolai Kochetov
08a7575984 Re-implement filtering a bit. 2023-11-28 16:17:35 +00:00
Han Fei
6733438adc Merge branch 'master' into hanfei/statistic 2023-11-28 16:35:00 +01:00
zvonand
c306d21b54 merge master + resolve conflicts 2023-11-28 15:51:21 +01:00
Han Fei
3aa4a6ae2c check integration test 2023-11-28 14:47:31 +01:00
Antonio Andelic
9dd4783675 Merge branch 'master' into fix-readonly-replica 2023-11-28 10:43:29 +00:00
vdimir
ae09d04578
Merge pull request #57154 from ClickHouse/vdimir/params_test_check_table 
Minor changes in test_check_table
 2023-11-28 11:10:26 +01:00
Han Fei
7bd2ea3add Merge branch 'master' into hanfei/statistic 2023-11-27 23:31:23 +01:00
Han Fei
8e1b61cc13 fix style 2023-11-27 22:03:05 +01:00
Nikolai Kochetov
e1b00b3ee0 Merge branch 'master' into SystemTablesFilterEngine 2023-11-27 18:54:21 +00:00
Han Fei
347efc669e support memory soft limit for keeper 2023-11-27 18:28:10 +01:00
Antonio Andelic
2ea44adefd Add test 2023-11-27 15:05:56 +00:00
Antonio Andelic
9707796869 Ignore comments when comparing column descriptions 2023-11-27 12:25:41 +00:00
MikhailBurdukov
6f19e8ebd1
Merge branch 'master' into native_copy_for_s3_disks 2023-11-27 14:25:36 +03:00
MikhailBurdukov
c10c30832c Review fix 2023-11-27 10:58:30 +00:00
Kruglov Pavel
b10e46b2bc
Merge pull request #57126 from Avogar/size-virtual-column 
Add _size virtual column to s3/file/hdfs/url/azureBlobStorage engines
 2023-11-27 10:12:18 +01:00
Bharat Nallan Chakravarthy
6454687fa6 Merge remote-tracking branch 'upstream/master' into ncb/hostname-system-log-tables 2023-11-26 16:24:18 -08:00
Azat Khuzhin
1168b7910b Update test_throttling with enabled s3_disable_checksum 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-11-26 20:03:33 +01:00
Vitaly Baranov
4fed61e8ca
Merge pull request #57146 from vitlibar/fix-test_replicated_merge_tree_encryption_codec_different_keys_2 
Fix test test_replicated_merge_tree_encryption_codec/test.py::test_different_keys
 2023-11-25 09:36:33 +01:00
Alexey Milovidov
c478acab42
Merge pull request #57206 from azat/tests/test_distributed_storage_configuration 
Fix test_distributed_storage_configuration flakiness
 2023-11-25 02:57:51 +01:00
Sema Checherinda
abd399705e
Merge pull request #57025 from CheSema/s3_engine_heavy_write_check_mem 
test_s3_engine_heavy_write_check_mem: turn test off
 2023-11-24 21:07:35 +01:00
Azat Khuzhin
d0b4917fd3 Fix test_distributed_storage_configuration flakiness 
It fails [1] due to extra reservation on disk2 for some system.*_log
tables, sure we can turn them off, but better to fix it explicitly with
a separate policy.

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/98cddf5312722e403dcea429639ac13dc6cada33/integration_tests__tsan__[2_6].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-11-24 20:28:18 +01:00
Alexey Milovidov
411660c267 Fix integration tests 2023-11-24 17:35:56 +01:00
Vitaly Baranov
5769a88b92
Merge pull request #57133 from vitlibar/change-default-for-wait_dictionaries_load_at_startup 
Change the default for wait_dictionaries_load_at_startup to true
 2023-11-24 17:09:05 +01:00
Vitaly Baranov
683827c6df Add comment. 2023-11-24 14:01:52 +01:00
vdimir
3b4f70a811
Minor changes in test_check_table 2023-11-23 16:24:11 +00:00
Vitaly Baranov
eabdc76e7c Fix test test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 2023-11-23 15:36:11 +01:00
Vitaly Baranov
5f461ff780 Change the default for "wait_dictionaries_load_at_startup" to true, 
and use this setting only if "dictionaries_lazy_load" is false.
 2023-11-23 14:45:42 +01:00
Aleksei Filatov
1a03f5f7f4 Merge remote-tracking branch 'upstream/master' into add_cancellation_point_for_moving_background_operation 2023-11-23 16:43:33 +03:00
avogar
e3afb4182b Fix tests 2023-11-23 11:10:41 +00:00
serxa
0fc25fbda8 Merge branch 'master' into async-loader-integration 2023-11-23 09:32:43 +00:00
jsc0218
aba2f0e4b2 Fix Style 2023-11-22 20:02:15 +00:00
jsc0218
a1bde09c9c move the test from stateless to integration 2023-11-22 19:13:13 +00:00
Kruglov Pavel
b84e3cf683
Merge branch 'master' into size-virtual-column 2023-11-22 19:25:00 +01:00
avogar
193f8a5d23 Add test for azure 2023-11-22 18:12:50 +00:00
zvonand
5153798aeb Introduced fileCluster table function 
Added fileCluster function
Added test and docs
 2023-11-22 15:06:04 +01:00
Kruglov Pavel
9fb1acc947
Merge pull request #56810 from Avogar/iceberg-metadata-files 
Iceberg metadata files
 2023-11-22 14:40:04 +01:00
Sema Checherinda
adf4ade00a
Update tests/integration/test_merge_tree_s3/test.py 2023-11-22 13:38:15 +01:00
Kseniia Sumarokova
e4f66b8469
Merge pull request #55158 from kssenii/fs-cache-improvement 
fs cache improvement for big reads
 2023-11-21 21:50:00 +01:00
Nikita Mikhaylov
087a0a6043
Merge pull request #57040 from ClickHouse/fix-rabbitmq-supress-tsan-in-test 
Fix: suppress TSAN in RabbitMQ test
 2023-11-21 17:52:36 +01:00
Sema Checherinda
63bd45fa5f
Update tests/integration/test_merge_tree_s3/test.py 2023-11-21 17:42:37 +01:00
vdimir
a139ae97eb
Merge pull request #52918 from ClickHouse/vdimir/s3_blob_log 
Add system table with blob storage operations log
 2023-11-21 17:40:42 +01:00
Vitaly Baranov
4052ae8bfd
Merge pull request #57007 from vitlibar/fix-dropping-tables-in-test_create_or_drop_tables_during_backup 
Fix dropping tables in test_create_or_drop_tables_during_backup
 2023-11-21 17:16:16 +01:00
Sema Checherinda
2e1c142d6d black style 2023-11-21 14:03:49 +01:00
Igor Nikonov
08b2740d0f Fix: supress test with tsan 2023-11-21 11:26:14 +00:00
Kseniia Sumarokova
d384762123
Merge branch 'master' into fs-cache-improvement 2023-11-21 11:24:52 +01:00
Kseniia Sumarokova
f960036d76
Merge pull request #56874 from ClickHouse/kssenii-patch-8 
Update test_storage_s3_queue/test.py
 2023-11-21 11:13:00 +01:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Sema Checherinda
524edd8460 turn test off 2023-11-21 00:08:20 +01:00
Alexey Milovidov
6a1955cb34 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-useless-install 2023-11-20 21:34:31 +01:00
Sema Checherinda
485f1834d8
Merge pull request #56938 from CheSema/lz4-buffering 
Lz4 compression: buffer block in a rare case
 2023-11-20 20:33:30 +01:00
Kseniia Sumarokova
9454b203e1
Merge branch 'master' into kssenii-patch-8 2023-11-20 20:19:21 +01:00
Vitaly Baranov
897cd06bcf Fix dropping tables in test "test_create_or_drop_tables_during_backup". 2023-11-20 19:31:53 +01:00
avogar
f537bad469 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-11-20 14:32:50 +00:00
Sema Checherinda
a7fc8d4b99 test_merge_tree_s3 counts errors, turn off s3_use_adaptive_timeouts 2023-11-20 15:04:14 +01:00
avogar
872556a5d4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-11-20 14:03:36 +00:00