Commit Graph

3564 Commits

Author SHA1 Message Date
Kruglov Pavel
22dbaec11a
Merge pull request #33142 from Avogar/fix-s3-write
Don't allow to write into S3 if path contains globs
2021-12-25 16:07:38 +03:00
kssenii
59dd5ba063 Validate config keys for external dictionaries 2021-12-25 14:26:35 +03:00
Azat Khuzhin
42ab02a810 Fix test_async_drain_connection flakiness
Sometimes [1] 5 seconds is not enough, since drain_timeout is 3 seconds,
and 2 seconds sometimes is not enough to do other things, especially
under sanitizers:

    E           Code: 159. DB::Exception: Received from 172.16.1.2:9000. DB::Exception: Timeout exceeded: elapsed 5.019254094 seconds, maximum: 5. Stack trace:

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/0/826f7cb0f53e20e67ef52800cb735bb88a6de658/integration_tests__thread__actions__[4/4].html
2021-12-25 11:55:57 +03:00
Azat Khuzhin
da211909e9 tests/integration: refactor test_async_drain_connection 2021-12-25 11:55:57 +03:00
Yatian Xu
3fa78c5642 test case for metrics: StorageBufferBytes 2021-12-25 00:43:45 -08:00
Yatian Xu
2e151a278d correct table name 2021-12-25 00:03:23 -08:00
Yatian Xu
02362a35b4 add test case for metrics: StorageBufferBytes 2021-12-25 00:02:02 -08:00
avogar
58966e90e9 Don't allow to write into S3 if path contains globs 2021-12-24 17:13:35 +03:00
Nikita Mikhaylov
29fdc422d0
Merge pull request #32974 from nikitamikhaylov/hdfsCluster
Merging #32400
2021-12-24 16:02:54 +03:00
Kevin Michel
ffc1fca296
Start/stop servers when listen_host/*_port changes
This allows starting and stopping separately each protocol server
without restarting ClickHouse.

This also allows adding or removing `listen_host` entries, which
start and stops servers for all enabled ports.

When stopping a server, the listening socket is immediately closed
(and available for another server).

Protocols with persistent connections try to wait for any currently
running query to finish before closing the connection, but idle
connection are closed quickly (depending on how often the protocol
is polled).

An extra ProfileEvent is added, `MainConfigLoads`, it is
incremented every time the configuration is reloaded. This helps
when trying to assess whether the new configuration was applied.
2021-12-24 08:26:02 +01:00
mergify[bot]
d4ec469ed1
Merge branch 'master' into query-kind-concurent_restriction 2021-12-24 07:02:00 +00:00
Kseniia Sumarokova
f0434f21ce
Merge pull request #33039 from kssenii/fix-azure-exception
Fix exception in azure write buffer desctructor
2021-12-23 10:26:40 +03:00
alexey-milovidov
26e651eaf4
Merge pull request #32916 from azat/jemalloc-percpu-fix
Disable percpu arena in jemalloc in case of non deterministic CPU count
2021-12-23 08:05:43 +03:00
kssenii
182a065c9b Add test 2021-12-22 23:47:16 +03:00
Kseniia Sumarokova
764bb1181f
Merge pull request #33017 from stigsb/table-overrides-drop-add-column-support
Remove the possibility of adding columns with table overrides
2021-12-22 22:30:21 +03:00
Azat Khuzhin
b2d4520870 tests/integration/test_jemalloc_percpu_arena: cover explicit narenas 2021-12-22 21:06:21 +03:00
Azat Khuzhin
c82701b1ef tests/integration/test_jemalloc_percpu_arena: add support of fast-test binary 2021-12-22 21:05:11 +03:00
Azat Khuzhin
6efa3b3667 Add a test for jemalloc percpu arena with different number of online CPUs 2021-12-22 21:05:11 +03:00
Kseniia Sumarokova
01f2e4c00d
Merge pull request #33037 from kssenii/fix-s3-empty-file
Fix s3 table function reading empty file
2021-12-22 18:17:29 +03:00
tavplubix
85d49bf557
Merge pull request #33044 from ClickHouse/tavplubix-patch-5
Decrease default timeout for integration test
2021-12-22 17:26:23 +03:00
tavplubix
d8fb293d57
Merge pull request #32900 from zzsmdfj/issue/#15182_MaterializeMySQL_support_bit_type
Add support for BIT data type in MaterializedMySQL
2021-12-22 15:58:29 +03:00
tavplubix
8775afc25e
Update pytest.ini 2021-12-22 14:53:40 +03:00
kssenii
11731cde92 Fix test 2021-12-22 13:57:08 +03:00
tavplubix
4ba76dd8e3
Update materialize_with_ddl.py 2021-12-22 12:19:51 +03:00
kssenii
d8ed4db6d4 Fix 2021-12-22 11:42:23 +03:00
Stig Bakken
1cc70ad524 Remove the possibility of adding columns with table overrides
(except ALIAS columns)

Supporting this properly requires more comprehensive changes.
2021-12-22 08:20:14 +01:00
Kseniia Sumarokova
484a4d62dd
Merge pull request #32835 from kssenii/fising-s3
Fix optimization with lazy seek for async reads from remote fs
2021-12-22 09:38:23 +03:00
Kseniia Sumarokova
b8f6ca470a
Merge pull request #32985 from kssenii/rabbit-fix-tests
RabbitMQ tests fix
2021-12-21 21:32:39 +03:00
ianton-ru
e6fd4bfb50
Merge branch 'master' into MDB-15474 2021-12-21 17:38:36 +03:00
Anton Ivashkin
0c0bf66334 Merge master 2021-12-21 17:27:54 +03:00
Haavard Kvaalen
9a2b0680a1 Fix weird permission on log directories
Several log directories for integration tests were created with full
permissions for "others" but no permission for the owner of the
directory.  This caused issues when trying to run integration tests
again without manually wiping files from previous runs.  The test would
fail with "PermissionError: [Errno 13] Permission denied: 'logs'".

The intention may have been to do the equivalent of 'chmod o+rwx logs',
but it instead did the equivalent of 'chmod 007 logs'.
2021-12-21 11:50:54 +01:00
kssenii
ebcb66a361 Fix 2021-12-21 10:58:02 +03:00
Kseniia Sumarokova
e0f2fe457d
Update test.py 2021-12-21 10:29:27 +03:00
Kseniia Sumarokova
d51b101244
Update test.py 2021-12-21 00:06:24 +03:00
kssenii
a064410a39 Fix 2021-12-21 00:03:55 +03:00
Anton Ivashkin
c724b074ae Remove zero-copy version converter 2021-12-20 20:23:25 +03:00
Vitaly Baranov
e61d3eef0c
Merge pull request #32747 from vitlibar/improve-grpc-compression
Improve gRPC compression support
2021-12-20 19:44:58 +03:00
Anton Ivashkin
f0b9a4327a Use table UUID in zero-copy shared label in ZooKeeper 2021-12-20 16:19:00 +03:00
Zhichang Yu
869cdbeb44 added table function hdfsCluster 2021-12-20 13:14:44 +00:00
Nikolai Kochetov
2e62f086a1
Merge pull request #32751 from ClickHouse/fix-32668
Apply some more optimizations to NO_QUERY ast.
2021-12-20 15:47:25 +03:00
Kseniia Sumarokova
967738ff8f
Merge pull request #32749 from kssenii/table-override-pg
Support Table Override clause for MaterializedPostgreSQL too
2021-12-20 13:11:47 +03:00
kssenii
0411fd6565 Better test 2021-12-19 21:24:52 +03:00
Vitaly Baranov
708439b036 Support customized compression for input/output data in gRPC protocol. 2021-12-19 16:29:44 +03:00
alexey-milovidov
d3a1ffd05d
Merge pull request #32529 from dmthuc/fix_groupBitmapAnd_function_distributed_table
fix groupBitmapAnd function on distributed table
2021-12-19 08:28:26 +03:00
alexey-milovidov
d4f3823f0e
Merge pull request #32885 from arenadata/ADQM-414
warm up for test_kafka_json_as_string_no_kdc
2021-12-19 07:54:30 +03:00
Azat Khuzhin
f93eb61946 tests/integration/cleanup_environment: fix obtaining PYTEST_CLEANUP_CONTAINERS 2021-12-18 13:21:18 +03:00
Dao
e3f16cb73f fix intergration test 2021-12-18 14:42:42 +08:00
Dao
5500d0e439 fix intergration test 2021-12-18 13:35:36 +08:00
Dao
21b8a11907 fix intergration test 2021-12-18 12:27:31 +08:00
Dao
8bbd006d42 enable zookeeper for integration test 2021-12-18 07:55:12 +08:00
Ilya Golshtein
587b663e56 a comment requested by reviewer 2021-12-17 22:49:40 +03:00
kssenii
dd902b6875 Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-17 22:03:58 +03:00
kssenii
dd9347263b Merge remote-tracking branch 'upstream/master' into update-test 2021-12-17 19:13:24 +03:00
kssenii
40c266840c Fixes 2021-12-17 18:16:29 +03:00
minhthucdao
001900655e
Merge branch 'master' into fix_groupBitmapAnd_function_distributed_table 2021-12-17 23:03:47 +08:00
Dao
52af96627c introduce new version and add test 2021-12-17 22:59:02 +08:00
Kseniia Sumarokova
70937088e9
Merge pull request #32886 from kssenii/azure-rename
Rename files and code from BlobStorage to AzureBlobStorage
2021-12-17 16:59:08 +03:00
alesapin
6bd7e425c6
Merge pull request #22535 from CurtizJ/sparse-serialization
Sparse serialization and ColumnSparse
2021-12-17 15:26:17 +03:00
Ilya Golshtein
2821e14492 typo fixed 2021-12-17 15:01:57 +03:00
kssenii
fcb0efcf7d Update test 2021-12-17 14:34:49 +03:00
zzsmdfj
4a0554c963 to #15182_MaterializeMySQL_support_bit_type-fix-flakey-test 2021-12-17 18:30:09 +08:00
kssenii
ed42620530 Rename BlobStorage to AzureBlobStorage 2021-12-17 12:24:05 +03:00
Vladimir C
fa011c66b7
Merge pull request #32846 from vdimir/flaky_test_s3_zero_copy_replication 2021-12-17 12:09:21 +03:00
vdimir
3f118f6734
Try fix flaky test: order for test_s3_zero_copy_replication 2021-12-17 12:08:39 +03:00
zzsmdfj
e9eed1f927 to #15182_MaterializeMySQL_support_bit_type 2021-12-17 15:03:24 +08:00
tavplubix
db01e94f66
Merge pull request #32802 from ClickHouse/fix_restart_unavailable_mysql
Fix restarting with unavailable MySQL
2021-12-17 08:57:15 +03:00
kssenii
8199b24b80 Add test 2021-12-17 01:55:15 +03:00
Ilya Golshtein
faaed87945 warm up for test_kafka_json_as_string_no_kdc 2021-12-17 01:24:21 +03:00
kssenii
97222be3d9 Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-17 01:14:28 +03:00
Alexander Tokmakov
3a8b2b15fc fix 2021-12-16 19:01:04 +03:00
Azat Khuzhin
ba2c9405da Fix test_storage_s3 flakiness 2021-12-16 07:56:43 +03:00
kssenii
7583c8007e Some review fixes 2021-12-16 00:38:46 +03:00
kssenii
dc1f0c58fa Merge remote-tracking branch 'upstream/master' into table-override-pg 2021-12-15 15:27:58 +03:00
Alexander Tokmakov
07bfbc26d6 fix restarting with unavailable mysql 2021-12-15 15:08:08 +03:00
mergify[bot]
3a7179756f
Merge branch 'master' into fix-32668 2021-12-15 08:31:10 +00:00
alexey-milovidov
4adf3b0285
Merge pull request #32740 from azat/integration-tests-fix-mysql-wait
tests/integration: fix waiting of mysql client container
2021-12-14 23:23:50 +03:00
tavplubix
2cf5456423
Merge pull request #32511 from ClickHouse/fix_attaching_gdb2
Try to fix attaching gdb 2
2021-12-14 21:10:00 +03:00
kssenii
47f549166d Table Override for PostgreSQL 2021-12-14 20:36:43 +03:00
Anton Popov
16312e7e4a Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-14 18:58:17 +03:00
Nikolai Kochetov
d394f0e753 Allpy some more optimizations to NO_QUERY ast. 2021-12-14 17:19:18 +03:00
alesapin
0d7bbc399b
Merge pull request #32739 from azat/integration-tests-fix-ch-wait
tests/integration: fix wait_start()/start_clickhouse()
2021-12-14 13:49:15 +03:00
alesapin
a02da27c6f
Merge pull request #32694 from ClickHouse/add-test_s3_zero_copy_concurrent_merge
Add test_s3_zero_copy_concurrent_merge
2021-12-14 12:49:21 +03:00
cmsxbc
0b8bc3f6de wait more times for clear background queries. 2021-12-14 07:37:38 +00:00
cmsxbc
b30e250eed add max_concurrent_select_queries and max_concurrent_insert_queries 2021-12-14 07:37:38 +00:00
Azat Khuzhin
ae122e430a tests/integration: fix waiting of mysql client container
healthy means that everything is OK and no need to wait.

And before it works only because mysql container was too slow to start
and wait_mysql_client_to_start() returns only when the status was
unhealthy/starting, which was obviously wrong.

Fixes: #20393 (cc @qoega)
2021-12-14 10:04:30 +03:00
Azat Khuzhin
1a134f5fc0 tests/integration: fix printing all available services in wait_mysql_client_to_start()
There is no docker-compose.yml in that place, since mysql client has
different name:

    2021-12-14 04:03:02 [ 862 ] DEBUG : Command:['docker-compose', 'ps', '--services', '--all'] (cluster.py:63, run_and_check)
    2021-12-14 04:03:03 [ 862 ] DEBUG : Stderr: (cluster.py:71, run_and_check)
    2021-12-14 04:03:03 [ 862 ] DEBUG : Stderr:        Can't find a suitable configuration file in this directory or any (cluster.py:71, run_and_check)
    2021-12-14 04:03:03 [ 862 ] DEBUG : Stderr:        parent. Are you in the right directory? (cluster.py:71, run_and_check)
    2021-12-14 04:03:03 [ 862 ] DEBUG : Stderr: (cluster.py:71, run_and_check)
    2021-12-14 04:03:03 [ 862 ] DEBUG : Stderr:        Supported filenames: docker-compose.yml, docker-compose.yaml (cluster.py:71, run_and_check)
    2021-12-14 04:03:03 [ 862 ] DEBUG : Stderr:         (cluster.py:71, run_and_check)
    2021-12-14 04:03:03 [ 862 ] DEBUG : Exitcode:1 (cluster.py:73, run_and_check)
2021-12-14 10:00:20 +03:00
Azat Khuzhin
01a57af774 tests/integration: fix wait_start()/start_clickhouse()
Execute query at least once, otherwise it may incorrectly detect that
the server is "alive".

Fixes: test_keeper_incorrect_config/test.py::test_duplicate_endpoint
CI: https://s3.amazonaws.com/clickhouse-test-reports/32536/559b01a2e43d09571195fb1cc08d83133f559203/integration_tests__asan__actions__[1/3].html
2021-12-14 09:45:46 +03:00
mergify[bot]
2b181d7107
Merge branch 'master' into mongo-extend-auth-2 2021-12-13 22:21:10 +00:00
Kseniia Sumarokova
a9ca84ff7d
Merge pull request #32325 from stigsb/table-overrides
Add TABLE OVERRIDE feature for MaterializedMySQL databases
2021-12-13 20:39:42 +03:00
kssenii
ddbeadd104 Done 2021-12-13 16:09:18 +00:00
Kseniia Sumarokova
78ed383f15
Merge pull request #31505 from ContentSquare/azure_blob_storage_dependency
Azure Blob Storage Disks
2021-12-13 18:28:48 +03:00
Nikolai Kochetov
31f8ff4e9c Add test_s3_zero_copy_concurrent_merge 2021-12-13 16:34:04 +03:00
Jakub Kuklis
77f339caaf Cache test as the first test in the file 2021-12-13 13:41:17 +01:00
Jakub Kuklis
38e35a0ef5 Removing any extra work and correcting the cache path in the cache test 2021-12-13 10:52:02 +01:00
kssenii
a8484ec06d Add test 2021-12-12 00:47:21 +03:00
mergify[bot]
1e28f7340e
Merge branch 'master' into kafka_protobuf_issue26643 2021-12-11 08:05:20 +00:00
Kseniia Sumarokova
7e0bc290da
Merge pull request #32460 from kssenii/mongo-extend-auth
Allow empty credentials for mongo engine
2021-12-11 10:42:15 +03:00
Jakub Kuklis
60e06ca34c Changing order of tests to check if that affects failing tests in the CI/CD 2021-12-10 22:04:38 +01:00
Jakub Kuklis
578aec97e9 Removing TODO and NOTE marks used mainly as hooks for review in the PR 2021-12-10 22:04:38 +01:00
Jakub Kuklis
69900ba417 Zero replication support test 2021-12-10 22:04:38 +01:00
Jakub Kuklis
b5db3a307f Cache test, removing spurious imports 2021-12-10 22:04:38 +01:00
Jakub Kuklis
7351325f13 Simple check for files in cloud 2021-12-10 22:04:38 +01:00
Jakub Kuklis
312e5b8c45 Abstracting utility functions from main integration tests for remote disks 2021-12-10 22:04:38 +01:00
Jakub Kuklis
1d5c69b366 Minor tweaks to the test, disabling another slow one 2021-12-10 22:04:38 +01:00
Jakub Kuklis
312ea92989 More integration tests, all pass, at least usually 2021-12-10 22:04:38 +01:00
Jakub Kuklis
afc64c14ce Merging test works correctly 2021-12-10 22:04:38 +01:00
Jakub Kuklis
6dca10e1c5 First integration tests, the one for merging fails for the moment 2021-12-10 22:04:38 +01:00
Jakub Kuklis
e58f74d8dc Connecting to Azurite and invoking simple INSERTs works in the integration test 2021-12-10 22:04:38 +01:00
Jakub Kuklis
8afe69ff93 Comming back to StorageSharedKeyCredential for the integration test, get 400 Bad requests from Azurite when trying to create a container now 2021-12-10 22:04:38 +01:00
Jakub Kuklis
f180a7ba5f Solved Invalid storage account error with connection strings 2021-12-10 22:04:38 +01:00
Jakub Kuklis
abd685e14c Installing azure-storage-blob in the integration tests docker, raising exception when not connected to Azurite, changing ports in the Azurite docker 2021-12-10 22:04:38 +01:00
Jakub Kuklis
008141de36 Enable authentication via StorageSharedKeyCredential 2021-12-10 22:04:38 +01:00
Jakub Kuklis
d51f979c7a Adding endpoint validation 2021-12-10 22:04:38 +01:00
Jakub Kuklis
910db6ca57 Code cleanup addressing PR remarks 2021-12-10 22:04:38 +01:00
Jakub Kuklis
639d8a2b38 Running Azurite docker in the integration test for Blob Storage support 2021-12-10 22:04:37 +01:00
Jakub Kuklis
5ed511f9da Working skeleton for Blob Storage integration test 2021-12-10 22:04:37 +01:00
Jakub Kuklis
24bafe4740 Starting an integration test for Blob Storage 2021-12-10 22:04:37 +01:00
Vitaly Baranov
b5b195f4e2
Merge branch 'master' into kafka_protobuf_issue26643 2021-12-10 23:22:35 +03:00
Alexander Tokmakov
dfa31a6577 fix 2021-12-10 22:38:34 +03:00
alesapin
a6e55b00e3
Revert "Revert "Split long tests into multiple checks"" 2021-12-10 18:39:02 +03:00
alesapin
dcbba460c1
Revert "Split long tests into multiple checks" 2021-12-10 18:38:42 +03:00
alesapin
a2aa0bc6cc
Merge pull request #32496 from ClickHouse/smaller_checks
Split long tests into multiple checks
2021-12-10 17:42:14 +03:00
vdimir
a0488d21d2
Fix test_prometheus_endpoint 2021-12-10 14:58:46 +03:00
alesapin
2c7bb56a2c Split integration tests into parts 2021-12-10 13:22:29 +03:00
alexey-milovidov
6ce2559078
Merge branch 'master' into feature/dotnet-integration-test 2021-12-10 01:25:47 +03:00
kssenii
f5a77fca39 Done 2021-12-09 14:44:25 +00:00
Anton Popov
d8367334a3 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-08 18:26:19 +03:00
Mikhail f. Shiryaev
03927f5fc6
Revert "graphite: split tagged/plain rollup rules (for merges perfomance)"
This reverts commit 303552f515.
2021-12-08 15:49:21 +01:00
tavplubix
4f46ac6b30
Remove LeaderElection (#32140)
* remove LeaderElection

* try fix tests

* Update test.py

* Update test.py
2021-12-07 19:55:55 +03:00
Mikhail f. Shiryaev
5fc20b3e6a
Merge pull request #25122 from msaf1980/rollup_rules_type
GraphiteMergeTree Rollup rules type
2021-12-07 10:18:08 +01:00
Stig Bakken
bb521cd363 Add TABLE OVERRIDE feature for MaterializedMySQL databases 2021-12-07 09:45:02 +01:00
Kseniia Sumarokova
7e261245d3
Merge pull request #32256 from taiyang-li/default_for_hdfs
Support default expression for storage hdfs and optimize fetching when source is column oriented
2021-12-06 23:32:01 +03:00
msaf1980
303552f515
graphite: split tagged/plain rollup rules (for merges perfomance) 2021-12-06 16:49:08 +01:00
Alexander Tokmakov
7797a72907 fix false-positive ciclyc dependency with XML dict 2021-12-06 16:35:43 +03:00
mergify[bot]
52904a4973
Merge branch 'master' into default_for_hdfs 2021-12-06 13:19:49 +00:00
Vladimir C
c502454088
Merge pull request #32185 from kitaisreal/range-hashed-dictionary-added-update-field-support 2021-12-06 10:57:47 +03:00
Maksim Kita
b9718b8450
Merge pull request #32118 from kitaisreal/fix-test-executable-storage-input
Fix flacky test test_executable_storage_input
2021-12-06 10:39:30 +03:00
taiyang-li
9d1e63be0f 1. support default expression for storage hdfs; 2. optimize storage hdfs when format is column oriented 2021-12-06 11:54:45 +08:00
Maksim Kita
4bbb02bbae RangeHashedDictionary added update_field support 2021-12-03 14:06:58 +03:00
Kruglov Pavel
17d6f539b3
Merge pull request #32019 from azat/dist-desc
Fix excessive DESC TABLE for remote() over identifier
2021-12-03 12:15:05 +03:00
mergify[bot]
51b8148cc0
Merge branch 'master' into fix-test-executable-storage-input 2021-12-03 09:11:05 +00:00
Kseniia Sumarokova
a8dcbc988a
Merge pull request #32129 from kssenii/fix-postgres-tests
fixing postgres tests
2021-12-03 09:15:20 +03:00
Anton Popov
f6be3d16fd
Merge pull request #24820 from kssenii/versioning
Versioning of aggregate function states
2021-12-03 01:41:44 +03:00
kssenii
06312a2df9 Fix 2021-12-02 22:48:21 +03:00
alesapin
45021bd35c
Merge pull request #32121 from ClickHouse/add_assertion_in_keeper
Add check for duplicate hostnames and IDs in KeeperConfig
2021-12-02 22:24:46 +03:00
tavplubix
4d96d6ab77
Merge pull request #32052 from stigsb/fix-flaky-create-table-like-test
Fix flaky integration test for MaterializedMySQL CREATE TABLE LIKE
2021-12-02 21:28:00 +03:00
Anton Ivashkin
98bae1b313 Fix tests 2021-12-02 16:25:45 +03:00
kssenii
64fcbe347e Update 2021-12-02 16:08:11 +03:00
alesapin
b8bffcb14c
Merge pull request #32115 from ClickHouse/fix_flaky_keeper_tests
Fix flaky keeper whitelist test
2021-12-02 15:41:17 +03:00
alesapin
437591f2fa Add check for duplicate hostnames and IDs in KeeperConfig 2021-12-02 14:46:33 +03:00