thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-20 00:30:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
5692db736c
ClickHouse/tests/config/install.sh

97 lines
4.9 KiB
Bash
Raw Normal View History

Install script for CI
2020-09-24 08:35:16 +00:00
#!/bin/bash
# script allows to install configs for clickhouse server and clients required
# for testing (stateless and stateful tests)
set -x -e
DEST_SERVER_PATH="${1:-/etc/clickhouse-server}"
DEST_CLIENT_PATH="${2:-/etc/clickhouse-client}"
SRC_PATH="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
Update install.sh
2020-09-24 10:22:00 +00:00
echo "Going to install test configs from $SRC_PATH into $DEST_SERVER_PATH"
Install script for CI
2020-09-24 08:35:16 +00:00
mkdir -p $DEST_SERVER_PATH/config.d/
mkdir -p $DEST_SERVER_PATH/users.d/
mkdir -p $DEST_CLIENT_PATH
fasttest fixup
2020-09-29 18:55:06 +00:00
ln -sf $SRC_PATH/config.d/zookeeper.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/listen.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/part_log.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/text_log.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/metric_log.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/custom_settings_prefixes.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/macros.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/disks.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/secure_ports.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/clusters.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/graphite.xml $DEST_SERVER_PATH/config.d/
ln -sf $SRC_PATH/config.d/database_atomic.xml $DEST_SERVER_PATH/config.d/
Ignore max concurrent queries in tests
2020-10-06 07:13:55 +00:00
ln -sf $SRC_PATH/config.d/max_concurrent_queries.xml $DEST_SERVER_PATH/config.d/
Mask password in data_path in the system.distribution_queue
2020-11-05 20:04:34 +00:00
ln -sf $SRC_PATH/config.d/test_cluster_with_incorrect_pw.xml $DEST_SERVER_PATH/config.d/
More renames
2021-03-29 08:24:56 +00:00
ln -sf $SRC_PATH/config.d/keeper_port.xml $DEST_SERVER_PATH/config.d/
Disable rotation of the logs in tests (on CI) Right now due to rotation the archive with the clickhouse-server.log maybe not full, for example: - not full - https://clickhouse-test-reports.s3.yandex.net/16947/caf5f98db2ae39dd911f7d1fc0a7c2cc382b1c53/functional_stateless_tests_(ubsan)/clickhouse-server.log (no messages about server start) - full - https://clickhouse-test-reports.s3.yandex.net/16993/d1f52dc72d417580c4088cf3880593176416bea2/functional_stateless_tests_(thread).html And sometimes rotated part may include relative part of the log, so it is better to disable it.
2020-11-18 21:55:37 +00:00
ln -sf $SRC_PATH/config.d/logging_no_rotate.xml $DEST_SERVER_PATH/config.d/
Add missing config link
2020-12-07 20:36:47 +00:00
ln -sf $SRC_PATH/config.d/tcp_with_proxy.xml $DEST_SERVER_PATH/config.d/
Add a test for custom TLD
2020-12-03 21:11:38 +00:00
ln -sf $SRC_PATH/config.d/top_level_domains_lists.xml $DEST_SERVER_PATH/config.d/
Use new top_level_domains_path
2020-12-08 20:58:27 +00:00
ln -sf $SRC_PATH/config.d/top_level_domains_path.xml $DEST_SERVER_PATH/config.d/
fasttest fixup
2020-09-29 18:55:06 +00:00
ln -sf $SRC_PATH/users.d/log_queries.xml $DEST_SERVER_PATH/users.d/
ln -sf $SRC_PATH/users.d/readonly.xml $DEST_SERVER_PATH/users.d/
ln -sf $SRC_PATH/users.d/access_management.xml $DEST_SERVER_PATH/users.d/
Database atomic sync drop detach
2020-10-13 14:29:24 +00:00
ln -sf $SRC_PATH/users.d/database_atomic_drop_detach_sync.xml $DEST_SERVER_PATH/users.d/
Enable OpenTelemetry tracing in functional tests
2020-11-19 15:58:33 +00:00
ln -sf $SRC_PATH/users.d/opentelemetry.xml $DEST_SERVER_PATH/users.d/
Fix flaky tests with distributed queries There was few attempts to fix this [1] and [2], but it still pops up sometimes, for example here [3]. [1]: 009f57fc2fc8 / #11211 [2]: cdb6bed8b98e / #14198 [3]: https://clickhouse-test-reports.s3.yandex.net/21318/38be9ff43ac4c46ce6e803fc125d910bde1d4c71/functional_stateless_tests_(release,_databasereplicated).html#fail1 Let's use more generic approach, and do not hide any errors (and see does increasing timeout will be enough).
2021-03-03 20:32:53 +00:00
ln -sf $SRC_PATH/users.d/remote_queries.xml $DEST_SERVER_PATH/users.d/
fasttest fixup
2020-09-29 18:55:06 +00:00
ln -sf $SRC_PATH/ints_dictionary.xml $DEST_SERVER_PATH/
ln -sf $SRC_PATH/strings_dictionary.xml $DEST_SERVER_PATH/
ln -sf $SRC_PATH/decimals_dictionary.xml $DEST_SERVER_PATH/
ln -sf $SRC_PATH/executable_dictionary.xml $DEST_SERVER_PATH/
Fixed tests
2021-03-01 13:57:47 +00:00
ln -sf $SRC_PATH/executable_pool_dictionary.xml $DEST_SERVER_PATH/
add config
2021-05-18 19:34:49 +00:00
ln -sf $SRC_PATH/transactions.xml $DEST_SERVER_PATH/
fasttest fixup
2020-09-29 18:55:06 +00:00
Use new top_level_domains_path
2020-12-08 20:58:27 +00:00
ln -sf $SRC_PATH/top_level_domains $DEST_SERVER_PATH/
Add a test for custom TLD
2020-12-03 21:11:38 +00:00
fasttest fixup
2020-09-29 18:55:06 +00:00
ln -sf $SRC_PATH/server.key $DEST_SERVER_PATH/
ln -sf $SRC_PATH/server.crt $DEST_SERVER_PATH/
ln -sf $SRC_PATH/dhparam.pem $DEST_SERVER_PATH/
Install script for CI
2020-09-24 08:35:16 +00:00
# Retain any pre-existing config and allow ClickHouse to load it if required
fasttest fixup
2020-09-29 18:55:06 +00:00
ln -sf --backup=simple --suffix=_original.xml \
Update install.sh
2020-09-24 10:22:00 +00:00
$SRC_PATH/config.d/query_masking_rules.xml $DEST_SERVER_PATH/config.d/
Install script for CI
2020-09-24 08:35:16 +00:00
if [[ -n "$USE_POLYMORPHIC_PARTS" ]] && [[ "$USE_POLYMORPHIC_PARTS" -eq 1 ]]; then
fasttest fixup
2020-09-29 18:55:06 +00:00
ln -sf $SRC_PATH/config.d/polymorphic_parts.xml $DEST_SERVER_PATH/config.d/
Install script for CI
2020-09-24 08:35:16 +00:00
fi
Merge branch 'master' into enable_atomic_database_by_default
2020-09-24 23:04:01 +00:00
if [[ -n "$USE_DATABASE_ORDINARY" ]] && [[ "$USE_DATABASE_ORDINARY" -eq 1 ]]; then
fasttest fixup
2020-09-29 18:55:06 +00:00
ln -sf $SRC_PATH/users.d/database_ordinary.xml $DEST_SERVER_PATH/users.d/
Install script for CI
2020-09-24 08:35:16 +00:00
fi
remove some debug code
2021-02-15 10:26:34 +00:00
if [[ -n "$USE_DATABASE_REPLICATED" ]] && [[ "$USE_DATABASE_REPLICATED" -eq 1 ]]; then
ln -sf $SRC_PATH/users.d/database_replicated.xml $DEST_SERVER_PATH/users.d/
try run two nodes with Replicated db and NuKeeper
2021-03-13 01:25:13 +00:00
ln -sf $SRC_PATH/config.d/database_replicated.xml $DEST_SERVER_PATH/config.d/
rm /etc/clickhouse-server/config.d/zookeeper.xml
rename other occurrances
2021-03-29 20:04:50 +00:00
rm /etc/clickhouse-server/config.d/keeper_port.xml
fix
2021-03-13 10:22:48 +00:00
# There is a bug in config reloading, so we cannot override macros using --macros.replica r2
# And we have to copy configs...
Don't fail if directories already exists
2021-04-07 20:00:14 +00:00
mkdir -p /etc/clickhouse-server1
mkdir -p /etc/clickhouse-server2
fix tests
2021-03-18 12:49:31 +00:00
chown clickhouse /etc/clickhouse-server1
fix
2021-03-13 10:22:48 +00:00
chown clickhouse /etc/clickhouse-server2
fix tests
2021-03-18 12:49:31 +00:00
chgrp clickhouse /etc/clickhouse-server1
fix
2021-03-13 10:22:48 +00:00
chgrp clickhouse /etc/clickhouse-server2
fix tests
2021-03-18 12:49:31 +00:00
sudo -u clickhouse cp -r /etc/clickhouse-server/* /etc/clickhouse-server1
fix
2021-03-13 10:22:48 +00:00
sudo -u clickhouse cp -r /etc/clickhouse-server/* /etc/clickhouse-server2
fix tests
2021-03-18 12:49:31 +00:00
rm /etc/clickhouse-server1/config.d/macros.xml
fix
2021-03-13 10:22:48 +00:00
rm /etc/clickhouse-server2/config.d/macros.xml
fix tests
2021-03-18 12:49:31 +00:00
sudo -u clickhouse cat /etc/clickhouse-server/config.d/macros.xml | sed "s|<replica>r1</replica>|<replica>r2</replica>|" > /etc/clickhouse-server1/config.d/macros.xml
sudo -u clickhouse cat /etc/clickhouse-server/config.d/macros.xml | sed "s|<shard>s1</shard>|<shard>s2</shard>|" > /etc/clickhouse-server2/config.d/macros.xml
fix
2021-03-13 10:22:48 +00:00
Don't fail if directories already exists
2021-04-07 20:00:14 +00:00
sudo mkdir -p /var/lib/clickhouse1
sudo mkdir -p /var/lib/clickhouse2
fix tests
2021-03-18 12:49:31 +00:00
sudo chown clickhouse /var/lib/clickhouse1
fix
2021-03-13 10:22:48 +00:00
sudo chown clickhouse /var/lib/clickhouse2
fix tests
2021-03-18 12:49:31 +00:00
sudo chgrp clickhouse /var/lib/clickhouse1
fix
2021-03-13 10:22:48 +00:00
sudo chgrp clickhouse /var/lib/clickhouse2
remove some debug code
2021-02-15 10:26:34 +00:00
fi
Install script for CI
2020-09-24 08:35:16 +00:00
ln -sf $SRC_PATH/client_config.xml $DEST_CLIENT_PATH/config.xml
Reference in New Issue Copy Permalink