mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-30 03:22:14 +00:00
b8b6d597ae
* Avoid OOM in perf tests
At some point perf tests started to fail for one setup on CI [1]:
/home/ubuntu/actions-runner/_work/_temp/f8fce7b1-8bc4-49c8-a203-c96867f4420a.sh: line 5: 1882659 Killed python3 performance_comparison_check.py "$CHECK_NAME"
Error: Process completed with exit code 137.
[1]: https://github.com/ClickHouse/ClickHouse/actions/runs/4230936986/jobs/7349818625
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Switch perf tests to ubuntu 22.04 for parallel with --memsuspend
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
---------
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
69 lines
2.7 KiB
Docker
69 lines
2.7 KiB
Docker
# docker build -t clickhouse/performance-comparison .
|
|
|
|
# Using ubuntu:22.04 over 20.04 as all other images, since:
|
|
# a) ubuntu 20.04 has too old parallel, and does not support --memsuspend
|
|
# b) anyway for perf tests it should not be important (backward compatiblity
|
|
# with older ubuntu had been checked lots of times in various tests)
|
|
FROM ubuntu:22.04
|
|
|
|
# ARG for quick switch to a given ubuntu mirror
|
|
ARG apt_archive="http://archive.ubuntu.com"
|
|
RUN sed -i "s|http://archive.ubuntu.com|$apt_archive|g" /etc/apt/sources.list
|
|
|
|
ENV LANG=C.UTF-8
|
|
ENV TZ=Europe/Moscow
|
|
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
|
|
|
|
RUN apt-get update \
|
|
&& DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \
|
|
bash \
|
|
curl \
|
|
dmidecode \
|
|
g++ \
|
|
gdb \
|
|
git \
|
|
gnuplot \
|
|
imagemagick \
|
|
libc6-dbg \
|
|
moreutils \
|
|
ncdu \
|
|
numactl \
|
|
p7zip-full \
|
|
parallel \
|
|
psmisc \
|
|
python3 \
|
|
python3-dev \
|
|
python3-pip \
|
|
python3-setuptools \
|
|
rsync \
|
|
tree \
|
|
tzdata \
|
|
vim \
|
|
wget \
|
|
rustc \
|
|
cargo \
|
|
ripgrep \
|
|
zstd \
|
|
&& pip3 --no-cache-dir install 'clickhouse-driver==0.2.1' scipy \
|
|
&& apt-get purge --yes python3-dev g++ \
|
|
&& apt-get autoremove --yes \
|
|
&& apt-get clean \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
COPY * /
|
|
|
|
# Bind everything to one NUMA node, if there's more than one. Theoretically the
|
|
# node #0 should be less stable because of system interruptions. We bind
|
|
# randomly to node 1 or 0 to gather some statistics on that. We have to bind
|
|
# both servers and the tmpfs on which the database is stored. How to do it
|
|
# through Yandex Sandbox API is unclear, but by default tmpfs uses
|
|
# 'process allocation policy', not sure which process but hopefully the one that
|
|
# writes to it, so just bind the downloader script as well. We could also try to
|
|
# remount it with proper options in Sandbox task.
|
|
# https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt
|
|
# Double-escaped backslashes are a tribute to the engineering wonder of docker --
|
|
# it gives '/bin/sh: 1: [bash,: not found' otherwise.
|
|
CMD ["bash", "-c", "node=$((RANDOM % $(numactl --hardware | sed -n 's/^.*available:\\(.*\\)nodes.*$/\\1/p'))); echo Will bind to NUMA node $node; numactl --cpunodebind=$node --membind=$node /entrypoint.sh"]
|
|
|
|
# docker run --network=host --volume <workspace>:/workspace --volume=<output>:/output -e PR_TO_TEST=<> -e SHA_TO_TEST=<> clickhouse/performance-comparison
|