With default KillMode=control-group, systemd will send signals to all
processes in cgroup and this will lead to server will be terminated
forcefully due to double signal.
2023.09.19 12:47:06.369090 [ 763 ] {} <Information> Application: Received termination signal (Terminated)
2023.09.19 12:47:06.369141 [ 762 ] {} <Debug> Application: Received termination signal.
2023.09.19 12:47:06.369215 [ 763 ] {} <Information> Application: Received termination signal (Terminated)
2023.09.19 12:47:06.369225 [ 763 ] {} <Information> Application: This is the second termination signal. Immediately terminate.
2023.09.19 12:47:06.400959 [ 761 ] {} <Information> Application: Child process exited normally with code 143.
Someone may naively think that, hey, I can change KillMode to
process/mixed, but this will not work either, because in this case
systemd cannot wait for the $MainPID (and main_pid_alien=true in
systemd's sources), because it is not a child of systemd, and this will
lead to double signal again:
2023.09.19 16:24:19.694473 [ 3118 ] {} <Information> Application: Received termination signal (Terminated)
2023.09.19 16:24:19.694894 [ 3118 ] {} <Information> Application: Received termination signal (Terminated)
2023.09.19 16:24:19.695060 [ 3118 ] {} <Information> Application: This is the second termination signal. Immediately terminate.
And this is because it sends signal firstly on a normal termnation and
then when it cleans up left over processes:
clickhouse-server.service: Found left-over process 3117 (clickhouse-serv) in control group while starting unit. Ignoring.
And yes, even though it prints "Ignoring" here (I guess it is related to
the fact that it can be ignored if the signal will not be handled)
Here is a proof of double signal by systemd:
# pgrep clickhouse-serv | xargs strace -e /kill -fp
strace: Process 3117 attached with 469 threads
[pid 3582] --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0} ---
[pid 3580] --- SIGCONT {si_signo=SIGCONT, si_code=SI_USER, si_pid=1, si_uid=0} ---
[pid 3582] --- SIGCONT {si_signo=SIGCONT, si_code=SI_USER, si_pid=1, si_uid=0} ---
[pid 3580] --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=1, si_uid=0} ---
^^^
[pid 3118] tgkill(3117, 3118, SIGTERM) = 0 # and this is a force termination
So yes, there is no other way except for disabling signal forwarding.
*Well, there is another way, but I guess it is will be unwelcome (even
though systemd can be configured in multiple ways right now, and there
is even systemd-oomd instead of clickhouse'es watchdog) - disable
watchdog completelly.*
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Default systemd's timeout for sending SIGKILL after SIGTERM is 1m30s
(TimeoutStopSec), which is can be not enough to wait for queries or
shutdown the storages.
And besides in this case shutdown_wait_unfinished server settings are
ignored.
So let's just disable this systemd logic and rely on
shutdown_wait_unfinished instead.
But note shutting down the storages can take a while, but it is better
to give it time instead of killing the process, since killing may lead
to data loss.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
CapabilityBoundingSet that contained in systemd unit before is about
allowing to set some capabilities, not about granting them.
To grant them you need to use AmbientCapabilities.
And if you do not use 'clickhouse install' then:
- IO priorities was unavailable (since they requires CAP_SYS_NICE)
- For taskstats the procfs was used instead of netlink
Not a big deal, but still.
Here how it had been tested:
$ systemd-run -p CapabilityBoundingSet=CAP_NET_ADMIN --shell
root:/etc (master)# capsh --print
Current: cap_net_admin=ep
Bounding set =cap_net_admin
Ambient set =
$ systemd-run -p User=azat -p CapabilityBoundingSet=CAP_NET_ADMIN --shell
azat:/etc$ capsh --print
Current: =
Bounding set =cap_net_admin
Ambient set =
$ systemd-run -p User=azat -p AmbientCapabilities=CAP_NET_ADMIN -p CapabilityBoundingSet=CAP_NET_ADMIN --shell
azat:/etc$ capsh --print
Current: cap_net_admin=eip
Bounding set =cap_net_admin
Ambient set =cap_net_admin
Note, if you are running it under root (without changing user) you don't
need to specify AmbientCapabilities additionally, because root has all
capabilities by default and they had been inherited.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
After ClickHouse became systemd aware (#43400), it waits not more then
TimeoutStartSec (1m30sec by default), while before it simply ensures
that the process is there.
And likely 1m30sec can be not enough for some cluster, and this will
lead to endless restarts.
At first I've increased it to 10min, but there was a comment about that
this is not enough, and I agree with this.
But I'm not sure that using "inifinity" is a good option, but I cannot
think of any downsides of this.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
