mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-16 20:53:27 +00:00
208 lines
6.9 KiB
Markdown
208 lines
6.9 KiB
Markdown
---
|
||
slug: /en/operations/allocation-profiling
|
||
sidebar_label: "Allocation profiling"
|
||
title: "Allocation profiling"
|
||
---
|
||
|
||
import Tabs from '@theme/Tabs';
|
||
import TabItem from '@theme/TabItem';
|
||
|
||
# Allocation profiling
|
||
|
||
ClickHouse uses [jemalloc](https://github.com/jemalloc/jemalloc) as its global allocator that comes with some tools for allocation sampling and profiling.
|
||
To make allocation profiling more convenient, `SYSTEM` commands are provided along 4LW commands in Keeper.
|
||
|
||
## Sampling allocations and flushing heap profiles
|
||
|
||
If we want to sample and profile allocations in `jemalloc`, we need to start ClickHouse/Keeper with profiling enabled using environment variable `MALLOC_CONF`.
|
||
|
||
```sh
|
||
MALLOC_CONF=background_thread:true,prof:true
|
||
```
|
||
|
||
`jemalloc` will sample allocation and store the information internally.
|
||
|
||
We can tell `jemalloc` to flush current profile by running:
|
||
|
||
<Tabs groupId="binary">
|
||
<TabItem value="clickhouse" label="ClickHouse">
|
||
|
||
SYSTEM JEMALLOC FLUSH PROFILE
|
||
|
||
</TabItem>
|
||
<TabItem value="keeper" label="Keeper">
|
||
|
||
echo jmfp | nc localhost 9181
|
||
|
||
</TabItem>
|
||
</Tabs>
|
||
|
||
By default, heap profile file will be generated in `/tmp/jemalloc_clickhouse._pid_._seqnum_.heap` where `_pid_` is the PID of ClickHouse and `_seqnum_` is the global sequence number for the current heap profile.
|
||
For Keeper, the default file is `/tmp/jemalloc_keeper._pid_._seqnum_.heap` following the same rules.
|
||
|
||
A different location can be defined by appending the `MALLOC_CONF` environment variable with `prof_prefix` option.
|
||
For example, if we want to generate profiles in `/data` folder where the prefix for filename will be `my_current_profile` we can run ClickHouse/Keeper with following environment variable:
|
||
```sh
|
||
MALLOC_CONF=background_thread:true,prof:true,prof_prefix:/data/my_current_profile
|
||
```
|
||
Generated file will append to prefix PID and sequence number.
|
||
|
||
## Analyzing heap profiles
|
||
|
||
After we generated heap profiles, we need to analyze them.
|
||
For that, we need to use `jemalloc`'s tool called [jeprof](https://github.com/jemalloc/jemalloc/blob/dev/bin/jeprof.in) which can be installed in multiple ways:
|
||
- installing `jemalloc` using system's package manager
|
||
- cloning [jemalloc repo](https://github.com/jemalloc/jemalloc) and running autogen.sh from the root folder that will provide you with `jeprof` script inside the `bin` folder
|
||
|
||
:::note
|
||
`jeprof` uses `addr2line` to generate stacktraces which can be really slow.
|
||
If that’s the case, we recommend installing an [alternative implementation](https://github.com/gimli-rs/addr2line) of the tool.
|
||
|
||
```
|
||
git clone https://github.com/gimli-rs/addr2line
|
||
cd addr2line
|
||
cargo b --examples -r
|
||
cp ./target/release/examples/addr2line path/to/current/addr2line
|
||
```
|
||
:::
|
||
|
||
There are many different formats to generate from the heap profile using `jeprof`.
|
||
We recommend to run `jeprof --help` to check usage and many different options the tool provides.
|
||
|
||
In general, `jeprof` command will look like this:
|
||
|
||
```sh
|
||
jeprof path/to/binary path/to/heap/profile --output_format [ > output_file]
|
||
```
|
||
|
||
If we want to compare which allocations happened between 2 profiles we can set the base argument:
|
||
|
||
```sh
|
||
jeprof path/to/binary --base path/to/first/heap/profile path/to/second/heap/profile --output_format [ > output_file]
|
||
```
|
||
|
||
For example:
|
||
|
||
- if we want to generate a text file with each procedure written per line:
|
||
|
||
```sh
|
||
jeprof path/to/binary path/to/heap/profile --text > result.txt
|
||
```
|
||
|
||
- if we want to generate a PDF file with call-graph:
|
||
|
||
```sh
|
||
jeprof path/to/binary path/to/heap/profile --pdf > result.pdf
|
||
```
|
||
|
||
### Generating flame graph
|
||
|
||
`jeprof` allows us to generate collapsed stacks for building flame graphs.
|
||
|
||
We need to use `--collapsed` argument:
|
||
|
||
```sh
|
||
jeprof path/to/binary path/to/heap/profile --collapsed > result.collapsed
|
||
```
|
||
|
||
After that, we can use many different tools to visualize collapsed stacks.
|
||
|
||
Most popular would be [FlameGraph](https://github.com/brendangregg/FlameGraph) which contains a script called `flamegraph.pl`:
|
||
|
||
```sh
|
||
cat result.collapsed | /path/to/FlameGraph/flamegraph.pl --color=mem --title="Allocation Flame Graph" --width 2400 > result.svg
|
||
```
|
||
|
||
Another interesting tool is [speedscope](https://www.speedscope.app/) that allows you to analyze collected stacks in a more interactive way.
|
||
|
||
## Controlling allocation profiler during runtime
|
||
|
||
If ClickHouse/Keeper were started with enabled profiler, they support additional commands for disabling/enabling allocation profiling during runtime.
|
||
Using those commands, it's easier to profile only specific intervals.
|
||
|
||
Disable profiler:
|
||
|
||
<Tabs groupId="binary">
|
||
<TabItem value="clickhouse" label="ClickHouse">
|
||
|
||
SYSTEM JEMALLOC DISABLE PROFILE
|
||
|
||
</TabItem>
|
||
<TabItem value="keeper" label="Keeper">
|
||
|
||
echo jmdp | nc localhost 9181
|
||
|
||
</TabItem>
|
||
</Tabs>
|
||
|
||
Enable profiler:
|
||
|
||
<Tabs groupId="binary">
|
||
<TabItem value="clickhouse" label="ClickHouse">
|
||
|
||
SYSTEM JEMALLOC ENABLE PROFILE
|
||
|
||
</TabItem>
|
||
<TabItem value="keeper" label="Keeper">
|
||
|
||
echo jmep | nc localhost 9181
|
||
|
||
</TabItem>
|
||
</Tabs>
|
||
|
||
It's also possible to control the initial state of the profiler by setting `prof_active` option which is enabled by default.
|
||
For example, if we don't want to sample allocations during startup but only after we enable the profiler, we can start ClickHouse/Keeper with following environment variable:
|
||
```sh
|
||
MALLOC_CONF=background_thread:true,prof:true,prof_active:false
|
||
```
|
||
|
||
and enable profiler at a later point.
|
||
|
||
## Additional options for profiler
|
||
|
||
`jemalloc` has many different options available related to profiler which can be controlled by modifying `MALLOC_CONF` environment variable.
|
||
For example, interval between allocation samples can be controlled with `lg_prof_sample`.
|
||
If you want to dump heap profile every N bytes you can enable it using `lg_prof_interval`.
|
||
|
||
We recommend to check `jemalloc`s [reference page](https://jemalloc.net/jemalloc.3.html) for such options.
|
||
|
||
## Other resources
|
||
|
||
ClickHouse/Keeper expose `jemalloc` related metrics in many different ways.
|
||
|
||
:::warning Warning
|
||
It's important to be aware that none of these metrics are synchronized with each other and values may drift.
|
||
:::
|
||
|
||
### System table `asynchronous_metrics`
|
||
|
||
```sql
|
||
SELECT *
|
||
FROM system.asynchronous_metrics
|
||
WHERE metric ILIKE '%jemalloc%'
|
||
FORMAT Vertical
|
||
```
|
||
|
||
[Reference](/en/operations/system-tables/asynchronous_metrics)
|
||
|
||
### System table `jemalloc_bins`
|
||
|
||
Contains information about memory allocations done via jemalloc allocator in different size classes (bins) aggregated from all arenas.
|
||
|
||
[Reference](/en/operations/system-tables/jemalloc_bins)
|
||
|
||
### Prometheus
|
||
|
||
All `jemalloc` related metrics from `asynchronous_metrics` are also exposed using Prometheus endpoint in both ClickHouse and Keeper.
|
||
|
||
[Reference](/en/operations/server-configuration-parameters/settings#prometheus)
|
||
|
||
### `jmst` 4LW command in Keeper
|
||
|
||
Keeper supports `jmst` 4LW command which returns [basic allocator statistics](https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Basic-Allocator-Statistics).
|
||
|
||
Example:
|
||
```sh
|
||
echo jmst | nc localhost 9181
|
||
```
|