jemalloc can show the following warning:
Number of CPUs detected is not deterministic. Per-CPU arena disabled
It will be shown if one of the following returns different number of
CPUs:
- _SC_NPROCESSORS_ONLN
- _SC_NPROCESSORS_CONF
- sched_getaffinity()
And actually for my CPU linux returns different numbers, because there
are more possible CPUs then online, from dmesg:
smpboot: Allowing 128 CPUs, 64 hotplug CPUs
And from sysfs:
# grep . /sys/devices/system/cpu/{possible,online,offline}
/sys/devices/system/cpu/possible:0-127
/sys/devices/system/cpu/online:0-63
/sys/devices/system/cpu/offline:64-127
From ACPI:
# acpidump -o acpi
# acpixtract -a acpi
# iasl -d *.dat
# grep -e 'Processor Enabled' apic.dsl | sort | uniq -c
64 Processor Enabled : 0
64 Processor Enabled : 1
So I guess this is the same as what happened in this perf run [1].
[1]: https://s3.amazonaws.com/clickhouse-test-reports/51360/5d43a64112711b339b82b1c0e8df7882546a1a3c/performance_comparison_[4_4]/report.html
P.S. personally I, just use cmdline=possible_cpus=64 to fix this for my
setup.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Add zookeeper name in endpoint id
When we migrate a replicated table from one zookeeper cluster to
another (the reason why we migration is that zookeeper's load is
too high), we will create a new table with the same zpath, but it
will fail and the old table will be in trouble.
Here is some infomation:
1.old table:
CREATE TABLE a1 (`id` UInt64)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/default/a1/{shard}', '{replica}')
ORDER BY (id);
2.new table:
CREATE TABLE a2 (`id` UInt64)
ENGINE = ReplicatedMergeTree('aux1:/clickhouse/tables/default/a1/{shard}', '{replica}')
ORDER BY (id);
3.error info:
<Error> executeQuery: Code: 220. DB::Exception: Duplicate interserver IO endpoint:
DataPartsExchange:/clickhouse/tables/default/a1/01/replicas/02.
(DUPLICATE_INTERSERVER_IO_ENDPOINT)
<Error> InterserverIOHTTPHandler: Code: 221. DB::Exception: No interserver IO endpoint
named DataPartsExchange:/clickhouse/tables/default/a1/01/replicas/02.
(NO_SUCH_INTERSERVER_IO_ENDPOINT)
* Revert "Add zookeeper name in endpoint id"
This reverts commit 9deb75b249619b7abdd38e3949ca8b3a76c9df8e.
* Add zookeeper name in endpoint id
When we migrate a replicated table from one zookeeper cluster to
another (the reason why we migration is that zookeeper's load is
too high), we will create a new table with the same zpath, but it
will fail and the old table will be in trouble.
* Fix incompatible with a new setting
* add a test, fix other issues
* Update 02442_auxiliary_zookeeper_endpoint_id.sql
* Update 02735_system_zookeeper_connection.reference
* Update 02735_system_zookeeper_connection.sql
* Update run.sh
* Remove the 'no-fasttest' tag
* Update 02442_auxiliary_zookeeper_endpoint_id.sql
---------
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
By default woboq uses $version.$minor.$patch, while clang simply uses
$version:
CMake Error at generator/CMakeLists.txt:77 (message):
Could not find any clang builtins headers in
/usr/lib/llvm-16/lib/clang/16.0.4/include
# ls -d /usr/lib/llvm-16/lib/clang/16/include
/usr/lib/llvm-16/lib/clang/16/include
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Otherwise you will get the following error:
CMake Error at /usr/lib/llvm-16/lib/cmake/clang/ClangTargets.cmake:792 (message):
The imported target "clangBasic" references the file
"/usr/lib/llvm-16/lib/libclangBasic.a"
but this file does not exist. Possible reasons include:
* The file was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and contained
"/usr/lib/llvm-16/lib/cmake/clang/ClangTargets.cmake"
but not all the files it references.
Call Stack (most recent call first):
/usr/lib/llvm-16/lib/cmake/clang/ClangConfig.cmake:20 (include)
generator/CMakeLists.txt:7 (Find_Package)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Avoid OOM in perf tests
At some point perf tests started to fail for one setup on CI [1]:
/home/ubuntu/actions-runner/_work/_temp/f8fce7b1-8bc4-49c8-a203-c96867f4420a.sh: line 5: 1882659 Killed python3 performance_comparison_check.py "$CHECK_NAME"
Error: Process completed with exit code 137.
[1]: https://github.com/ClickHouse/ClickHouse/actions/runs/4230936986/jobs/7349818625
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Switch perf tests to ubuntu 22.04 for parallel with --memsuspend
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
---------
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
This reverts commit 24d5be1669, reversing
changes made to a756b4be27.
After #38717 there is no need in --no-stress, since there will be no
such checks for previous releases that fails.
Reverts: #38212
Requires: #38717
Previously due to using of threading.Thread, exception had been
ignored, and perf test simply continues, yes it will show some errors,
but it also may show that the test became faster, while it is because
the underlying table was empty.
Replace threading.Thread with a SafeThread (inline implementation) that
simply rethrow exception in join.
Here is an example of such report [1], look at the ip_trie test.
[1]: https://s3.amazonaws.com/clickhouse-test-reports/45654/2101b66570cbb9eb9a492afa8ab82d562c34336b/performance_comparison_[3/4]/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Replicate poco into base/poco/
* De-register poco submodule
* Build poco from ClickHouse
* Exclude poco from stylecheck
* Exclude poco from whitespace check
* Exclude poco from typo check
* Remove x bit from sources/headers (the style check complained)
* Exclude poco from duplicate include check
* Fix fasttest
* Remove contrib/poco-cmake/*
* Simplify poco build descriptions
* Remove poco stuff not used by ClickHouse
* Glob poco sources
* Exclude poco from clang-tidy
