As it turns out, docker does not pass through the sysctls, so adjust
this for know users of unprivileged ports (>32K):
- HDFS
- kafka
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
The problem is that MinIO uses dynamic port for console address by
default, which may be below ip_local_port_range, and in this case it ca
be reused in a short time, sicne first MinIO allocates the socket, then
close this socket, and only after try to bind to it.
And even though this is a problem of MinIO I'm not a go developer to fix
it.
v2: use long notation of the 127.0.0.1 (that version of MinIO on CI
cannot handle 127.1, while 2023-07-21T21-12-44Z can)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
There was some cases when some patches to the datetime code leads to
flaky tests, due to the tests itself had been runned using regular
timezone (TZ).
But if you will this tests with something "specific" (that is not
strictly defined around 1970 year), those tests will fail.
So to catch such issues in the PRs itself, let's randomize
session_timezone as well.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
libunwind is reentrant and signal safe, and works faster then then
gcc_eh (plus it has some custom patches for problems that have been
found during it's usage in ClickHouse).
gcc_eh may be missing in the system (if gcc was not installed), and
even if it exists clickhouse uses -nodefaultlibs, so some care should be
made to make it work.
Also this library is tiny and there shouln't be any problem to require
it always (there is already tendency to require some contrib libraries,
i.e. poco).
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
jemalloc can show the following warning:
Number of CPUs detected is not deterministic. Per-CPU arena disabled
It will be shown if one of the following returns different number of
CPUs:
- _SC_NPROCESSORS_ONLN
- _SC_NPROCESSORS_CONF
- sched_getaffinity()
And actually for my CPU linux returns different numbers, because there
are more possible CPUs then online, from dmesg:
smpboot: Allowing 128 CPUs, 64 hotplug CPUs
And from sysfs:
# grep . /sys/devices/system/cpu/{possible,online,offline}
/sys/devices/system/cpu/possible:0-127
/sys/devices/system/cpu/online:0-63
/sys/devices/system/cpu/offline:64-127
From ACPI:
# acpidump -o acpi
# acpixtract -a acpi
# iasl -d *.dat
# grep -e 'Processor Enabled' apic.dsl | sort | uniq -c
64 Processor Enabled : 0
64 Processor Enabled : 1
So I guess this is the same as what happened in this perf run [1].
[1]: https://s3.amazonaws.com/clickhouse-test-reports/51360/5d43a64112711b339b82b1c0e8df7882546a1a3c/performance_comparison_[4_4]/report.html
P.S. personally I, just use cmdline=possible_cpus=64 to fix this for my
setup.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Add zookeeper name in endpoint id
When we migrate a replicated table from one zookeeper cluster to
another (the reason why we migration is that zookeeper's load is
too high), we will create a new table with the same zpath, but it
will fail and the old table will be in trouble.
Here is some infomation:
1.old table:
CREATE TABLE a1 (`id` UInt64)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/default/a1/{shard}', '{replica}')
ORDER BY (id);
2.new table:
CREATE TABLE a2 (`id` UInt64)
ENGINE = ReplicatedMergeTree('aux1:/clickhouse/tables/default/a1/{shard}', '{replica}')
ORDER BY (id);
3.error info:
<Error> executeQuery: Code: 220. DB::Exception: Duplicate interserver IO endpoint:
DataPartsExchange:/clickhouse/tables/default/a1/01/replicas/02.
(DUPLICATE_INTERSERVER_IO_ENDPOINT)
<Error> InterserverIOHTTPHandler: Code: 221. DB::Exception: No interserver IO endpoint
named DataPartsExchange:/clickhouse/tables/default/a1/01/replicas/02.
(NO_SUCH_INTERSERVER_IO_ENDPOINT)
* Revert "Add zookeeper name in endpoint id"
This reverts commit 9deb75b249619b7abdd38e3949ca8b3a76c9df8e.
* Add zookeeper name in endpoint id
When we migrate a replicated table from one zookeeper cluster to
another (the reason why we migration is that zookeeper's load is
too high), we will create a new table with the same zpath, but it
will fail and the old table will be in trouble.
* Fix incompatible with a new setting
* add a test, fix other issues
* Update 02442_auxiliary_zookeeper_endpoint_id.sql
* Update 02735_system_zookeeper_connection.reference
* Update 02735_system_zookeeper_connection.sql
* Update run.sh
* Remove the 'no-fasttest' tag
* Update 02442_auxiliary_zookeeper_endpoint_id.sql
---------
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
By default woboq uses $version.$minor.$patch, while clang simply uses
$version:
CMake Error at generator/CMakeLists.txt:77 (message):
Could not find any clang builtins headers in
/usr/lib/llvm-16/lib/clang/16.0.4/include
# ls -d /usr/lib/llvm-16/lib/clang/16/include
/usr/lib/llvm-16/lib/clang/16/include
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Otherwise you will get the following error:
CMake Error at /usr/lib/llvm-16/lib/cmake/clang/ClangTargets.cmake:792 (message):
The imported target "clangBasic" references the file
"/usr/lib/llvm-16/lib/libclangBasic.a"
but this file does not exist. Possible reasons include:
* The file was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and contained
"/usr/lib/llvm-16/lib/cmake/clang/ClangTargets.cmake"
but not all the files it references.
Call Stack (most recent call first):
/usr/lib/llvm-16/lib/cmake/clang/ClangConfig.cmake:20 (include)
generator/CMakeLists.txt:7 (Find_Package)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
