Commit Graph

1767 Commits

Author SHA1 Message Date
Azat Khuzhin
16165d9498 Improve error messages for decompressor
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-17 13:10:40 +02:00
Azat Khuzhin
c374653c7b Revert "Merge pull request #52138 from azat/decompressor-inode"
This reverts commit 6524031348, reversing
changes made to 9bf114f9a3.

This was not a good idea, since the underlying problem was that
`/proc/self/exe` was pointing to `qemu-$ARCH-static` (because the code
uses realpath() over normal interface readlink(), which is not caught by
the qemu linux-user).

And this means that later, it will try to overwrite incorrect binary and
then execute some garbage.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-17 13:10:29 +02:00
Azat Khuzhin
20a671b8cf Skip protection from double decompression if inode from maps cannot be obtained
Under some circumstances, like using qemu-$ARCH-static, /proc/self/maps
will not contain information about /proc/self/exe.

Well, strictly speaking it does contains, however qemu will not pass it
to the user program:

<details>

<summary>strace</summary>

    $ sudo strace -s10000 -f arch-chroot . /qemu-riscv64-static /clickhouse
    ...
    execve("/qemu-riscv64-static", ["/qemu-riscv64-static", "/clickhouse"], 0x7fffffffe458 /* 20 vars */) = 0
    readlinkat(AT_FDCWD, "/proc/self/exe", "/qemu-riscv64-static", 4096) = 20
    openat(AT_FDCWD, "/proc/self/maps", O_RDONLY|O_CLOEXEC) = 4
    [pid  3126] read(4, "00010000-00111000 r--p 00000000 fe:01 30312571                           /clickhouse\n00111000-00119000 r--p 00100000 fe:01 30312571                           /clickhouse\n00119000-0011a000 rw-p 00108000 fe:01 30312571                           /clickhouse\n0011a000-0013d000 rw-p 00000000 00:00 0 \n4000000000-4000001000 ---p 00000000 00:00 0 \n4000001000-4000801000 rw-p 00000000 00:00 0 \n4000801000-400081a000 r--p 00000000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081a000-400081b000 ---p 00000000 00:00 0 \n400081b000-400081c000 r--p 00019000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081c000-400081e000 rw-p 0001a000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081e000-400081f000 r--p 00000000 00:00 0 \n400081f000-4000922000 r--p 00000000 fe:01 30316935 /lib/riscv64-linux-gnu/libc-2.32.so\n4000922000-4000926000 r--p 00102000 fe:01 30316935                       /lib/riscv64-linux-gnu/libc-2.32.so\n4000926000-4000928000 rw-p 00106000 fe:01 30316935                       /lib/riscv64-linux-gnu/libc-2.32.so\n4000928000-400092d000 rw-p 00000000 00:00 0 \n400092d000-40009af000 r--p 00000000 fe:01 30316943                       /lib/riscv64-linux-gnu/libm-2.32.so\n40009af000-40009b0000 r--p 00081000 fe:01 30316943                       /lib/riscv64-linux-gnu/libm-2.32.so\n40009b0000-40009b1000 rw-p 00082000 fe:01 30316943 /lib/riscv64-linux-gnu/libm-2.32.so\n40009b1000-40009c5000 r--p 00000000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c5000-40009c6000 r--p 00013000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c6000-40009c7000 rw-p 00014000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c7000-40009cb000 rw-p 00000000 00:00 0 \n40009cb000-40009cd000 r--p 00000000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009cd000-40009ce000 r--p 00001000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009ce000-40009cf000 rw-p 00002000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009cf000-40009d1000 rw-p 00000000 00:00 0 \n7fffe8000000-7fffeffff000 rwxp 00000000 00:00 0 \n7fffeffff000-7ffff0000000 ---p 00000000 00:00 0 \n7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0 \n7ffff0021000-7ffff4000000 ---p 00000000 00:00 0 \n7ffff6b4b000-7ffff6b5b000 rw-p 00000000 00:00 0 \n7ffff71ff000-7ffff7200000 ---p 00000000 00:00 0 \n7ffff7200000-7ffff7a00000 rw-p 00000000 00:00 0\n7ffff7a00000-7ffff7a3c000 r--p 00000000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7a3c000-7ffff7c74000 r-xp 0003c000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7c74000-7ffff7d77000 r--p 00274000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7d77000-7ffff7dce000 r--p 00377000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7dce000-7ffff7df7000 rw-p 003ce000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7df7000-7ffff7e0c000 rw-p 00000000 00:00 0                          [heap]\n7ffff7e0c000-7ffff7e70000 rw-p 00000000 00:00 0                          [heap]\n7ffff7f42000-7ffff7ff9000 rw-p 00000000 00:00 0 \n7ffff7ff9000-7ffff7ffd000 r--p 00000000 00:00 0                          [vvar]\n7ffff7ffd000-7ffff7fff000 r-xp 00000000 00:00 0                          [vdso]\n7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]\nffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]\n", 4096) = 3608
    [pid  3126] read(4, "", 1024)           = 0
    [pid  3126] close(4)                    = 0
    [pid  3126] write(3, "10000-111000 r-xp 00000000 fe:01 30312571", 41) = 41
    [pid  3126] write(3, "                                /clickhouse\n", 44) = 44
    [pid  3126] write(3, "111000-119000 r--p 00100000 fe:01 30312571", 42) = 42
    [pid  3126] write(3, "                               /clickhouse\n", 43) = 43
    [pid  3126] write(3, "119000-11a000 rw-p 00108000 fe:01 30312571", 42) = 42
    [pid  3126] write(3, "                               /clickhouse\n", 43) = 43
    [pid  3126] write(3, "11a000-13d000 rw-p 00000000 00:00 0", 35) = 35
    [pid  3126] write(3, "                                      \n", 39) = 39
    [pid  3126] write(3, "4000000000-4000001000 ---p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "4000001000-4000801000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              [stack]\n", 38) = 38
    [pid  3126] write(3, "4000801000-400081a000 r-xp 00000000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57 <unfinished ...>
    [pid  3127] <... clock_nanosleep resumed>0x7ffff79ff060) = 0
    [pid  3126] <... write resumed>)        = 57
    [pid  3127] clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=10000000},  <unfinished ...>
    [pid  3126] write(3, "400081a000-400081b000 ---p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400081b000-400081c000 r--p 00019000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57) = 57
    [pid  3126] write(3, "400081c000-400081e000 rw-p 0001a000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57) = 57
    [pid  3126] write(3, "400081e000-400081f000 r-xp 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400081f000-4000922000 r-xp 00000000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000922000-4000926000 r--p 00102000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000926000-4000928000 rw-p 00106000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000928000-400092d000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400092d000-40009af000 r-xp 00000000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009af000-40009b0000 r--p 00081000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009b0000-40009b1000 rw-p 00082000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009b1000-40009c5000 r-xp 00000000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c5000-40009c6000 r--p 00013000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c6000-40009c7000 rw-p 00014000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c7000-40009cb000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "40009cb000-40009cd000 r-xp 00000000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009cd000-40009ce000 r--p 00001000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009ce000-40009cf000 rw-p 00002000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009cf000-40009d1000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-16 08:17:14 +02:00
robot-ch-test-poll
93b40c490f
Merge pull request #51945 from arenadata/ADQM-950-B
Implement support of date/time format specifiers in log and error log file names
2023-07-14 18:54:27 +02:00
robot-clickhouse
8916e7d69b Update version_date.tsv and changelogs after v23.3.8.21-lts 2023-07-13 08:49:48 +00:00
robot-clickhouse
798f31a612 Update version_date.tsv and changelogs after v23.4.6.25-stable 2023-07-12 12:36:57 +00:00
Victor Krasnov
e311d31d87 Merge branch 'master' into ADQM-950-B 2023-07-10 15:50:09 +08:00
Alexey Milovidov
dd1c528d2f
Merge pull request #51985 from azat/cleanup-remote-servers
[RFC] Cleanup remote_servers in dist config.xml
2023-07-10 01:24:06 +03:00
Azat Khuzhin
2db092f9d8 Cleanup remote_servers in dist config.xml
At first, there was no such amount of clusters in dist config, they
added when someone need to write some new cluster for tests.

So let's move them to the clusters.xml that is deployed only for tests,
and leave only default cluster.

And cleanup also some configs that had been copied from dist config in
the repo (about test_config_* integration tests, this should be OK,
since there are more_clusters.xml as well, that covers additional
cases).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-09 16:55:27 +02:00
Alexey Milovidov
29f625e7bb Add a check to validate that the stateful tests are stateful 2023-07-09 03:43:18 +02:00
robot-clickhouse
45d36b736a Update version_date.tsv and changelogs after v23.6.2.18-stable 2023-07-08 23:14:02 +00:00
Victor Krasnov
3edee4174c Add AWST time zone abbreviation to the ignore list 2023-07-07 11:34:03 +08:00
Alexey Milovidov
685f2949b7
Revert "Fix build" 2023-07-07 02:53:13 +03:00
Alexey Milovidov
c47b32b17a Fix build 2023-07-07 01:52:18 +02:00
Alexey Milovidov
48eb30de51 Fix build 2023-07-07 01:42:29 +02:00
Vitaly Baranov
8dd1b7b8b9
Merge branch 'master' into add-hex-functions-for-cityhash 2023-07-03 18:00:33 +02:00
Vitaly Baranov
35d1b8448b
Merge branch 'master' into add-hex-functions-for-cityhash 2023-07-02 01:32:54 +02:00
Antonio Andelic
eba60dd81e
Merge branch 'master' into keeper-with-disks 2023-07-01 20:59:16 +02:00
robot-clickhouse
8ed1ec49d1 Update version_date.tsv and changelogs after v23.6.1.1524-stable 2023-06-30 15:21:13 +00:00
DanRoscigno
904c533a84 spelling list 2023-06-30 09:32:54 -04:00
Antonio Andelic
2058b1346e Merge branch 'master' into keeper-with-disks 2023-06-30 13:22:33 +00:00
robot-clickhouse
f48de18640 Update version_date.tsv and changelogs after v23.4.5.22-stable 2023-06-29 20:59:01 +00:00
DanRoscigno
a8172ca5d2 update spelling list 2023-06-29 14:47:23 -04:00
robot-clickhouse
5656d18690 Update version_date.tsv and changelogs after v23.5.4.25-stable 2023-06-29 13:36:55 +00:00
Vitaly Baranov
0cccba62cf Support getHexUIntLowercase() with CityHash_v1_0_2::uint128 parameter. 2023-06-29 15:29:37 +02:00
Vitaly Baranov
fb6243ec9d
Merge pull request #51040 from vitlibar/reorder-part-checksum-halves
Show halves of checksums in correct order
2023-06-29 12:04:51 +02:00
Mikhail f. Shiryaev
d8f4e45b21
Merge pull request #51548 from ClickHouse/auto/v23.3.6.7-lts
Update version_date.tsv and changelogs after v23.3.6.7-lts
2023-06-28 20:19:17 +02:00
robot-clickhouse
9c2a9a60ea Update version_date.tsv and changelogs after v23.3.6.7-lts 2023-06-28 16:35:41 +00:00
Vitaly Baranov
5570863676
Merge branch 'master' into reorder-part-checksum-halves 2023-06-28 17:25:34 +02:00
Nikita Mikhaylov
d24c5ab01f
Merge branch 'master' into fix-ip-aggregate-state 2023-06-27 14:11:26 +02:00
Dan Roscigno
bcb106e138
add missing aggregate functions (#51443)
* add missing aggregate functions

* add sparkBar
2023-06-27 01:39:47 +02:00
Vitaly Baranov
f1f0daa654 Show halves of checksums in "system.parts", "system.projection_parts" and error messages in the correct order. 2023-06-25 17:17:56 +02:00
Dan Roscigno
cd8cdd4e9b
Update aspell-dict.txt 2023-06-23 16:20:56 -04:00
DanRoscigno
4d92fc915f revert to master 2023-06-23 15:16:57 -04:00
DanRoscigno
4d5a4f342d Merge branch 'docs-s3-engine-partition' of github.com:DanRoscigno/ClickHouse into docs-s3-engine-partition 2023-06-23 10:28:23 -04:00
DanRoscigno
3a0db35d16 allow on-prem 2023-06-23 10:01:54 -04:00
Andrey Zvonov
d4316c7189
Merge branch 'master' into zvonand-implicit-tz 2023-06-22 17:23:20 +02:00
robot-clickhouse
df2284b807 Update version_date.tsv and changelogs after v23.3.5.9-lts 2023-06-22 11:56:00 +00:00
Robert Schulze
57070a6a6e
Ignore "modularization" 2023-06-21 18:18:08 +00:00
Andrey Zvonov
07191ce10d
Merge branch 'master' into zvonand-implicit-tz 2023-06-20 00:24:53 +02:00
Yakov Olkhovskiy
4004be74dc
Merge branch 'master' into fix-ip-aggregate-state 2023-06-19 10:58:06 -04:00
robot-clickhouse
fbcec61aec Update version_date.tsv and changelogs after v22.8.19.10-lts 2023-06-17 14:01:58 +00:00
Yakov Olkhovskiy
cf301324fb
Merge branch 'master' into fix-ip-aggregate-state 2023-06-16 12:49:17 -04:00
zvonand
79222bace2 fix style 2023-06-16 11:11:47 +02:00
zvonand
4155d13d69 merge master and resolve conflict 2023-06-16 02:20:32 +02:00
Yakov Olkhovskiy
35b7474ccf
add MapState to aspell-dict.txt 2023-06-15 10:22:58 -04:00
Kseniia Sumarokova
f1f8b302bf
Merge pull request #50934 from azat/tests/fix-query_log
Fix tests sanity checks and avoid dropping system.query_log table
2023-06-15 12:44:23 +02:00
Antonio Andelic
4825286b0c
Merge branch 'master' into keeper-with-disks 2023-06-14 10:48:53 +02:00
Azat Khuzhin
bb971fd7b7 check-style: allow {database} for ReplicatedMergeTree as well
CLICKHOUSE_TEST_ZOOKEEPER_PREFIX is a {test_name}_{database}, but
actually {database} should be enough, since it is uniq for each test run.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-13 19:45:51 +02:00
Azat Khuzhin
945981e5f6 Fix tests sanity checks
In #43147 the "tests" had been added to EXCLUDE_DIRS, and the reason for
this is that there was some C++ code to ignore [1], however it also
ignores snaity check for query_log.

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/43147/63de577172ee024a08e76db69f5000568673db48/style_check.html

v2: check-style: ignore $EXCLUDE_DIRS for some other sanity checks of tests
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-06-13 19:45:51 +02:00