thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-28 02:21:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
119,502 Commits 9,188 Branches 1,782 Tags 2.2 GiB
ed97284bfa
Commit Graph

1782 Commits

Author SHA1 Message Date
robot-ch-test-poll2
d816d8b92f
Merge pull request #52198 from azat/qemu-compressor 
Fix self extracting binaries under qemu linux-user (qemu-$ARCH-static)
 2023-07-18 12:02:47 +02:00
Alexey Milovidov
f4e095b502
Merge pull request #52149 from ucasfl/array_concat_agg 
Add array_concat_agg for compatibility with BigQuery
 2023-07-18 03:03:27 +03:00
vdimir
5de1cfee7d
Merge pull request #51735 from arenadata/ADQM-976 2023-07-17 16:24:11 +02:00
Azat Khuzhin
1fb7605fb4 Fix self extracting binaries under qemu linux-user (qemu-$ARCH-static) 
The problem was that the decompressor uses realpath(/proc/self/exe)
instead of readlink(/proc/self/exe), while realpath() does lots of
trickerly [1] which leads to bypassing qemu linux-user override [2] of
/proc/self/exe to the executable with with it had been called -- and
the reason for this is that the getpid() after unshare returns 1, while
reading /proc/self returns the pid that was before unshare (from the
chroot) [3].

  [1]: 4290aed051/stdlib/canonicalize.c (L223)
  [2]: ed8ad9728a/linux-user/syscall.c (L8634)
  [3]: https://gist.github.com/azat/fcbd8b6c26afd505ae5f3387fc15f0e2

But note, that even after this patch qemu without binfmt will not work,
due to internally the code calls execv() while qemu does not handle it
(see [4]).

  [4]: https://patchwork.kernel.org/project/qemu-devel/patch/1453091602-21843-1-git-send-email-petrosagg@gmail.com/

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-17 13:10:40 +02:00
Azat Khuzhin
16165d9498 Improve error messages for decompressor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-17 13:10:40 +02:00
Azat Khuzhin
c374653c7b Revert "Merge pull request #52138 from azat/decompressor-inode" 
This reverts commit 6524031348, reversing
changes made to 9bf114f9a3.

This was not a good idea, since the underlying problem was that
`/proc/self/exe` was pointing to `qemu-$ARCH-static` (because the code
uses realpath() over normal interface readlink(), which is not caught by
the qemu linux-user).

And this means that later, it will try to overwrite incorrect binary and
then execute some garbage.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-17 13:10:29 +02:00
FFFFFFFHHHHHHH
0895e47629
Merge branch 'master' into jaccard_similarity 2023-07-17 10:33:10 +08:00
flynn
7f80f7dcb0
Merge branch 'master' into array_concat_agg 2023-07-17 01:15:24 +08:00
flynn
6899070f95 fix 2023-07-16 09:54:13 +00:00
Azat Khuzhin
20a671b8cf Skip protection from double decompression if inode from maps cannot be obtained 
Under some circumstances, like using qemu-$ARCH-static, /proc/self/maps
will not contain information about /proc/self/exe.

Well, strictly speaking it does contains, however qemu will not pass it
to the user program:

<details>

<summary>strace</summary>

    $ sudo strace -s10000 -f arch-chroot . /qemu-riscv64-static /clickhouse
    ...
    execve("/qemu-riscv64-static", ["/qemu-riscv64-static", "/clickhouse"], 0x7fffffffe458 /* 20 vars */) = 0
    readlinkat(AT_FDCWD, "/proc/self/exe", "/qemu-riscv64-static", 4096) = 20
    openat(AT_FDCWD, "/proc/self/maps", O_RDONLY|O_CLOEXEC) = 4
    [pid  3126] read(4, "00010000-00111000 r--p 00000000 fe:01 30312571                           /clickhouse\n00111000-00119000 r--p 00100000 fe:01 30312571                           /clickhouse\n00119000-0011a000 rw-p 00108000 fe:01 30312571                           /clickhouse\n0011a000-0013d000 rw-p 00000000 00:00 0 \n4000000000-4000001000 ---p 00000000 00:00 0 \n4000001000-4000801000 rw-p 00000000 00:00 0 \n4000801000-400081a000 r--p 00000000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081a000-400081b000 ---p 00000000 00:00 0 \n400081b000-400081c000 r--p 00019000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081c000-400081e000 rw-p 0001a000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081e000-400081f000 r--p 00000000 00:00 0 \n400081f000-4000922000 r--p 00000000 fe:01 30316935 /lib/riscv64-linux-gnu/libc-2.32.so\n4000922000-4000926000 r--p 00102000 fe:01 30316935                       /lib/riscv64-linux-gnu/libc-2.32.so\n4000926000-4000928000 rw-p 00106000 fe:01 30316935                       /lib/riscv64-linux-gnu/libc-2.32.so\n4000928000-400092d000 rw-p 00000000 00:00 0 \n400092d000-40009af000 r--p 00000000 fe:01 30316943                       /lib/riscv64-linux-gnu/libm-2.32.so\n40009af000-40009b0000 r--p 00081000 fe:01 30316943                       /lib/riscv64-linux-gnu/libm-2.32.so\n40009b0000-40009b1000 rw-p 00082000 fe:01 30316943 /lib/riscv64-linux-gnu/libm-2.32.so\n40009b1000-40009c5000 r--p 00000000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c5000-40009c6000 r--p 00013000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c6000-40009c7000 rw-p 00014000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c7000-40009cb000 rw-p 00000000 00:00 0 \n40009cb000-40009cd000 r--p 00000000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009cd000-40009ce000 r--p 00001000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009ce000-40009cf000 rw-p 00002000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009cf000-40009d1000 rw-p 00000000 00:00 0 \n7fffe8000000-7fffeffff000 rwxp 00000000 00:00 0 \n7fffeffff000-7ffff0000000 ---p 00000000 00:00 0 \n7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0 \n7ffff0021000-7ffff4000000 ---p 00000000 00:00 0 \n7ffff6b4b000-7ffff6b5b000 rw-p 00000000 00:00 0 \n7ffff71ff000-7ffff7200000 ---p 00000000 00:00 0 \n7ffff7200000-7ffff7a00000 rw-p 00000000 00:00 0\n7ffff7a00000-7ffff7a3c000 r--p 00000000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7a3c000-7ffff7c74000 r-xp 0003c000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7c74000-7ffff7d77000 r--p 00274000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7d77000-7ffff7dce000 r--p 00377000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7dce000-7ffff7df7000 rw-p 003ce000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7df7000-7ffff7e0c000 rw-p 00000000 00:00 0                          [heap]\n7ffff7e0c000-7ffff7e70000 rw-p 00000000 00:00 0                          [heap]\n7ffff7f42000-7ffff7ff9000 rw-p 00000000 00:00 0 \n7ffff7ff9000-7ffff7ffd000 r--p 00000000 00:00 0                          [vvar]\n7ffff7ffd000-7ffff7fff000 r-xp 00000000 00:00 0                          [vdso]\n7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]\nffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]\n", 4096) = 3608
    [pid  3126] read(4, "", 1024)           = 0
    [pid  3126] close(4)                    = 0
    [pid  3126] write(3, "10000-111000 r-xp 00000000 fe:01 30312571", 41) = 41
    [pid  3126] write(3, "                                /clickhouse\n", 44) = 44
    [pid  3126] write(3, "111000-119000 r--p 00100000 fe:01 30312571", 42) = 42
    [pid  3126] write(3, "                               /clickhouse\n", 43) = 43
    [pid  3126] write(3, "119000-11a000 rw-p 00108000 fe:01 30312571", 42) = 42
    [pid  3126] write(3, "                               /clickhouse\n", 43) = 43
    [pid  3126] write(3, "11a000-13d000 rw-p 00000000 00:00 0", 35) = 35
    [pid  3126] write(3, "                                      \n", 39) = 39
    [pid  3126] write(3, "4000000000-4000001000 ---p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "4000001000-4000801000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              [stack]\n", 38) = 38
    [pid  3126] write(3, "4000801000-400081a000 r-xp 00000000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57 <unfinished ...>
    [pid  3127] <... clock_nanosleep resumed>0x7ffff79ff060) = 0
    [pid  3126] <... write resumed>)        = 57
    [pid  3127] clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=10000000},  <unfinished ...>
    [pid  3126] write(3, "400081a000-400081b000 ---p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400081b000-400081c000 r--p 00019000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57) = 57
    [pid  3126] write(3, "400081c000-400081e000 rw-p 0001a000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57) = 57
    [pid  3126] write(3, "400081e000-400081f000 r-xp 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400081f000-4000922000 r-xp 00000000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000922000-4000926000 r--p 00102000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000926000-4000928000 rw-p 00106000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000928000-400092d000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400092d000-40009af000 r-xp 00000000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009af000-40009b0000 r--p 00081000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009b0000-40009b1000 rw-p 00082000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009b1000-40009c5000 r-xp 00000000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c5000-40009c6000 r--p 00013000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c6000-40009c7000 rw-p 00014000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c7000-40009cb000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "40009cb000-40009cd000 r-xp 00000000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009cd000-40009ce000 r--p 00001000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009ce000-40009cf000 rw-p 00002000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009cf000-40009d1000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-16 08:17:14 +02:00
robot-ch-test-poll
93b40c490f
Merge pull request #51945 from arenadata/ADQM-950-B 
Implement support of date/time format specifiers in log and error log file names
 2023-07-14 18:54:27 +02:00
robot-clickhouse
8916e7d69b Update version_date.tsv and changelogs after v23.3.8.21-lts 2023-07-13 08:49:48 +00:00
robot-clickhouse
798f31a612 Update version_date.tsv and changelogs after v23.4.6.25-stable 2023-07-12 12:36:57 +00:00
Robert Schulze
521c993eff
Merge branch 'master' into jaccard_similarity 2023-07-11 10:36:18 +02:00
Dmitry Kardymon
0cd4b753d6 Merge remote-tracking branch 'origin/master' into ADQM-976 2023-07-10 13:27:45 +00:00
Victor Krasnov
e311d31d87 Merge branch 'master' into ADQM-950-B 2023-07-10 15:50:09 +08:00
Alexey Milovidov
dd1c528d2f
Merge pull request #51985 from azat/cleanup-remote-servers 
[RFC] Cleanup remote_servers in dist config.xml
 2023-07-10 01:24:06 +03:00
Azat Khuzhin
2db092f9d8 Cleanup remote_servers in dist config.xml 
At first, there was no such amount of clusters in dist config, they
added when someone need to write some new cluster for tests.

So let's move them to the clusters.xml that is deployed only for tests,
and leave only default cluster.

And cleanup also some configs that had been copied from dist config in
the repo (about test_config_* integration tests, this should be OK,
since there are more_clusters.xml as well, that covers additional
cases).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-09 16:55:27 +02:00
Alexey Milovidov
29f625e7bb Add a check to validate that the stateful tests are stateful 2023-07-09 03:43:18 +02:00
robot-clickhouse
45d36b736a Update version_date.tsv and changelogs after v23.6.2.18-stable 2023-07-08 23:14:02 +00:00
Robert Schulze
08b2441784
Merge branch 'master' into jaccard_similarity 2023-07-07 12:41:09 +02:00
Victor Krasnov
3edee4174c Add AWST time zone abbreviation to the ignore list 2023-07-07 11:34:03 +08:00
Alexey Milovidov
685f2949b7
Revert "Fix build" 2023-07-07 02:53:13 +03:00
Alexey Milovidov
c47b32b17a Fix build 2023-07-07 01:52:18 +02:00
Alexey Milovidov
48eb30de51 Fix build 2023-07-07 01:42:29 +02:00
Robert Schulze
01caf205bd
Merge branch 'master' into jaccard_similarity 2023-07-04 10:39:52 +02:00
Robert Schulze
d738ae00a3
Merge remote-tracking branch 'rschu1ze/master' into jaccard_similarity 2023-07-03 18:34:18 +00:00
Vitaly Baranov
8dd1b7b8b9
Merge branch 'master' into add-hex-functions-for-cityhash 2023-07-03 18:00:33 +02:00
Dmitry Kardymon
16ab84d804 Style fix 2023-07-03 11:50:26 +00:00
Dmitry Kardymon
1a40e30797 Add initcapUtf8: impl + tests 2023-07-03 11:37:55 +00:00
FFFFFFFHHHHHHH
91d091a806 fix style 2023-07-02 18:58:58 +08:00
Vitaly Baranov
35d1b8448b
Merge branch 'master' into add-hex-functions-for-cityhash 2023-07-02 01:32:54 +02:00
Antonio Andelic
eba60dd81e
Merge branch 'master' into keeper-with-disks 2023-07-01 20:59:16 +02:00
robot-clickhouse
8ed1ec49d1 Update version_date.tsv and changelogs after v23.6.1.1524-stable 2023-06-30 15:21:13 +00:00
DanRoscigno
904c533a84 spelling list 2023-06-30 09:32:54 -04:00
Antonio Andelic
2058b1346e Merge branch 'master' into keeper-with-disks 2023-06-30 13:22:33 +00:00
robot-clickhouse
f48de18640 Update version_date.tsv and changelogs after v23.4.5.22-stable 2023-06-29 20:59:01 +00:00
DanRoscigno
a8172ca5d2 update spelling list 2023-06-29 14:47:23 -04:00
robot-clickhouse
5656d18690 Update version_date.tsv and changelogs after v23.5.4.25-stable 2023-06-29 13:36:55 +00:00
Vitaly Baranov
0cccba62cf Support getHexUIntLowercase() with CityHash_v1_0_2::uint128 parameter. 2023-06-29 15:29:37 +02:00
Vitaly Baranov
fb6243ec9d
Merge pull request #51040 from vitlibar/reorder-part-checksum-halves 
Show halves of checksums in correct order
 2023-06-29 12:04:51 +02:00
Mikhail f. Shiryaev
d8f4e45b21
Merge pull request #51548 from ClickHouse/auto/v23.3.6.7-lts 
Update version_date.tsv and changelogs after v23.3.6.7-lts
 2023-06-28 20:19:17 +02:00
robot-clickhouse
9c2a9a60ea Update version_date.tsv and changelogs after v23.3.6.7-lts 2023-06-28 16:35:41 +00:00
Vitaly Baranov
5570863676
Merge branch 'master' into reorder-part-checksum-halves 2023-06-28 17:25:34 +02:00
Nikita Mikhaylov
d24c5ab01f
Merge branch 'master' into fix-ip-aggregate-state 2023-06-27 14:11:26 +02:00
Dan Roscigno
bcb106e138
add missing aggregate functions (#51443) 
* add missing aggregate functions

* add sparkBar
 2023-06-27 01:39:47 +02:00
Vitaly Baranov
f1f0daa654 Show halves of checksums in "system.parts", "system.projection_parts" and error messages in the correct order. 2023-06-25 17:17:56 +02:00
Dan Roscigno
cd8cdd4e9b
Update aspell-dict.txt 2023-06-23 16:20:56 -04:00
DanRoscigno
4d92fc915f revert to master 2023-06-23 15:16:57 -04:00
DanRoscigno
4d5a4f342d Merge branch 'docs-s3-engine-partition' of github.com:DanRoscigno/ClickHouse into docs-s3-engine-partition 2023-06-23 10:28:23 -04:00