thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-10 01:25:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
134,289 Commits 9,099 Branches 1,774 Tags 2 GiB
3a4b6a2379
Commit Graph

67 Commits

Author SHA1 Message Date
Yakov Olkhovskiy
648fa52583 move instead copy 2024-02-06 18:48:57 +00:00
Antonio Andelic
fce276c441 Print info while decompressing 2023-11-06 09:29:02 +00:00
Robert Schulze
e4c081652d
Suppress more checks 2023-09-26 11:01:35 +00:00
Robert Schulze
7b585da613
Fix clang-tidy 2023-09-22 10:12:11 +00:00
Robert Schulze
50c51c2854
Fix new clang-tidy-s 2023-09-20 22:17:39 +00:00
Azat Khuzhin
1fb7605fb4 Fix self extracting binaries under qemu linux-user (qemu-$ARCH-static) 
The problem was that the decompressor uses realpath(/proc/self/exe)
instead of readlink(/proc/self/exe), while realpath() does lots of
trickerly [1] which leads to bypassing qemu linux-user override [2] of
/proc/self/exe to the executable with with it had been called -- and
the reason for this is that the getpid() after unshare returns 1, while
reading /proc/self returns the pid that was before unshare (from the
chroot) [3].

  [1]: 4290aed051/stdlib/canonicalize.c (L223)
  [2]: ed8ad9728a/linux-user/syscall.c (L8634)
  [3]: https://gist.github.com/azat/fcbd8b6c26afd505ae5f3387fc15f0e2

But note, that even after this patch qemu without binfmt will not work,
due to internally the code calls execv() while qemu does not handle it
(see [4]).

  [4]: https://patchwork.kernel.org/project/qemu-devel/patch/1453091602-21843-1-git-send-email-petrosagg@gmail.com/

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-17 13:10:40 +02:00
Azat Khuzhin
16165d9498 Improve error messages for decompressor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-17 13:10:40 +02:00
Azat Khuzhin
c374653c7b Revert "Merge pull request #52138 from azat/decompressor-inode" 
This reverts commit 6524031348, reversing
changes made to 9bf114f9a3.

This was not a good idea, since the underlying problem was that
`/proc/self/exe` was pointing to `qemu-$ARCH-static` (because the code
uses realpath() over normal interface readlink(), which is not caught by
the qemu linux-user).

And this means that later, it will try to overwrite incorrect binary and
then execute some garbage.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-17 13:10:29 +02:00
Azat Khuzhin
20a671b8cf Skip protection from double decompression if inode from maps cannot be obtained 
Under some circumstances, like using qemu-$ARCH-static, /proc/self/maps
will not contain information about /proc/self/exe.

Well, strictly speaking it does contains, however qemu will not pass it
to the user program:

<details>

<summary>strace</summary>

    $ sudo strace -s10000 -f arch-chroot . /qemu-riscv64-static /clickhouse
    ...
    execve("/qemu-riscv64-static", ["/qemu-riscv64-static", "/clickhouse"], 0x7fffffffe458 /* 20 vars */) = 0
    readlinkat(AT_FDCWD, "/proc/self/exe", "/qemu-riscv64-static", 4096) = 20
    openat(AT_FDCWD, "/proc/self/maps", O_RDONLY|O_CLOEXEC) = 4
    [pid  3126] read(4, "00010000-00111000 r--p 00000000 fe:01 30312571                           /clickhouse\n00111000-00119000 r--p 00100000 fe:01 30312571                           /clickhouse\n00119000-0011a000 rw-p 00108000 fe:01 30312571                           /clickhouse\n0011a000-0013d000 rw-p 00000000 00:00 0 \n4000000000-4000001000 ---p 00000000 00:00 0 \n4000001000-4000801000 rw-p 00000000 00:00 0 \n4000801000-400081a000 r--p 00000000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081a000-400081b000 ---p 00000000 00:00 0 \n400081b000-400081c000 r--p 00019000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081c000-400081e000 rw-p 0001a000 fe:01 30316932                       /lib/riscv64-linux-gnu/ld-2.32.so\n400081e000-400081f000 r--p 00000000 00:00 0 \n400081f000-4000922000 r--p 00000000 fe:01 30316935 /lib/riscv64-linux-gnu/libc-2.32.so\n4000922000-4000926000 r--p 00102000 fe:01 30316935                       /lib/riscv64-linux-gnu/libc-2.32.so\n4000926000-4000928000 rw-p 00106000 fe:01 30316935                       /lib/riscv64-linux-gnu/libc-2.32.so\n4000928000-400092d000 rw-p 00000000 00:00 0 \n400092d000-40009af000 r--p 00000000 fe:01 30316943                       /lib/riscv64-linux-gnu/libm-2.32.so\n40009af000-40009b0000 r--p 00081000 fe:01 30316943                       /lib/riscv64-linux-gnu/libm-2.32.so\n40009b0000-40009b1000 rw-p 00082000 fe:01 30316943 /lib/riscv64-linux-gnu/libm-2.32.so\n40009b1000-40009c5000 r--p 00000000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c5000-40009c6000 r--p 00013000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c6000-40009c7000 rw-p 00014000 fe:01 30316946                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n40009c7000-40009cb000 rw-p 00000000 00:00 0 \n40009cb000-40009cd000 r--p 00000000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009cd000-40009ce000 r--p 00001000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009ce000-40009cf000 rw-p 00002000 fe:01 30316939                       /lib/riscv64-linux-gnu/libdl-2.32.so\n40009cf000-40009d1000 rw-p 00000000 00:00 0 \n7fffe8000000-7fffeffff000 rwxp 00000000 00:00 0 \n7fffeffff000-7ffff0000000 ---p 00000000 00:00 0 \n7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0 \n7ffff0021000-7ffff4000000 ---p 00000000 00:00 0 \n7ffff6b4b000-7ffff6b5b000 rw-p 00000000 00:00 0 \n7ffff71ff000-7ffff7200000 ---p 00000000 00:00 0 \n7ffff7200000-7ffff7a00000 rw-p 00000000 00:00 0\n7ffff7a00000-7ffff7a3c000 r--p 00000000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7a3c000-7ffff7c74000 r-xp 0003c000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7c74000-7ffff7d77000 r--p 00274000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7d77000-7ffff7dce000 r--p 00377000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7dce000-7ffff7df7000 rw-p 003ce000 fe:01 30316953                   /qemu-riscv64-static\n7ffff7df7000-7ffff7e0c000 rw-p 00000000 00:00 0                          [heap]\n7ffff7e0c000-7ffff7e70000 rw-p 00000000 00:00 0                          [heap]\n7ffff7f42000-7ffff7ff9000 rw-p 00000000 00:00 0 \n7ffff7ff9000-7ffff7ffd000 r--p 00000000 00:00 0                          [vvar]\n7ffff7ffd000-7ffff7fff000 r-xp 00000000 00:00 0                          [vdso]\n7ffffffde000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]\nffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]\n", 4096) = 3608
    [pid  3126] read(4, "", 1024)           = 0
    [pid  3126] close(4)                    = 0
    [pid  3126] write(3, "10000-111000 r-xp 00000000 fe:01 30312571", 41) = 41
    [pid  3126] write(3, "                                /clickhouse\n", 44) = 44
    [pid  3126] write(3, "111000-119000 r--p 00100000 fe:01 30312571", 42) = 42
    [pid  3126] write(3, "                               /clickhouse\n", 43) = 43
    [pid  3126] write(3, "119000-11a000 rw-p 00108000 fe:01 30312571", 42) = 42
    [pid  3126] write(3, "                               /clickhouse\n", 43) = 43
    [pid  3126] write(3, "11a000-13d000 rw-p 00000000 00:00 0", 35) = 35
    [pid  3126] write(3, "                                      \n", 39) = 39
    [pid  3126] write(3, "4000000000-4000001000 ---p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "4000001000-4000801000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              [stack]\n", 38) = 38
    [pid  3126] write(3, "4000801000-400081a000 r-xp 00000000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57 <unfinished ...>
    [pid  3127] <... clock_nanosleep resumed>0x7ffff79ff060) = 0
    [pid  3126] <... write resumed>)        = 57
    [pid  3127] clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=10000000},  <unfinished ...>
    [pid  3126] write(3, "400081a000-400081b000 ---p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400081b000-400081c000 r--p 00019000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57) = 57
    [pid  3126] write(3, "400081c000-400081e000 rw-p 0001a000 fe:01 30316932", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/ld-2.32.so\n", 57) = 57
    [pid  3126] write(3, "400081e000-400081f000 r-xp 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400081f000-4000922000 r-xp 00000000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000922000-4000926000 r--p 00102000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000926000-4000928000 rw-p 00106000 fe:01 30316935", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libc-2.32.so\n", 59) = 59
    [pid  3126] write(3, "4000928000-400092d000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "400092d000-40009af000 r-xp 00000000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009af000-40009b0000 r--p 00081000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009b0000-40009b1000 rw-p 00082000 fe:01 30316943", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libm-2.32.so\n", 59) = 59
    [pid  3126] write(3, "40009b1000-40009c5000 r-xp 00000000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c5000-40009c6000 r--p 00013000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c6000-40009c7000 rw-p 00014000 fe:01 30316946", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libpthread-2.32.so\n", 65) = 65
    [pid  3126] write(3, "40009c7000-40009cb000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31
    [pid  3126] write(3, "40009cb000-40009cd000 r-xp 00000000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009cd000-40009ce000 r--p 00001000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009ce000-40009cf000 rw-p 00002000 fe:01 30316939", 50) = 50
    [pid  3126] write(3, "                       /lib/riscv64-linux-gnu/libdl-2.32.so\n", 60) = 60
    [pid  3126] write(3, "40009cf000-40009d1000 rw-p 00000000 00:00 0", 43) = 43
    [pid  3126] write(3, "                              \n", 31) = 31

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-07-16 08:17:14 +02:00
Yakov Olkhovskiy
07a3988749
geteuid is enough 2023-03-08 08:26:07 -05:00
Yakov Olkhovskiy
aacd05e34b
bugfix + review suggestion 2023-03-06 21:24:04 -05:00
Yakov Olkhovskiy
a8ceab1366
NOLINT for getenv 2023-03-03 09:24:58 -05:00
Yakov Olkhovskiy
df41a83ddb set uid gid to file's original 2023-03-02 14:49:49 +00:00
Yakov Olkhovskiy
566a0e166f preserve uid gid if running with sudo 2023-03-02 00:42:02 +00:00
Azat Khuzhin
6587d11885 Check return value of close() in self-extracting-executable 
And also add description into perror() over callign simply
perror(nullptr).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-07 11:28:22 +01:00
Yakov Olkhovskiy
c8f635e40b WSL1 inconsistency 2023-01-16 18:32:41 +00:00
Azat Khuzhin
15bcd6250a Fix -Wshorten-64-to-32 for darwin builds 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:42 +02:00
Azat Khuzhin
8414ea0691 Fix -Wshorten-64-to-32 in self extracting executable 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-21 13:25:42 +02:00
Yakov Olkhovskiy
1338f7d65e disable inode lock for macos and freebsd 2022-09-05 15:41:52 +00:00
Yakov Olkhovskiy
a9cab86a73 typo 2022-08-30 01:46:17 +00:00
Yakov Olkhovskiy
7a85a943f1 fix close lock on exec, refactoring, comments added 2022-08-30 01:11:04 +00:00
Yakov Olkhovskiy
ae82c68725 truncate lock 2022-08-25 10:52:56 +00:00
Yakov Olkhovskiy
6c1a7205e3 fix style 2022-08-24 21:47:37 +00:00
Yakov Olkhovskiy
f043d03053 fix style 2022-08-24 21:28:52 +00:00
Yakov Olkhovskiy
6afd9c176b inode based singleton 2022-08-24 21:05:00 +00:00
Yakov Olkhovskiy
d11d494587
Merge pull request #40426 from ClickHouse/util-compressor-exec 
Self-extracting: add --exec param, allow compressed to be renamed
 2022-08-23 11:20:28 -04:00
Yakov Olkhovskiy
088a969500 comments added 2022-08-22 12:14:29 +00:00
Yakov Olkhovskiy
47d5a64f13 fix style 2022-08-21 22:06:11 +00:00
Yakov Olkhovskiy
32717a844e allow empty --exec to signify no executable to run, add some stats printout 2022-08-21 21:39:24 +00:00
Yakov Olkhovskiy
5be782a7e6 fix buffer overflow 2022-08-20 21:02:36 +00:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
Yakov Olkhovskiy
d4facec3fa typo 2022-08-20 14:03:03 +00:00
Yakov Olkhovskiy
151afb16e2 add EOL 2022-08-19 20:57:12 +00:00
Yakov Olkhovskiy
d81ec25027 add --exec param, allow compressed to be renamed 2022-08-19 20:43:10 +00:00
Yakov Olkhovskiy
2c3732ed3a extract real path of executable instead of argv[0] 2022-08-08 17:56:16 -04:00
Yakov Olkhovskiy
9dfd650a7c
Merge pull request #39763 from ClickHouse/util-self-extracting-exec 
Self-extracting - run resulting executable with execvp
 2022-08-05 08:13:15 -04:00
Yakov Olkhovskiy
ba86959b72 Merge branch 'util-self-extracting-exec' of https://github.com/ClickHouse/ClickHouse into util-self-extracting-exec 2022-08-03 13:15:06 -04:00
Yakov Olkhovskiy
53530a5fa4 deadcode/clang-tidy fight solution 2022-08-03 13:11:11 -04:00
Yakov Olkhovskiy
5a5d028154
remove global variables 2022-08-03 03:34:51 -04:00
Yakov Olkhovskiy
504180d7d6
stat is different for macos 2022-08-02 15:39:11 -04:00
Yakov Olkhovskiy
e65cef79ad some refactoring 2022-08-01 14:32:32 -04:00
Yakov Olkhovskiy
8820774fe3 do not rename original file - just remove 2022-08-01 13:39:26 -04:00
Yakov Olkhovskiy
2a074288f0 style fix 2022-08-01 11:06:54 -04:00
Yakov Olkhovskiy
c79893f4ef remove call to sh, use random names suffix for temporaries 2022-08-01 10:58:08 -04:00
Yakov Olkhovskiy
3aeb525036 style fix 2022-08-01 01:45:02 -04:00
Yakov Olkhovskiy
7a2d969850 fix clang tidy unused return 2022-08-01 01:20:40 -04:00
Yakov Olkhovskiy
c534bdd42f clang tidy treats call to system as security issue - replace with fork/exec 2022-08-01 01:13:50 -04:00
Yakov Olkhovskiy
3731a415e6 run resulting executable with execvp 2022-07-31 20:07:33 -04:00
Yakov Olkhovskiy
250f19378d free compression and decompression contexts 2022-07-25 15:22:48 -04:00
Yakov Olkhovskiy
a85818f654 some refactoring 2022-07-24 21:13:00 -04:00