Commit Graph

106124 Commits

Author SHA1 Message Date
kssenii
8d20af8127 Fix 2023-01-20 13:34:23 +01:00
Robert Schulze
7e6d3163b1
Initial inverted index docs 2023-01-20 12:12:20 +00:00
Azat Khuzhin
bdeb5514c5 Fix ASan builds for glibc 2.36+ (use RTLD_NEXT for ThreadFuzzer interceptors)
Recently I noticed that clickhouse compiled with ASan does not work with
newer glibc 2.36+, before I though that this was only about compiling
with old but using new, however that was not correct, ASan simply does
not work with glibc 2.36+.

Here is a simple reproducer [1]:

    $ cat > test-asan.cpp <<EOL
    #include <pthread.h>
    int main()
    {
        // something broken in ASan in interceptor for __pthread_mutex_lock
        // and only since glibc 2.36, and for pthread_mutex_lock everything is OK
        pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
        return __pthread_mutex_lock(&mutex);
    }
    EOL
    $ clang -g3 -o test-asan test-asan.cpp -fsanitize=address
    $ ./test-asan
    AddressSanitizer:DEADLYSIGNAL
    =================================================================
    ==15659==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0x7fffffffccb0 sp 0x7fffffffcb98 T0)
    ==15659==Hint: pc points to the zero page.
    ==15659==The signal is caused by a READ memory access.
    ==15659==Hint: address points to the zero page.
        #0 0x0  (<unknown module>)
        #1 0x7ffff7cda28f  (/usr/lib/libc.so.6+0x2328f) (BuildId: 1e94beb079e278ac4f2c8bce1f53091548ea1584)

    AddressSanitizer can not provide additional info.
    SUMMARY: AddressSanitizer: SEGV (<unknown module>)
    ==15659==ABORTING

  [1]: https://gist.github.com/azat/af073e57a248e04488b21068643f079e

I've started observing glibc code, there was some changes in glibc, that
moves pthread functions out from libpthread.so.0 into libc.so.6
(somewhere between 2.31 and 2.35), but
the problem pops up only with 2.36, 2.35 works fine.

After this I've looked into changes between 2.35 and 2.36, and found
this patch [2] - "dlsym: Make RTLD_NEXT prefer default version
definition [BZ #14932]", that fixes this bug [3].

  [2]: https://sourceware.org/git/?p=glibc.git;a=commit;h=efa7936e4c91b1c260d03614bb26858fbb8a0204
  [3]: https://sourceware.org/bugzilla/show_bug.cgi?id=14932

The problem with using DL_LOOKUP_RETURN_NEWEST flag for RTLD_NEXT is
that it does not resolve hidden symbols (and __pthread_mutex_lock is
indeed hidden).

Here is a sample that will show the difference [4]:

    $ cat > test-dlsym.c <<EOL
    #define _GNU_SOURCE
    #include <dlfcn.h>
    #include <stdio.h>

    int main()
    {
        void *p = dlsym(RTLD_NEXT, "__pthread_mutex_lock");
        printf("__pthread_mutex_lock: %p (via RTLD_NEXT)\n", p);
        return 0;
    }
    EOL

    # glibc 2.35: __pthread_mutex_lock: 0x7ffff7e27f70 (via RTLD_NEXT)
    # glibc 2.36: __pthread_mutex_lock: (nil) (via RTLD_NEXT)

  [4]: https://gist.github.com/azat/3b5f2ae6011bef2ae86392cea7789eb7

But ThreadFuzzer uses internal symbols to wrap
pthread_mutex_lock/pthread_mutex_unlock, which are intercepted by ASan
and this leads to NULL dereference.

The fix was obvious - just use dlsym(RTLD_NEXT), however on older
glibc's this leads to endless recursion (see commits in the code). But
only for jemalloc [5], and even though sanitizers does not uses jemalloc
the code of ThreadFuzzer is generic and I don't want to guard it with
more preprocessors macros.

  [5]: https://gist.github.com/azat/588d9c72c1e70fc13ebe113197883aa2

So we have to use RTLD_NEXT only for ASan.

There is also one more interesting issue, if you will compile with clang
that itself had been compiled with newer libc (i.e. 2.36), you will get
the following error:

    $ podman run --privileged -v $PWD/.cmake-asan/programs:/root/bin -e PATH=/bin:/root/bin -e --rm -it ubuntu-dev-v3 clickhouse
    ==1==ERROR: AddressSanitizer failed to allocate 0x0 (0) bytes of SetAlternateSignalStack (error code: 22)
    ...
    ==1==End of process memory map.
    AddressSanitizer: CHECK failed: sanitizer_common.cpp:53 "((0 && "unable to mmap")) != (0)" (0x0, 0x0) (tid=1)
        <empty stack>

The problem is that since GLIBC_2.31, `SIGSTKSZ` is a call to
`getconf(_SC_MINSIGSTKSZ)`, but older glibc does not have it, so `-1`
will be returned and used as `SIGSTKSZ` instead.

The workaround to disable alternative stack:

    $ podman run --privileged -v $PWD/.cmake-asan/programs:/root/bin -e PATH=/bin:/root/bin -e ASAN_OPTIONS=use_sigaltstack=0 --rm -it ubuntu-dev-v3 clickhouse client --version
    ClickHouse client version 22.13.1.1.

Fixes: #43426
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-20 13:09:13 +01:00
Nikolai Kochetov
1e29993aef Fixing build 2023-01-20 11:55:20 +00:00
Maksim Kita
3e08a98f16
Merge pull request #45388 from azat/dict/remove-preallocate
Remove PREALLOCATE for HASHED/SPARSE_HASHED dictionaries
2023-01-20 14:51:25 +03:00
Anton Popov
9c0ba7c7ca
Merge pull request #45432 from CurtizJ/allow-json-extract-int-from-float
Allow to convert float stored in string field to integer in `JSONExtract`
2023-01-20 12:35:06 +01:00
Kseniia Sumarokova
c066b9bddd
Update SwapHelper.h 2023-01-20 12:19:19 +01:00
Maksim Kita
e067a55b78 Fixed tests 2023-01-20 12:19:16 +01:00
Sema Checherinda
09f3a5c599 add a comment, add a check, fix test 2023-01-20 12:10:31 +01:00
Maksim Kita
23e26032ca
Merge pull request #45399 from aalexfvk/alexfvk/mdb-21326_fix_system_dictionaries_when_dictionary_with_bad_structure
Fix select from system.dictionaries when there is dictionary with bad structure
2023-01-20 13:36:32 +03:00
Maksim Kita
758c8f2776
Merge branch 'master' into dict/remove-preallocate 2023-01-20 13:15:37 +03:00
Maksim Kita
e6ee5554d1 Fixed tests 2023-01-20 11:15:13 +01:00
Azat Khuzhin
1f9a65b875 Modernize InternalTextLogsQueue::getPriorityName()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-20 11:09:35 +01:00
Azat Khuzhin
fc276abadd Fix log level "Test" for send_logs_level in client
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-01-20 11:09:35 +01:00
Antonio Andelic
0ad37ad286
Merge pull request #45320 from stigsb/system_tables_volume_config
Add <storage_policy> config parameter for system logs
2023-01-20 10:27:57 +01:00
Robert Schulze
5ec6d89d43
Merge pull request #38667 from ClibMouse/ftsearch
Inverted Indices Implementation
2023-01-20 10:18:05 +01:00
SmitaRKulkarni
6aa63414db
Merge pull request #45072 from ClickHouse/43891_Disallow_concurrent_backups_and_restores
Added settings to disallow concurrent backups and restores
2023-01-20 09:17:20 +01:00
Aleksei Filatov
42549e89f2 [rev 2] Fix review notes 2023-01-20 09:37:49 +03:00
Nikolai Kochetov
3e00d18498 Merge branch 'master' into fix-disabled-two-level-agg 2023-01-19 20:54:04 +00:00
Nikolay Degterinsky
dd7fef11a2 Add default granularity 2023-01-19 20:52:38 +00:00
Nikolai Kochetov
d24be2712e Fix disabled two-level aggregation from HTTP 2023-01-19 20:50:27 +00:00
Robert Schulze
c7d12564cb
Merge pull request #45444 from ClickHouse/index-docs
Docs: Beautify section on secondary index types
2023-01-19 19:36:45 +01:00
Robert Schulze
099f50c0e3
Remove redundancy 2023-01-19 18:35:19 +00:00
Maksim Kita
3363f7c718 Added GroupingFunctionsResolvePass 2023-01-19 19:06:02 +01:00
Maksim Kita
506f91b841 Fixed tests 2023-01-19 19:05:49 +01:00
Maksim Kita
2c56b0b2b9 Planner small fixes 2023-01-19 19:05:49 +01:00
Kseniia Sumarokova
ad4a9d2880
Update SwapHelper.h 2023-01-19 18:58:09 +01:00
kssenii
f56f515392 Fix 2023-01-19 18:45:06 +01:00
Robert Schulze
d8144ffafc
Merge branch 'master' into index-docs 2023-01-19 18:05:27 +01:00
Robert Schulze
c9988f4411
Mini fix 2023-01-19 17:04:17 +00:00
Anton Popov
089d1f5b62 fix fuzzer 2023-01-19 17:03:24 +00:00
Robert Schulze
7355e3bc77
Docs: Beautify section on secondary index types 2023-01-19 17:00:31 +00:00
kssenii
4ce8950712 Minor changes 2023-01-19 17:53:10 +01:00
robot-clickhouse-ci-1
13d4d40568
Merge pull request #45325 from ClickHouse/download-builds
Improve manual to get binaries from CI
2023-01-19 17:49:52 +01:00
larryluogit
52ae33dba7
Merge branch 'master' into ftsearch 2023-01-19 11:34:11 -05:00
Robert Schulze
bfcb5e7915
Fix typos 2023-01-19 16:08:21 +00:00
avogar
c34c0aa22e Fix comments 2023-01-19 16:03:46 +00:00
Yakov Olkhovskiy
8313bc8384 fix test with correct error codes 2023-01-19 15:55:30 +00:00
Han Fei
3007507a8b
Merge pull request #45428 from hanfei1991/hanfei/fix-empty-expressions
fix regexp logical error in stress tests
2023-01-19 16:39:39 +01:00
Kruglov Pavel
9820beae68
Apply suggestions from code review
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
2023-01-19 16:11:13 +01:00
Anton Popov
4ca359d57b
Merge pull request #45418 from CurtizJ/fix-disk-encrypted
Fix reading from encrypted disk with passed file size
2023-01-19 16:11:08 +01:00
Anton Popov
7f2e37860d allow to convert float stored in string field to integer in JSONExtract 2023-01-19 14:24:55 +00:00
Aleksei Filatov
afada0ecb3 Fix review notes 2023-01-19 17:02:57 +03:00
Alexander Tokmakov
7bb65cc002
Update StorageReplicatedMergeTree.cpp 2023-01-19 16:45:41 +03:00
Igor Nikonov
d0ce804bfc Fix: dynamic_cast -> typeid_cast for SortingStep 2023-01-19 13:40:21 +00:00
Dan Roscigno
245899c0f6
Merge pull request #45411 from DanRoscigno/docs-support_batch_delete
add support_batch_delete to mergetree docs
2023-01-19 08:03:33 -05:00
Han Fei
94336a9b66 fix typo 2023-01-19 13:55:29 +01:00
Dan Roscigno
b77346ea08
Merge branch 'master' into docs-support_batch_delete 2023-01-19 07:49:12 -05:00
Nikita Taranov
fdea042991
Add delete by ttl for zookeeper_log (#45419) 2023-01-19 13:42:16 +01:00
Vladimir C
5818dec4cc
Update 02534_join_prewhere_bug_44062.sql 2023-01-19 13:33:23 +01:00