thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 20:02:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
66,426 Commits 9,330 Branches 1,783 Tags 2.1 GiB
660e824851
Commit Graph

66426 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolae Vartolomei
660e824851 Missed one server_died.set() 2021-06-21 15:13:23 +01:00
Nicolae Vartolomei
115edd3e42 Fix hang and incorrect exit code returned from clickhouse-test 
Variables aren't shared when using multiprocessing, use shared memory
instead
https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects.

There appears to be a deadlock when multiple threads try to send
sigterm signal at the same time. Avoid it by making sure sigterm is sent
only once for the process group.
 2021-06-21 12:37:36 +01:00
alexey-milovidov
8ba6a5393f
Merge pull request #25509 from azat/fix-altstack 
Fix alternative stack for SIGSEGV handling
 2021-06-21 12:10:00 +03:00
alexey-milovidov
d2c86e52e6
Update checkStackSize.cpp 2021-06-21 12:09:14 +03:00
alexey-milovidov
f37faf02fa
Update checkStackSize.cpp 2021-06-21 12:08:30 +03:00
alexey-milovidov
9a4125dd7c
Update ThreadStatus.cpp 2021-06-21 12:06:41 +03:00
alesapin
7c7e96f26c
Merge pull request #25526 from ClickHouse/revert-25459-add_run_id_option 
Revert "Add run-id option to integration tests"
 2021-06-21 11:10:53 +03:00
alesapin
feecf398fb
Revert "Add run-id option to integration tests" 2021-06-21 11:10:44 +03:00
Kruglov Pavel
a54cbef5b7
Merge pull request #24341 from Avogar/arrow 
Support complex types in Arrow/Parquet/ORC
 2021-06-21 11:05:57 +03:00
alesapin
65ce3929b8
Merge pull request #25459 from ClickHouse/add_run_id_option 
Add run-id option to integration tests
 2021-06-21 10:21:24 +03:00
alexey-milovidov
7cd86086c2
Merge pull request #25521 from azat/fuzzer-catch-TOO_DEEP_RECURSION 
Catch TOO_DEEP_RECURSION in fuzzer for formatted query too
 2021-06-21 02:03:55 +03:00
Azat Khuzhin
fa29fe6ba9 Catch TOO_DEEP_RECURSION for formatted query too 2021-06-21 00:39:59 +03:00
Anton Popov
d8b6f15ef4
Merge pull request #23027 from azat/distributed-push-down-limit 
Add ability to push down LIMIT for distributed queries
 2021-06-20 23:08:50 +03:00
alexey-milovidov
77b7cec8c0
Merge pull request #25498 from vzakaznikov/enable_rbac_testflows_tests 
Enabling TestFlows RBAC tests.
 2021-06-20 21:40:21 +03:00
Azat Khuzhin
12eadc2b5e Add a guard page for alternative signal handling stack 2021-06-20 20:08:50 +03:00
Vitaliy Zakaznikov
8a56419a9a Fixing alter modify tests to explicitly set the default value to '0' so that 
we don't have empty strings that are not guaranteed to
convert to 0.
 2021-06-20 10:06:12 -04:00
alexey-milovidov
bf482f75a7
Merge pull request #25427 from azat/replxx-container-overflow 
Fix container-overflow in replxx during incremental search (Ctrl-R)
 2021-06-20 14:10:58 +03:00
Azat Khuzhin
4dcbf3a5c5 Replace __pthread_get_minstack() with a const 2021-06-20 14:03:04 +03:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652) 
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
 2021-06-20 11:24:43 +03:00
alexey-milovidov
3a2adfb602
Merge pull request #25494 from azat/formatting-NOT-fix 
Stable NOT chain formatting
 2021-06-20 03:59:10 +03:00
alexey-milovidov
7189e1034b
Merge pull request #25510 from azat/fuzzer-catch-parser-exception 
Catch "Maximum parse depth" error in fuzzer
 2021-06-20 03:58:56 +03:00
Azat Khuzhin
9a3a1397e6 Fix alternative stack for SIGSEGV handling 
Previous stack size of 4096 wasn't enough, and this lead to that on
SIGSEGV signal handler may overwrite some TLS data (like in the
following example [1]) and in real binary "current_thread" was
overwritten, and this leads to lack of query-id on SIGSEGV, like here
[2].

  [1]: https://gist.github.com/azat/2ee360fd4f2828d363b0926431afacc6
  [2]: https://clickhouse-test-reports.s3.yandex.net/24411/79563953201ce6249a8fd49e22be3902ca4ee43a/fuzzer_ubsan/report.html#fail1

Fix this by using __pthread_get_minstack() that takes TLS block into
account. Yes it is private, but let's try (like rust tried).

This patch will also decrease TLS, since now the per-thread stack will
be allocated in the heap, not on the stack of each thread.

Refs: https://sourceware.org/legacy-ml/libc-alpha/2012-06/msg00335.html
Refs: https://maskray.me/blog/2021-02-14-all-about-thread-local-storage
Refs: https://sourceware.org/bugzilla/show_bug.cgi?id=11787
 2021-06-19 22:47:01 +03:00
Azat Khuzhin
2bc9664b52 Catch "Maximum parse depth" error in fuzzer 
This exception should not fail the fuzzer check like here [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/25494/c6339b4c08ad8f50bf34831367de9a1ea90b1e09/fuzzer_msan/report.html#fail1
 2021-06-19 21:34:37 +03:00
Azat Khuzhin
0286673f8f Fix __pthread_get_minstack() 
This is the function that should take into account TLS block, and 1MB is
too high, since it will be used for sigaltstack() on SIGSEGV

v0: copy-paste glibc __pthread_get_minstack()
v2: return static 16K instead of 1MB
 2021-06-19 21:27:02 +03:00
Azat Khuzhin
57960746e2 Add getStackSize() helper 2021-06-19 20:09:01 +03:00
Azat Khuzhin
efb6433a39 Fix container-overflow in replxx during incremental search (Ctrl-R) 
ASAN report:

    =================================================================
    ==7686==ERROR: AddressSanitizer: container-overflow on address 0x6200000bf080 at pc 0x00002a787e79 bp 0x7fffffffa2f0 sp 0x7fffffffa2e8
    READ of size 4 at 0x6200000bf080 thread T0
        0 0x2a787e78 in replxx::calculate_displayed_length(char32_t const*, int) obj-x86_64-linux-gnu/../contrib/replxx/src/util.cxx:66:15
        1 0x2a75786c in replxx::Replxx::ReplxxImpl::dynamicRefresh(replxx::Prompt&, char32_t*, int, int) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:2201:3
        2 0x2a7453f0 in replxx::Replxx::ReplxxImpl::incremental_history_search(char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:2008:3
        3 0x2a73eecc in replxx::Replxx::ReplxxImpl::action(unsigned long long, replxx::Replxx::ACTION_RESULT (replxx::Replxx::ReplxxImpl::* const&)(char32_t), char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:1246:29
        4 0x2a73eecc in replxx::Replxx::ReplxxImpl::invoke(replxx::Replxx::ACTION, char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:318:70
        5 0x2a74ed29 in std::__1::__function::__policy_func<replxx::Replxx::ACTION_RESULT (char32_t)>::operator()(char32_t&&) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16
        6 0x2a74ed29 in std::__1::function<replxx::Replxx::ACTION_RESULT (char32_t)>::operator()(char32_t) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12
        7 0x2a74ed29 in replxx::Replxx::ReplxxImpl::get_input_line() obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx🔢11
        8 0x2a74dd3c in replxx::Replxx::ReplxxImpl::input(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:580:8
        9 0x2a2a4075 in ReplxxLineReader::readOneLine(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../base/common/ReplxxLineReader.cpp:112:29
        10 0x2a29b499 in LineReader::readLine(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../base/common/LineReader.cpp:81:26
        11 0xb580f02 in DB::Client::mainImpl() obj-x86_64-linux-gnu/../programs/client/Client.cpp:665:33
        12 0xb575825 in DB::Client::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/client/Client.cpp:300:20
        13 0x2a3aff25 in Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8
        14 0xb54c810 in mainEntryClickHouseClient(int, char**) obj-x86_64-linux-gnu/../programs/client/Client.cpp:2702:23
        15 0xb326d8a in main obj-x86_64-linux-gnu/../programs/main.cpp:360:12
        16 0x7ffff7dcbb24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)
        17 0xb2794ad in _start (/src/ch/tmp/upstream/clickhouse-asan+0xb2794ad)

    0x6200000bf080 is located 0 bytes inside of 3672-byte region [0x6200000bf080,0x6200000bfed8)
    allocated by thread T0 here:
        0 0xb3231dd in operator new(unsigned long) (/src/ch/tmp/upstream/clickhouse-asan+0xb3231dd)
        1 0x2a75fb15 in void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/new:235:10
        2 0x2a75fb15 in std::__1::__libcpp_allocate(unsigned long, unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/new:261:10
        3 0x2a75fb15 in std::__1::allocator<char32_t>::allocate(unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:840:38
        4 0x2a75fb15 in std::__1::allocator_traits<std::__1::allocator<char32_t> >::allocate(std::__1::allocator<char32_t>&, unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:468:21
        5 0x2a75fb15 in std::__1::vector<char32_t, std::__1::allocator<char32_t> >::__vallocate(unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:993:37
        6 0x2a75fb15 in std::__1::enable_if<(__is_cpp17_forward_iterator<char32_t*>::value) && (is_constructible<char32_t, std::__1::iterator_traits<char32_t*>::reference>::value), void>::type std::__1::vector<char32_t, std::__1::allocator<char32_t> >::assign<char32_t*>(char32_t*, char32_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:1460:9
        7 0x2a745242 in std::__1::vector<char32_t, std::__1::allocator<char32_t> >::operator=(std::__1::vector<char32_t, std::__1::allocator<char32_t> > const&) obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:1405:9
        8 0x2a745242 in replxx::UnicodeString::assign(replxx::UnicodeString const&) obj-x86_64-linux-gnu/../contrib/replxx/src/unicodestring.hxx:83:9
        9 0x2a745242 in replxx::Replxx::ReplxxImpl::incremental_history_search(char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:1993:24
        10 0x2a73eecc in replxx::Replxx::ReplxxImpl::action(unsigned long long, replxx::Replxx::ACTION_RESULT (replxx::Replxx::ReplxxImpl::* const&)(char32_t), char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:1246:29
        11 0x2a73eecc in replxx::Replxx::ReplxxImpl::invoke(replxx::Replxx::ACTION, char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:318:70
        12 0x2a74ed29 in std::__1::__function::__policy_func<replxx::Replxx::ACTION_RESULT (char32_t)>::operator()(char32_t&&) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16
        13 0x2a74ed29 in std::__1::function<replxx::Replxx::ACTION_RESULT (char32_t)>::operator()(char32_t) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12
        14 0x2a74ed29 in replxx::Replxx::ReplxxImpl::get_input_line() obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx🔢11
        15 0x2a74dd3c in replxx::Replxx::ReplxxImpl::input(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:580:8
        16 0x2a2a4075 in ReplxxLineReader::readOneLine(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../base/common/ReplxxLineReader.cpp:112:29
        17 0x2a29b499 in LineReader::readLine(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../base/common/LineReader.cpp:81:26
        18 0xb580f02 in DB::Client::mainImpl() obj-x86_64-linux-gnu/../programs/client/Client.cpp:665:33
        19 0xb575825 in DB::Client::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/client/Client.cpp:300:20
        20 0x2a3aff25 in Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8
        21 0xb54c810 in mainEntryClickHouseClient(int, char**) obj-x86_64-linux-gnu/../programs/client/Client.cpp:2702:23
        22 0xb326d8a in main obj-x86_64-linux-gnu/../programs/main.cpp:360:12
        23 0x7ffff7dcbb24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)

    HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
    If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
    SUMMARY: AddressSanitizer: container-overflow obj-x86_64-linux-gnu/../contrib/replxx/src/util.cxx:66:15 in replxx::calculate_displayed_length(char32_t const*, int)
    Shadow bytes around the buggy address:
      0x0c408000fdc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      0x0c408000fdd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c408000fde0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c408000fdf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c408000fe00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
    =>0x0c408000fe10:[fc]fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe20: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe30: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe40: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe50: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe60: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
      Shadow gap:              cc
    ==7686==ABORTING

Refs: https://github.com/ClickHouse-Extras/replxx/pull/16

v2: fix test, do not use /dev/null since it client will lock it
 2021-06-19 18:41:33 +03:00
alexey-milovidov
f7571a2d3b
Update CHANGELOG.md 2021-06-19 13:32:18 +03:00
Guo Wei (William)
7767ce5442
Update README.md (#25507) 2021-06-19 11:50:57 +03:00
Azat Khuzhin
c6339b4c08 Stable NOT chain formatting 2021-06-19 08:03:36 +03:00
robot-clickhouse
4b1b612cfd Update version_date.tsv after release 21.6.5.37 2021-06-19 02:58:45 +03:00
Vitaliy Zakaznikov
8615e31c88 Fixing extended precision data types SRS name. 2021-06-18 18:35:35 -04:00
alexey-milovidov
d1348f2275
Merge pull request #25419 from nikitamikhaylov/copier-sharding-key 
Fix segfault when sharding_key is absent in config
 2021-06-19 01:23:44 +03:00
alexey-milovidov
55c06feeab
Merge pull request #25474 from qoega/update-flatbuffers 
Update flatbuffers to v.2.0.0
 2021-06-19 01:22:42 +03:00
Vitaliy Zakaznikov
6567b819bd Enabling TestFlows RBAC tests. 2021-06-18 17:29:33 -04:00
Alexey Milovidov
29b80af6cc Skip RocksDB tests for Ordinary database 2021-06-19 00:09:50 +03:00
alesapin
49cd16e74f Better 2021-06-18 22:08:12 +03:00
Vladimir
1c91a8d2c2
Merge pull request #25454 from vdimir/issue-23351-v2 2021-06-18 21:56:46 +03:00
Kseniia Sumarokova
4fe722d1a4
Merge pull request #25431 from vitlibar/no-underscore_before-preprocessed_configs 
Fix underscore before preprocessed configs.
 2021-06-18 17:53:03 +03:00
Kseniia Sumarokova
5974f96f0c
Merge pull request #23656 from kssenii/progress-bar 
Progress indication for File table engine in clickhouse-local and insert from file in clickhouse-client
 2021-06-18 17:46:54 +03:00
Vladimir
27f0fdb539
Merge pull request #25292 from Tiaonmmn/patch-1 
Update formats.md
 2021-06-18 17:46:02 +03:00
mergify[bot]
c723dd7d40
Merge branch 'master' into arrow 2021-06-18 12:18:57 +00:00
alexey-milovidov
e4d80f53fa
Merge pull request #25311 from ClickHouse/fix-bad-test-client_print_query_id 
Fix flaky test 01520_client_print_query_id and others.
 2021-06-18 14:41:29 +03:00
vdimir
0ebca58e86
Check column in removeColumnNullability 2021-06-18 14:06:55 +03:00
Vladimir
4de0d1a593
Merge pull request #24885 from Jokser/disk-s3-read-error-fix 2021-06-18 13:38:59 +03:00
Yatsishin Ilya
b47ce8de89 update 2021-06-18 13:34:15 +03:00
Maksim Kita
822f77be74
Merge pull request #25458 from kitaisreal/low-cardinality-interval-type 
DataTypeLowCardinality support DataTypeInterval tests
 2021-06-18 13:16:42 +03:00
alesapin
f0490ad7c1 Better formatting 2021-06-18 12:24:04 +03:00
alesapin
8a5bc2e7bc Add run-id option to integration tests 2021-06-18 12:23:22 +03:00
Maksim Kita
180f0aa1e0 Added tests for DataTypeLowCardinality support DataTypeInterval 2021-06-18 11:49:40 +03:00
Vladimir
6af272891a
Merge pull request #24870 from azat/mv-dist-join-fix 2021-06-18 11:18:43 +03:00