thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-24 16:42:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
66,544 Commits 9,168 Branches 1,781 Tags 2.1 GiB
2949cd1e6f
Commit Graph

66544 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alina Terekhova
d09ad74c12 Tiret fix 2021-06-20 14:37:28 +02:00
Alina Terekhova
11d350e16d UInt link fixes + small edits 2021-06-20 14:23:18 +02:00
alexey-milovidov
bf482f75a7
Merge pull request #25427 from azat/replxx-container-overflow 
Fix container-overflow in replxx during incremental search (Ctrl-R)
 2021-06-20 14:10:58 +03:00
Azat Khuzhin
4dcbf3a5c5 Replace __pthread_get_minstack() with a const 2021-06-20 14:03:04 +03:00
Mike Kot
4c391f8e99
SYSTEM RESTORE REPLICA replica [ON CLUSTER cluster] (#13652) 
* initial commit: add setting and stub

* typo

* added test stub

* fix

* wip merging new integration test and code proto

* adding steps interpreters

* adding firstly proposed solution (moving parts etc)

* added checking zookeeper path existence

* fixing the include

* fixing and sorting includes

* fixing outdated struct

* fix the name

* added ast ptr as level of indirection

* fix ref

* updating the changes

* working on test stub

* fix iterator -> reference

* revert rocksdb submodule update

* fixed show privileges test

* updated the test stub

* replaced rand() with thread_local_rng(), updated the tests

updated the test

fixed test config path

test fix

removed error messages

fixed the test

updated the test

fixed string literal

fixed literal

typo: =

* fixed the empty replica error message

* updated the test and the code with logs

* updated the possible test cases, updated

* added the code/test milestone comments

* updated the test (added more testcases)

* replaced native assert with CH one

* individual replicas recursive delete fix

* updated the AS db.name AST

* two small logging fixes

* manually generated AST fixes

* Updated the test, added the possible algo change

* Some thoughts about optimizing the solution:

ALTER MOVE PARTITION .. TO TABLE -> move to detached/ + ALTER ... ATTACH

* fix

* Removed the replica sync in test as it's invalid

* Some test tweaks

* tmp

* Rewrote the algo by using the executeQuery instead of

hand-crafting the ASTPtr.

Two questions still active.

* tr: logging active parts

* Extracted the parts moving algo into a separate helper function

* Fixed the test data and the queries slightly

* Replaced query to system.parts to direct invocation,

started building the test that breaks on various parts.

* Added the case for tables when at least one replica is alive

* Updated the test to test replicas restoration by detaching/attaching

* Altered the test to check restoration without replica restart

* Added the tables swap in the start if the server failed last time

* Hotfix when only /replicas/replica... path was deleted

* Restore ZK paths while creating a replicated MergeTree table

* Updated the docs, fixed the algo for individual replicas restoration case

* Initial parts table storage fix, tests sync fix

* Reverted individual replica restoration to general algo

* Slightly optimised getDataParts

* Trying another solution with parts detaching

* Rewrote algo without any steps, added ON CLUSTER support

* Attaching parts from other replica on restoration

* Getting part checksums from ZK

* Removed ON CLUSTER, finished working solution

* Multiple small changes after review

* Fixing parallel test

* Supporting rewritten form on cluster

* Test fix

* Moar logging

* Using source replica as checksum provider

* improve test, remove some code from parser

* Trying solution with move to detached + forget

* Moving all parts (not only Committed) to detached

* Edited docs for RESTORE REPLICA

* Re-merging

* minor fixes

Co-authored-by: Alexander Tokmakov <avtokmakov@yandex-team.ru>
 2021-06-20 11:24:43 +03:00
alexey-milovidov
3a2adfb602
Merge pull request #25494 from azat/formatting-NOT-fix 
Stable NOT chain formatting
 2021-06-20 03:59:10 +03:00
alexey-milovidov
7189e1034b
Merge pull request #25510 from azat/fuzzer-catch-parser-exception 
Catch "Maximum parse depth" error in fuzzer
 2021-06-20 03:58:56 +03:00
Azat Khuzhin
9a3a1397e6 Fix alternative stack for SIGSEGV handling 
Previous stack size of 4096 wasn't enough, and this lead to that on
SIGSEGV signal handler may overwrite some TLS data (like in the
following example [1]) and in real binary "current_thread" was
overwritten, and this leads to lack of query-id on SIGSEGV, like here
[2].

  [1]: https://gist.github.com/azat/2ee360fd4f2828d363b0926431afacc6
  [2]: https://clickhouse-test-reports.s3.yandex.net/24411/79563953201ce6249a8fd49e22be3902ca4ee43a/fuzzer_ubsan/report.html#fail1

Fix this by using __pthread_get_minstack() that takes TLS block into
account. Yes it is private, but let's try (like rust tried).

This patch will also decrease TLS, since now the per-thread stack will
be allocated in the heap, not on the stack of each thread.

Refs: https://sourceware.org/legacy-ml/libc-alpha/2012-06/msg00335.html
Refs: https://maskray.me/blog/2021-02-14-all-about-thread-local-storage
Refs: https://sourceware.org/bugzilla/show_bug.cgi?id=11787
 2021-06-19 22:47:01 +03:00
Azat Khuzhin
2bc9664b52 Catch "Maximum parse depth" error in fuzzer 
This exception should not fail the fuzzer check like here [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/25494/c6339b4c08ad8f50bf34831367de9a1ea90b1e09/fuzzer_msan/report.html#fail1
 2021-06-19 21:34:37 +03:00
Azat Khuzhin
0286673f8f Fix __pthread_get_minstack() 
This is the function that should take into account TLS block, and 1MB is
too high, since it will be used for sigaltstack() on SIGSEGV

v0: copy-paste glibc __pthread_get_minstack()
v2: return static 16K instead of 1MB
 2021-06-19 21:27:02 +03:00
Azat Khuzhin
57960746e2 Add getStackSize() helper 2021-06-19 20:09:01 +03:00
Azat Khuzhin
efb6433a39 Fix container-overflow in replxx during incremental search (Ctrl-R) 
ASAN report:

    =================================================================
    ==7686==ERROR: AddressSanitizer: container-overflow on address 0x6200000bf080 at pc 0x00002a787e79 bp 0x7fffffffa2f0 sp 0x7fffffffa2e8
    READ of size 4 at 0x6200000bf080 thread T0
        0 0x2a787e78 in replxx::calculate_displayed_length(char32_t const*, int) obj-x86_64-linux-gnu/../contrib/replxx/src/util.cxx:66:15
        1 0x2a75786c in replxx::Replxx::ReplxxImpl::dynamicRefresh(replxx::Prompt&, char32_t*, int, int) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:2201:3
        2 0x2a7453f0 in replxx::Replxx::ReplxxImpl::incremental_history_search(char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:2008:3
        3 0x2a73eecc in replxx::Replxx::ReplxxImpl::action(unsigned long long, replxx::Replxx::ACTION_RESULT (replxx::Replxx::ReplxxImpl::* const&)(char32_t), char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:1246:29
        4 0x2a73eecc in replxx::Replxx::ReplxxImpl::invoke(replxx::Replxx::ACTION, char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:318:70
        5 0x2a74ed29 in std::__1::__function::__policy_func<replxx::Replxx::ACTION_RESULT (char32_t)>::operator()(char32_t&&) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16
        6 0x2a74ed29 in std::__1::function<replxx::Replxx::ACTION_RESULT (char32_t)>::operator()(char32_t) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12
        7 0x2a74ed29 in replxx::Replxx::ReplxxImpl::get_input_line() obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx🔢11
        8 0x2a74dd3c in replxx::Replxx::ReplxxImpl::input(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:580:8
        9 0x2a2a4075 in ReplxxLineReader::readOneLine(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../base/common/ReplxxLineReader.cpp:112:29
        10 0x2a29b499 in LineReader::readLine(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../base/common/LineReader.cpp:81:26
        11 0xb580f02 in DB::Client::mainImpl() obj-x86_64-linux-gnu/../programs/client/Client.cpp:665:33
        12 0xb575825 in DB::Client::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/client/Client.cpp:300:20
        13 0x2a3aff25 in Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8
        14 0xb54c810 in mainEntryClickHouseClient(int, char**) obj-x86_64-linux-gnu/../programs/client/Client.cpp:2702:23
        15 0xb326d8a in main obj-x86_64-linux-gnu/../programs/main.cpp:360:12
        16 0x7ffff7dcbb24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)
        17 0xb2794ad in _start (/src/ch/tmp/upstream/clickhouse-asan+0xb2794ad)

    0x6200000bf080 is located 0 bytes inside of 3672-byte region [0x6200000bf080,0x6200000bfed8)
    allocated by thread T0 here:
        0 0xb3231dd in operator new(unsigned long) (/src/ch/tmp/upstream/clickhouse-asan+0xb3231dd)
        1 0x2a75fb15 in void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/new:235:10
        2 0x2a75fb15 in std::__1::__libcpp_allocate(unsigned long, unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/new:261:10
        3 0x2a75fb15 in std::__1::allocator<char32_t>::allocate(unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:840:38
        4 0x2a75fb15 in std::__1::allocator_traits<std::__1::allocator<char32_t> >::allocate(std::__1::allocator<char32_t>&, unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/__memory/allocator_traits.h:468:21
        5 0x2a75fb15 in std::__1::vector<char32_t, std::__1::allocator<char32_t> >::__vallocate(unsigned long) obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:993:37
        6 0x2a75fb15 in std::__1::enable_if<(__is_cpp17_forward_iterator<char32_t*>::value) && (is_constructible<char32_t, std::__1::iterator_traits<char32_t*>::reference>::value), void>::type std::__1::vector<char32_t, std::__1::allocator<char32_t> >::assign<char32_t*>(char32_t*, char32_t*) obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:1460:9
        7 0x2a745242 in std::__1::vector<char32_t, std::__1::allocator<char32_t> >::operator=(std::__1::vector<char32_t, std::__1::allocator<char32_t> > const&) obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:1405:9
        8 0x2a745242 in replxx::UnicodeString::assign(replxx::UnicodeString const&) obj-x86_64-linux-gnu/../contrib/replxx/src/unicodestring.hxx:83:9
        9 0x2a745242 in replxx::Replxx::ReplxxImpl::incremental_history_search(char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:1993:24
        10 0x2a73eecc in replxx::Replxx::ReplxxImpl::action(unsigned long long, replxx::Replxx::ACTION_RESULT (replxx::Replxx::ReplxxImpl::* const&)(char32_t), char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:1246:29
        11 0x2a73eecc in replxx::Replxx::ReplxxImpl::invoke(replxx::Replxx::ACTION, char32_t) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:318:70
        12 0x2a74ed29 in std::__1::__function::__policy_func<replxx::Replxx::ACTION_RESULT (char32_t)>::operator()(char32_t&&) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16
        13 0x2a74ed29 in std::__1::function<replxx::Replxx::ACTION_RESULT (char32_t)>::operator()(char32_t) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12
        14 0x2a74ed29 in replxx::Replxx::ReplxxImpl::get_input_line() obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx🔢11
        15 0x2a74dd3c in replxx::Replxx::ReplxxImpl::input(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../contrib/replxx/src/replxx_impl.cxx:580:8
        16 0x2a2a4075 in ReplxxLineReader::readOneLine(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../base/common/ReplxxLineReader.cpp:112:29
        17 0x2a29b499 in LineReader::readLine(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../base/common/LineReader.cpp:81:26
        18 0xb580f02 in DB::Client::mainImpl() obj-x86_64-linux-gnu/../programs/client/Client.cpp:665:33
        19 0xb575825 in DB::Client::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/client/Client.cpp:300:20
        20 0x2a3aff25 in Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8
        21 0xb54c810 in mainEntryClickHouseClient(int, char**) obj-x86_64-linux-gnu/../programs/client/Client.cpp:2702:23
        22 0xb326d8a in main obj-x86_64-linux-gnu/../programs/main.cpp:360:12
        23 0x7ffff7dcbb24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)

    HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
    If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
    SUMMARY: AddressSanitizer: container-overflow obj-x86_64-linux-gnu/../contrib/replxx/src/util.cxx:66:15 in replxx::calculate_displayed_length(char32_t const*, int)
    Shadow bytes around the buggy address:
      0x0c408000fdc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      0x0c408000fdd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c408000fde0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c408000fdf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c408000fe00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
    =>0x0c408000fe10:[fc]fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe20: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe30: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe40: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe50: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
      0x0c408000fe60: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
      Shadow gap:              cc
    ==7686==ABORTING

Refs: https://github.com/ClickHouse-Extras/replxx/pull/16

v2: fix test, do not use /dev/null since it client will lock it
 2021-06-19 18:41:33 +03:00
alexey-milovidov
f7571a2d3b
Update CHANGELOG.md 2021-06-19 13:32:18 +03:00
Guo Wei (William)
7767ce5442
Update README.md (#25507) 2021-06-19 11:50:57 +03:00
Azat Khuzhin
c6339b4c08 Stable NOT chain formatting 2021-06-19 08:03:36 +03:00
robot-clickhouse
4b1b612cfd Update version_date.tsv after release 21.6.5.37 2021-06-19 02:58:45 +03:00
Vitaliy Zakaznikov
8615e31c88 Fixing extended precision data types SRS name. 2021-06-18 18:35:35 -04:00
alexey-milovidov
d1348f2275
Merge pull request #25419 from nikitamikhaylov/copier-sharding-key 
Fix segfault when sharding_key is absent in config
 2021-06-19 01:23:44 +03:00
alexey-milovidov
55c06feeab
Merge pull request #25474 from qoega/update-flatbuffers 
Update flatbuffers to v.2.0.0
 2021-06-19 01:22:42 +03:00
Vitaliy Zakaznikov
6567b819bd Enabling TestFlows RBAC tests. 2021-06-18 17:29:33 -04:00
Alexey Milovidov
29b80af6cc Skip RocksDB tests for Ordinary database 2021-06-19 00:09:50 +03:00
alesapin
49cd16e74f Better 2021-06-18 22:08:12 +03:00
Vladimir
1c91a8d2c2
Merge pull request #25454 from vdimir/issue-23351-v2 2021-06-18 21:56:46 +03:00
Dmitriy
ccad32fe96 Translate to Russian 
Выполнил перевод на русский язык.
 2021-06-18 21:20:53 +03:00
Alina Terekhova
fdc729764b Edits 2021-06-18 19:53:52 +02:00
Kseniia Sumarokova
4fe722d1a4
Merge pull request #25431 from vitlibar/no-underscore_before-preprocessed_configs 
Fix underscore before preprocessed configs.
 2021-06-18 17:53:03 +03:00
Kseniia Sumarokova
5974f96f0c
Merge pull request #23656 from kssenii/progress-bar 
Progress indication for File table engine in clickhouse-local and insert from file in clickhouse-client
 2021-06-18 17:46:54 +03:00
Vladimir
27f0fdb539
Merge pull request #25292 from Tiaonmmn/patch-1 
Update formats.md
 2021-06-18 17:46:02 +03:00
mergify[bot]
c723dd7d40
Merge branch 'master' into arrow 2021-06-18 12:18:57 +00:00
alexey-milovidov
e4d80f53fa
Merge pull request #25311 from ClickHouse/fix-bad-test-client_print_query_id 
Fix flaky test 01520_client_print_query_id and others.
 2021-06-18 14:41:29 +03:00
vdimir
0ebca58e86
Check column in removeColumnNullability 2021-06-18 14:06:55 +03:00
Vladimir
4de0d1a593
Merge pull request #24885 from Jokser/disk-s3-read-error-fix 2021-06-18 13:38:59 +03:00
Yatsishin Ilya
b47ce8de89 update 2021-06-18 13:34:15 +03:00
Maksim Kita
822f77be74
Merge pull request #25458 from kitaisreal/low-cardinality-interval-type 
DataTypeLowCardinality support DataTypeInterval tests
 2021-06-18 13:16:42 +03:00
alesapin
69f3b0028a Merge branch 'add_run_id_option' into update_buffer_size_in_nuraft 2021-06-18 12:41:02 +03:00
alesapin
f0490ad7c1 Better formatting 2021-06-18 12:24:04 +03:00
alesapin
8a5bc2e7bc Add run-id option to integration tests 2021-06-18 12:23:22 +03:00
Maksim Kita
180f0aa1e0 Added tests for DataTypeLowCardinality support DataTypeInterval 2021-06-18 11:49:40 +03:00
Vladimir
6af272891a
Merge pull request #24870 from azat/mv-dist-join-fix 2021-06-18 11:18:43 +03:00
vdimir
b25f6ba5b6
Remove assumeMutable from removeColumnNullability 2021-06-18 11:13:53 +03:00
Vladimir
fab7c9c7f6
Merge pull request #24061 from vdimir/issue-23351 2021-06-18 11:07:28 +03:00
Vladimir
51e5a019ed
Merge pull request #25410 from vdimir/issue-21730 2021-06-18 11:00:26 +03:00
kssenii
f7029b168b Generate ya make 2021-06-18 10:56:24 +03:00
Maksim Kita
1b53719d0f
Merge pull request #25446 from ClickHouse/add-test-8417 
Add a test for #8417
 2021-06-18 10:20:09 +03:00
Maksim Kita
2792f302fd
Merge pull request #25294 from gyuton/gyuton-DOCSUP-10300-Edit-and-translate-to-Russian 
DOCSUP-10300: Edited and translated to Russian
 2021-06-18 10:11:10 +03:00
Maksim Kita
6c0dfae880
Update external-dicts-dict-sources.md 2021-06-18 10:10:52 +03:00
kssenii
bdc8fe1ca0 Merge branch 'master' of github.com:ClickHouse/ClickHouse into progress-bar 2021-06-18 09:33:29 +03:00
Nikolai Kochetov
b2620e1def
Merge pull request #24405 from amosbird/replacemorealias 
Rewrite more alias columns
 2021-06-18 09:10:31 +03:00
Vitaly Baranov
f766d74fb6 Fix testflows LDAP tests. 2021-06-18 06:18:40 +03:00
Vitaly Baranov
fc3d77bd66 Fix underscore before preprocessed configs. 2021-06-18 06:14:11 +03:00