thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
74,861 Commits 9,328 Branches 1,783 Tags 2.1 GiB
27ca943863
Commit Graph

74861 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexander Tokmakov
27ca943863 add fault injection in ZooKeeper client 2021-10-22 18:59:38 +03:00
Alexander Tokmakov
6e296d0342 refactor ctor arguments 2021-10-21 13:44:52 +03:00
Vladimir C
49685f0b9a
Merge pull request #30292 from Joeywzr/trim_dev 2021-10-19 14:49:43 +03:00
Nikita Mikhaylov
ea2a647724
Merge pull request #30192 from azat/granula-adjust-LOGICAL_ERROR-fix 
Fix reading from MergeTree with max_read_buffer_size=0
 2021-10-19 13:25:55 +03:00
Ilya Yatsishin
4396daeb1f
Merge pull request #30064 from azat/client-print-profile-events 2021-10-19 12:42:09 +03:00
vdimir
3dfbc80f0b
Add cases to test replaceRegexpAll_bug 2021-10-19 12:38:11 +03:00
Kruglov Pavel
1e2ceeb2e7
Merge pull request #29291 from Avogar/capnproto 
Add CapnProto output format, refactor CapnProto input format
 2021-10-19 11:54:55 +03:00
Kruglov Pavel
de2b75abf0
Merge pull request #30053 from Avogar/clickhouse-local 
Remove unused headers and handle exception 'unrecognised option' in clickhouse-local
 2021-10-19 11:52:15 +03:00
Nikolai Kochetov
cc7a3ea9ff
Merge pull request #30322 from ClickHouse/update-mergine-sorted-memory-optimization 
Update memory optimisation for MergingSorted.
 2021-10-19 11:12:50 +03:00
alesapin
53ad965f8e
Merge pull request #30170 from ClickHouse/debug_keeper 
Start keeper asynchronously in some cases
 2021-10-19 10:14:36 +03:00
Azat Khuzhin
424bf6fcf4 client: add ability to print raw profile events 
This can be useful for debugging and for testing (since you will not
need to obtain query_id and look at query_log).

v2:
- mark test as long
- add option to docs
- add type of profile event into logs too
v3:
- resolve conflicts
- and fix onProfileEvents callback
v4:
- add --print-profile-events separate switch
 2021-10-19 00:54:38 +03:00
Azat Khuzhin
f7b76373ce Move ProfileEvents packet type from TCPHandler into ProfileEventsExt 2021-10-19 00:54:38 +03:00
alexey-milovidov
670f25db12
Merge pull request #30181 from Biron-BI/antlr_window_function 
add support of window function in antlr grammar
 2021-10-18 23:30:12 +03:00
Nikita Mikhaylov
75c16b8f89
Merge pull request #30284 from Algunenano/sequential_tests 
Adjust resource heavy tests
 2021-10-18 22:42:14 +03:00
Nikita Mikhaylov
d004fd96ed
Merge pull request #30318 from azat/fix-shared-build 
Move SquashingTransform to Interpreters (to fix split build)
 2021-10-18 22:41:01 +03:00
alexey-milovidov
885c4daf94
Merge pull request #30191 from azat/aio-O_DIRECT-fix 
Fix pread_fake_async/pread_threadpool with min_bytes_to_use_direct_io
 2021-10-18 22:29:31 +03:00
alexey-milovidov
1c64a656f4
Update adopters.md 2021-10-18 21:56:43 +03:00
alexey-milovidov
0d2ce2e711
Update adopters.md 2021-10-18 21:56:04 +03:00
Azat Khuzhin
e576fd17bd Do not allow reading to empty buffer in MergeTreeReaderStream 
Changelog:
- Verify that all rows was read in MergeTreeReaderWide
- Ignore some exceptions for Wide parts
- Take max_rows_to_read into account
- Verify that requested rows is not zero (otherwise it is too tricky)
- Simply verify that buffer is not empty
 2021-10-18 21:19:34 +03:00
Azat Khuzhin
cd4b33c8c9 Verify that all rows was read in MergeTreeReaderCompact 
v0: Use fmt-like style exception in MergeTreeReaderCompact
v2: Update the check
 2021-10-18 21:19:34 +03:00
Azat Khuzhin
41e6df0f4b Remove unused offset_columns from MergeTreeReaderWide::readRows() 2021-10-18 21:19:34 +03:00
Azat Khuzhin
4e3910e564 Add a test for adjusting last granula with max_read_buffer_size=0 2021-10-18 21:19:34 +03:00
Nikolai Kochetov
7962eddf60 Review fixes. 2021-10-18 19:18:09 +03:00
michon470
3c8e963420
DOCSUP-15198: output_format_csv_null_representation setting translation (#29977) 
* Перевод без изменений содержания

* в ру-версию добавлены ngrambf_v1 и tokenbf_v1

* Update mergetree.md

* Update docs/ru/operations/settings/settings.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>

* Update docs/ru/operations/settings/settings.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>

* Update docs/ru/operations/settings/settings.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>

* Update docs/ru/operations/settings/settings.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>

* Update docs/ru/operations/settings/settings.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>

* Update docs/ru/engines/table-engines/mergetree-family/mergetree.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>

* Update docs/ru/engines/table-engines/mergetree-family/mergetree.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>

* Update docs/ru/engines/table-engines/mergetree-family/mergetree.md

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>

* Corrections and translation

* in EN ver. lines 349-351 were included into the codeblock -- moved them to the proper place
* ...

Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
 2021-10-18 15:40:26 +03:00
tavplubix
837670fd85
Merge pull request #30320 from azat/clickhouse-test-longer-wait 
Increase default wait of the server start in clickhouse-test
 2021-10-18 15:11:32 +03:00
Vladimir C
4a4700424e
Merge pull request #30230 from vdimir/CLOUDSUPPORT-92764 2021-10-18 14:46:02 +03:00
Kruglov Pavel
6350957709
Fix special build 2021-10-18 14:30:02 +03:00
Kruglov Pavel
003955b59f
Merge branch 'master' into clickhouse-local 2021-10-18 14:24:08 +03:00
avogar
a28b048415 Fix test 2021-10-18 14:22:27 +03:00
vdimir
042e61febf
Allow identifiers staring with numbers in multiple joins 2021-10-18 13:41:40 +03:00
Nikolai Kochetov
6e479b301a Update memory optimisation for MergingSorted. 2021-10-18 12:54:12 +03:00
alesapin
fa342c961a Merge branch 'debug_keeper' of github.com:ClickHouse/ClickHouse into debug_keeper 2021-10-18 12:14:06 +03:00
alesapin
6d24ca4c3a Review fixes 2021-10-18 12:13:24 +03:00
Anton Popov
71b474793c
Merge pull request #30174 from CurtizJ/better-interfaces 
Better interfaces for `IDataType` and `ISerialization`
 2021-10-18 12:07:04 +03:00
Azat Khuzhin
09b782a52e Increase default wait of the server start in clickhouse-test 
Set --server-check-retries to 90 (and this is ~45 seconds), since right
now sometimes it is not enough [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/30191/0e34a9d550cfe6924fe575871f36c44dd44acdaa/functional_stateless_tests_(memory).html#fail1

And the reason I guess is clickhouse-test had been rewritten to
http.client in #30065, and since now it does not need to execute
clickhouse-client binary, which in debug/sanitizers builds can take also
sometime.
That said that with clickhouse-client for hung check it was not 15
seconds, but more (each clickhouse-client requires 0.6sec with
sanitizers for simple SELECT 1, while w/o 0.1second, also too much
should be optimized)
 2021-10-18 10:46:02 +03:00
Azat Khuzhin
6d4af3bac1 Move SquashingTransform to Interpreters (to fix split build) 
clickhouse_common_io requires clickhouse_core:

    ld.lld: error: undefined symbol: DB::blocksHaveEqualStructure(DB::Block const&, DB::Block const&)
    >>> referenced by SquashingTransform.cpp:92 (/src/ch/clickhouse/src/Common/SquashingTransform.cpp:92)
    >>>               src/CMakeFiles/clickhouse_common_io.dir/Common/SquashingTransform.cpp.o:(void DB::SquashingTransform::append<DB::Block&&>(DB::Block&&))
    >>> referenced by SquashingTransform.cpp:92 (/src/ch/clickhouse/src/Common/SquashingTransform.cpp:92)
    >>>               src/CMakeFiles/clickhouse_common_io.dir/Common/SquashingTransform.cpp.o:(void DB::SquashingTransform::append<DB::Block const&>(DB::Block const&))

while clickhouse_core requires clickhouse_common_io:

    "clickhouse_core" of type SHARED_LIBRARY
      depends on "roaring" (weak)
      depends on "clickhouse_common_io" (weak)
      depends on "clickhouse_common_config" (weak)
      depends on "clickhouse_common_zookeeper" (weak)
      depends on "clickhouse_dictionaries_embedded" (weak)
      depends on "clickhouse_parsers" (weak)

Follow-up for: #30247 (cc @KochetovNicolai)
 2021-10-18 10:28:36 +03:00
alesapin
f1fe96e194 Merge branch 'master' into debug_keeper 2021-10-18 10:16:07 +03:00
alesapin
158b4c26b7
Merge pull request #30134 from ClickHouse/fix-network-bytes-test 
Fix flaky test 01939_network_send_bytes_metrics
 2021-10-18 10:11:59 +03:00
alexey-milovidov
452eee7922
Merge pull request #30315 from ClickHouse/no-recursive-submodules 
Remove recursive submodules
 2021-10-18 05:02:40 +03:00
Alexey Milovidov
2cef9983fa Remove recursive submodules 2021-10-18 02:29:04 +03:00
Alexey Milovidov
f032fb71f0 Minor modification in hardware benchmark 2021-10-18 02:09:41 +03:00
alexey-milovidov
9f21c98f2a
Merge pull request #30312 from den-crane/patch-35 
Doc. Path to ssd_cache
 2021-10-18 00:33:55 +03:00
Azat Khuzhin
0e34a9d550 Add MSan instrumentation for preadv2 
MSan report [1]:

    ==10672==WARNING: MemorySanitizer: use-of-uninitialized-value
        3 0x30dd6005 in DB::CompressionCodecFactory::get(unsigned char) const obj-x86_64-linux-gnu/../src/Compression/CompressionFactory.cpp:105:65
        4 0x30dab4d5 in DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&, bool) obj-x86_64-linux-gnu/../src/Compression/CompressedReadBufferBase.cpp:127:53
        5 0x30db54af in DB::CompressedReadBufferFromFile::readBig(char*, unsigned long) obj-x86_64-linux-gnu/../src/Compression/CompressedReadBufferFromFile.cpp:119:38

      Uninitialized value was stored to memory at
        0 0x9ef4e99 in __msan_memcpy (/src/ch/tmp/30191/clickhouse-msan+0x9ef4e99)
        1 0xa0c1999 in DB::ReadBuffer::read() obj-x86_64-linux-gnu/../src/IO/ReadBuffer.h:173:13
        2 0xa0c1999 in DB::ReadBuffer::readStrict() obj-x86_64-linux-gnu/../src/IO/ReadBuffer.h:184:27
        3 0x30daafd1 in DB::CompressedReadBufferBase::readCompressedData() obj-x86_64-linux-gnu/../src/Compression/CompressedReadBufferBase.cpp:120:20
        4 0x30db54af in DB::CompressedReadBufferFromFile::readBig() obj-x86_64-linux-gnu/../src/Compression/CompressedReadBufferFromFile.cpp:119:38

      Uninitialized value was created by a heap allocation
        0 0x9ef5285 in posix_memalign (/src/ch/tmp/30191/clickhouse-msan+0x9ef5285)
        1 0xa0cb98d in Allocator<false, false>::allocNoTrack(unsigned long, unsigned long) obj-x86_64-linux-gnu/../src/Common/Allocator.h:235:27
        2 0xa0caf0d in Allocator<false, false>::alloc(unsigned long, unsigned long) obj-x86_64-linux-gnu/../src/Common/Allocator.h:96:16
        3 0xa0caf0d in DB::Memory<Allocator<false, false> >::alloc() obj-x86_64-linux-gnu/../src/IO/BufferWithOwnMemory.h:126:49
        4 0xa0ca4a1 in DB::Memory<Allocator<false, false> >::Memory() obj-x86_64-linux-gnu/../src/IO/BufferWithOwnMemory.h:43:9
        5 0xa0ca4a1 in DB::BufferWithOwnMemory<DB::SeekableReadBuffer>::BufferWithOwnMemory(unsigned long, char*, unsigned long) obj-x86_64-linux-gnu/../src/IO/BufferWithOwnMemory.h:153:29
        6 0xa0ca4a1 in DB::ReadBufferFromFileBase::ReadBufferFromFileBase() obj-x86_64-linux-gnu/../src/IO/ReadBufferFromFileBase.cpp:11:7
        7 0xa59d980 in DB::AsynchronousReadBufferFromFileDescriptor::AsynchronousReadBufferFromFileDescriptor() obj-x86_64-linux-gnu/../src/IO/AsynchronousReadBufferFromFileDescriptor.h:40:11

    SUMMARY: MemorySanitizer: use-of-uninitialized-value obj-x86_64-linux-gnu/../contrib/libcxx/include/__hash_table:116:10 in std::__1::__constrain_hash(unsigned long, unsigned long)

  [1]: https://clickhouse-test-reports.s3.yandex.net/30191/18bb0e7d8d47dd694390f3f7b2ecc921a167afac/fuzzer_msan/report.html#fail1
 2021-10-17 23:54:58 +03:00
Azat Khuzhin
14baed927f Add extensive test for various read settings 
The following settings had been covered:
- min_bytes_to_use_direct_io
- local_filesystem_read_method
- local_filesystem_read_prefetch
- read_priority
- max_read_buffer_size
 2021-10-17 23:54:58 +03:00
Azat Khuzhin
fb54d6b6e0 BufferWithOwnMemory: take reallocs into account 2021-10-17 23:54:58 +03:00
Azat Khuzhin
348033d8bd BufferWithOwnMemory: make size aligned not capacity 
This will fix pread_fake_async/pread_threadpool with min_bytes_to_use_direct_io
 2021-10-17 23:54:58 +03:00
Azat Khuzhin
9cc1178ebc BufferWithOwnMemory: do not try to align if buffer already aligned 2021-10-17 23:54:57 +03:00
Azat Khuzhin
34810456e1 Fix alignment for prefetch in AsynchronousReadBufferFromFileDescriptor 2021-10-17 23:54:57 +03:00
Denny Crane
084938fa8c
Update external-dicts-dict-layout.md 2021-10-17 16:28:22 -03:00
Denny Crane
eb66442feb
Update external-dicts-dict-layout.md 2021-10-17 16:21:21 -03:00