Commit Graph

109 Commits

Author SHA1 Message Date
Azat Khuzhin
98e3a99a88 Do not catch exceptions during final flush in writers destructors
Since this hides real problems, since destructor does final flush and if
it fails, then data will be lost.

One of such examples if MEMORY_LIMIT_EXCEEDED exception, so lock
exceptions from destructors, by using
MemoryTracker::LockExceptionInThread to block these exception, and allow
others (so std::terminate will be called, since this is c++11 with
noexcept for destructors by default).

Here is an example, that leads to empty block in the distributed batch:

    2021.01.21 12:43:18.619739 [ 46468 ] {7bd60d75-ebcb-45d2-874d-260df9a4ddac} <Error> virtual DB::CompressedWriteBuffer::~CompressedWriteBuffer(): Code: 241, e.displayText() = DB::Exception: Memory limit (for user) exceeded: would use 332.07 GiB (attempt to allocate chunk of 4355342 bytes), maximum: 256.00 GiB, Stack trace (when copying this message, always include the lines below):

    0. DB::Exception::Exception<>() @ 0x86f7b88 in /usr/bin/clickhouse
    ...
    4. void DB::PODArrayBase<>::resize<>(unsigned long) @ 0xe9e878d in /usr/bin/clickhouse
    5. DB::CompressedWriteBuffer::nextImpl() @ 0xe9f0296 in /usr/bin/clickhouse
    6. DB::CompressedWriteBuffer::~CompressedWriteBuffer() @ 0xe9f0415 in /usr/bin/clickhouse
    7. DB::DistributedBlockOutputStream::writeToShard() @ 0xf6bed4a in /usr/bin/clickhouse
2021-02-05 01:31:45 +03:00
Alexey Milovidov
5f13fdddc0 Remove useless code from BigInt 2021-01-26 21:22:40 +03:00
Alexey Milovidov
cc11e7c2f2 Whitespace 2021-01-25 00:23:06 +03:00
alexey-milovidov
ca825f36f1
Merge pull request #18853 from azat/dist-send-checksums
Improve checksum checks for async INSERT into Distributed on the sender
2021-01-15 17:12:54 +03:00
ygrek
8f2a830d83
add zstd long range option (#17184)
* add zstd long compression option

* tests: add zstd long read-write test

Co-authored-by: Joris Giovannangeli <joris.giovannangeli@ahrefs.com>
Co-authored-by: ip <igor@ahrefs.com>
2021-01-13 16:22:59 +03:00
Azat Khuzhin
946a126799 Add CheckingCompressedReadBuffer
Buffer for reading from a compressed file with just checking checksums
of the compressed blocks, without any decompression, so result can be
proxied.
2021-01-10 21:23:42 +03:00
alexey-milovidov
6be8338584
Update CompressedReadBufferBase.cpp 2021-01-08 22:41:24 +03:00
Azat Khuzhin
f2ff785460 Do not check bit flips for big buffers (since the size can be corrupted) 2021-01-08 02:26:17 +03:00
Alexey Milovidov
1572d2122b Respect network_compression_method in async INSERT into Distributed table 2021-01-06 03:41:34 +03:00
Alexey Milovidov
438f0f971b Fix the issue with async Distributed INSERTs and network_compression_method #18741 2021-01-06 03:24:42 +03:00
Alexey Milovidov
24f4fa6edf Follow Arcadia ya.make rules 2020-11-17 00:16:50 +03:00
Alexey Milovidov
3df04ce0c2 Follow Arcadia ya.make rules 2020-11-16 21:24:58 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Alexey Milovidov
b56486510f Improve Arcadia 2020-11-05 12:55:01 +03:00
Maxim Akhmedov
3627fabfb9 Remove -g0 form Arcadia build settings. 2020-10-29 17:37:23 +03:00
tavplubix
a1f1db753b
Update CompressedReadBufferBase.cpp 2020-10-28 18:23:10 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell 2020-10-27 12:04:03 +01:00
Ivan
1d170f5745
ASTTableIdentifier Part #1: improve internal representation of ASTIdentifier name (#16149)
* Use only |name_parts| as primary name source

* Restore legacy logic for table restoration

* Fix build

* Fix tests

* Add pytest server config

* Fix tests

* Fixes due to review
2020-10-24 21:46:10 +03:00
alesapin
50eba27d1a
Update CompressionFactory.h 2020-09-23 10:24:25 +03:00
alesapin
fd394f699c Better comments 2020-09-22 15:49:55 +03:00
alesapin
bcea1a4207 Remove redundant change 2020-09-21 18:55:21 +03:00
alesapin
8517dda807 First working test 2020-09-21 17:47:10 +03:00
alesapin
4debccb7fe Almost working version 2020-09-21 17:22:13 +03:00
alesapin
a3e999784b Buildable code 2020-09-21 14:24:10 +03:00
alesapin
7fc5a2dae1 Some intermediate stage 2 2020-09-21 10:18:23 +03:00
alesapin
e96a3ac5f3 DataType in enumerate streams 2020-09-18 14:37:58 +03:00
alesapin
5f9e7ed169
Merge pull request #14834 from ClickHouse/fix_compression_codec_perf
Fix performance degradation during parts selection for merge
2020-09-17 10:09:41 +03:00
alesapin
18398c1510 Fix style 2020-09-16 12:08:39 +03:00
alesapin
78cb39ff92
Update ICompressionCodec.h 2020-09-16 11:21:39 +03:00
alesapin
af8ed46261
Update CompressionCodecGorilla.cpp 2020-09-16 11:20:39 +03:00
alesapin
cf45eb1545
Update CompressionCodecDoubleDelta.cpp 2020-09-16 11:20:22 +03:00
alesapin
e3299d9249
Update CompressionCodecMultiple.cpp 2020-09-16 11:19:59 +03:00
alesapin
da386990fe
Update ICompressionCodec.cpp 2020-09-16 11:18:42 +03:00
alesapin
74c0b6eb76
Update CompressionCodecGorilla.cpp 2020-09-16 10:00:41 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build (#14809) 2020-09-15 12:55:57 +03:00
alesapin
e96256d367 Trying to improve performance 2020-09-14 22:15:25 +03:00
Anton Popov
32135d96f9
Merge pull request #12183 from CurtizJ/polymorphic-parts-2
Support codecs in compact parts
2020-09-09 12:34:13 +03:00
Anton Popov
a419267dc6 minor fixes 2020-09-08 19:28:49 +03:00
Alexey Milovidov
e3924b8057 Fix "Arcadia" 2020-09-08 01:14:13 +03:00
Anton Popov
68913eab62 better reading from compact parts with differents codecs 2020-09-04 15:48:55 +03:00
Anton Popov
d5da58918e create less compressed streams while writing compact parts 2020-09-04 01:04:46 +03:00
Anton Popov
98011f6fc3 Merge remote-tracking branch 'upstream/master' into HEAD 2020-09-03 17:53:05 +03:00
alesapin
6f16c08191 Fix ya.make 2020-08-28 12:29:07 +03:00
alesapin
d8d7c46711 Remove ya.make update 2020-08-28 12:27:59 +03:00
alesapin
77faf9587f Better interface 2020-08-28 12:07:20 +03:00
alesapin
2fc80189af Add default compression codec to merge tree data part 2020-08-26 18:29:46 +03:00
alesapin
4326c9c971 Fix delta 2020-08-26 12:16:32 +03:00
alesapin
249c4b4a94 Less strange code and one method 2020-08-26 11:59:02 +03:00
alesapin
21d78f8513 Better codec description representation 2020-08-26 11:45:13 +03:00