Alexey Milovidov
2a0cba2b9f
Merge branch 'master' into async-reads
2021-08-01 01:25:13 +03:00
PHO
6425dd001a
Add a codec AES_128_GCM_SIV for encrypting columns on disk ( #19896 )
...
* Add a codec Encrypted() for encrypting columns on disk
While this is implemented as a compression codec, it does not actually compress data. It instead encrypts data on disk. The key is obtained by executing a user-specified command at the server startup, or if it's not specified the codec refuses to process any data. For now the only supported cipher is 'AES-128-GCM-SIV'.
2021-07-30 12:12:33 +03:00
Alexey Milovidov
5c3f5cb35a
Experiment with asynchronous readers
2021-07-26 03:34:36 +03:00
Alexey Milovidov
21fb6ddea5
Remove AIO for sequential reads
2021-07-05 23:02:24 +03:00
Stas Kelvich
9a84033db5
Fix typo in hardware failure error message
2021-06-22 12:48:15 +03:00
Yuriy Chernyshov
c0cb5387f4
Use zstd/include in ya.make
2021-06-09 23:02:39 +03:00
Alexey Milovidov
8d85145725
Remove experimental compression codecs
2021-06-06 02:26:52 +03:00
Alexey Milovidov
273226de32
Remove string parameter for Density
2021-05-24 06:43:25 +03:00
Alexey Milovidov
4290b48a61
Fix clang-tidy
2021-05-24 06:00:57 +03:00
Alexey Milovidov
5412242908
Fix error
2021-05-24 02:38:46 +03:00
Alexey Milovidov
70d9f5c154
Fix error
2021-05-24 02:36:50 +03:00
Alexey Milovidov
8d2fa6cf50
Allow to build without experimental compression libraries and remove from fasttest
2021-05-24 01:18:12 +03:00
Alexey Milovidov
57a3f132d7
Fix style
2021-05-24 00:48:54 +03:00
Alexey Milovidov
40d4f0678f
Remove overload (harmful)
2021-05-23 04:25:06 +03:00
Alexey Milovidov
fdc5bb72f5
Minor modification
2021-05-23 04:14:29 +03:00
Alexey Milovidov
3057bbe831
Remove header files
2021-05-23 04:12:30 +03:00
Alexey Milovidov
deb68b15da
Fix bad dependencies in code
2021-05-23 04:03:42 +03:00
Alexey Milovidov
77414b50e6
Remove copy-paste
2021-05-23 03:55:16 +03:00
Alexey Milovidov
1a091c7a17
Fix unbalanced whitespace
2021-05-23 03:32:22 +03:00
Alexey Milovidov
a25edc192c
Fix whitespace
2021-05-23 03:31:58 +03:00
Alexey Milovidov
b0df9286f0
Fix whitespace
2021-05-23 03:30:53 +03:00
Alexey Milovidov
ecc2e98265
Fix wrong intent in exception message
2021-05-23 03:30:17 +03:00
Alexey Milovidov
4cfabd2675
Fix style, fix typo and remove extra code
2021-05-23 03:21:48 +03:00
Alexey Milovidov
0da00f6497
Fix style, fix typo and remove extra code
2021-05-23 03:18:24 +03:00
Alexey Milovidov
e330fe4bb5
Fix extra whitespace in exception messages
2021-05-23 03:15:11 +03:00
alexey-milovidov
1ed30f7246
Merge branch 'master' into issue-16775
2021-05-22 17:14:23 +03:00
alesapin
2b62ce9044
Add smoke test for local address
2021-05-18 11:35:51 +03:00
Alexey Milovidov
604daa9581
Fix a bunch of warnings from PVS-Studio
2021-05-08 19:09:17 +03:00
Alexey Milovidov
45dea86395
More fixes for PVS-Studio
2021-05-08 18:35:09 +03:00
Alexey Milovidov
47a4c101b9
Merge branch 'master' into normalize-bigint
2021-05-08 16:41:15 +03:00
mwish
9328ec9c50
[Init] initialize commit
2021-05-08 18:07:39 +08:00
fibersel
f01b53bafb
add newline
2021-05-06 16:47:23 +03:00
fibersel
94b159c510
add newlines
2021-05-06 16:34:27 +03:00
fibersel
cb53bbb7b0
add experimental codecs flag, add integration test for experimental codecs
2021-05-06 14:57:22 +03:00
Alexey Milovidov
1fda866881
Merge branch 'master' into normalize-bigint
2021-05-02 21:24:45 +03:00
Maksim Kita
318c4bb80d
Add examples folder filter to ya.make.in
2021-04-30 11:25:52 +03:00
Nikita Mikhaylov
9f55424250
move to examples everywhere
2021-04-27 01:51:42 +03:00
Alexey Milovidov
649550c5ab
Attempt to normalize big integers
2021-04-25 12:30:43 +03:00
Alexey Milovidov
8f01af62d9
Merge branch 'master' into normalize-bigint
2021-04-25 06:57:44 +03:00
Anton Popov
ea82e7725f
Merge pull request #21562 from CurtizJ/serialization-refactoring-4
...
Refactoring of data types serialization
2021-03-29 16:36:44 +03:00
alexey-milovidov
7ab2890c4d
Merge pull request #22206 from ClickHouse/mmap-cache
...
Add cache for mmap IO
2021-03-29 05:17:55 +03:00
Maksim Kita
459d00f999
Fixed tests
2021-03-28 22:42:34 +03:00
Alexey Milovidov
2a8ac01cdb
Rename as suggested by Kita
2021-03-28 22:24:28 +03:00
Maksim Kita
608d37deed
CachedCompressedReadBuffer fix cache usage
2021-03-28 21:32:38 +03:00
Alexey Milovidov
50f712e198
Integrate mmap cache to the infrastructure
2021-03-28 04:10:30 +03:00
Alexey Milovidov
0e3571478d
Code review changes
2021-03-27 02:42:22 +03:00
Alexey Milovidov
e55f7e6333
Fix error
2021-03-26 02:21:59 +03:00
Alexey Milovidov
ad66c4a916
Fix error
2021-03-26 01:56:24 +03:00
Alexey Milovidov
a8ce138788
Speedup codec NONE
2021-03-25 23:08:47 +03:00
Anton Popov
6a15431be7
Merge remote-tracking branch 'upstream/master' into HEAD
2021-03-25 15:57:35 +03:00
Alexey Milovidov
9ca0566132
Minor modification
2021-03-17 02:08:33 +03:00
Anton Popov
ed42437219
Merge remote-tracking branch 'upstream/master' into HEAD
2021-03-13 01:41:26 +03:00
Yatsishin Ilya
7b69ef92be
gcc old cast check failed
2021-03-10 18:04:34 +03:00
Anton Popov
bc417cf54a
refactoring of serializations
2021-03-09 17:46:52 +03:00
Yatsishin Ilya
71d745b4ec
implicit const conversion
2021-03-09 17:31:54 +03:00
Azat Khuzhin
98e3a99a88
Do not catch exceptions during final flush in writers destructors
...
Since this hides real problems, since destructor does final flush and if
it fails, then data will be lost.
One of such examples if MEMORY_LIMIT_EXCEEDED exception, so lock
exceptions from destructors, by using
MemoryTracker::LockExceptionInThread to block these exception, and allow
others (so std::terminate will be called, since this is c++11 with
noexcept for destructors by default).
Here is an example, that leads to empty block in the distributed batch:
2021.01.21 12:43:18.619739 [ 46468 ] {7bd60d75-ebcb-45d2-874d-260df9a4ddac} <Error> virtual DB::CompressedWriteBuffer::~CompressedWriteBuffer(): Code: 241, e.displayText() = DB::Exception: Memory limit (for user) exceeded: would use 332.07 GiB (attempt to allocate chunk of 4355342 bytes), maximum: 256.00 GiB, Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception<>() @ 0x86f7b88 in /usr/bin/clickhouse
...
4. void DB::PODArrayBase<>::resize<>(unsigned long) @ 0xe9e878d in /usr/bin/clickhouse
5. DB::CompressedWriteBuffer::nextImpl() @ 0xe9f0296 in /usr/bin/clickhouse
6. DB::CompressedWriteBuffer::~CompressedWriteBuffer() @ 0xe9f0415 in /usr/bin/clickhouse
7. DB::DistributedBlockOutputStream::writeToShard() @ 0xf6bed4a in /usr/bin/clickhouse
2021-02-05 01:31:45 +03:00
Alexey Milovidov
093108bf81
Normalize BigInt implementation
2021-01-27 03:54:57 +03:00
Alexey Milovidov
5f13fdddc0
Remove useless code from BigInt
2021-01-26 21:22:40 +03:00
Alexey Milovidov
cc11e7c2f2
Whitespace
2021-01-25 00:23:06 +03:00
alexey-milovidov
ca825f36f1
Merge pull request #18853 from azat/dist-send-checksums
...
Improve checksum checks for async INSERT into Distributed on the sender
2021-01-15 17:12:54 +03:00
ygrek
8f2a830d83
add zstd long range option ( #17184 )
...
* add zstd long compression option
* tests: add zstd long read-write test
Co-authored-by: Joris Giovannangeli <joris.giovannangeli@ahrefs.com>
Co-authored-by: ip <igor@ahrefs.com>
2021-01-13 16:22:59 +03:00
Azat Khuzhin
946a126799
Add CheckingCompressedReadBuffer
...
Buffer for reading from a compressed file with just checking checksums
of the compressed blocks, without any decompression, so result can be
proxied.
2021-01-10 21:23:42 +03:00
alexey-milovidov
6be8338584
Update CompressedReadBufferBase.cpp
2021-01-08 22:41:24 +03:00
Azat Khuzhin
f2ff785460
Do not check bit flips for big buffers (since the size can be corrupted)
2021-01-08 02:26:17 +03:00
Alexey Milovidov
1572d2122b
Respect network_compression_method in async INSERT into Distributed table
2021-01-06 03:41:34 +03:00
Alexey Milovidov
438f0f971b
Fix the issue with async Distributed INSERTs and network_compression_method #18741
2021-01-06 03:24:42 +03:00
Alexey Milovidov
24f4fa6edf
Follow Arcadia ya.make rules
2020-11-17 00:16:50 +03:00
Alexey Milovidov
3df04ce0c2
Follow Arcadia ya.make rules
2020-11-16 21:24:58 +03:00
Alexander Tokmakov
b94cc5c4e5
remove more stringstreams
2020-11-10 21:22:26 +03:00
Alexey Milovidov
fd84d16387
Fix "server failed to start" error
2020-11-07 03:14:53 +03:00
Alexey Milovidov
b56486510f
Improve Arcadia
2020-11-05 12:55:01 +03:00
Maxim Akhmedov
3627fabfb9
Remove -g0 form Arcadia build settings.
2020-10-29 17:37:23 +03:00
tavplubix
a1f1db753b
Update CompressedReadBufferBase.cpp
2020-10-28 18:23:10 +03:00
Mikhail Filimonov
41971e073a
Fix typos reported by codespell
2020-10-27 12:04:03 +01:00
Ivan
1d170f5745
ASTTableIdentifier Part #1 : improve internal representation of ASTIdentifier name ( #16149 )
...
* Use only |name_parts| as primary name source
* Restore legacy logic for table restoration
* Fix build
* Fix tests
* Add pytest server config
* Fix tests
* Fixes due to review
2020-10-24 21:46:10 +03:00
alesapin
50eba27d1a
Update CompressionFactory.h
2020-09-23 10:24:25 +03:00
alesapin
fd394f699c
Better comments
2020-09-22 15:49:55 +03:00
alesapin
bcea1a4207
Remove redundant change
2020-09-21 18:55:21 +03:00
alesapin
8517dda807
First working test
2020-09-21 17:47:10 +03:00
alesapin
4debccb7fe
Almost working version
2020-09-21 17:22:13 +03:00
alesapin
a3e999784b
Buildable code
2020-09-21 14:24:10 +03:00
alesapin
7fc5a2dae1
Some intermediate stage 2
2020-09-21 10:18:23 +03:00
alesapin
e96a3ac5f3
DataType in enumerate streams
2020-09-18 14:37:58 +03:00
alesapin
5f9e7ed169
Merge pull request #14834 from ClickHouse/fix_compression_codec_perf
...
Fix performance degradation during parts selection for merge
2020-09-17 10:09:41 +03:00
alesapin
18398c1510
Fix style
2020-09-16 12:08:39 +03:00
alesapin
78cb39ff92
Update ICompressionCodec.h
2020-09-16 11:21:39 +03:00
alesapin
af8ed46261
Update CompressionCodecGorilla.cpp
2020-09-16 11:20:39 +03:00
alesapin
cf45eb1545
Update CompressionCodecDoubleDelta.cpp
2020-09-16 11:20:22 +03:00
alesapin
e3299d9249
Update CompressionCodecMultiple.cpp
2020-09-16 11:19:59 +03:00
alesapin
da386990fe
Update ICompressionCodec.cpp
2020-09-16 11:18:42 +03:00
alesapin
74c0b6eb76
Update CompressionCodecGorilla.cpp
2020-09-16 10:00:41 +03:00
Artem Zuikov
51ba12c2c3
Try speedup build ( #14809 )
2020-09-15 12:55:57 +03:00
alesapin
e96256d367
Trying to improve performance
2020-09-14 22:15:25 +03:00
Anton Popov
32135d96f9
Merge pull request #12183 from CurtizJ/polymorphic-parts-2
...
Support codecs in compact parts
2020-09-09 12:34:13 +03:00
Anton Popov
a419267dc6
minor fixes
2020-09-08 19:28:49 +03:00
Alexey Milovidov
e3924b8057
Fix "Arcadia"
2020-09-08 01:14:13 +03:00
Anton Popov
68913eab62
better reading from compact parts with differents codecs
2020-09-04 15:48:55 +03:00
Anton Popov
d5da58918e
create less compressed streams while writing compact parts
2020-09-04 01:04:46 +03:00
Anton Popov
98011f6fc3
Merge remote-tracking branch 'upstream/master' into HEAD
2020-09-03 17:53:05 +03:00
alesapin
6f16c08191
Fix ya.make
2020-08-28 12:29:07 +03:00
alesapin
d8d7c46711
Remove ya.make update
2020-08-28 12:27:59 +03:00
alesapin
77faf9587f
Better interface
2020-08-28 12:07:20 +03:00
alesapin
2fc80189af
Add default compression codec to merge tree data part
2020-08-26 18:29:46 +03:00
alesapin
4326c9c971
Fix delta
2020-08-26 12:16:32 +03:00
alesapin
249c4b4a94
Less strange code and one method
2020-08-26 11:59:02 +03:00
alesapin
21d78f8513
Better codec description representation
2020-08-26 11:45:13 +03:00
alesapin
e5bc5ea419
Fix compression codec
2020-08-25 22:30:52 +03:00
alesapin
9ca8c5e77a
Rename DefaultCompression to simple Default
2020-08-25 18:53:06 +03:00
alesapin
7bf0b74293
Fix style
2020-08-25 18:22:21 +03:00
alesapin
1180bad3a7
Add ability to specify DefaultCompression codec which correspond to settings specified in config.xml
2020-08-25 18:02:32 +03:00
Artem Zuikov
becc186c91
Add support for extended precision integers and decimals ( #13097 )
2020-08-19 14:52:17 +03:00
Alexey Milovidov
edd89a8610
Fix half of typos
2020-08-08 03:47:03 +03:00
Alexey Milovidov
6f690b7c0d
Normalize ya.make files, fix "Arcadia" build
2020-08-02 16:57:38 +03:00
Nikita Mikhaylov
4d49d2c671
another removes
2020-07-30 13:31:14 +03:00
Nikita Mikhaylov
1d6c2ba689
rewrite other test + remove useless includes
2020-07-29 20:37:57 +03:00
Anton Popov
fbec940e0f
fix reading and check query for compact parts with different codecs
2020-07-11 02:33:36 +03:00
Anton Popov
24f627e52c
fix reading from compact parts with different codecs
2020-07-10 18:57:10 +03:00
Alexey Milovidov
fcdcb3cb1e
Remove useless code
2020-07-10 04:58:27 +03:00
Alexey Milovidov
380f748358
Fix issues
2020-07-09 08:08:41 +03:00
Alexey Milovidov
b48d375ec5
Fix bad code
2020-07-09 08:00:53 +03:00
Alexey Milovidov
c82711b013
Fix build
2020-07-09 07:56:38 +03:00
Alexey Milovidov
a01a61c179
Remove another chunk of unneeded code
2020-07-09 04:07:18 +03:00
Alexey Milovidov
3cc9fd5b8f
Remove something obviously wrong
2020-07-09 04:02:20 +03:00
Alexey Milovidov
ea970fd57c
Remove bad ugliness
2020-07-09 04:00:16 +03:00
Alexey Milovidov
45e706aa8c
Fix bad code, once again
2020-07-09 03:48:51 +03:00
Alexey Milovidov
105a35a6ae
Fix strange things
2020-07-09 03:47:16 +03:00
Alexey Milovidov
0c8e12affd
Remove ridiculous code
2020-07-09 03:46:00 +03:00
Alexey Milovidov
9dc91e3ee4
Fix outstandingly wrong code
2020-07-09 03:35:14 +03:00
Alexey Milovidov
4b8c827f84
Fix wrong exception code in codecs Delta, DoubleDelta #12110
2020-07-09 03:19:02 +03:00
Azat Khuzhin
f3ab0aa081
gtest_compressionCodec: suppress non instantiated gtest warning
...
gtest reports:
[ RUN ] GoogleTestVerification.UninstantiatedParameterizedTestSuite<CodecTestPerformance>
../src/Compression/tests/gtest_compressionCodec.cpp:590: Failure
Parameterized test suite CodecTestPerformance is defined via TEST_P, but never instantiated. None of the test cases will run. Either no INSTANTIATE_TEST_SUITE_P is provided or the only ones provided expand to nothing.
Ideally, TEST_P definitions should only ever be included as part of binaries that intend to use them. (As opposed to, for example, being placed in a library that may be linked in to get other utilities.)
To suppress this error for this test suite, insert the following line (in a non-header) in the namespace it is defined in:
GTEST_ALLOW_UNINSTANTIATED_PARAMETERIZED_TEST(CodecTestPerformance);
[ FAILED ] GoogleTestVerification.UninstantiatedParameterizedTestSuite<CodecTestPerformance> (0 ms)
2020-07-07 02:00:08 +03:00
Azat Khuzhin
0a24d9eabf
gtest_compressionCodec: fix lack of operator<< for char8_t
2020-07-06 23:34:40 +03:00
Azat Khuzhin
dab5b5ad7b
gtest_compressionCodec: use fmt over boost::format
...
boost::format is not compiled under gcc10:
from ../src/Compression/tests/gtest_compressionCodec.cpp:14:
/usr/include/boost/format/alt_sstream_impl.hpp: In instantiation of ‘boost::io::basic_altstringbuf<Ch, Tr, Alloc>::int_type boost::io::basic_altstringbuf<Ch, Tr, Alloc>::overflow(boost::io::basic_altstringbuf<Ch, Tr, Alloc>::int_type) [with Ch = char; Tr = std::char_traits<char>; Alloc = std::allocator<char>; boost::io::basic_altstringbuf<Ch, Tr, Alloc>::int_type = int]’:
/usr/include/boost/format/alt_sstream_impl.hpp:227:9: required from here
/usr/include/boost/format/alt_sstream_impl.hpp:261:45: error: no matching function for call to ‘std::allocator<char>::allocate(std::size_t&, char*)’
261 | newptr = alloc_.allocate(new_size, is_allocated_? oldptr : 0);
(although this is system-wide boost, it is pretty recent - 1.72)
2020-07-06 23:34:40 +03:00
Azat Khuzhin
a276d0da4f
gtest_compressionCodec: is_trivial+is_standard_layout over deprecated is_pod
2020-07-06 23:34:40 +03:00
Alexey Milovidov
1462a66d1e
Fix typos
2020-06-27 22:05:00 +03:00
alexey-milovidov
4f2ed7a5e3
Merge branch 'master' into arch-chooser
2020-06-10 22:31:19 +03:00
alexey-milovidov
3d2254bc99
Merge pull request #11471 from Enmk/Codec_test_fixes
...
Fixed using nullptr source and dest buffers in codecs, fixed test
2020-06-07 02:46:09 +03:00
Vasily Nemkov
8996f47124
Revived Alexey Milovidov's fixes
2020-06-06 12:52:56 +03:00
Alexey Milovidov
a73b360c80
Update comment
2020-06-06 11:45:54 +03:00
Alexey Milovidov
cbf71d66e6
Fix undefined behaviour of DoubleDelta
2020-06-06 11:43:51 +03:00
alexey-milovidov
f984cb750e
Update gtest_compressionCodec.cpp
2020-06-06 01:06:45 +03:00
Vasily Nemkov
05ae8c405b
Fixed using nullptr source and dest buffers in codecs, fixed test
2020-06-05 22:08:52 +03:00
alexey-milovidov
0d30d1005f
Merge pull request #11448 from ClickHouse/fix-codecs-compression-size
...
Try fix return compressed size for codecs.
2020-06-05 04:43:58 +03:00
alexey-milovidov
1fd94de4b4
Update CompressionCodecGorilla.cpp
2020-06-04 23:51:07 +03:00
alexey-milovidov
f24a1f5210
Update CompressionCodecDoubleDelta.cpp
2020-06-04 23:50:31 +03:00
Alexey Milovidov
f7398a9c09
Fix clang-tidy
2020-06-04 23:39:26 +03:00
Nikolai Kochetov
133037c410
Try fix return compressed size for codecs.
2020-06-04 20:56:53 +03:00
Alexey Milovidov
18999e4e7c
Temporarily disable gtest_compressionCodec
2020-06-04 20:56:15 +03:00
alesapin
7fac00ae15
Fix segfault with wrong codecs arguments
2020-06-04 11:55:56 +03:00
Alexey Milovidov
f1ee4b85f0
Fix unit test with UBSan
2020-06-02 02:54:17 +03:00
Dmitrii Kovalkov
37d13d4bce
Compilable
2020-05-29 07:31:59 +02:00
Alexey Milovidov
eacff92d0e
Progress on task
2020-05-23 22:35:08 +03:00
alexey-milovidov
c54f6d5a66
Merge pull request #10546 from kyprizel/master
...
Lexer, ast, checksum and column specs fuzzers added
2020-05-05 14:50:55 +03:00
Alexey Milovidov
a198dce505
Fix double space
2020-05-04 17:55:37 +03:00
Alexey Milovidov
87fb6bf081
Better exception message
2020-05-04 17:46:32 +03:00
Alexey Milovidov
301c6137e4
Better exception messages; fixed error; updated tests
2020-05-04 17:25:54 +03:00
alexey-milovidov
6b0bffcdab
Update CompressionFactory.cpp
2020-05-04 03:47:41 +03:00
Alexey Milovidov
99c18c5a09
Check for suspicious codecs #4966
2020-05-04 03:11:49 +03:00
Eldar Zaitov
670ed4bf1b
lexer, ast, checksum and column specs fuzzers added
2020-04-28 02:12:18 +03:00
Eldar Zaitov
d7a037c26b
split fuzzers and sanitizers
2020-04-25 10:12:38 +03:00
Alexey Milovidov
2b569cf260
Added clarification in exception message #10405
2020-04-23 21:31:37 +03:00
Alexey Milovidov
be22a4b94e
Checkpoint
2020-04-22 08:39:31 +03:00
alexey-milovidov
17e7d4d88a
Merge pull request #10307 from abyss7/arcadia-4
...
Changes for auto-sync with Arcadia
2020-04-17 05:08:34 +03:00
Ivan Lezhankin
e230632645
Changes required for auto-sync with Arcadia
2020-04-16 15:31:57 +03:00
Alexey Milovidov
cdeda4ab91
Fix usage of max_parser_depth setting; remove harmful default function arguments
2020-04-16 04:06:10 +03:00
Ivan Lezhankin
06446b4f08
dbms/ → src/
2020-04-03 18:14:31 +03:00