Commit Graph

26 Commits

Author SHA1 Message Date
vdimir
0f1a7c252d
better TemporaryDataOnDisk 2022-09-29 09:51:46 +00:00
vdimir
ac39bbb3f1
[wip] Common interface for temporary data on disk 2022-09-29 09:51:40 +00:00
Vladimir C
f0e4088c84
Merge pull request #41154 from azat/fix-sorted-block-writer 2022-09-13 19:14:45 +02:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Azat Khuzhin
763bb18f98 Fix SIGSEGV in SortedBlocksWriter in case of empty block
CI found one issue [1].

Here is the stack trace for invalid read:

<details>

<summary>stack trace</summary>

```
    0: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long(this="") const at string:1445:22
    1: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(this="", __str="") at string:1927
    2: DB::TemporaryFileLazySource::TemporaryFileLazySource(this=0x00007f3aec105f58, path_="", header_=0x00007f38ffd93b40) at TemporaryFileLazySource.cpp:11
    3: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] DB::TemporaryFileLazySource* std::__1::construct_at<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, DB::TemporaryFileLazySource*>(__args=0x00007f38ffd91560) at construct_at.h:38:50
    4: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] void std::__1::allocator_traits<std::__1::allocator<DB::TemporaryFileLazySource> >::construct<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void, void>(__args=0x00007f38ffd91560) at allocator_traits.h:298
    5: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::__shared_ptr_emplace<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block>(this=0x00007f3aec105f40, __args=0x00007f38ffd91560) at shared_ptr.h:293
    6: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::allocate_shared<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:954
    7: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::make_shared<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:963
    8: DB::SortedBlocksWriter::streamFromFile(this=<unavailable>, file=<unavailable>) const at SortedBlocksWriter.cpp:238
    9: DB::SortedBlocksWriter::premerge(this=<unavailable>) at SortedBlocksWriter.cpp:209:32
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/stress_test__asan_.html

So the problem here is that there was empty unique_ptr<> reference to
temporary file, because of empty block that accepted by
SortedBlocksWriter::insert(), but insert() is not a problem the problem
is premerge() that steals blocks from insert() and do not have check
that there are some rows. However this check exists in
SortedBlocksWriter::flush(), and in that case temporary file is not
created.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 19:52:38 +02:00
Azat Khuzhin
57146c9361 Fix typos in SortedBlocksWriter
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 19:52:38 +02:00
vdimir
91788f29e8
Upd TemporaryFileOnDisk 2022-08-24 16:15:54 +00:00
vdimir
7194df1184
Move back TemporaryFile -> TemporaryFileOnDisk 2022-08-24 16:14:11 +00:00
vdimir
51c44424cc
More metrics for temp files 2022-08-24 16:14:09 +00:00
vdimir
cd4038d511
Use TemporaryFileOnDisk instead of Poco::TemporaryFile 2022-08-24 16:14:08 +00:00
Maksim Kita
bdc21737d5 MergeTree merge disable batch optimization 2022-07-05 16:15:00 +02:00
avogar
975ba9146f Fix tests naming, remove code duplication 2022-02-07 14:12:19 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
Nikolai Kochetov
fd14faeae2 Remove DataStreams folder. 2021-10-15 23:18:20 +03:00
Nikolai Kochetov
341553febd Fix build. 2021-09-16 20:40:42 +03:00
Nikolai Kochetov
b997214620 Rename QueryPipeline to QueryPipelineBuilder. 2021-09-14 20:48:18 +03:00
Nikolai Kochetov
80e0e24448 Fix unit test and style. 2021-07-23 12:29:53 +03:00
Nikolai Kochetov
fd754430eb Remove more streams. 2021-07-22 19:05:52 +03:00
Alexey Milovidov
261a220227 Remove some code 2021-07-17 21:06:46 +03:00
Kruglov Pavel
caef103837
Merge branch 'master' into Add_IStoragePolicy_interface 2021-01-29 14:00:12 +03:00
Nikolai Kochetov
87246ea91b Fix header for MergeJoin with constants. 2021-01-26 15:37:42 +03:00
kreuzerkrieg
29a2ef3089 Add IStoragePolicy interface 2021-01-26 10:55:28 +02:00
Gleb Novikov
7f5b6fba78 Generic volume is coming...
1. SingleDiskVolume for temporary volumes
2. Generic VolumePtr in StoragePolicies
3. Removed max_data_part_size in system.storage_policies, added volume_type
2020-07-26 21:27:36 +03:00
Artem Zuikov
857d84bf7c
Buffer left blocks optimisation for MergeJoin (#10601) 2020-06-16 23:13:18 +03:00
Gleb Novikov
c637d99e07 Volumes and storages refactoring:
1. Moved Volume to separate file
  2. Created IVolume interface and implemented current behaviour in implementation of new interface — VolumeJBOD
  3. Replaced all old volume usages with new VolumeJBOD. Where it is unnecessary to have JBOD — left just IVolume.
  4. Removed old Volume completely
  5. Moved StoragePolicy to separated files
  6. Moved DiskSelector to separated files
  7. Removed DiskSpaceMonitor file
2020-05-04 23:15:38 +03:00
Artem Zuikov
e34a9457e6
Improve right table insert for Partial MergeJoin on disk (#10467) 2020-04-28 16:55:50 +03:00