Commit Graph

80680 Commits

Author SHA1 Message Date
Alexey Milovidov
c4713ee34b Normalize left, right, leftUTF8, rightUTF8 functions 2022-01-05 06:46:31 +03:00
freedomDR
af601f98d3 doc fix bloomfilter 2022-01-05 03:37:54 +00:00
Alexey Milovidov
ff779b6409 Fix error in #33124 2022-01-05 06:33:28 +03:00
Alexey Milovidov
bf59114cee Make "left" and "right" real functions 2022-01-05 05:32:40 +03:00
Alexey Milovidov
6f40d3a342 Parser simplification (incomplete) 2022-01-05 05:30:01 +03:00
alexey-milovidov
48eec3d8f8
Merge pull request #33366 from ClickHouse/aggregate-dependencies
Aggregate functions to calculate dependencies between columns
2022-01-05 02:50:25 +03:00
Denny Crane
fbf8dd53d8
Update date-time-functions.md 2022-01-04 18:56:14 -04:00
Denny Crane
466078031d
Update date-time-functions.md 2022-01-04 18:53:01 -04:00
Alexey Milovidov
0696c1971f Fix clang-tidy 2022-01-04 23:52:10 +03:00
alexey-milovidov
1f82ab2382
Update SECURITY.md 2022-01-04 23:40:35 +03:00
alexey-milovidov
03eadbb3d8
Merge pull request #33399 from ClickHouse/remove-obsolete-builder
Remove obsolete and unsupported Docker Builder #33345
2022-01-04 23:16:55 +03:00
alexey-milovidov
09f2cd1f95
Merge pull request #33372 from azat/pread-eof
Do not try to read pass EOF (to workaround a bug in a kernel)
2022-01-04 22:54:10 +03:00
Alexey Milovidov
2d3f53ec5c Merge branch 'dotnet-integration-test' of github.com:yandex/ClickHouse into dotnet-integration-test 2022-01-04 22:49:58 +03:00
Alexey Milovidov
184033b782 Trying to make it work 2022-01-04 22:49:41 +03:00
Maksim Kita
fb9224fd61
Merge pull request #33388 from kitaisreal/dictionary-structure-fixes
DictionaryStructure fixes
2022-01-04 21:41:46 +03:00
mergify[bot]
7cfa01cdba
Merge branch 'master' into dotnet-integration-test 2022-01-04 17:16:02 +00:00
Alexey Milovidov
adc577c52f Remove obsolete and unsupported Docker Builder #33345 2022-01-04 20:03:59 +03:00
alexey-milovidov
9e9e6b7922
Merge pull request #33341 from azat/local-memory-tracking
clickhouse-local: track memory under --max_memory_usage_in_client option
2022-01-04 19:43:41 +03:00
alexey-milovidov
f80b83022f
Merge pull request #33335 from ClickHouse/interval-length-sum-allow-negative
Allow negative intervals in function `intervalLengthSum`
2022-01-04 19:36:11 +03:00
mergify[bot]
327e9572bb
Merge branch 'master' into dotnet-integration-test 2022-01-04 16:35:28 +00:00
alexey-milovidov
5616000568
Merge pull request #33397 from iladriano/patch-1
Update success.html
2022-01-04 19:29:54 +03:00
Adri Fernandez
ef16346af0
Update success.html 2022-01-04 08:20:53 -08:00
Maksim Kita
c6121f6c2e
Merge pull request #33386 from azat/DROP-INSERT-UAF-fix-v2
Fix possible use-after-free for INSERT into MV with concurrent DROP
2022-01-04 18:53:53 +03:00
Maksim Kita
53e45c34e3
Merge pull request #33373 from den-crane/patch-54
Doc. default values for max_concurrent_queries settings
2022-01-04 14:06:41 +03:00
Maksim Kita
c84193ac67 DictionaryStructure fixes 2022-01-04 14:02:46 +03:00
Azat Khuzhin
5ed7440381 Fix possible use-after-free for INSERT into MV with concurrent DROP
ASan founds [1]:

    ==553==
    ERROR: AddressSanitizer: heap-use-after-free on address 0x61e004694080 at pc 0x000029150af2 bp 0x7f70b3f8ef10 sp 0x7f70b3f8ef08
    READ of size 8 at 0x61e004694080 thread T477 (QueryPipelineEx)
        0 0x29150af1 in DB::MergeTreeDataWriter::writeTempPart() >
        1 0x293b8e43 in DB::MergeTreeSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeSink.cpp:27:65
        2 0x29dac73b in DB::SinkToStorage::onConsume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5
        3 0x29c72dd2 in DB::ExceptionKeepingTransform::work()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Processors/Transforms/ExceptionKeepingTransform.cpp:151:51

    0x61e004694080 is located 2048 bytes inside of 2480-byte region [0x61e004693880,0x61e004694230)
    freed by thread T199 (BgSchPool) here:
        ...
        4 0x26220f20 in DB::DatabaseCatalog::TableMarkedAsDropped::~TableMarkedAsDropped() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.h:248:12
        5 0x26220f20 in DB::DatabaseCatalog::dropTableDataTask() obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:908:1

      [1]: https://s3.amazonaws.com/clickhouse-test-reports/33201/4f04d6af61eabf4899eb8188150dc862aaab80fc/stress_test__address__actions_.html

There was a fix in #32572, but it was not complete (yes it reduced the
race window a lot, but not completely), since the inner table still can
go away after the INSERT chain was built, to fix this obtain the
reference earlier.

Follow-up for: #32572 (cc @tavplubix)
2022-01-04 13:34:01 +03:00
Azat Khuzhin
2cdc5b37e3 Do not try to read pass EOF (to workaround a bug in a kernel)
For unaligned offset pread() may return EINVAL even if the offset pass
EOF, although it should not, since otherwise there is no abiliity to
rely on read() == 0 is EOF (with pread() loop).

Here is a reproducer for the problem on 4.9.0-12-amd64:

    $ head -c27 /dev/urandom > /tmp/pread.issue
    $ xfs_io
    xfs_io> open -d /tmp/pread.issue
    xfs_io> pread 1000 4096
    pread: Invalid argument

And this is how it should work:

    xfs_io> pread 29 4096
    read 0/4096 bytes at offset 29

Note, here I use interactive mode since we had old xfs_io that does not
allow to execute multiple commands at once, and to avoid EMFILE issue

Here is some history of a patches that affects this behaviour in the
linux kernel:

- the issue had been introduced in
  torvalds/linux@9fe55eea7e v3.14
  ("Fix race when checking i_size on direct i/o read")
- an attempt to fix it had been made in
  torvalds/linux@74cedf9b6c v4.4
  ("direct-io: Fix negative return from dio read beyond eof")
- but this wasn't enough, since alignment check was earlier, so
  eventually fixed in
  torvalds/linux@41b21af388 v5.10
  ("direct-io: defer alignment check until after the EOF check")

Someone may ask why CI does not shows the issue, since:
- it had 4.19 kernel when CI was in yandex
- now it has 5.4 when CI is in AWS
Since both of those kernels does not have the last patch.

But, this bug requires the following conditions to met:
- index_granularity_bytes=0
- min_merge_bytes_to_use_direct_io=1
Which was not covered by CI yet.
2022-01-04 10:53:17 +03:00
Azat Khuzhin
f0f2a9cc40 Add a regression test for pread pass EOF returns EINVAL 2022-01-04 10:53:17 +03:00
Azat Khuzhin
71a99ab27c Accept real file size in createReadBufferFromFileBase()
Right now streams relies on correct file size not the number of bytes
that will be read from the stream, to overcome one bug in the linux
kernel that may return EIINVAL for pread() with offset pass the EOF.

v2: Swap read_hint and file_size (since it is easy to miss something)

Before the first argument to readFile()/createReadBufferFromFileBase()
was read_hint, not the file_size, and let's preserve the order, since
it is easy to miss something

This will also fix 02051_read_settings test automatically because now
MergeTreeReaderStream will pass estimated_sum_mark_range_bytes to
read_hint not file_size, previously it cause on of the following errors:
- Attempt to read after EOF w/ O_DIRECT
- and LOGICAL_ERROR while adjusting granulas w/o O_DIRECT

This will also improve zero-length reads guard (via
ReadBufferFromEmptyFile), that had been added in #30190

v3: fix for other storages that wasn't enabled in fast-test
v4: ignore ENOENT/ENOTSUP in readFile
2022-01-04 10:53:17 +03:00
Azat Khuzhin
e548dae8d9 Pass size of the file to the readers
v2: s/size/file_size/g
2022-01-04 10:53:17 +03:00
Alexey Milovidov
e80f4008fa Fix idiotic test 2022-01-04 00:50:41 +03:00
mergify[bot]
a30c10a235
Merge branch 'master' into interval-length-sum-allow-negative 2022-01-03 20:53:55 +00:00
alexey-milovidov
b5ee8cb694
Merge pull request #33374 from den-crane/patch-55
Doc. No need to double backslashes (\\)
2022-01-03 23:51:08 +03:00
Denny Crane
7261e121dd
Update string-search-functions.md 2022-01-03 16:19:54 -04:00
Denny Crane
bb713f1487
Update settings.md 2022-01-03 16:01:18 -04:00
Denny Crane
f764e26c7b
Update settings.md 2022-01-03 15:59:52 -04:00
Denny Crane
9114d127e0
default values for max_concurrent_queries settings 2022-01-03 15:48:22 -04:00
Alexey Milovidov
7bd4d1a8d8 Fix UBSan 2022-01-03 22:24:03 +03:00
Alexey Milovidov
8aa26fd04a Fix UBSan 2022-01-03 22:23:34 +03:00
Alexey Milovidov
ac84015f01 Make at least some sense 2022-01-03 22:18:57 +03:00
Alexey Milovidov
04a39317c1 Add check 2022-01-03 21:39:04 +03:00
Nikita Mikhaylov
659f502ce5
Merge pull request #33346 from amosbird/blobstoragebuildfix2 2022-01-03 21:17:09 +04:00
alexey-milovidov
6efe73fc1b
Merge pull request #26559 from nikitamikhaylov/merging-dev-sum-bug
Merging #26074
2022-01-03 19:46:54 +03:00
alexey-milovidov
cb19d6f519
Merge pull request #29953 from amosbird/saturatetime
Saturate date/datetime to zero.
2022-01-03 19:44:33 +03:00
Alexey Milovidov
c7f65c2d31 Merge branch 'aggregate-dependencies' of github.com:yandex/ClickHouse into aggregate-dependencies 2022-01-03 18:29:46 +03:00
alexey-milovidov
0c5485fd3a
Update AggregateFunctionContingencyCoefficient.cpp 2022-01-03 16:45:15 +03:00
alexey-milovidov
8608dbb8e7
Fix CLion 2022-01-03 16:45:02 +03:00
alexey-milovidov
6e9684149b
Update wide_integer_impl.h 2022-01-03 16:43:58 +03:00
mergify[bot]
9e7102542b
Merge branch 'master' into aggregate-dependencies 2022-01-03 13:42:34 +00:00
alesapin
9619b0fd96
Merge pull request #33336 from ClickHouse/fix-diff-encoding
Fix (?) encoding issue in #33331
2022-01-03 15:17:22 +03:00