Commit Graph

83635 Commits

Author SHA1 Message Date
cnmade
a7dda35f7f Translate zh/engines/table-engines/integrations/sqlite: rename old file 2022-02-19 15:11:43 +08:00
bharatnc
233b4486ed add tests 2022-02-18 22:42:39 -08:00
bharatnc
b7678c1bc1 fix s2 functions having const columns 2022-02-18 22:42:27 -08:00
Azat Khuzhin
fef5f146e7 Fix ENOENT with fsync_part_directory and Vertical merge
fsync of the temporary part directory is superfluous anyway, and besides
that directory is not exists at that time, that will lead to ENOENT
error:

    2022.02.18 17:02:51.634565 [ 35639 ] {} <Error> void DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(DB::TaskRuntimeDataPtr) [Queue = DB::MergeMutateRuntimeQueue]: Code: 107. DB::ErrnoException: Cannot open file /var/lib/clickhouse/data/system/text_log/tmp_merge_202202_1864_3192_14/, errno: 2, strerror: No such file or directory. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

    0. DB::Exception::Exception() @ 0xb26ecfa in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    1. DB::throwFromErrnoWithPath() @ 0xb2700ea in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    2. DB::LocalDirectorySyncGuard::LocalDirectorySyncGuard() @ 0x14905531 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    3. DB::DiskLocal::getDirectorySyncGuard() const @ 0x148af3e3 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug
    4. DB::MergeTask::ExecuteAndFinalizeHorizontalPart::prepare() @ 0x157bef13 in /usr/lib/debug/.build-id/01/8c328bd4858d67.debug

Note, that IMergeTreeDataPart::renameTo() anyway will have fsync for the
directory.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-19 07:50:59 +03:00
bharatnc
d9acbf480c fix - skip fast tests 2022-02-18 18:45:15 -08:00
bharatnc
8076dd8cf5 add tests 2022-02-18 16:35:07 -08:00
bharatnc
7c9a581af6 convert const cols to full cols 2022-02-18 16:34:50 -08:00
Mikhail f. Shiryaev
d2cbc5af40
Add patch type for stable/lts releases 2022-02-19 00:13:07 +01:00
Mikhail f. Shiryaev
df9165bbff
Refactoring of Release, add Repo
- Do not pass Namespace to Release.do
- Decouple repository to another class
2022-02-19 00:13:00 +01:00
Mikhail f. Shiryaev
0960fbad52
Set selease_branch only once as Release property 2022-02-19 00:10:21 +01:00
Mikhail f. Shiryaev
d2a8cff508
Add release_branch checker 2022-02-19 00:01:42 +01:00
Mikhail f. Shiryaev
0f1565a3a2
Clean-up the code a little bit 2022-02-19 00:01:39 +01:00
alexey-milovidov
9630087131
Update app.py 2022-02-19 00:26:43 +03:00
alexey-milovidov
37fbf7c329
Merge branch 'master' into add-trusted-taiyang-li 2022-02-19 00:25:52 +03:00
Nikolai Kochetov
40a48c5670 Enable async writes to S3. 2022-02-18 18:32:04 +00:00
Nikolai Kochetov
e4d5db6161
Merge pull request #34717 from azat/merge-mutate-memory-tracker
Fix possible memory_tracker use-after-free (for async s3 writes) for merges/mutations
2022-02-18 19:28:43 +01:00
Maksim Kita
3dd6b91cbe
Merge pull request #34731 from kitaisreal/performance-tests-fix-h3
Performance tests fix H3
2022-02-18 19:21:38 +01:00
kssenii
acf9a092a9 Review fixes 2022-02-18 19:09:54 +01:00
Vladimir C
fcec52f49c
Merge pull request #34612 from vdimir/fix_aarch64/01681_arg_min_max_if_fix
Adjust 01681_arg_min_max_if_fix
2022-02-18 18:35:52 +01:00
Vladimir C
9b7d011ee7
Merge pull request #34529 from vdimir/join-nullable-on-pipeline
Apply join_use_nulls on types before join
2022-02-18 18:34:44 +01:00
alexey-milovidov
b9a377c890
Update build.md 2022-02-18 20:01:44 +03:00
Maksim Kita
80b0efb367 Performance tests fix H3 2022-02-18 15:57:54 +00:00
Maksim Kita
f3dca476ae Updated documentation 2022-02-18 15:35:18 +00:00
Maksim Kita
0f9f30cc9e Added tests 2022-02-18 15:21:11 +00:00
Vladimir C
a249819c52
Merge pull request #34730 from ucasfl/name
rename name in ci check
2022-02-18 16:17:11 +01:00
Maksim Kita
187af5b8b9
Merge pull request #34699 from DevTeamBK/missing_noexcept
Adding noexcept for move constructor SharedContextHolder
2022-02-18 16:15:06 +01:00
vdimir
963e859f92
Ignore result in 01681_arg_min_max_if_fix 2022-02-18 14:10:06 +00:00
vdimir
7012489d97
Revert "Check float to integer overflow on aarch64"
This reverts commit 8a7100701b.
2022-02-18 14:07:35 +00:00
feng lv
3270e5f636 rename name in ci check 2022-02-18 14:06:16 +00:00
vdimir
6ca154858c
upd TableJoin::applyKeyConvertToTable 2022-02-18 13:40:57 +00:00
Azat Khuzhin
65e9b4879d Fix possible memory_tracker use-after-free for merges/mutations
There are two possible cases for execution merges/mutations:
1) from background thread
2) from OPTIMIZE TABLE query

1) is pretty simple, it's memory tracking structure is as follow:

    current_thread::memory_tracker = level=Thread / description="(for thread)" ==
      background_thread_memory_tracker = level=Thread / description="(for thread)"
    current_thread::memory_tracker.parent = level=Global / description="(total)"

  So as you can see it is pretty simple and MemoryTrackerThreadSwitcher
  does not do anything icky for this case.

2) is complex, it's memory tracking structure is as follow:

    current_thread::memory_tracker = level=Thread / description="(for thread)"
    current_thread::memory_tracker.parent = level=Process / description="(for query)" ==
      background_thread_memory_tracker = level=Process / description="(for query)"

  Before this patch to track memory (and related things, like sampling,
  profiling and so on) for OPTIMIZE TABLE query dirty hacks was done to
  do this, since current_thread memory_tracker was of Thread scope, that
  does not have any limits.

  And so if will change parent for it to Merge/Mutate memory tracker
  (which also does not have some of settings) it will not be correctly
  tracked.

  To address this Merge/Mutate was set as parent not to the
  current_thread memory_tracker but to it's parent, since it's scope is
  Process with all settings.

  But that parent's memory_tracker is the memory_tracker of the
  thread_group, and so if you will have nested ThreadPool inside
  merge/mutate (this is the case for s3 async writes, which has been
  added in #33291) you may get use-after-free of memory_tracker.

  Consider the following example:

    MemoryTrackerThreadSwitcher()
      thread_group.memory_tracker.parent = merge_list_entry->memory_tracker
      (see also background_thread_memory_tracker above)

    CurrentThread::attachTo()
      current_thread.memory_tracker.parent = thread_group.memory_tracker

    CurrentThread::detachQuery()
      current_thread.memory_tracker.parent = thread_group.memory_tracker.parent
      # and this is equal to merge_list_entry->memory_tracker

    ~MemoryTrackerThreadSwitcher()
      thread_group.memory_tracker = thread_group.memory_tracker.parent

  So after the following we will get incorrect memory_tracker (from the
  mege_list_entry) when the next job in that ThreadPool will not have
  thread_group, since in this case it will not try to update the
  current_thread.memory_tracker.parent and use-after-free will happens.

So to address the (2) issue, settings from the parent memory_tracker
should be copied to the merge_list_entry->memory_tracker, to avoid
playing with parent memory tracker.

Note, that settings from the query (OPTIMIZE TABLE) is not available at
that time, so it cannot be used (instead of parent's memory tracker
settings).

v2: remove memory_tracker.setOrRaiseHardLimit() from settings

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-18 16:23:54 +03:00
vdimir
6eda8ef914
use isNullable function in join_common/convertTypeToNullable 2022-02-18 13:09:11 +00:00
vdimir
4ce64891f9
Add taiyang-li to trusted contributors 2022-02-18 13:59:47 +01:00
Amos Bird
f459e8fc95
Less getMark calls 2022-02-18 19:55:19 +08:00
Jakub Kuklis
0aa21a2133 Document behaviour of quantile variants for empty numeric sequences 2022-02-18 11:48:59 +01:00
Maksim Kita
652980eec3
Merge pull request #34714 from azat/fix-custom_partitioning_local
Fix flaky 00502_custom_partitioning_local
2022-02-18 11:27:08 +01:00
Amos Bird
56197cf31e
Allow only to specify --log-file 2022-02-18 18:14:58 +08:00
Nikolay Degterinsky
4b316b5233
Merge pull request #34716 from cnmade/PR202202181640
Translate zh/sql-reference/statements/alter/order-by
2022-02-18 12:35:54 +03:00
Azat Khuzhin
c204ac7744 Export MemoryTracker::getHardLimit()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-02-18 12:05:12 +03:00
cnmade
658a23af50 Translate zh/sql-reference/statements/alter/order-by: remove space 2022-02-18 16:53:16 +08:00
cnmade
4d913f328b Translate zh/sql-reference/statements/alter/order-by: remove bak file 2022-02-18 16:50:58 +08:00
cnmade
3643e9e397 Translate zh/sql-reference/statements/alter/order-by: translate to zh 2022-02-18 16:50:19 +08:00
cnmade
9d300809a4 Translate zh/sql-reference/statements/alter/order-by: reimport old file 2022-02-18 16:43:16 +08:00
cnmade
1c92f4f724 Translate zh/sql-reference/statements/alter/order-by: rename old file 2022-02-18 16:40:16 +08:00
Cody Baker
cc6ef4f4b3
Add new team members to company page (#34701) 2022-02-18 11:21:00 +03:00
Nikolay Degterinsky
ff52ecea8d
Merge pull request #34706 from cnmade/PR202202181040
Translate zh/sql-reference/statements/alter/delete
2022-02-18 10:43:44 +03:00
Nikolay Degterinsky
4346404ae5
Merge pull request #34705 from cnmade/PR202202181030
sync translate to zh: engines/table-engines/special/memory: sync inde…
2022-02-18 10:41:02 +03:00
alesapin
efb9a6d0fa
Merge pull request #34584 from bigo-sg/keerpSnapMemOpt
Keeper memory optimization by not holding snapshot in memory
2022-02-18 10:34:31 +03:00
Nikolay Degterinsky
c09275f0da
Merge pull request #34368 from usurai/sw
Add startsWith & endsWith function for arrays
2022-02-18 08:34:48 +03:00
cnmade
df14daaba6 Translate zh/sql-reference/statements/alter/delete: delete old file 2022-02-18 10:40:31 +08:00