thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
93,282 Commits 9,326 Branches 1,784 Tags 2.1 GiB
a8da5d96fc
Commit Graph

3777 Commits

Author SHA1 Message Date
Nikolai Kochetov
147a819221 Refactor a little bit more. 2022-05-31 14:43:38 +00:00
yaqi-zhao
a2857491c4 add avx512 support for mergetreereader 2022-05-30 20:53:00 -04:00
mergify[bot]
55913cf8e1
Merge branch 'master' into turn_on_s3_tests 2022-05-30 20:52:40 +00:00
HeenaBansal2009
b7eb6bbd38 Fixed clang-tidy-CheckTriviallyCopyableMove-errors 2022-05-30 11:09:03 -07:00
Alexander Gololobov
e2dd6f6249 Removed prewhere_info.alias_actions 2022-05-30 19:58:23 +02:00
Nikolai Kochetov
5b4658aa5e Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-30 09:47:35 +00:00
alesapin
c32b6076fb Remove stranges from code 2022-05-30 00:12:33 +02:00
alesapin
0e8ab36913 Merge branch 'master' into turn_on_s3_tests 2022-05-29 14:37:10 +02:00
alesapin
d2cdbf3956 Fix refactoring issue 2022-05-29 14:09:49 +02:00
alesapin
9dc81e5cc8
Merge pull request #37598 from ClickHouse/revert-37545-revert-37424-fix_fetching_part_deadlock 
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part""
 2022-05-29 13:54:34 +02:00
Alexey Milovidov
97606c324c
Merge pull request #37574 from azat/mt-tiny-refactor 
Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm()
 2022-05-29 07:59:57 +03:00
Alexey Milovidov
73e2e63414
Merge pull request #37612 from ClickHouse/clang-tidy-14 
Fix clang-tidy-14, part 1
 2022-05-29 03:16:32 +03:00
Anton Popov
1d9b3be7da
Merge pull request #37536 from CurtizJ/profile-events-for-part-types 
Add profile events for introspection of part types
 2022-05-28 14:25:21 +02:00
Alexey Milovidov
c50791dd3b Fix clang-tidy-14, part 1 2022-05-27 22:52:14 +02:00
mergify[bot]
f5ee337bab
Merge branch 'master' into revert-37545-revert-37424-fix_fetching_part_deadlock 2022-05-27 16:52:00 +00:00
alesapin
5a296aec01 Fix build 2022-05-27 16:34:16 +02:00
alesapin
be1c3c132b Fix some trash 2022-05-27 16:08:49 +02:00
alesapin
6d6779f17a
Merge pull request #37139 from ClickHouse/i_object_storage 
Separate object storage operations from disks
 2022-05-27 13:59:50 +02:00
alesapin
c79600c4c8 Fix build 2022-05-27 13:44:29 +02:00
alesapin
841858ec30
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part"" 2022-05-27 13:13:36 +02:00
Azat Khuzhin
c6c60364ae Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm() 
In favor of
MergeTask::ExecuteAndFinalizeHorizontalPart::chooseMergeAlgorithm()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 21:20:50 +03:00
Azat Khuzhin
73a99d4eee Improve error message for skipped/expired columns 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
f0cac5417a More optional skipping of fully expired columns (by TTL) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
46a94f395d Move the check into IMergedBlockOutputStream::removeEmptyColumnsFromPart() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
4288d09a85 Do not write expired columns by TTL after merge w/o TTL 
Usually second merge do not perform TTL, since everything is up to date,
however in this case TTLTransform is not used, and hence expired_columns
will not be filled for new part, and so those columns will be written
with default values.

Avoid this, by manually filling expired_columns.

Here is a simpler reproducer:

Simple reproducer:

    ```sql
    create table ttl_02262 (date Date, key Int, value String TTL date + interval 1 month) engine=MergeTree order by key settings min_bytes_for_wide_part=0, min_rows_for_wide_part=0;
    insert into ttl_02262 values ('2010-01-01', 2010, 'foo');
    ```

    ```sh
    # ls -l .server/data/default/ttl_02262/all_*
    .server/data/default/ttl_02262/all_1_1_0:
    total 48
    -rw-r----- 1 root root 335 May 26 14:19 checksums.txt
    -rw-r----- 1 root root  76 May 26 14:19 columns.txt
    -rw-r----- 1 root root   1 May 26 14:19 count.txt
    -rw-r----- 1 root root  28 May 26 14:19 date.bin
    -rw-r----- 1 root root  48 May 26 14:19 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:19 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:19 key.bin
    -rw-r----- 1 root root  48 May 26 14:19 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:19 primary.idx
    -rw-r----- 1 root root  99 May 26 14:19 ttl.txt
    -rw-r----- 1 root root  30 May 26 14:19 value.bin
    -rw-r----- 1 root root  48 May 26 14:19 value.mrk2
    ```

    ```sql
    optimize table ttl_02262 final;
    ```

    ```sh
    .server/data/default/ttl_02262/all_1_1_1:
    total 40
    -rw-r----- 1 root root 279 May 26 14:19 checksums.txt
    -rw-r----- 1 root root  61 May 26 14:19 columns.txt
    -rw-r----- 1 root root   1 May 26 14:19 count.txt
    -rw-r----- 1 root root  28 May 26 14:19 date.bin
    -rw-r----- 1 root root  48 May 26 14:19 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:19 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:19 key.bin
    -rw-r----- 1 root root  48 May 26 14:19 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:19 primary.idx
    -rw-r----- 1 root root  81 May 26 14:19 ttl.txt
    ```

    ```sql
    optimize table ttl_02262 final;
    ```

    ```sh
    .server/data/default/ttl_02262/all_1_1_2:
    total 48
    -rw-r----- 1 root root 349 May 26 14:20 checksums.txt
    -rw-r----- 1 root root  76 May 26 14:20 columns.txt
    -rw-r----- 1 root root   1 May 26 14:20 count.txt
    -rw-r----- 1 root root  28 May 26 14:20 date.bin
    -rw-r----- 1 root root  48 May 26 14:20 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:20 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:20 key.bin
    -rw-r----- 1 root root  48 May 26 14:20 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:20 primary.idx
    -rw-r----- 1 root root  81 May 26 14:20 ttl.txt
    -rw-r----- 1 root root  27 May 26 14:20 value.bin
    -rw-r----- 1 root root  48 May 26 14:20 value.mrk2
    ```

And now we have `value.*` for all_1_1_2, this should not happen.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
8328d7068b Fix updating of MergeTreeDataPartTTLInfo::finished 
Previously you cannot distinguish non-initialized finished with
initialized to false, so update() cannot do the correct thing.

Rename the field to avoid hidden usage.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
0de1a64436 Log empty parts in IMergedBlockOutputStream::removeEmptyColumnsFromPart() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:09 +03:00
Azat Khuzhin
dc9ca3d70c
Fix LOGICAL_ERROR in getMaxSourcePartsSizeForMerge during merges (#37413) 2022-05-26 14:14:58 +02:00
Nikolai Kochetov
84f97b53de Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-26 11:07:45 +00:00
Anton Popov
f488efd27e fix tests 2022-05-26 00:03:31 +00:00
alesapin
8f1aac0ce4 Fix merge with master 2022-05-26 00:44:45 +02:00
alesapin
1db9cf480b Merge remote-tracking branch 'origin/master' into i_object_storage 2022-05-25 22:50:22 +02:00
Alexander Tokmakov
47820c216d
Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part" 2022-05-25 23:10:33 +03:00
alesapin
c7b16065e1 Merge with master 2022-05-25 21:47:05 +02:00
Nikolai Kochetov
6d4a26afac Update ReadProgressCallback. 2022-05-25 19:45:48 +00:00
alesapin
6f5c86e55e Merge branch 'master' into i_object_storage 2022-05-25 20:49:01 +02:00
alesapin
51868a9a4f
Merge pull request #37424 from metahys/fix_fetching_part_deadlock 
(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part
 2022-05-25 20:15:41 +02:00
alesapin
0a3597da72
Merge pull request #34915 from ianton-ru/MDB-16962 
Fix collision of S3 operation log revision
 2022-05-25 18:15:31 +02:00
Anton Popov
16e839ac71 add profile events for introspection of part types 2022-05-25 14:54:49 +00:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
mergify[bot]
73662b4436
Merge branch 'master' into fix_fetching_part_deadlock 2022-05-25 14:22:35 +00:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
alesapin
9a19309e69 Slightly better fix 2022-05-24 14:46:29 +02:00
alesapin
3ca7a8831b Revert "fix deadlock during fetching part" 
This reverts commit 6ae8a26fae.
 2022-05-24 14:26:06 +02:00
alesapin
a5ba6bca95 Merge branch 'master' into metahys-fix_fetching_part_deadlock 2022-05-24 14:07:47 +02:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Alexander Tokmakov
6bc68c0cbc Merge branch 'master' into fixes_for_transactions 2022-05-23 18:49:21 +02:00
metahys
6ae8a26fae fix deadlock during fetching part 2022-05-23 20:08:04 +08:00
mergify[bot]
747aa5575c
Merge branch 'master' into remove-useless-code-2 2022-05-22 17:41:57 +00:00
alesapin
c8d92b87c8 Merge branch 'master' into i_object_storage 2022-05-22 12:16:10 +02:00
Robert Schulze
19d53c14fa
Merge pull request #37382 from ClickHouse/wc++98-compat-extra-semi 
Enable -Wc++98-compat-extra-semi
 2022-05-22 09:40:45 +02:00
Alexander Tokmakov
44f2d4529a better waiting, add fault injection 2022-05-20 22:10:01 +02:00
Nikolai Kochetov
56feef01e7 Move some resources 2022-05-20 19:49:31 +00:00
Anton Popov
cb0e6c2718 mark all operators bool() as explicit 2022-05-20 15:29:54 +00:00
Robert Schulze
0f6715bd91
Follow-up to PR #37300: semicolon warnings 
In PR #37300, Alexej asked why we the compiler does not warn about
unnecessary semicolons, e.g.

  f()
  {
  }; // <-- here

The answer is surprising: In C++98, above syntax was disallowed but by
most compilers accepted it regardless. C++>11 introduced "empty
declarations" which made the syntax legal.

The previous behavior can be restored using flag
-Wc++98-compat-extra-semi. This finds many useless semicolons which were
removed in this change. Unfortunately, there are also false positives
which would require #pragma-s and HAS_* logic (--> check_flags.cmake) to
suppress. In the end, -Wc++98-compat-extra-semi comes with extra effort
for little benefit. Therefore, this change only fixes some semicolons
but does not enable the flag.
 2022-05-20 15:06:34 +02:00
Alexander Tokmakov
12bbb7de87 fix race on TID allocation 2022-05-20 12:41:44 +02:00
alesapin
654b27e307 Merge branch 'master' into i_object_storage 2022-05-20 11:56:13 +02:00
Robert Schulze
b475fbc9a7
Merge pull request #37300 from ClickHouse/cmake-cleanup-pt3 
Various cmake cleanups
 2022-05-20 10:02:36 +02:00
alesapin
92c15ec97c Get rid of IDiskRemote 2022-05-19 20:07:15 +02:00
Alexander Tokmakov
f787dc7097
Revert "Fix mutations in tables with columns of type Object" 2022-05-19 13:24:48 +03:00
Vitaly Baranov
7c8956ca8d
Merge pull request #36879 from tonickkozlov/tonickkozlov/lazy-marks-loading 
Lazy marks loading
 2022-05-18 22:56:46 +02:00
alesapin
e99c90118e Merge branch 'master' into i_object_storage 2022-05-18 12:36:22 +02:00
Kruglov Pavel
aaace46da2
Merge pull request #37266 from CurtizJ/fix-mutations-with-object 
Fix mutations in tables with columns of type `Object`
 2022-05-18 12:19:55 +02:00
alesapin
4d4634b12d
Merge pull request #37164 from kssenii/ficx-async-reads-assertions 
fix async reads from remote fs internal setting not being always turned on
 2022-05-17 11:29:14 +02:00
Robert Schulze
43945cea1b
Fixing some warnings 2022-05-16 20:59:27 +02:00
Anton Popov
b6c5ab4fcf fix mutations in tables with columns of type Object 2022-05-16 18:26:53 +00:00
alesapin
ce6d5f7dcc Merge branch 'master' into i_object_storage 2022-05-16 17:42:22 +02:00
kssenii
f7493965b7 Remove additional check for merge tree 2022-05-16 11:19:57 +02:00
kssenii
49201f217f Revert 2022-05-16 11:17:56 +02:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies 2022-05-16 10:12:50 +02:00
Robert Schulze
859f2f3391
Enclose large if block in {} 2022-05-15 14:49:20 +02:00
alesapin
cc304a3178 Merge branch 'master' into i_object_storage 2022-05-13 23:44:00 +02:00
alesapin
1c082717a4 Relax log level for some checks in check thread 2022-05-13 23:29:36 +02:00
Kseniia Sumarokova
0f07c41f19
Update MergeTreeIOSettings.h 2022-05-13 13:26:47 +02:00
kssenii
cde5cecc4d One more 2022-05-12 23:57:25 +02:00
kssenii
e5466a40fd Make sure async reads from remote fs related internal setting is turned on when needed 2022-05-12 19:00:19 +02:00
Dmitry Novik
df071bb701
Merge pull request #37129 from ClickHouse/enable-overcommit-for-user 
Update documentation and defaults for memory overcommit
 2022-05-12 14:38:43 +02:00
alesapin
5dcd25be23 Initial implementation 2022-05-12 00:04:54 +02:00
Maksim Kita
437d70d4da Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
4e7d10297b Fixed style 2022-05-11 21:59:51 +02:00
Maksim Kita
ce92f6aab1 Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
8ceb63ee6c Added JIT compilation of SortDescription 2022-05-11 21:59:51 +02:00
Dmitry Novik
17608b3d93 Update documentation and defaults for memory overcommit 2022-05-11 16:18:41 +00:00
Anton Kozlov
23e6792898 Lazy marks loading 2022-05-11 10:15:56 +00:00
Anton Popov
aec30c4076
Merge pull request #37053 from CurtizJ/remove-streams-comments 
Remove last mentions of data streams
 2022-05-10 13:38:13 +02:00
Anton Popov
e911900054 remove last mentions of data streams 2022-05-09 19:15:24 +00:00
Anton Ivashkin
6a6505c224 Merge master 2022-05-09 17:21:34 +03:00
alesapin
28e492bc17 Followup fix 2022-05-09 15:21:33 +02:00
alesapin
018ed10684 Add test 2022-05-09 15:21:21 +02:00
Robert Schulze
61cbcbf073
Enable clang-tidy readability-misleading-indentation 
Official docs:

  Correct indentation helps to understand code. Mismatch of the
  syntactical structure and the indentation of the code may hide serious
  problems.
 2022-05-08 19:12:01 +02:00
alesapin
2f650366a6 Merge branch 'master' into better_delete_outdated_parts 2022-05-08 12:55:42 +02:00
alesapin
46712f1d98 Fix forgotten parts in cleanup thread 2022-05-08 00:53:55 +02:00
Alexey Milovidov
fdbb5b75b2
Merge branch 'master' into minor-renames 2022-05-07 14:18:50 +03:00
Alexey Milovidov
4c6f964421
Merge pull request #36909 from ClickHouse/fixes_for_replicated_merge_tree 
Some fixes for replicated merge tree
 2022-05-06 00:44:21 +03:00
Alexander Tokmakov
45284108ba fix race with check thread 2022-05-05 14:16:26 +02:00
Nikolai Kochetov
d6735f06c7 Fixing spaces. 2022-05-05 09:24:34 +00:00
Nikolai Kochetov
5574a3cc43 Fixing style. 2022-05-05 09:23:23 +00:00
Nikolai Kochetov
b3d38301c9 Fix some more tests. 2022-05-05 09:19:12 +00:00
taiyang-li
d673bd096e fix issue: https://github.com/ClickHouse/ClickHouse/issues/36671 2022-05-05 14:48:33 +08:00
Robert Schulze
c48d9a4174
Merge pull request #36844 from rschu1ze/shared_ptr_helper3 
Remove inherited create() method + disallow copying of storages
 2022-05-04 19:16:30 +02:00
Alexander Tokmakov
af9b4a5b9c fix intersecting parts 2022-05-04 16:22:06 +02:00
Antonio Andelic
808ea6dc65
Merge pull request #36888 from amosbird/projection-fix-one 
Fix incorrect query result when doing constant aggregation
 2022-05-04 10:34:40 +02:00
mergify[bot]
64084b5e32
Merge branch 'master' into shared_ptr_helper3 2022-05-03 20:46:16 +00:00
Nikolai Kochetov
fc3645a483 Fix some tests. 2022-05-03 19:32:24 +00:00
Amos Bird
7febf57439
Fix minmax_count_projection constant aggregation 2022-05-04 03:07:59 +08:00
Nikolai Kochetov
49e613a666 Fixing build. 2022-05-03 17:58:47 +00:00
Sergei Trifonov
1bcae160a0
Update MergeTreeBackgroundExecutor.h 2022-05-03 18:50:05 +02:00
Nikolai Kochetov
35095191eb Merge branch 'master' into refactor-something-in-part-volumes 2022-05-03 17:51:47 +02:00
Nikolai Kochetov
4a20e4f37e Fixing build. 2022-05-03 15:48:05 +00:00
Alexander Gololobov
367898da5e
Merge pull request #35318 from weeds085490/feat/add_part_offset 
feat(...): [LWD] support getting _part_offset of a row
 2022-05-03 13:27:02 +02:00
Nikolai Kochetov
540fafbe74
Merge pull request #36856 from ClickHouse/repro-lc-and-s3 
Reproduce and a little bit better fix for LC dict right offset.
 2022-05-03 11:32:08 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable 
... IStorage has deleted copy ctor / assignment already
 2022-05-03 09:07:08 +02:00
Nikolai Kochetov
b10decc987 Fix style and formatting. 2022-05-02 18:59:43 +00:00
Nikolai Kochetov
3d9b87af09 Reproduce and a little bit better fix for LC dict right offset. 2022-05-02 18:06:47 +00:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Kseniia Sumarokova
66f62b2ba6
Merge pull request #36632 from DevTeamBK/Fix-Coverity-Issues 
Minor Coverity defects fixes
 2022-04-30 15:11:40 +02:00
mergify[bot]
cc08ccb420
Merge branch 'master' into remove-useless-code-2 2022-04-30 12:48:15 +00:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase 
Reorganize source files so that base won't depend on Common
 2022-04-30 09:40:58 +03:00
Nikolai Kochetov
5807ca47b0
Merge pull request #36425 from ClickHouse/pool-size-hot-reload 
Allow to increase number of threads and tasks for background executors
 2022-04-29 17:24:12 +02:00
HeenaBansal2009
631b0154c0 Reverted change from ReadBufferFromRemoteFSGather.cpp 2022-04-29 07:21:35 -07:00
Anton Popov
19428353e7
Merge pull request #35797 from azat/nested-columns-fixes 
RFC: Fix ALTER DROP COLUMN of nested column with compact parts
 2022-04-29 15:09:05 +02:00
alesapin
c72596bae6
Merge pull request #36763 from kssenii/fix-async-reads 
Fix some issues with low cardinality and async reads from remote fs
 2022-04-29 12:18:19 +02:00
Alexey Milovidov
9fb1d92ff5
Merge pull request #36707 from CurtizJ/fix-vertical-merges 
Fix vertical merges in wide parts
 2022-04-29 10:09:50 +03:00
Azat Khuzhin
bf312c2a5b Fix ALTER DROP COLUMN of nested column with compact parts 
ALTER DROP COLUMN of nested column did not requires mutation before, and
so it leaves nested column as-is, and in case of compact parts
subsequent alter, that requires mutation, will trigger READ_COLUMN of
that nested column (because it exists in part), but it will fail because
there is no such column in the table already.

Here is example of such a failure on CI - [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/35459/52099b23a1cb9a7ff036c5c60aa037c999b333ef/stateless_tests__thread__actions__[1/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 07:12:34 +03:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
kssenii
84490d5198 Fix 2022-04-28 18:33:25 +02:00
HeenaBansal2009
3ac9434c57 Merge branch 'master' into Fix-Coverity-Issues 2022-04-28 08:14:50 -07:00
alesapin
af980ef4ef
Merge pull request #36371 from Joeywzr/add_query_level_settingss 
Add query level settings: parts_to_delay_insert and parts_to_throw_insert
 2022-04-28 11:08:14 +02:00
Anton Popov
eb810116a4 fix vertical merges in wide parts 2022-04-27 13:42:47 +00:00
Nikolai Kochetov
e44af67fee Merge branch 'master' into refactor-something-in-part-volumes 2022-04-26 21:08:00 +02:00
HeenaBansal2009
4961caec13 Merge branch 'master' into Fix-Coverity-Issues 2022-04-26 11:43:44 -07:00
Alexander Tokmakov
b1949ee901
Merge pull request #36680 from ClickHouse/better_logs_virtual_parts 
Better logs for virtual parts
 2022-04-26 21:36:20 +03:00
Alexander Tokmakov
d149ac8bd7 better logs for virtual parts 2022-04-26 16:58:40 +02:00
HeenaBansal2009
13a9a8fd5e Style check fix 2022-04-26 07:13:09 -07:00
HeenaBansal2009
267f426605 Minor Coverity defetcs fixes 2022-04-26 07:13:09 -07:00
Alexander Tokmakov
190becdcc3
Merge pull request #36644 from ClickHouse/fix_some_flaky_tests 
Fix some flaky tests
 2022-04-26 13:40:14 +03:00
Anton Popov
e051751012
Merge pull request #36637 from CurtizJ/dynamic-columns-11 
Fix merges of wide parts with type `Object`
 2022-04-26 10:34:04 +02:00
Alexey Milovidov
7683901689
Merge pull request #36630 from azat/mutator-cleanup 
Tiny Mutator code cleanup
 2022-04-26 08:24:50 +03:00
Memo
c38a4b4255
Update src/Storages/MergeTree/MergeTreeData.h 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-04-26 10:26:18 +08:00
mergify[bot]
a5aab53b70
Merge branch 'master' into add_query_level_settingss 2022-04-25 21:41:36 +00:00
mergify[bot]
178f244c8b
Merge branch 'master' into dynamic-columns-11 2022-04-25 21:36:56 +00:00
alesapin
92296484e7
Merge pull request #36348 from rschu1ze/erase_if3 
Replace remove-erase idiom by C++20 erase()/erase_if()
 2022-04-25 23:34:18 +02:00
Alexander Tokmakov
19a376b9f7 fix assertions 2022-04-25 22:41:46 +02:00
Anton Popov
d28b1559e6 fix merges of wide parts with type Object 2022-04-25 16:44:24 +00:00
mergify[bot]
265398d1b6
Merge branch 'master' into feat/add_part_offset 2022-04-25 15:58:16 +00:00
Azat Khuzhin
8b544e26d3 Move some functions from MergeTreeDataMergerMutator to MutateTask 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-25 18:16:38 +03:00
Azat Khuzhin
615d35e233 Remove dead function MergeTreeDataMergerMutator::splitMutationCommands() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-25 18:16:38 +03:00
Azat Khuzhin
8440fec3db Fix coding alignment in MutateTask 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-25 18:16:38 +03:00
Nikolai Kochetov
0fa63a8d65
Merge pull request #36549 from ClickHouse/interpolate-alias-fix 
Avoid projections if INTERPOLATE with expressions is present
 2022-04-25 13:18:00 +02:00
Alexander Gololobov
c484a1c269 Preserve order of virtual columns in the header and in data chunks when adding _part_offset together with other virtual columns 2022-04-25 12:10:56 +02:00
alesapin
edc78247f0
Merge pull request #36299 from ClickHouse/fix_trash 
Trying to fix some trash in zero copy replication
 2022-04-24 13:19:43 +02:00
Nikolai Kochetov
e7e6125b46 More fixes 2022-04-22 20:38:18 +00:00
alesapin
f183eabaa2 Return normal settings 2022-04-22 22:36:55 +02:00
Nikolai Kochetov
1805d60917 Fix mark cache. 2022-04-22 20:29:14 +00:00
alesapin
ebdd53ef8f Fix shared fetch 2022-04-22 19:18:18 +02:00
Nikolai Kochetov
8c00692844 Part 8 2022-04-22 16:58:09 +00:00
Nikita Mikhaylov
e475849761 Style and Docs update 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
bf6d3af536 Better 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
31ccb9c1c3 Done 2022-04-22 13:56:16 +00:00
Yakov Olkhovskiy
e4af79c2a0 avoid projections if INTERPOLATE with expressions is present 2022-04-22 09:47:59 -04:00
alesapin
64383bcb0a Better 2022-04-22 15:35:25 +02:00
xiedeyantu
550460c541 fix log should print from path 2022-04-22 18:30:40 +08:00
mergify[bot]
cd8fa75546
Merge branch 'master' into fix_trash 2022-04-21 21:45:31 +00:00
Nikolai Kochetov
9133e398b8 Part 7 2022-04-21 19:19:13 +00:00
Maksim Kita
57444fc7d3
Merge pull request #36444 from rschu1ze/clang-tidy-fixes 
Clang tidy fixes
 2022-04-21 16:11:27 +02:00
Dmitry Novik
77a82cc090
Merge pull request #35631 from amosbird/projection-fix1 
Fix broken SET reuse during projection analysis.
 2022-04-21 15:32:52 +02:00
alesapin
5465415751 Fix replace/move partition with zero copy replication 2022-04-21 14:39:12 +02:00
alesapin
cbfe00bb47 Fix tidy 2022-04-21 00:30:13 +02:00
alesapin
c14e2e0b96 Fix more 2022-04-20 21:08:26 +02:00
alesapin
6ee0df251f Add ditry settings 2022-04-20 15:11:03 +02:00
alesapin
41b57f78cf Revert extreme settings 2022-04-20 13:59:12 +02:00
alesapin
5334522d5c Extreme settings 2022-04-20 13:33:42 +02:00
alesapin
2eee79dc2a Review fixes 2022-04-20 13:26:20 +02:00
alesapin
40c15222f8 Merge branch 'master' into fix_trash 2022-04-20 12:45:49 +02:00
Antonio Andelic
d77678d1d1
Merge pull request #36410 from xinhuitian/move_down_data_range_construction 
Move down data_range construction in filterMarksUsingIndex and filterMarksUsingMergedIndex
 2022-04-20 12:18:05 +02:00
Alexander Tokmakov
1d30a97fd2 Merge branch 'master' into remove-useless-code-2 2022-04-20 11:45:56 +02:00
tavplubix
2c54381520
Merge pull request #36200 from ClickHouse/fix_create_as_replicated_default_args 
Fix CREATE AS for ReplicatedMergeTree with default arguments
 2022-04-20 12:01:13 +03:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings 
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
 2022-04-20 10:29:05 +02:00
Robert Schulze
b6d7367538
Merge remote-tracking branch 'origin/master' into erase_if3 
Conflicts:
- Interpreters/ActionsDAG.cpp
 2022-04-20 10:02:59 +02:00
Antonio Andelic
bbb0be6a44
Merge pull request #36402 from rschu1ze/clang-tidy-contains 
Activate clang-tidy warning "readability-container-contains"
 2022-04-20 08:15:46 +02:00
Nikolai Kochetov
bcbab2ead8 Part 6. 2022-04-19 19:34:41 +00:00
alesapin
aea7c7755a Fix style 2022-04-19 17:25:08 +02:00
alesapin
7cb7c120cc Less ugly 2022-04-19 15:53:10 +02:00
alesapin
cc06bc3d99 Add some clarifications 2022-04-19 14:01:30 +02:00
alesapin
ee8d26ff0e Better test and fix for move 2022-04-19 13:27:55 +02:00
alesapin
8ade9fba52 Fix 2022-04-19 11:56:14 +02:00
xinhuitian
5d0957e669 move down data range construction in filterMarksUsingIndex and filterMarksUsingMergedIndex 2022-04-19 13:26:33 +08:00
alesapin
bd7b3847c1 Some code 2022-04-19 01:09:09 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
mergify[bot]
daca157111
Merge branch 'master' into fix_storage_distributed_ttl 2022-04-18 17:31:59 +00:00
Memo
a17f102fe6 fix style 2022-04-18 20:31:40 +08:00
alesapin
a97754f462 Fix 2022-04-18 13:39:09 +02:00
alesapin
c4b70f3ed3 Merge branch 'master' into fix_trash 2022-04-18 12:20:09 +02:00
Memo
b3adf150b5 add_query_level_settings 2022-04-18 12:15:41 +08:00
Alexey Milovidov
36595e4206
Merge pull request #36320 from ClickHouse/fix-clang-tidy-14 
Fix clang-tidy-14 (part 1)
 2022-04-18 07:02:10 +03:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
Robert Schulze
1e1df8e101
Replace remove-erase idiom by C++20 erase()/erase_if() 
- makes the code less verbose while being as efficient
 2022-04-17 12:04:47 +02:00
Alexey Milovidov
294efeccfe Fix clang-tidy-14 (part 1) 2022-04-16 04:54:04 +02:00
alesapin
1706ae9e15 Some trash implementation 2022-04-15 18:36:23 +02:00
alesapin
5a8419a48e Remove more trash 2022-04-15 17:05:17 +02:00
alesapin
eb7593f786 Remove more trash 2022-04-15 16:24:38 +02:00
Alexander Tokmakov
4ce0b280e6 wait for node to disappear instead of removing 2022-04-14 15:31:05 +02:00
Anton Popov
305dd57262
Merge branch 'master' into fix_storage_distributed_ttl 2022-04-14 14:51:15 +02:00
tavplubix
3af2384066
Merge pull request #36190 from azat/revert-27002-mutation-stuck 
Revert "Fix possible mutation stuck due to race with DROP_RANGE"
 2022-04-14 15:03:35 +03:00
Alexander Tokmakov
66fdf35dfd remove outdated parts immediately on drop partition 2022-04-13 18:01:22 +02:00
Alexander Tokmakov
40fea2b833 do not unfold {uuid} macro 2022-04-13 16:51:59 +02:00
Azat Khuzhin
6dbc138e76 Remove duplicated parts removal from mutation. 
This commit reverts 701e2ffd79 ("Fix
possible mutation stuck due to race with DROP_RANGE"), and also adds
some clarification comments.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-13 16:09:17 +03:00
Nikolai Kochetov
76870ad92a Part 5 2022-04-12 18:59:49 +00:00
mergify[bot]
0b3c15c07a
Merge branch 'master' into projection-fix1 2022-04-12 13:49:28 +00:00
Alexander Tokmakov
457a9e9691 fixes for ReplicatedMergeTree 2022-04-12 14:14:26 +02:00
Alexey Milovidov
a54c01cf72 Remove useless code in ReplicatedMergeTreeRestartingThread 2022-04-11 00:44:30 +02:00
Denny Crane
383076ca0e
fix typo 2022-04-10 16:42:43 -03:00
Nikolai Kochetov
7c4b288638 Part 4 2022-04-08 18:56:08 +00:00
Alexander Tokmakov
141fbc6980 Merge branch 'master' into mvcc_prototype 2022-04-08 13:38:11 +02:00
Alexander Tokmakov
49c35f3261 Merge branch 'master' into mvcc_prototype 2022-04-08 13:34:40 +02:00
Anton Popov
a16ac67c0c
Merge pull request #36034 from amosbird/better-projection2 
Use storage_snapshot for projection analysis
 2022-04-08 13:22:53 +02:00
alesapin
8ec802bc62
Merge pull request #35475 from kssenii/remote-fs-cache-improvements 
Allow to write remote fs cache on all write operations. Add `system.remote_filesystem_cache` table. Add `drop remote filesystem cache (<path>)` query. Add `system.remote_data_paths` table.
 2022-04-08 12:06:26 +02:00
Alexander Tokmakov
6a46da93ae Merge branch 'master' into mvcc_prototype 2022-04-07 23:22:19 +02:00
Nikolai Kochetov
bc3e1ec3f2 Part 3 2022-04-07 17:44:49 +00:00
kssenii
5dce2f18b5 Better 2022-04-07 18:48:35 +02:00
Nikolai Kochetov
5cbec37907 Part 2 2022-04-07 11:58:38 +00:00
Amos Bird
a071bde59e
Use storage_snapshot for projection analysis 2022-04-07 19:56:31 +08:00
Alexander Tokmakov
8290ffa88d Merge branch 'master' into mvcc_prototype 2022-04-07 13:50:42 +02:00
Amos Bird
1238babb6f
Make SelectQueryInfo pseudo-copyable 2022-04-07 17:46:50 +08:00
Amos Bird
9cf5935604
Fix broken SET reuse during projection analysis. 2022-04-07 17:46:49 +08:00
jewisliu
77edd41b2e [Improvement] improvement in PARTITION ALL 
1. ASTPartition::formatImpl should output ALL while executing ALTER TABLE t DETACH PARTITION ALL
2. prohibit PARTITION ALL excepte DETACH PARTITION ALL
 2022-04-07 17:37:01 +08:00
Nikolai Kochetov
5a1392a8e3 Try refactor something (1) 2022-04-05 19:12:48 +00:00
Nikolai Kochetov
def33323a0
Merge pull request #35788 from amosbird/better-projection1 
Improve minmax_count_projection
 2022-04-05 19:59:45 +02:00
Alexander Tokmakov
1fe50ad201 Merge branch 'master' into mvcc_prototype 2022-04-05 14:38:02 +02:00
alesapin
eaa72ef002
Merge pull request #35904 from ClickHouse/more_logs_on_unsuccessful_part_removal 
More logs on unsuccessful part removal
 2022-04-05 11:29:08 +02:00
Alexander Tokmakov
da00beaf7f Merge branch 'master' into mvcc_prototype 2022-04-05 11:14:42 +02:00
Amos Bird
5bc09550d8
Fix tests 2022-04-05 15:56:39 +08:00
Amos Bird
35a8bb2a9b
add comment 2022-04-05 15:56:38 +08:00
Amos Bird
163664fad7
Improve minmax_count_projection 2022-04-05 15:56:37 +08:00
Vladimir C
54b4049ea5
Merge pull request #35794 from awakeljw/fork_chmaster 2022-04-05 09:44:45 +02:00
alesapin
d04c48e67a Fix build 2022-04-04 20:14:09 +02:00
Anton Popov
bd93be65aa
Merge pull request #35820 from azat/columns-ttl-fix 
Avoid processing per-column TTL multiple times
 2022-04-04 16:42:50 +02:00
Anton Popov
d08d4a2437
Merge pull request #35806 from CurtizJ/dynamic-columns-8 
Fix inserts to columns of type `Object` in partitioned tables
 2022-04-04 16:16:54 +02:00
alesapin
3ccf99c3d7
Merge pull request #35884 from ClickHouse/don_t_allow_to_reset_settings 
Forbid to reset non existing settings
 2022-04-04 15:27:34 +02:00
Alexander Tokmakov
a2167f12b8 Merge branch 'master' into mvcc_prototype 2022-04-04 14:24:23 +02:00
Nikita Taranov
bd89fcafdb
Make SortDescription::column_name always non-empty (#35805) 2022-04-04 14:17:15 +02:00
alesapin
9b75ef6ce9 Fix build 2022-04-04 14:03:16 +02:00
alesapin
885447342c More logs on unsuccessful part removal 2022-04-04 13:17:33 +02:00
alesapin
803a1a2a9c Fix tests and check 2022-04-04 11:19:02 +02:00
Anton Popov
ae53aae106 fix clang-tidy 2022-04-04 08:48:31 +00:00
alesapin
a8c1ccdb20 Forbit to reset non existing settings 2022-04-03 18:29:24 +02:00
alesapin
91453fe4d6
Merge pull request #35759 from chen9t/fix_dirty_metacache 
Clean dirty meta cache when part is broken during part loading phase
 2022-04-03 13:33:46 +02:00
Azat Khuzhin
195b4c47ea Avoid processing per-column TTL multiple times 
Before this patch ttl.txt will not be written for per-column TTLs, and
hence it will be calculated every time after server restart of
DETACH/ATTACH cycle (note, that it will work w/o restart since in-memory
representation will avoid this).

v2: convert test to .sh to get correct current database over default for MV
v3: extract UUID to avoid error like in [1]:
    [ 490 ] {} <Error> void DB::SystemLog<DB::TextLogElement>::flushImpl(const std::vector<LogElement> &, uint64_t) []: Code: 349. DB::Exception: Cannot convert NULL value to non-Nullable type: While processing query_id LIKE concat('%', CAST(_CAST(NULL, 'Nullable(UUID)') AS uuid, 'String'), '%'): while pushing to view test_0hc2ro.this_text_log (c64e5af4-059e-4330-a728-354ecf83c031). (CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN)
    [1]: https://s3.amazonaws.com/clickhouse-test-reports/35820/a512d322b024d37d2f1082c4833f59f86057555f/stateless_tests_flaky_check__address__actions_.html
v4: add no-parallel to avoid issues with disappeared underlying table while pushing to text_log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-02 15:19:39 +03:00
kssenii
6c8e073a61 Merge master 2022-04-01 16:48:29 +02:00
Alexander Tokmakov
6591d1ceb7 Merge branch 'master' into mvcc_prototype 2022-04-01 15:38:46 +02:00
Kseniia Sumarokova
04729ecc70
Merge pull request #35798 from xiedeyantu/patch-1 
Delete duplicate code
 2022-04-01 10:43:31 +02:00
jewisliu
9a4cc78dfb support ALTER TABLE t DETACH PARTITION ALL syntax 2022-04-01 13:13:07 +08:00
Alexander Tokmakov
63e8d73ba4 better message for incorrect *MergeTree arguments 2022-03-31 17:08:30 +02:00
Anton Popov
42acb1dc29 fix inserts to columns of type Object in partitioned tables 2022-03-31 13:30:01 +00:00
xiedeyantu
d141dbc388
Delete duplicate code 
Delete duplicate code
 2022-03-31 19:34:30 +08:00
Alexander Tokmakov
5a50ad9de3 Merge branch 'master' into mvcc_prototype 2022-03-31 11:35:04 +02:00
chen9t
b40a69f59a Code style 2022-03-31 10:10:05 +08:00
chen9t
5cb2301e39 Clean dirty meta cache when part is broken during part loading phase 2022-03-30 20:40:20 +08:00
Anton Popov
1cba31c305
Merge pull request #35687 from CurtizJ/dynamic-columns-7 
Fix reading from type object
 2022-03-30 00:21:32 +02:00
Alexander Tokmakov
287d858fda Merge branch 'master' into mvcc_prototype 2022-03-29 16:24:12 +02:00
Dmitry Novik
8f935a72d6
Merge pull request #33230 from CurtizJ/read_in_order_max_rows_to_read 
Proper handle of 'max_rows_to_read' in case of reading in order of sorting key
 2022-03-29 15:16:34 +02:00
taiyang-li
834ac3a1b9 Merge remote-tracking branch 'ck/master' into rocksdb_metacache 2022-03-29 18:32:06 +08:00
Nikolai Kochetov
86690f18ee
Merge pull request #35667 from amosbird/projection-fix2 
Another fix of projection part check
 2022-03-29 10:22:01 +02:00
Amos Bird
3ab8fc1e7b
Fix projection part check 
Fix projection part check when projection part type is different from
parent part type.
 2022-03-29 10:39:07 +08:00
kssenii
44157e5c80 Merge master 2022-03-29 02:19:32 +02:00
Alexander Tokmakov
208b242188 Merge branch 'master' into mvcc_prototype 2022-03-28 19:58:06 +02:00
Anton Popov
0e4af89f69 fix reading from type object 2022-03-28 17:23:34 +00:00
mergify[bot]
f09ebea2d8
Merge branch 'master' into read_in_order_max_rows_to_read 2022-03-28 13:30:34 +00:00
taiyang-li
38702be40c Merge remote-tracking branch 'ck/master' into rocksdb_metacache 2022-03-28 21:03:22 +08:00
Alexander Tokmakov
9ec0951de5 update comment for mismatching checksums 2022-03-25 15:54:05 +01:00
kssenii
6769b84638 Merge master 2022-03-24 20:47:43 +01:00
Nikolai Kochetov
283e20a9a5
Merge pull request #35395 from amosbird/distributedmultiplejoin 
Validate some thoughts over making sets
 2022-03-24 10:30:26 +01:00
kssenii
d4161b5925 Add optin read_from_cache_if_exists_otherwise_bypass_cache (for merges) 2022-03-23 20:24:00 +01:00
kssenii
d2a3cfe5dc Cache on all write operations 2022-03-23 19:14:33 +01:00
kssenii
c2c7f365ce Merge master 2022-03-23 16:58:35 +01:00
kssenii
8c1be8950e Improve test 2022-03-23 16:55:18 +01:00
ianton-ru
32c6390400
Merge branch 'master' into MDB-16962 2022-03-23 16:53:30 +03:00
Anton Ivashkin
66c026d0ac Fix collision of S3 operation log revision 2022-03-23 16:48:05 +03:00
taiyang-li
6fe4c3d17e some minor changes 2022-03-23 12:13:42 +08:00
taiyang-li
8dbf1c60e7 merge master and fix conflict 2022-03-23 11:36:50 +08:00
Anton Popov
f693eba568 fix tests with approx rows 2022-03-22 14:30:40 +00:00
Alexander Tokmakov
3c762f566d Merge branch 'master' into mvcc_prototype 2022-03-21 20:16:29 +01:00
mergify[bot]
9b9959aae0
Merge branch 'master' into read_in_order_max_rows_to_read 2022-03-21 17:33:37 +00:00
Vitaly Baranov
72a2f31984
Merge pull request #33985 from vitlibar/improve-backup-and-restore 
Improve backup and restore (experimental)
 2022-03-21 17:59:51 +01:00
tavplubix
9226ea7b70
Merge pull request #35467 from ClickHouse/resubmit_21474 
Resubmit #21474
 2022-03-21 19:15:28 +03:00
Nikolai Kochetov
5e239762c6
Merge pull request #35334 from amosbird/fixpartitionpruneroverflow 
Fix incorrect partition pruning when constant in predicate has no supertype of related key columns
 2022-03-21 17:05:19 +01:00
kssenii
afd0c64a1a Better 2022-03-21 15:04:16 +01:00
kssenii
eabbce69a7 Add system drop remote filesystem cache query, allow to write cache on insert 2022-03-21 12:32:44 +01:00
Alexander Tokmakov
7cba97aeab Merge branch 'master' into resubmit_21474 2022-03-21 12:09:00 +01:00
mergify[bot]
e5a5ab2a40
Merge branch 'master' into distributedmultiplejoin 2022-03-21 10:00:51 +00:00
kssenii
59b7394caf WriteSettings 2022-03-21 09:52:48 +01:00
kssenii
1d80262a2a Add write settings, file cache settings 2022-03-21 09:50:50 +01:00
Amos Bird
243de091bb
Validate some thoughts over making sets 2022-03-21 10:58:44 +08:00
Alexander Tokmakov
9e05b12d2c Merge branch 'master' into mvcc_prototype 2022-03-20 22:42:26 +01:00
Vitaly Baranov
ce25afb2e9 Storages and databases are hollow by default now. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7f89b98308 Rework BackupSettings and RestoreSettings a little, pass StorageRestoreSettings to storages. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
258a472001 Shorter names: rename IRestoreFromBackupTask -> IRestoreTask. 2022-03-20 20:02:15 +01:00
Vitaly Baranov
7a63feb3f7 Make restore tasks explicit. 2022-03-20 20:01:31 +01:00
Alexey Milovidov
9b05801e0f
Merge pull request #35284 from CurtizJ/fix-mutations-sparse-columns 
Fix mutations in tables with enabled sparse columns
 2022-03-20 21:55:59 +03:00
alesapin
1bd5736e34 Fix build one more time 2022-03-19 20:00:08 +01:00