thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 20:02:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
97,002 Commits 9,330 Branches 1,783 Tags 2.1 GiB
81c98dadd2
Commit Graph

4097 Commits

Author SHA1 Message Date
alesapin
612c4571d5 Split method into smaller 2022-06-24 15:41:09 +02:00
alesapin
7517e1f4d5 Remove some complexity 2022-06-24 15:24:02 +02:00
alesapin
c57e07cb49 Style 2022-06-24 14:51:47 +02:00
alesapin
37310dc9df Simpler 2022-06-24 14:10:15 +02:00
alesapin
36b810b076 Remove unused parameter 2022-06-24 13:42:36 +02:00
alesapin
af1a9d18ab Remove transaction argument 2022-06-24 13:34:00 +02:00
alesapin
9910395823 Simplify method signature 2022-06-24 13:19:29 +02:00
Alexander Tokmakov
dbf2763788 automatically convert system database to Atomic 2022-06-23 21:38:43 +02:00
alesapin
2bdedf5e0c Some changes 2022-06-23 18:21:46 +02:00
mergify[bot]
174d8577fa
Merge branch 'master' into merge_tree_reader 2022-06-23 15:16:51 +00:00
kssenii
f7b329ee57 Merge master 2022-06-23 14:56:48 +02:00
alesapin
5661280ef6 Remove some read methods 2022-06-23 14:01:26 +02:00
Alexander Tokmakov
f00e6b5a7a deprecate old MergeTree syntax 2022-06-23 11:24:54 +02:00
alesapin
1b495ec8ad Fix style 2022-06-23 00:17:37 +02:00
kssenii
e40d9bcf55 Merge master 2022-06-22 23:28:52 +02:00
Anton Popov
52db1b35a1 improve performace of insertion to columns of type JSON 2022-06-22 17:45:51 +00:00
Alexander Gololobov
0ee47363d4 Fixed includes 2022-06-22 19:08:18 +02:00
Alexander Gololobov
8138a3a36e Cleanups based on review comments 2022-06-22 19:08:01 +02:00
alesapin
ca0299e927 Ban projections for zero-copy replication in a right way 2022-06-22 19:01:46 +02:00
Alexander Gololobov
e5b55b965b Removed incorrect check 2022-06-22 17:23:09 +02:00
Alexander Gololobov
dbc6d1a159 Cleanups 2022-06-22 17:23:09 +02:00
Alexander Gololobov
aea5bc7527 Removed unused should_reorder flag 2022-06-22 17:23:09 +02:00
Alexander Gololobov
5427d5c51b Apply filter at row level security step 2022-06-22 17:23:09 +02:00
Alexander Gololobov
b3922461b3 Properly handle empty actions 2022-06-22 17:23:09 +02:00
Alexander Gololobov
ba89c3954c Do not add the same vitrual if it has been added by prev_reader 2022-06-22 17:23:09 +02:00
Alexander Gololobov
e5c0ebc7b5 Handle read from wide part when no columns are requested 2022-06-22 17:23:09 +02:00
Alexander Gololobov
a9e3b8d29e Don't read the same columns again 2022-06-22 17:23:09 +02:00
Alexander Gololobov
4e426c63cc Debuging test failures 2022-06-22 17:23:09 +02:00
Alexander Gololobov
6a26325fab Test dirty hacks for multiple PREWHERE steps 2022-06-22 17:23:05 +02:00
Alexander Gololobov
4360fd9798 Vector of structs instead of multiple vectors 2022-06-22 17:17:42 +02:00
Alexander Gololobov
87b669f439 Intermediate changes 2022-06-22 17:17:42 +02:00
Alexander Gololobov
159ab765fb Moved estimateNumRows from lambda to function 2022-06-22 17:17:42 +02:00
Alexander Gololobov
64a2f3734b Protect ReadResult internals from MergeTreeRangeReader clients 2022-06-22 17:17:42 +02:00
Nikolai Kochetov
cc6fdfe0eb
Merge pull request #36555 from ClickHouse/refactor-something-in-part-volumes 
Separate data storage abstraction for MergeTree
 2022-06-22 11:13:36 +02:00
kssenii
4178abc0a5 Merge master 2022-06-21 22:38:51 +02:00
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa 
Support for Clang Thread Safety Analysis (TSA)
 2022-06-21 20:19:33 +02:00
Nikolai Kochetov
dcf6e0d060 Roll back some changes from IPartMetadataManager 2022-06-21 13:05:49 +00:00
Nikolai Kochetov
854d148d73 Add some comments. Remove some commented code. 2022-06-21 12:31:02 +00:00
Nikolai Kochetov
46c4fd6cf7 Fixing build. 2022-06-21 11:15:06 +00:00
Kruglov Pavel
3c5a627bd7
Merge pull request #38239 from CurtizJ/fix-reading-from-s3 
Fix reading from s3 in some corner cases
 2022-06-21 12:57:55 +02:00
Nikolai Kochetov
1e8c9ecd4c Merge branch 'master' into refactor-something-in-part-volumes 2022-06-21 12:37:21 +02:00
alesapin
fafcb876be
Merge pull request #38182 from ClickHouse/disk_transaction 
Disk transaction
 2022-06-21 12:28:49 +02:00
Nikolai Kochetov
b8d27aa8dd
Merge pull request #37469 from azat/projections-optimize_aggregation_in_order 
Implement in order aggregation (optimize_aggregation_in_order) for projections for tables with fully materialized projections
 2022-06-21 12:17:35 +02:00
Nikolai Kochetov
e45d552f4d Do not create empty folder for in-memory parts. 2022-06-21 09:59:46 +00:00
Nikolai Kochetov
5232c17b89 Fix style 2022-06-21 07:30:45 +00:00
Nikolai Kochetov
3429a5e07a Fix 2022-06-21 07:26:43 +00:00
Nikolai Kochetov
dccf90b1ea Cleanup. 2022-06-20 18:18:17 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Anton Popov
59a9ce6def fix reading from s3 in some corner cases 2022-06-20 15:24:38 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
Nikolai Kochetov
39b02d4ab8 Continure resolving conflicts. 2022-06-20 15:39:05 +02:00
Nikolai Kochetov
7452d04e3a Merge branch 'master' into refactor-something-in-part-volumes 2022-06-20 15:31:02 +02:00
Nikolai Kochetov
ef8b967c3f Fix more tests. 2022-06-20 13:07:14 +00:00
kssenii
90be49faa4 Merge master 2022-06-20 14:26:56 +02:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6 
Backup Improvements 6
 2022-06-20 14:13:30 +02:00
alesapin
08b5dd091b Fix double negative 2022-06-19 17:12:04 +02:00
Vitaly Baranov
8a7c970ce0 Fix style. 2022-06-19 15:58:26 +02:00
alesapin
50801e41c5 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-19 14:05:46 +02:00
Alexander Tokmakov
83adf56383 fix race 2022-06-17 18:13:57 +02:00
Alexander Tokmakov
e9c2157f01 Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-17 15:57:30 +02:00
Alexander Tokmakov
5ffc38d9b9 fix style check, add comments 2022-06-17 13:47:14 +02:00
alesapin
e5b405701a Merge branch 'master' into disk_transaction 2022-06-17 12:16:12 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache 
Extract some diff from pr #36171
 2022-06-17 12:07:31 +02:00
Alexander Tokmakov
39c0219c11 fixes 2022-06-16 19:41:32 +02:00
Anton Popov
13ec7e3092
Merge pull request #37978 from CurtizJ/fix-sparse-s3 
Fix reading of sparse columns from s3
 2022-06-16 15:32:54 +02:00
alesapin
379fff7318 Merge branch 'master' into disk_transaction 2022-06-16 14:36:19 +02:00
Azat Khuzhin
4ff82eb9d5 Implement optimize_aggregation_in_order for projections 
v2: use real column name instead of aliases from GROUP BY
    Fixes the following error in 01710_projection_aggregation_in_order:

        Not found column a in block. There are only columns: toStartOfHour(ts), sum(value). (NOT_FOUND_COLUMN_IN_BLOCK)
v2.1: Get back support for projected and non-projected parts
v2.2: merge tests and rename
v3: Reduce copy-paste for optimize_aggregation_in_order for projections
v4: rebase on top of QueryPlanResourceHolder
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
14f9491619 MergeTreeData: preserve order for group_by_elements_order_descr 
This is required for proper optimize_aggregation_in_order for
projections.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
4694929623 Implement merging only for AggregatingStep 
v2: fill AggregateColumnsConstData only for only_merge
    (fixes 01291_aggregation_in_order and some other tests)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
682c93e2e1 Use proper setting compile_aggregate_expressions in MergeTreeDataSelectExecutor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:35 +03:00
Alexander Tokmakov
6ddfec47d2 maybe fix livelock on queue processing 2022-06-15 22:22:27 +02:00
Alexander Tokmakov
5e2b42c85a Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-15 20:47:02 +02:00
Vitaly Baranov
1198e86295 Fix storing temporary tables and skipping system tables while making a backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
21f3bed435 Simplify path calculations in backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
592f568f83 Move backup/restore code to storages and databases - part 2. 2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57 Move backup/restore code to storages and databases - part 1. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21 Rework collecting replicated parts. 2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Daniel Kutenin
a769dea8ef
Merge branch 'ClickHouse:master' into master 2022-06-15 19:12:10 +01:00
Nikolai Kochetov
25b9990737 Fixing build 2022-06-15 16:32:22 +00:00
Alexander Tokmakov
2ac72319bd
Merge pull request #37185 from amosbird/projection-fix-three 
Fix possible heap-use-after-free error when reading system.projection_parts and system.projection_parts_columns
 2022-06-15 19:00:10 +03:00
kssenii
416c6008ee Merge master 2022-06-15 16:50:25 +02:00
kssenii
500f49972b Extract diff from PR 36171 2022-06-15 16:40:18 +02:00
Alexander Tokmakov
e4d4d0a854 fix race on queue processing 2022-06-15 16:21:26 +02:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Danila Kutenin
08e3f77a9c Optimize most important parts with NEON SIMD 
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.

That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
 2022-06-15 13:19:29 +00:00
alesapin
1b66166501 Merge branch 'master' into disk_transaction 2022-06-15 13:54:10 +02:00
alesapin
b557c1dd80 Interm stage 2022-06-15 13:51:21 +02:00
Nikolai Kochetov
2ee349f29c Fix test_encrypted_disk 2022-06-15 11:42:59 +00:00
Nikolai Kochetov
a274bec0fa Fix test_encrypted_disk 2022-06-15 11:41:08 +00:00
Alexander Tokmakov
56f2121c1a
Revert "Add backoff to merges in replicated queue if always_fetch_merged_part is enabled" 2022-06-15 13:04:01 +03:00
alesapin
af1cd745e1
Merge pull request #37975 from kssenii/clean-up-broken-detached 
Clean up broken detached parts after timeout
 2022-06-14 20:53:31 +02:00
Amos Bird
d5a7a5be8e
Fix use-after-free in system.projection_parts 2022-06-14 23:41:42 +08:00
alesapin
3b832a25f6
Merge pull request #37995 from ClickHouse/fix_queue 
Add backoff to merges in replicated queue if `always_fetch_merged_part` is enabled
 2022-06-14 11:38:44 +02:00
kssenii
0d2dae0046 Add const, rename settings 2022-06-14 00:47:03 +02:00
kssenii
74cffaf186 Merge master 2022-06-14 00:45:19 +02:00
kssenii
4813b90281 Merge master 2022-06-13 21:32:00 +02:00
kssenii
34e63b20fb Review fixes 2022-06-13 21:27:54 +02:00
Anton Popov
e6517164f2 add test and comment 2022-06-13 17:30:58 +00:00
Antonio Andelic
b9cf6fe367 Merge branch 'master' into pp-system-unfreeze 2022-06-13 14:46:30 +00:00
alesapin
d984763b95
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-06-13 15:10:20 +02:00
kssenii
debe467ee7 Review fixes 2022-06-13 14:25:42 +02:00
kssenii
4859e11c27 Review fixes 2022-06-12 01:15:10 +02:00
alesapin
2a81904fbc Revert "Revert "Fix bug with removal of moving parts"" 
This reverts commit c0581bf741.
 2022-06-12 01:08:24 +02:00
kssenii
aaf9a46f07 Merge master 2022-06-11 15:56:05 +02:00
alesapin
c0581bf741 Revert "Fix bug with removal of moving parts" 
This reverts commit e32133ff8f.
 2022-06-10 19:35:57 +02:00
alesapin
301eac7d59 Comment 2022-06-10 19:34:03 +02:00
alesapin
ffff1c22a6 Add backoff for merges when always_fetch_merged_part turned on 2022-06-10 19:33:28 +02:00
alesapin
e32133ff8f Fix bug with removal of moving parts 2022-06-10 18:47:06 +02:00
kssenii
fd6c84b831 Review fixes 2022-06-10 14:23:13 +02:00
Anton Popov
6a0b305a14 fix reading of sparse columns from s3 2022-06-10 12:09:37 +00:00
kssenii
498f389c21 Merge master 2022-06-10 12:49:10 +02:00
kssenii
0fd2c4b2c0 Continuation 2022-06-10 12:33:29 +02:00
kssenii
7a2676c7ab Clean up broken detached parts with timeout 2022-06-10 12:27:57 +02:00
Kseniia Sumarokova
508676ab4e
Merge pull request #37804 from ClickHouse/metadata_storage 
Metadata storage abstraction for Disks
 2022-06-10 11:43:28 +02:00
Alexander Tokmakov
780f7c87c7
Merge pull request #36113 from ClickHouse/remove-useless-code-2 
Remove useless code in ReplicatedMergeTreeRestartingThread
 2022-06-09 17:56:28 +03:00
Vadim Volodin
637d293fbd Add SYSTEM UNFREEZE query 2022-06-08 15:21:14 +03:00
alesapin
81dca513de Review fixes 2022-06-08 13:28:33 +02:00
Nikolai Kochetov
5bc9b32025 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-08 11:10:06 +00:00
alesapin
b88e1a2bef Merge branch 'master' into metadata_storage 2022-06-08 13:02:21 +02:00
alesapin
9fdc783eac
Merge pull request #37906 from ClickHouse/remove_tmp_directories_on_replicated 
Remove temporary `delete_tmp_` on clickhouse server start
 2022-06-08 12:35:51 +02:00
alesapin
e32d36d790 Proper fix 2022-06-07 17:58:32 +02:00
Nikolai Kochetov
678d978acf Merge branch 'master' into refactor-something-in-part-volumes 2022-06-07 15:23:00 +00:00
mergify[bot]
2d01abf871
Merge branch 'master' into revert-37647-Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-07 13:32:30 +00:00
Nikita Taranov
d09ad5fcd5
Fix build 
https://s3.amazonaws.com/clickhouse-test-reports/36593/a4b9614385e0ecd1654d15f507159678758ec557/fast_test__actions_/build_log.txt
 2022-06-07 13:56:03 +02:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type" 2022-06-07 13:53:10 +02:00
Anton Popov
ef6f5a6500
Merge pull request #37570 from azat/column-ttl-expired-fix 
Do not write expired columns by TTL after subsequent merges
 2022-06-07 13:05:03 +02:00
mergify[bot]
dd0603a595
Merge branch 'master' into metadata_storage 2022-06-06 18:59:45 +00:00
Anton Popov
d40b23272e
Merge pull request #37755 from CurtizJ/fix-mutations-again 
Return back #37266
 2022-06-06 20:22:03 +02:00
Robert Schulze
2d87af2a15
Merge pull request #37647 from DevTeamBK/Fix-all-CheckTriviallyCopyableMove-Errors 
Fix errors of CheckTriviallyCopyableMove type
 2022-06-05 19:58:47 +02:00
alesapin
a25a01af16 Fix WAL 2022-06-05 16:15:11 +02:00
alesapin
38df94666b Fix merge tree wal 2022-06-05 15:09:39 +02:00
alesapin
a2704bc541 Trying to fix WAL log 2022-06-05 01:05:15 +02:00
Alexey Milovidov
f860407af5
Merge pull request #37777 from ClickHouse/avx512_tail_zero 
Use multitarget framework for numZerosInTail implementation
 2022-06-04 01:46:56 +03:00
Nikolai Kochetov
3f67abc3b0 Fix some tests. 2022-06-03 18:49:12 +00:00
mergify[bot]
ddf7210ecc
Merge branch 'master' into remove-useless-code-2 2022-06-03 13:58:45 +00:00
Maksim Kita
8469e0d10d
ExecutableLambdaAdapter forward job and callback (#37830) 2022-06-03 15:54:47 +02:00
alesapin
ac03e3af3e
Merge branch 'master' into metadata_storage 2022-06-03 14:42:21 +02:00
Anton Popov
5cf2c67410
Merge branch 'master' into fix-mutations-again 2022-06-03 14:17:05 +02:00
Nikolai Kochetov
8991f39412 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 17:00:08 +00:00
alesapin
46bccae078 Trying to create metadata layer 2022-06-02 18:09:40 +02:00
Nikolai Kochetov
25c8a437ea Fixing build. 2022-06-02 15:00:07 +00:00
HeenaBansal2009
e3080f2a97 Merge remote-tracking branch 'origin' into Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-02 07:30:08 -07:00
mergify[bot]
fbfaaa9c74
Merge branch 'master' into fix-mutations-again 2022-06-02 11:57:19 +00:00
Alexander Tokmakov
45657d203a
Merge pull request #37763 from ClickHouse/fix_replace_range_again 
Fix flaky 01154_move_partition_long
 2022-06-02 14:15:16 +03:00
Alexander Gololobov
0b004b907a Added AVX2 implementation of numZerosInTail 2022-06-02 13:01:13 +02:00
Nikolai Kochetov
fc726c236f
Merge pull request #37661 from ClickHouse/no_alias_actions 
Trying to remove prewhere_info.alias_actions as they seem to always be included in prewhere_actions
 2022-06-02 12:31:04 +02:00
Nikolai Kochetov
89c5855d20 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-02 12:19:07 +02:00
Nikita Mikhaylov
d34e051c69
Support for simultaneous read from local and remote parallel replica (#37204) 2022-06-02 11:46:33 +02:00
Nikolai Kochetov
edac3d6714 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 09:36:20 +00:00
Alexander Gololobov
d8e4dcf9e2 Use multitarget framework for numZerosInTail implementation 2022-06-02 11:23:44 +02:00
mergify[bot]
88bed9de20
Merge branch 'master' into fix_replace_range_again 2022-06-02 07:05:44 +00:00
Alexander Gololobov
a0cf902d49
Merge pull request #37588 from yaqi-zhao/avx512_tail_zero 
add avx512 support for mergetree reader
 2022-06-02 08:51:52 +03:00
lthaooo
6632616733
Fix TTL merge scheduling bug (#36387) 2022-06-01 21:09:53 +02:00
Alexander Tokmakov
06f80770b8 fix stuck REPALCE_RANGE 2022-06-01 20:11:53 +02:00
Sema Checherinda
2626a49616 FR: Expose what triggered the merge in system.part_log #26255 2022-06-01 16:58:06 +02:00
kssenii
aa60de8f63 Merge master 2022-06-01 15:32:01 +02:00
Anton Popov
cae2478b3f Revert "Merge pull request #37355 from ClickHouse/revert-37266-fix-mutations-with-object" 
This reverts commit a53cfa9fca, reversing
changes made to 9acb42fcdb.
 2022-06-01 10:57:20 +00:00
Alexander Gololobov
26609a1875 Style fixes 2022-05-31 21:41:10 +02:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00
Nikolai Kochetov
147a819221 Refactor a little bit more. 2022-05-31 14:43:38 +00:00
yaqi-zhao
a2857491c4 add avx512 support for mergetreereader 2022-05-30 20:53:00 -04:00
mergify[bot]
55913cf8e1
Merge branch 'master' into turn_on_s3_tests 2022-05-30 20:52:40 +00:00
kssenii
bdfd166eb1 Merge master 2022-05-30 21:31:47 +02:00
HeenaBansal2009
b7eb6bbd38 Fixed clang-tidy-CheckTriviallyCopyableMove-errors 2022-05-30 11:09:03 -07:00
Alexander Gololobov
e2dd6f6249 Removed prewhere_info.alias_actions 2022-05-30 19:58:23 +02:00
Nikolai Kochetov
5b4658aa5e Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-30 09:47:35 +00:00
alesapin
c32b6076fb Remove stranges from code 2022-05-30 00:12:33 +02:00
alesapin
0e8ab36913 Merge branch 'master' into turn_on_s3_tests 2022-05-29 14:37:10 +02:00
alesapin
d2cdbf3956 Fix refactoring issue 2022-05-29 14:09:49 +02:00
alesapin
9dc81e5cc8
Merge pull request #37598 from ClickHouse/revert-37545-revert-37424-fix_fetching_part_deadlock 
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part""
 2022-05-29 13:54:34 +02:00
Alexey Milovidov
97606c324c
Merge pull request #37574 from azat/mt-tiny-refactor 
Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm()
 2022-05-29 07:59:57 +03:00
Alexey Milovidov
73e2e63414
Merge pull request #37612 from ClickHouse/clang-tidy-14 
Fix clang-tidy-14, part 1
 2022-05-29 03:16:32 +03:00
Anton Popov
1d9b3be7da
Merge pull request #37536 from CurtizJ/profile-events-for-part-types 
Add profile events for introspection of part types
 2022-05-28 14:25:21 +02:00
Alexey Milovidov
c50791dd3b Fix clang-tidy-14, part 1 2022-05-27 22:52:14 +02:00
mergify[bot]
f5ee337bab
Merge branch 'master' into revert-37545-revert-37424-fix_fetching_part_deadlock 2022-05-27 16:52:00 +00:00
kssenii
25b1ed526d Merge master 2022-05-27 16:50:45 +02:00
alesapin
5a296aec01 Fix build 2022-05-27 16:34:16 +02:00
alesapin
be1c3c132b Fix some trash 2022-05-27 16:08:49 +02:00
alesapin
6d6779f17a
Merge pull request #37139 from ClickHouse/i_object_storage 
Separate object storage operations from disks
 2022-05-27 13:59:50 +02:00
alesapin
c79600c4c8 Fix build 2022-05-27 13:44:29 +02:00
alesapin
841858ec30
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part"" 2022-05-27 13:13:36 +02:00
Azat Khuzhin
c6c60364ae Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm() 
In favor of
MergeTask::ExecuteAndFinalizeHorizontalPart::chooseMergeAlgorithm()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 21:20:50 +03:00
Azat Khuzhin
73a99d4eee Improve error message for skipped/expired columns 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
f0cac5417a More optional skipping of fully expired columns (by TTL) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
46a94f395d Move the check into IMergedBlockOutputStream::removeEmptyColumnsFromPart() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
4288d09a85 Do not write expired columns by TTL after merge w/o TTL 
Usually second merge do not perform TTL, since everything is up to date,
however in this case TTLTransform is not used, and hence expired_columns
will not be filled for new part, and so those columns will be written
with default values.

Avoid this, by manually filling expired_columns.

Here is a simpler reproducer:

Simple reproducer:

    ```sql
    create table ttl_02262 (date Date, key Int, value String TTL date + interval 1 month) engine=MergeTree order by key settings min_bytes_for_wide_part=0, min_rows_for_wide_part=0;
    insert into ttl_02262 values ('2010-01-01', 2010, 'foo');
    ```

    ```sh
    # ls -l .server/data/default/ttl_02262/all_*
    .server/data/default/ttl_02262/all_1_1_0:
    total 48
    -rw-r----- 1 root root 335 May 26 14:19 checksums.txt
    -rw-r----- 1 root root  76 May 26 14:19 columns.txt
    -rw-r----- 1 root root   1 May 26 14:19 count.txt
    -rw-r----- 1 root root  28 May 26 14:19 date.bin
    -rw-r----- 1 root root  48 May 26 14:19 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:19 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:19 key.bin
    -rw-r----- 1 root root  48 May 26 14:19 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:19 primary.idx
    -rw-r----- 1 root root  99 May 26 14:19 ttl.txt
    -rw-r----- 1 root root  30 May 26 14:19 value.bin
    -rw-r----- 1 root root  48 May 26 14:19 value.mrk2
    ```

    ```sql
    optimize table ttl_02262 final;
    ```

    ```sh
    .server/data/default/ttl_02262/all_1_1_1:
    total 40
    -rw-r----- 1 root root 279 May 26 14:19 checksums.txt
    -rw-r----- 1 root root  61 May 26 14:19 columns.txt
    -rw-r----- 1 root root   1 May 26 14:19 count.txt
    -rw-r----- 1 root root  28 May 26 14:19 date.bin
    -rw-r----- 1 root root  48 May 26 14:19 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:19 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:19 key.bin
    -rw-r----- 1 root root  48 May 26 14:19 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:19 primary.idx
    -rw-r----- 1 root root  81 May 26 14:19 ttl.txt
    ```

    ```sql
    optimize table ttl_02262 final;
    ```

    ```sh
    .server/data/default/ttl_02262/all_1_1_2:
    total 48
    -rw-r----- 1 root root 349 May 26 14:20 checksums.txt
    -rw-r----- 1 root root  76 May 26 14:20 columns.txt
    -rw-r----- 1 root root   1 May 26 14:20 count.txt
    -rw-r----- 1 root root  28 May 26 14:20 date.bin
    -rw-r----- 1 root root  48 May 26 14:20 date.mrk2
    -rw-r----- 1 root root  10 May 26 14:20 default_compression_codec.txt
    -rw-r----- 1 root root  30 May 26 14:20 key.bin
    -rw-r----- 1 root root  48 May 26 14:20 key.mrk2
    -rw-r----- 1 root root   8 May 26 14:20 primary.idx
    -rw-r----- 1 root root  81 May 26 14:20 ttl.txt
    -rw-r----- 1 root root  27 May 26 14:20 value.bin
    -rw-r----- 1 root root  48 May 26 14:20 value.mrk2
    ```

And now we have `value.*` for all_1_1_2, this should not happen.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
8328d7068b Fix updating of MergeTreeDataPartTTLInfo::finished 
Previously you cannot distinguish non-initialized finished with
initialized to false, so update() cannot do the correct thing.

Rename the field to avoid hidden usage.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:10 +03:00
Azat Khuzhin
0de1a64436 Log empty parts in IMergedBlockOutputStream::removeEmptyColumnsFromPart() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-26 20:14:09 +03:00
Azat Khuzhin
dc9ca3d70c
Fix LOGICAL_ERROR in getMaxSourcePartsSizeForMerge during merges (#37413) 2022-05-26 14:14:58 +02:00
Nikolai Kochetov
84f97b53de Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-26 11:07:45 +00:00
Anton Popov
f488efd27e fix tests 2022-05-26 00:03:31 +00:00
alesapin
8f1aac0ce4 Fix merge with master 2022-05-26 00:44:45 +02:00
alesapin
1db9cf480b Merge remote-tracking branch 'origin/master' into i_object_storage 2022-05-25 22:50:22 +02:00
kssenii
50cd358c0e Merge master 2022-05-25 22:21:28 +02:00
Alexander Tokmakov
47820c216d
Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part" 2022-05-25 23:10:33 +03:00
alesapin
c7b16065e1 Merge with master 2022-05-25 21:47:05 +02:00
Nikolai Kochetov
6d4a26afac Update ReadProgressCallback. 2022-05-25 19:45:48 +00:00
kssenii
b89a4edc04 Merge master 2022-05-25 20:55:04 +02:00
alesapin
6f5c86e55e Merge branch 'master' into i_object_storage 2022-05-25 20:49:01 +02:00
alesapin
51868a9a4f
Merge pull request #37424 from metahys/fix_fetching_part_deadlock 
(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part
 2022-05-25 20:15:41 +02:00
alesapin
0a3597da72
Merge pull request #34915 from ianton-ru/MDB-16962 
Fix collision of S3 operation log revision
 2022-05-25 18:15:31 +02:00
Anton Popov
16e839ac71 add profile events for introspection of part types 2022-05-25 14:54:49 +00:00
Nikolai Kochetov
1b85f2c1d6 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-25 16:27:40 +02:00
mergify[bot]
73662b4436
Merge branch 'master' into fix_fetching_part_deadlock 2022-05-25 14:22:35 +00:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
alesapin
9a19309e69 Slightly better fix 2022-05-24 14:46:29 +02:00
alesapin
3ca7a8831b Revert "fix deadlock during fetching part" 
This reverts commit 6ae8a26fae.
 2022-05-24 14:26:06 +02:00
alesapin
a5ba6bca95 Merge branch 'master' into metahys-fix_fetching_part_deadlock 2022-05-24 14:07:47 +02:00
Nikolai Kochetov
fd97a9d885 Move some resources 2022-05-23 19:47:32 +00:00
Alexander Tokmakov
6bc68c0cbc Merge branch 'master' into fixes_for_transactions 2022-05-23 18:49:21 +02:00
metahys
6ae8a26fae fix deadlock during fetching part 2022-05-23 20:08:04 +08:00
mergify[bot]
747aa5575c
Merge branch 'master' into remove-useless-code-2 2022-05-22 17:41:57 +00:00
alesapin
c8d92b87c8 Merge branch 'master' into i_object_storage 2022-05-22 12:16:10 +02:00
Robert Schulze
19d53c14fa
Merge pull request #37382 from ClickHouse/wc++98-compat-extra-semi 
Enable -Wc++98-compat-extra-semi
 2022-05-22 09:40:45 +02:00
kssenii
9aa20193d5 Merge master 2022-05-22 00:56:20 +02:00
Alexander Tokmakov
44f2d4529a better waiting, add fault injection 2022-05-20 22:10:01 +02:00
Nikolai Kochetov
56feef01e7 Move some resources 2022-05-20 19:49:31 +00:00
Anton Popov
cb0e6c2718 mark all operators bool() as explicit 2022-05-20 15:29:54 +00:00
Robert Schulze
0f6715bd91
Follow-up to PR #37300: semicolon warnings 
In PR #37300, Alexej asked why we the compiler does not warn about
unnecessary semicolons, e.g.

  f()
  {
  }; // <-- here

The answer is surprising: In C++98, above syntax was disallowed but by
most compilers accepted it regardless. C++>11 introduced "empty
declarations" which made the syntax legal.

The previous behavior can be restored using flag
-Wc++98-compat-extra-semi. This finds many useless semicolons which were
removed in this change. Unfortunately, there are also false positives
which would require #pragma-s and HAS_* logic (--> check_flags.cmake) to
suppress. In the end, -Wc++98-compat-extra-semi comes with extra effort
for little benefit. Therefore, this change only fixes some semicolons
but does not enable the flag.
 2022-05-20 15:06:34 +02:00
Alexander Tokmakov
12bbb7de87 fix race on TID allocation 2022-05-20 12:41:44 +02:00
alesapin
654b27e307 Merge branch 'master' into i_object_storage 2022-05-20 11:56:13 +02:00
Robert Schulze
b475fbc9a7
Merge pull request #37300 from ClickHouse/cmake-cleanup-pt3 
Various cmake cleanups
 2022-05-20 10:02:36 +02:00
alesapin
92c15ec97c Get rid of IDiskRemote 2022-05-19 20:07:15 +02:00
Alexander Tokmakov
f787dc7097
Revert "Fix mutations in tables with columns of type Object" 2022-05-19 13:24:48 +03:00
Vitaly Baranov
7c8956ca8d
Merge pull request #36879 from tonickkozlov/tonickkozlov/lazy-marks-loading 
Lazy marks loading
 2022-05-18 22:56:46 +02:00
alesapin
e99c90118e Merge branch 'master' into i_object_storage 2022-05-18 12:36:22 +02:00
Kruglov Pavel
aaace46da2
Merge pull request #37266 from CurtizJ/fix-mutations-with-object 
Fix mutations in tables with columns of type `Object`
 2022-05-18 12:19:55 +02:00
kssenii
e4e6e44e86 Merge master 2022-05-17 15:47:33 +02:00
alesapin
4d4634b12d
Merge pull request #37164 from kssenii/ficx-async-reads-assertions 
fix async reads from remote fs internal setting not being always turned on
 2022-05-17 11:29:14 +02:00
kssenii
6a51dea791 Merge master 2022-05-16 22:13:22 +02:00
kssenii
5e3410f60a Better 2022-05-16 22:09:11 +02:00
Robert Schulze
43945cea1b
Fixing some warnings 2022-05-16 20:59:27 +02:00
Anton Popov
b6c5ab4fcf fix mutations in tables with columns of type Object 2022-05-16 18:26:53 +00:00
alesapin
ce6d5f7dcc Merge branch 'master' into i_object_storage 2022-05-16 17:42:22 +02:00
kssenii
f3c1f3796b Some fixes 2022-05-16 16:16:38 +02:00
Vladimir Makarov
f809e2a79f fix error 2022-05-16 09:31:37 +00:00
kssenii
f7493965b7 Remove additional check for merge tree 2022-05-16 11:19:57 +02:00
kssenii
49201f217f Revert 2022-05-16 11:17:56 +02:00
Robert Schulze
e3cfec5b09
Merge remote-tracking branch 'origin/master' into clangtidies 2022-05-16 10:12:50 +02:00
Robert Schulze
859f2f3391
Enclose large if block in {} 2022-05-15 14:49:20 +02:00
Vladimir Makarov
69e2f8668e fix 2022-05-15 12:29:33 +00:00
Vladimir Makarov
d6512d257f fixes 2022-05-15 08:57:57 +00:00
Vladimir Makarov
e502ab2f55 disable mac os and try to fix sanitizer error 2022-05-15 08:25:46 +00:00
Vladimir Makarov
bdb6691405 fix style 2022-05-14 21:25:00 +00:00
VVMak
1b9834548d
Merge branch 'master' into annoy-2 2022-05-15 00:01:28 +03:00
Vladimir Makarov
59e0ef9641 remove extra headers 2022-05-14 18:00:34 +00:00
Vladimir Makarov
1ba0bcaffe fix includes and add docs 2022-05-14 17:53:45 +00:00
Vladimir Makarov
3187fdddb8 Merge branch 'ANNIndex-Documentation' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-05-14 14:21:19 +00:00
Vladimir Makarov
699f183ea0 small fix and comment about metrics 2022-05-14 10:47:05 +00:00
Vladimir Makarov
d1880f3c10 fix cmake and style 2022-05-14 10:27:05 +00:00
Vladimir Makarov
316858ee39 fix style 2022-05-14 08:55:07 +00:00
Vladimir Makarov
8f8c622e7c fix 2022-05-14 08:24:54 +00:00
alesapin
cc304a3178 Merge branch 'master' into i_object_storage 2022-05-13 23:44:00 +02:00
Vladimir Makarov
f4ae3150fa add optimization 2022-05-13 21:34:34 +00:00
alesapin
1c082717a4 Relax log level for some checks in check thread 2022-05-13 23:29:36 +02:00
Hakob Saghatelyan
e3726f9ec7 LIMIT restriction on query support added 2022-05-13 21:01:54 +00:00
Hakob Saghatelyan
3c601e5785 add Set support for ann_index_params setting 2022-05-13 20:59:56 +00:00
Vladimir Makarov
48222b1fd5 add select params 2022-05-13 20:42:50 +00:00
kssenii
58faaf0ea3 Merge master 2022-05-13 18:51:07 +02:00
Kseniia Sumarokova
0f07c41f19
Update MergeTreeIOSettings.h 2022-05-13 13:26:47 +02:00
Vladimir Makarov
d9054df684 rm extra files 2022-05-13 10:43:33 +00:00
Vladimir Makarov
5b69e4cc43 add cmake flags 2022-05-13 10:37:04 +00:00
kssenii
cde5cecc4d One more 2022-05-12 23:57:25 +02:00
kssenii
e5466a40fd Make sure async reads from remote fs related internal setting is turned on when needed 2022-05-12 19:00:19 +02:00
Dmitry Novik
df071bb701
Merge pull request #37129 from ClickHouse/enable-overcommit-for-user 
Update documentation and defaults for memory overcommit
 2022-05-12 14:38:43 +02:00
Vladimir Makarov
753d49d62f fix condition and add params 2022-05-12 12:30:18 +00:00
VVMak
6adbd516cc
Annoy tests (#58) 
* Added diskann to contrib

* Made cmake changes cleaner

* Implemented DiskANN aggregator

* Renamed index from Simple to DiskANN

* Implemented index aggregator

* Implemented index serialization

* Added condition to DiskANN index

* Implemented condition for diskann

* removed maybe_unused_attr

* Condition

* added some metrics and comparison operators

* Added common condition

* Added tests

* enchanced some tests

* Added inddex to tests

* Updated functional tests

* Added annoy tests

Co-authored-by: Danila Mishin <mishin.dk@phystech.edu>
Co-authored-by: Hakob Saghatelyan <sagatelyan.aa@phystech.edu>
 2022-05-12 13:49:31 +03:00
alesapin
5dcd25be23 Initial implementation 2022-05-12 00:04:54 +02:00
Maksim Kita
437d70d4da Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
4e7d10297b Fixed style 2022-05-11 21:59:51 +02:00
Maksim Kita
ce92f6aab1 Fixed tests 2022-05-11 21:59:51 +02:00
Maksim Kita
8ceb63ee6c Added JIT compilation of SortDescription 2022-05-11 21:59:51 +02:00
Dmitry Novik
17608b3d93 Update documentation and defaults for memory overcommit 2022-05-11 16:18:41 +00:00
Anton Kozlov
23e6792898 Lazy marks loading 2022-05-11 10:15:56 +00:00
Anton Popov
aec30c4076
Merge pull request #37053 from CurtizJ/remove-streams-comments 
Remove last mentions of data streams
 2022-05-10 13:38:13 +02:00
Anton Popov
e911900054 remove last mentions of data streams 2022-05-09 19:15:24 +00:00
Anton Ivashkin
6a6505c224 Merge master 2022-05-09 17:21:34 +03:00
alesapin
28e492bc17 Followup fix 2022-05-09 15:21:33 +02:00
alesapin
018ed10684 Add test 2022-05-09 15:21:21 +02:00
Vladimir Makarov
e8c7278c0d fix condition 2022-05-08 21:24:58 +00:00
VVMak
10bf7eea2c
Condition hot fix (#57) 
* Condition

* added some metrics and comparison operators

* Fix typo

* Update CommonCondition.h

* Added LpDistance and fixed bugs

* Limit support added, OrderBy query type support added, settings support added

* Metric support in alwaysUnknownOrTrue, cosmetic changes, documentation for the class

* Cosmetic changes

* Description corrected

* Update Settings.h

* hot fix

* misprints corrected

* Update CommonANNIndexes.cpp

Co-authored-by: Hakob Saghatelyan <sagatelyan.aa@phystech.edu>
Co-authored-by: Filatenkov Artur <58165623+FArthur-cmd@users.noreply.github.com>
 2022-05-08 23:58:59 +03:00
Vladimir Makarov
5f37e01339 condition 2022-05-08 19:30:33 +00:00
Robert Schulze
61cbcbf073
Enable clang-tidy readability-misleading-indentation 
Official docs:

  Correct indentation helps to understand code. Mismatch of the
  syntactical structure and the indentation of the code may hide serious
  problems.
 2022-05-08 19:12:01 +02:00
alesapin
2f650366a6 Merge branch 'master' into better_delete_outdated_parts 2022-05-08 12:55:42 +02:00
alesapin
46712f1d98 Fix forgotten parts in cleanup thread 2022-05-08 00:53:55 +02:00
Alexey Milovidov
fdbb5b75b2
Merge branch 'master' into minor-renames 2022-05-07 14:18:50 +03:00
Alexey Milovidov
4c6f964421
Merge pull request #36909 from ClickHouse/fixes_for_replicated_merge_tree 
Some fixes for replicated merge tree
 2022-05-06 00:44:21 +03:00
Alexander Tokmakov
45284108ba fix race with check thread 2022-05-05 14:16:26 +02:00
Nikolai Kochetov
d6735f06c7 Fixing spaces. 2022-05-05 09:24:34 +00:00
Nikolai Kochetov
5574a3cc43 Fixing style. 2022-05-05 09:23:23 +00:00
Nikolai Kochetov
b3d38301c9 Fix some more tests. 2022-05-05 09:19:12 +00:00
taiyang-li
d673bd096e fix issue: https://github.com/ClickHouse/ClickHouse/issues/36671 2022-05-05 14:48:33 +08:00
Vladimir Makarov
f53a3be4a8 add condition 2022-05-04 21:38:11 +00:00
Robert Schulze
c48d9a4174
Merge pull request #36844 from rschu1ze/shared_ptr_helper3 
Remove inherited create() method + disallow copying of storages
 2022-05-04 19:16:30 +02:00
Alexander Tokmakov
af9b4a5b9c fix intersecting parts 2022-05-04 16:22:06 +02:00
Antonio Andelic
808ea6dc65
Merge pull request #36888 from amosbird/projection-fix-one 
Fix incorrect query result when doing constant aggregation
 2022-05-04 10:34:40 +02:00
mergify[bot]
64084b5e32
Merge branch 'master' into shared_ptr_helper3 2022-05-03 20:46:16 +00:00
Nikolai Kochetov
fc3645a483 Fix some tests. 2022-05-03 19:32:24 +00:00
Amos Bird
7febf57439
Fix minmax_count_projection constant aggregation 2022-05-04 03:07:59 +08:00
Nikolai Kochetov
49e613a666 Fixing build. 2022-05-03 17:58:47 +00:00
Sergei Trifonov
1bcae160a0
Update MergeTreeBackgroundExecutor.h 2022-05-03 18:50:05 +02:00
Nikolai Kochetov
35095191eb Merge branch 'master' into refactor-something-in-part-volumes 2022-05-03 17:51:47 +02:00
Nikolai Kochetov
4a20e4f37e Fixing build. 2022-05-03 15:48:05 +00:00
Alexander Gololobov
367898da5e
Merge pull request #35318 from weeds085490/feat/add_part_offset 
feat(...): [LWD] support getting _part_offset of a row
 2022-05-03 13:27:02 +02:00
Nikolai Kochetov
540fafbe74
Merge pull request #36856 from ClickHouse/repro-lc-and-s3 
Reproduce and a little bit better fix for LC dict right offset.
 2022-05-03 11:32:08 +02:00
Robert Schulze
777b5bc15b
Don't let storages inherit from boost::noncopyable 
... IStorage has deleted copy ctor / assignment already
 2022-05-03 09:07:08 +02:00
Nikolai Kochetov
b10decc987 Fix style and formatting. 2022-05-02 18:59:43 +00:00
Nikolai Kochetov
3d9b87af09 Reproduce and a little bit better fix for LC dict right offset. 2022-05-02 18:06:47 +00:00
Robert Schulze
330212e0f4
Remove inherited create() method + disallow copying 
The original motivation for this commit was that shared_ptr_helper used
std::shared_ptr<>() which does two heap allocations instead of
make_shared<>() which does a single allocation. Turned out that
1. the affected code (--> Storages/) is not on a hot path (rendering the
performance argument moot ...)
2. yet copying Storage objects is potentially dangerous and was
   previously allowed.

Hence, this change

- removes shared_ptr_helper and as a result all inherited create() methods,

- instead, Storage objects are now created using make_shared<>() by the
  caller (for that to work, many constructors had to be made public), and

- all Storage classes were marked as noncopyable using boost::noncopyable.

In sum, we are (likely) not making things faster but the code becomes
cleaner and harder to misuse.
 2022-05-02 08:46:52 +02:00
Kseniia Sumarokova
66f62b2ba6
Merge pull request #36632 from DevTeamBK/Fix-Coverity-Issues 
Minor Coverity defects fixes
 2022-04-30 15:11:40 +02:00
mergify[bot]
cc08ccb420
Merge branch 'master' into remove-useless-code-2 2022-04-30 12:48:15 +00:00
Alexey Milovidov
1ddb04b992
Merge pull request #36715 from amosbird/refactorbase 
Reorganize source files so that base won't depend on Common
 2022-04-30 09:40:58 +03:00
Nikolai Kochetov
5807ca47b0
Merge pull request #36425 from ClickHouse/pool-size-hot-reload 
Allow to increase number of threads and tasks for background executors
 2022-04-29 17:24:12 +02:00
HeenaBansal2009
631b0154c0 Reverted change from ReadBufferFromRemoteFSGather.cpp 2022-04-29 07:21:35 -07:00
Anton Popov
19428353e7
Merge pull request #35797 from azat/nested-columns-fixes 
RFC: Fix ALTER DROP COLUMN of nested column with compact parts
 2022-04-29 15:09:05 +02:00
alesapin
c72596bae6
Merge pull request #36763 from kssenii/fix-async-reads 
Fix some issues with low cardinality and async reads from remote fs
 2022-04-29 12:18:19 +02:00
Alexey Milovidov
9fb1d92ff5
Merge pull request #36707 from CurtizJ/fix-vertical-merges 
Fix vertical merges in wide parts
 2022-04-29 10:09:50 +03:00
Azat Khuzhin
bf312c2a5b Fix ALTER DROP COLUMN of nested column with compact parts 
ALTER DROP COLUMN of nested column did not requires mutation before, and
so it leaves nested column as-is, and in case of compact parts
subsequent alter, that requires mutation, will trigger READ_COLUMN of
that nested column (because it exists in part), but it will fail because
there is no such column in the table already.

Here is example of such a failure on CI - [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/35459/52099b23a1cb9a7ff036c5c60aa037c999b333ef/stateless_tests__thread__actions__[1/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-29 07:12:34 +03:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
kssenii
84490d5198 Fix 2022-04-28 18:33:25 +02:00
HeenaBansal2009
3ac9434c57 Merge branch 'master' into Fix-Coverity-Issues 2022-04-28 08:14:50 -07:00
alesapin
af980ef4ef
Merge pull request #36371 from Joeywzr/add_query_level_settingss 
Add query level settings: parts_to_delay_insert and parts_to_throw_insert
 2022-04-28 11:08:14 +02:00
Anton Popov
eb810116a4 fix vertical merges in wide parts 2022-04-27 13:42:47 +00:00
Nikolai Kochetov
e44af67fee Merge branch 'master' into refactor-something-in-part-volumes 2022-04-26 21:08:00 +02:00
HeenaBansal2009
4961caec13 Merge branch 'master' into Fix-Coverity-Issues 2022-04-26 11:43:44 -07:00
Alexander Tokmakov
b1949ee901
Merge pull request #36680 from ClickHouse/better_logs_virtual_parts 
Better logs for virtual parts
 2022-04-26 21:36:20 +03:00
Alexander Tokmakov
d149ac8bd7 better logs for virtual parts 2022-04-26 16:58:40 +02:00
HeenaBansal2009
13a9a8fd5e Style check fix 2022-04-26 07:13:09 -07:00
HeenaBansal2009
267f426605 Minor Coverity defetcs fixes 2022-04-26 07:13:09 -07:00
kssenii
0c43b7b08a Merge master 2022-04-26 16:10:47 +02:00
kssenii
a395532d54 Add support for exact right bounded reads for some other buffers 2022-04-26 12:56:40 +02:00
Alexander Tokmakov
190becdcc3
Merge pull request #36644 from ClickHouse/fix_some_flaky_tests 
Fix some flaky tests
 2022-04-26 13:40:14 +03:00
Anton Popov
e051751012
Merge pull request #36637 from CurtizJ/dynamic-columns-11 
Fix merges of wide parts with type `Object`
 2022-04-26 10:34:04 +02:00
Alexey Milovidov
7683901689
Merge pull request #36630 from azat/mutator-cleanup 
Tiny Mutator code cleanup
 2022-04-26 08:24:50 +03:00
Memo
c38a4b4255
Update src/Storages/MergeTree/MergeTreeData.h 
Co-authored-by: alesapin <alesapin@gmail.com>
 2022-04-26 10:26:18 +08:00
mergify[bot]
a5aab53b70
Merge branch 'master' into add_query_level_settingss 2022-04-25 21:41:36 +00:00
mergify[bot]
178f244c8b
Merge branch 'master' into dynamic-columns-11 2022-04-25 21:36:56 +00:00
alesapin
92296484e7
Merge pull request #36348 from rschu1ze/erase_if3 
Replace remove-erase idiom by C++20 erase()/erase_if()
 2022-04-25 23:34:18 +02:00
Alexander Tokmakov
19a376b9f7 fix assertions 2022-04-25 22:41:46 +02:00
Anton Popov
d28b1559e6 fix merges of wide parts with type Object 2022-04-25 16:44:24 +00:00
mergify[bot]
265398d1b6
Merge branch 'master' into feat/add_part_offset 2022-04-25 15:58:16 +00:00
Azat Khuzhin
8b544e26d3 Move some functions from MergeTreeDataMergerMutator to MutateTask 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-25 18:16:38 +03:00
Azat Khuzhin
615d35e233 Remove dead function MergeTreeDataMergerMutator::splitMutationCommands() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-25 18:16:38 +03:00
Azat Khuzhin
8440fec3db Fix coding alignment in MutateTask 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-04-25 18:16:38 +03:00
Nikolai Kochetov
0fa63a8d65
Merge pull request #36549 from ClickHouse/interpolate-alias-fix 
Avoid projections if INTERPOLATE with expressions is present
 2022-04-25 13:18:00 +02:00
Alexander Gololobov
c484a1c269 Preserve order of virtual columns in the header and in data chunks when adding _part_offset together with other virtual columns 2022-04-25 12:10:56 +02:00
alesapin
edc78247f0
Merge pull request #36299 from ClickHouse/fix_trash 
Trying to fix some trash in zero copy replication
 2022-04-24 13:19:43 +02:00
Nikolai Kochetov
e7e6125b46 More fixes 2022-04-22 20:38:18 +00:00
alesapin
f183eabaa2 Return normal settings 2022-04-22 22:36:55 +02:00
Nikolai Kochetov
1805d60917 Fix mark cache. 2022-04-22 20:29:14 +00:00
alesapin
ebdd53ef8f Fix shared fetch 2022-04-22 19:18:18 +02:00
Nikolai Kochetov
8c00692844 Part 8 2022-04-22 16:58:09 +00:00
Nikita Mikhaylov
e475849761 Style and Docs update 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
bf6d3af536 Better 2022-04-22 13:56:16 +00:00
Nikita Mikhaylov
31ccb9c1c3 Done 2022-04-22 13:56:16 +00:00
Yakov Olkhovskiy
e4af79c2a0 avoid projections if INTERPOLATE with expressions is present 2022-04-22 09:47:59 -04:00
alesapin
64383bcb0a Better 2022-04-22 15:35:25 +02:00
xiedeyantu
550460c541 fix log should print from path 2022-04-22 18:30:40 +08:00
mergify[bot]
cd8fa75546
Merge branch 'master' into fix_trash 2022-04-21 21:45:31 +00:00
Nikolai Kochetov
9133e398b8 Part 7 2022-04-21 19:19:13 +00:00
Maksim Kita
57444fc7d3
Merge pull request #36444 from rschu1ze/clang-tidy-fixes 
Clang tidy fixes
 2022-04-21 16:11:27 +02:00
Dmitry Novik
77a82cc090
Merge pull request #35631 from amosbird/projection-fix1 
Fix broken SET reuse during projection analysis.
 2022-04-21 15:32:52 +02:00
alesapin
5465415751 Fix replace/move partition with zero copy replication 2022-04-21 14:39:12 +02:00
alesapin
cbfe00bb47 Fix tidy 2022-04-21 00:30:13 +02:00
alesapin
c14e2e0b96 Fix more 2022-04-20 21:08:26 +02:00
alesapin
6ee0df251f Add ditry settings 2022-04-20 15:11:03 +02:00
alesapin
41b57f78cf Revert extreme settings 2022-04-20 13:59:12 +02:00
alesapin
5334522d5c Extreme settings 2022-04-20 13:33:42 +02:00
alesapin
2eee79dc2a Review fixes 2022-04-20 13:26:20 +02:00
alesapin
40c15222f8 Merge branch 'master' into fix_trash 2022-04-20 12:45:49 +02:00
Antonio Andelic
d77678d1d1
Merge pull request #36410 from xinhuitian/move_down_data_range_construction 
Move down data_range construction in filterMarksUsingIndex and filterMarksUsingMergedIndex
 2022-04-20 12:18:05 +02:00
Alexander Tokmakov
1d30a97fd2 Merge branch 'master' into remove-useless-code-2 2022-04-20 11:45:56 +02:00
tavplubix
2c54381520
Merge pull request #36200 from ClickHouse/fix_create_as_replicated_default_args 
Fix CREATE AS for ReplicatedMergeTree with default arguments
 2022-04-20 12:01:13 +03:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings 
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
 2022-04-20 10:29:05 +02:00
Robert Schulze
b6d7367538
Merge remote-tracking branch 'origin/master' into erase_if3 
Conflicts:
- Interpreters/ActionsDAG.cpp
 2022-04-20 10:02:59 +02:00
Antonio Andelic
bbb0be6a44
Merge pull request #36402 from rschu1ze/clang-tidy-contains 
Activate clang-tidy warning "readability-container-contains"
 2022-04-20 08:15:46 +02:00
Nikolai Kochetov
bcbab2ead8 Part 6. 2022-04-19 19:34:41 +00:00
alesapin
aea7c7755a Fix style 2022-04-19 17:25:08 +02:00
alesapin
7cb7c120cc Less ugly 2022-04-19 15:53:10 +02:00
alesapin
cc06bc3d99 Add some clarifications 2022-04-19 14:01:30 +02:00
alesapin
ee8d26ff0e Better test and fix for move 2022-04-19 13:27:55 +02:00
alesapin
8ade9fba52 Fix 2022-04-19 11:56:14 +02:00
xinhuitian
5d0957e669 move down data range construction in filterMarksUsingIndex and filterMarksUsingMergedIndex 2022-04-19 13:26:33 +08:00
alesapin
bd7b3847c1 Some code 2022-04-19 01:09:09 +02:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains" 
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
 2022-04-18 23:53:11 +02:00
mergify[bot]
daca157111
Merge branch 'master' into fix_storage_distributed_ttl 2022-04-18 17:31:59 +00:00
Memo
a17f102fe6 fix style 2022-04-18 20:31:40 +08:00
alesapin
a97754f462 Fix 2022-04-18 13:39:09 +02:00
alesapin
c4b70f3ed3 Merge branch 'master' into fix_trash 2022-04-18 12:20:09 +02:00
Memo
b3adf150b5 add_query_level_settings 2022-04-18 12:15:41 +08:00
Alexey Milovidov
36595e4206
Merge pull request #36320 from ClickHouse/fix-clang-tidy-14 
Fix clang-tidy-14 (part 1)
 2022-04-18 07:02:10 +03:00
Alexey Milovidov
242919eddd Remove abbreviation 2022-04-18 01:02:49 +02:00
Robert Schulze
1e1df8e101
Replace remove-erase idiom by C++20 erase()/erase_if() 
- makes the code less verbose while being as efficient
 2022-04-17 12:04:47 +02:00
Alexey Milovidov
294efeccfe Fix clang-tidy-14 (part 1) 2022-04-16 04:54:04 +02:00
alesapin
1706ae9e15 Some trash implementation 2022-04-15 18:36:23 +02:00
Vladimir Makarov
48545bc390 Revert "part besides reader" 
This reverts commit 0914c1bb1e.
 2022-04-15 15:45:59 +00:00
Vladimir Makarov
0914c1bb1e part besides reader 2022-04-15 15:43:30 +00:00
alesapin
5a8419a48e Remove more trash 2022-04-15 17:05:17 +02:00