thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-17 11:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity
93,282 Commits 9,326 Branches 1,784 Tags 2.1 GiB
a8da5d96fc
Commit Graph

3777 Commits

Author SHA1 Message Date
alesapin
c57e07cb49 Style 2022-06-24 14:51:47 +02:00
alesapin
37310dc9df Simpler 2022-06-24 14:10:15 +02:00
alesapin
36b810b076 Remove unused parameter 2022-06-24 13:42:36 +02:00
alesapin
af1a9d18ab Remove transaction argument 2022-06-24 13:34:00 +02:00
alesapin
9910395823 Simplify method signature 2022-06-24 13:19:29 +02:00
Alexander Tokmakov
dbf2763788 automatically convert system database to Atomic 2022-06-23 21:38:43 +02:00
alesapin
2bdedf5e0c Some changes 2022-06-23 18:21:46 +02:00
mergify[bot]
174d8577fa
Merge branch 'master' into merge_tree_reader 2022-06-23 15:16:51 +00:00
alesapin
5661280ef6 Remove some read methods 2022-06-23 14:01:26 +02:00
Alexander Tokmakov
f00e6b5a7a deprecate old MergeTree syntax 2022-06-23 11:24:54 +02:00
alesapin
1b495ec8ad Fix style 2022-06-23 00:17:37 +02:00
Anton Popov
52db1b35a1 improve performace of insertion to columns of type JSON 2022-06-22 17:45:51 +00:00
Alexander Gololobov
0ee47363d4 Fixed includes 2022-06-22 19:08:18 +02:00
Alexander Gololobov
8138a3a36e Cleanups based on review comments 2022-06-22 19:08:01 +02:00
alesapin
ca0299e927 Ban projections for zero-copy replication in a right way 2022-06-22 19:01:46 +02:00
Alexander Gololobov
e5b55b965b Removed incorrect check 2022-06-22 17:23:09 +02:00
Alexander Gololobov
dbc6d1a159 Cleanups 2022-06-22 17:23:09 +02:00
Alexander Gololobov
aea5bc7527 Removed unused should_reorder flag 2022-06-22 17:23:09 +02:00
Alexander Gololobov
5427d5c51b Apply filter at row level security step 2022-06-22 17:23:09 +02:00
Alexander Gololobov
b3922461b3 Properly handle empty actions 2022-06-22 17:23:09 +02:00
Alexander Gololobov
ba89c3954c Do not add the same vitrual if it has been added by prev_reader 2022-06-22 17:23:09 +02:00
Alexander Gololobov
e5c0ebc7b5 Handle read from wide part when no columns are requested 2022-06-22 17:23:09 +02:00
Alexander Gololobov
a9e3b8d29e Don't read the same columns again 2022-06-22 17:23:09 +02:00
Alexander Gololobov
4e426c63cc Debuging test failures 2022-06-22 17:23:09 +02:00
Alexander Gololobov
6a26325fab Test dirty hacks for multiple PREWHERE steps 2022-06-22 17:23:05 +02:00
Alexander Gololobov
4360fd9798 Vector of structs instead of multiple vectors 2022-06-22 17:17:42 +02:00
Alexander Gololobov
87b669f439 Intermediate changes 2022-06-22 17:17:42 +02:00
Alexander Gololobov
159ab765fb Moved estimateNumRows from lambda to function 2022-06-22 17:17:42 +02:00
Alexander Gololobov
64a2f3734b Protect ReadResult internals from MergeTreeRangeReader clients 2022-06-22 17:17:42 +02:00
Nikolai Kochetov
cc6fdfe0eb
Merge pull request #36555 from ClickHouse/refactor-something-in-part-volumes 
Separate data storage abstraction for MergeTree
 2022-06-22 11:13:36 +02:00
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa 
Support for Clang Thread Safety Analysis (TSA)
 2022-06-21 20:19:33 +02:00
Nikolai Kochetov
dcf6e0d060 Roll back some changes from IPartMetadataManager 2022-06-21 13:05:49 +00:00
Nikolai Kochetov
854d148d73 Add some comments. Remove some commented code. 2022-06-21 12:31:02 +00:00
Nikolai Kochetov
46c4fd6cf7 Fixing build. 2022-06-21 11:15:06 +00:00
Kruglov Pavel
3c5a627bd7
Merge pull request #38239 from CurtizJ/fix-reading-from-s3 
Fix reading from s3 in some corner cases
 2022-06-21 12:57:55 +02:00
Nikolai Kochetov
1e8c9ecd4c Merge branch 'master' into refactor-something-in-part-volumes 2022-06-21 12:37:21 +02:00
alesapin
fafcb876be
Merge pull request #38182 from ClickHouse/disk_transaction 
Disk transaction
 2022-06-21 12:28:49 +02:00
Nikolai Kochetov
b8d27aa8dd
Merge pull request #37469 from azat/projections-optimize_aggregation_in_order 
Implement in order aggregation (optimize_aggregation_in_order) for projections for tables with fully materialized projections
 2022-06-21 12:17:35 +02:00
Nikolai Kochetov
e45d552f4d Do not create empty folder for in-memory parts. 2022-06-21 09:59:46 +00:00
Nikolai Kochetov
5232c17b89 Fix style 2022-06-21 07:30:45 +00:00
Nikolai Kochetov
3429a5e07a Fix 2022-06-21 07:26:43 +00:00
Nikolai Kochetov
dccf90b1ea Cleanup. 2022-06-20 18:18:17 +00:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-20 18:11:30 +00:00
Anton Popov
59a9ce6def fix reading from s3 in some corner cases 2022-06-20 15:24:38 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa 2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA) 
- TSA is a static analyzer build by Google which finds race conditions
  and deadlocks at compile time.

- It works by associating a shared member variable with a
  synchronization primitive that protects it. The compiler can then
  check at each access if proper locking happened before. A good
  introduction are [0] and [1].

- TSA requires some help by the programmer via annotations. Luckily,
  LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
  std::shared_mutex and std::scoped_lock. This commit enables them
  (--> contrib/libcxx-cmake/CMakeLists.txt).

- Further, this commit adds convenience macros for the low-level
  annotations for use in ClickHouse (--> base/defines.h). For
  demonstration, they are leveraged in a few places.

- As we compile with "-Wall -Wextra -Weverything", the required compiler
  flag "-Wthread-safety-analysis" was already enabled. Negative checks
  are an experimental feature of TSA and disabled
  (--> cmake/warnings.cmake). Compile times did not increase noticeably.

- TSA is used in a few places with simple locking. I tried TSA also
  where locking is more complex. The problem was usually that it is
  unclear which data is protected by which lock :-(. But there was
  definitely some weird code where locking looked broken. So there is
  some potential to find bugs.

*** Limitations of TSA besides the ones listed in [1]:

- The programmer needs to know which lock protects which piece of shared
  data. This is not always easy for large classes.

- Two synchronization primitives used in ClickHouse are not annotated in
  libcxx:
  (1) std::unique_lock: A releaseable lock handle often together with
      std::condition_variable, e.g. in solve producer-consumer problems.
  (2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
      considered a design flaw + typically it is slower than a standard
      mutex. In this commit, one std::recursive_mutex was converted to
      std::mutex and annotated with TSA.

- For free-standing functions (e.g. helper functions) which are passed
  shared data members, it can be tricky to specify the associated lock.
  This is because the annotations use the normal C++ rules for symbol
  resolution.

[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
 2022-06-20 16:13:25 +02:00
Nikolai Kochetov
39b02d4ab8 Continure resolving conflicts. 2022-06-20 15:39:05 +02:00
Nikolai Kochetov
7452d04e3a Merge branch 'master' into refactor-something-in-part-volumes 2022-06-20 15:31:02 +02:00
Nikolai Kochetov
ef8b967c3f Fix more tests. 2022-06-20 13:07:14 +00:00
Vitaly Baranov
06c4082a7c
Merge pull request #37358 from vitlibar/backup-improvements-6 
Backup Improvements 6
 2022-06-20 14:13:30 +02:00
alesapin
08b5dd091b Fix double negative 2022-06-19 17:12:04 +02:00
Vitaly Baranov
8a7c970ce0 Fix style. 2022-06-19 15:58:26 +02:00
alesapin
50801e41c5 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-19 14:05:46 +02:00
Alexander Tokmakov
83adf56383 fix race 2022-06-17 18:13:57 +02:00
Alexander Tokmakov
e9c2157f01 Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-17 15:57:30 +02:00
Alexander Tokmakov
5ffc38d9b9 fix style check, add comments 2022-06-17 13:47:14 +02:00
alesapin
e5b405701a Merge branch 'master' into disk_transaction 2022-06-17 12:16:12 +02:00
alesapin
1c7a7da487
Merge pull request #38088 from kssenii/diff-with-cache 
Extract some diff from pr #36171
 2022-06-17 12:07:31 +02:00
Alexander Tokmakov
39c0219c11 fixes 2022-06-16 19:41:32 +02:00
Anton Popov
13ec7e3092
Merge pull request #37978 from CurtizJ/fix-sparse-s3 
Fix reading of sparse columns from s3
 2022-06-16 15:32:54 +02:00
alesapin
379fff7318 Merge branch 'master' into disk_transaction 2022-06-16 14:36:19 +02:00
Azat Khuzhin
4ff82eb9d5 Implement optimize_aggregation_in_order for projections 
v2: use real column name instead of aliases from GROUP BY
    Fixes the following error in 01710_projection_aggregation_in_order:

        Not found column a in block. There are only columns: toStartOfHour(ts), sum(value). (NOT_FOUND_COLUMN_IN_BLOCK)
v2.1: Get back support for projected and non-projected parts
v2.2: merge tests and rename
v3: Reduce copy-paste for optimize_aggregation_in_order for projections
v4: rebase on top of QueryPlanResourceHolder
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
14f9491619 MergeTreeData: preserve order for group_by_elements_order_descr 
This is required for proper optimize_aggregation_in_order for
projections.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
4694929623 Implement merging only for AggregatingStep 
v2: fill AggregateColumnsConstData only for only_merge
    (fixes 01291_aggregation_in_order and some other tests)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:36 +03:00
Azat Khuzhin
682c93e2e1 Use proper setting compile_aggregate_expressions in MergeTreeDataSelectExecutor 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 09:58:35 +03:00
Alexander Tokmakov
6ddfec47d2 maybe fix livelock on queue processing 2022-06-15 22:22:27 +02:00
Alexander Tokmakov
5e2b42c85a Merge branch 'master' into fix_flaky_tests_with_transactions 2022-06-15 20:47:02 +02:00
Vitaly Baranov
1198e86295 Fix storing temporary tables and skipping system tables while making a backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
21f3bed435 Simplify path calculations in backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
592f568f83 Move backup/restore code to storages and databases - part 2. 2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57 Move backup/restore code to storages and databases - part 1. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21 Rework collecting replicated parts. 2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982 Restore parts of MergeTree in correct order. 2022-06-15 20:28:40 +02:00
Daniel Kutenin
a769dea8ef
Merge branch 'ClickHouse:master' into master 2022-06-15 19:12:10 +01:00
Nikolai Kochetov
25b9990737 Fixing build 2022-06-15 16:32:22 +00:00
Alexander Tokmakov
2ac72319bd
Merge pull request #37185 from amosbird/projection-fix-three 
Fix possible heap-use-after-free error when reading system.projection_parts and system.projection_parts_columns
 2022-06-15 19:00:10 +03:00
kssenii
500f49972b Extract diff from PR 36171 2022-06-15 16:40:18 +02:00
Alexander Tokmakov
e4d4d0a854 fix race on queue processing 2022-06-15 16:21:26 +02:00
Nikolai Kochetov
2a9a63ac7b Merge branch 'master' into refactor-something-in-part-volumes 2022-06-15 15:35:26 +02:00
Danila Kutenin
08e3f77a9c Optimize most important parts with NEON SIMD 
First part, updated most UTF8, hashing, memory and codecs. Except
utf8lower and upper, maybe a little later.

That includes huge amount of research with movemask dealing. Exact
details and blog post TBD.
 2022-06-15 13:19:29 +00:00
alesapin
1b66166501 Merge branch 'master' into disk_transaction 2022-06-15 13:54:10 +02:00
alesapin
b557c1dd80 Interm stage 2022-06-15 13:51:21 +02:00
Nikolai Kochetov
2ee349f29c Fix test_encrypted_disk 2022-06-15 11:42:59 +00:00
Nikolai Kochetov
a274bec0fa Fix test_encrypted_disk 2022-06-15 11:41:08 +00:00
Alexander Tokmakov
56f2121c1a
Revert "Add backoff to merges in replicated queue if always_fetch_merged_part is enabled" 2022-06-15 13:04:01 +03:00
alesapin
af1cd745e1
Merge pull request #37975 from kssenii/clean-up-broken-detached 
Clean up broken detached parts after timeout
 2022-06-14 20:53:31 +02:00
Amos Bird
d5a7a5be8e
Fix use-after-free in system.projection_parts 2022-06-14 23:41:42 +08:00
alesapin
3b832a25f6
Merge pull request #37995 from ClickHouse/fix_queue 
Add backoff to merges in replicated queue if `always_fetch_merged_part` is enabled
 2022-06-14 11:38:44 +02:00
kssenii
0d2dae0046 Add const, rename settings 2022-06-14 00:47:03 +02:00
kssenii
74cffaf186 Merge master 2022-06-14 00:45:19 +02:00
kssenii
34e63b20fb Review fixes 2022-06-13 21:27:54 +02:00
Anton Popov
e6517164f2 add test and comment 2022-06-13 17:30:58 +00:00
Antonio Andelic
b9cf6fe367 Merge branch 'master' into pp-system-unfreeze 2022-06-13 14:46:30 +00:00
alesapin
d984763b95
Update src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-06-13 15:10:20 +02:00
kssenii
debe467ee7 Review fixes 2022-06-13 14:25:42 +02:00
kssenii
4859e11c27 Review fixes 2022-06-12 01:15:10 +02:00
alesapin
2a81904fbc Revert "Revert "Fix bug with removal of moving parts"" 
This reverts commit c0581bf741.
 2022-06-12 01:08:24 +02:00
alesapin
c0581bf741 Revert "Fix bug with removal of moving parts" 
This reverts commit e32133ff8f.
 2022-06-10 19:35:57 +02:00
alesapin
301eac7d59 Comment 2022-06-10 19:34:03 +02:00
alesapin
ffff1c22a6 Add backoff for merges when always_fetch_merged_part turned on 2022-06-10 19:33:28 +02:00
alesapin
e32133ff8f Fix bug with removal of moving parts 2022-06-10 18:47:06 +02:00
kssenii
fd6c84b831 Review fixes 2022-06-10 14:23:13 +02:00
Anton Popov
6a0b305a14 fix reading of sparse columns from s3 2022-06-10 12:09:37 +00:00
kssenii
498f389c21 Merge master 2022-06-10 12:49:10 +02:00
kssenii
7a2676c7ab Clean up broken detached parts with timeout 2022-06-10 12:27:57 +02:00
Kseniia Sumarokova
508676ab4e
Merge pull request #37804 from ClickHouse/metadata_storage 
Metadata storage abstraction for Disks
 2022-06-10 11:43:28 +02:00
Alexander Tokmakov
780f7c87c7
Merge pull request #36113 from ClickHouse/remove-useless-code-2 
Remove useless code in ReplicatedMergeTreeRestartingThread
 2022-06-09 17:56:28 +03:00
Vadim Volodin
637d293fbd Add SYSTEM UNFREEZE query 2022-06-08 15:21:14 +03:00
alesapin
81dca513de Review fixes 2022-06-08 13:28:33 +02:00
Nikolai Kochetov
5bc9b32025 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-08 11:10:06 +00:00
alesapin
b88e1a2bef Merge branch 'master' into metadata_storage 2022-06-08 13:02:21 +02:00
alesapin
9fdc783eac
Merge pull request #37906 from ClickHouse/remove_tmp_directories_on_replicated 
Remove temporary `delete_tmp_` on clickhouse server start
 2022-06-08 12:35:51 +02:00
alesapin
e32d36d790 Proper fix 2022-06-07 17:58:32 +02:00
Nikolai Kochetov
678d978acf Merge branch 'master' into refactor-something-in-part-volumes 2022-06-07 15:23:00 +00:00
mergify[bot]
2d01abf871
Merge branch 'master' into revert-37647-Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-07 13:32:30 +00:00
Nikita Taranov
d09ad5fcd5
Fix build 
https://s3.amazonaws.com/clickhouse-test-reports/36593/a4b9614385e0ecd1654d15f507159678758ec557/fast_test__actions_/build_log.txt
 2022-06-07 13:56:03 +02:00
Anton Popov
df6882d2b9
Revert "Fix errors of CheckTriviallyCopyableMove type" 2022-06-07 13:53:10 +02:00
Anton Popov
ef6f5a6500
Merge pull request #37570 from azat/column-ttl-expired-fix 
Do not write expired columns by TTL after subsequent merges
 2022-06-07 13:05:03 +02:00
mergify[bot]
dd0603a595
Merge branch 'master' into metadata_storage 2022-06-06 18:59:45 +00:00
Anton Popov
d40b23272e
Merge pull request #37755 from CurtizJ/fix-mutations-again 
Return back #37266
 2022-06-06 20:22:03 +02:00
Robert Schulze
2d87af2a15
Merge pull request #37647 from DevTeamBK/Fix-all-CheckTriviallyCopyableMove-Errors 
Fix errors of CheckTriviallyCopyableMove type
 2022-06-05 19:58:47 +02:00
alesapin
a25a01af16 Fix WAL 2022-06-05 16:15:11 +02:00
alesapin
38df94666b Fix merge tree wal 2022-06-05 15:09:39 +02:00
alesapin
a2704bc541 Trying to fix WAL log 2022-06-05 01:05:15 +02:00
Alexey Milovidov
f860407af5
Merge pull request #37777 from ClickHouse/avx512_tail_zero 
Use multitarget framework for numZerosInTail implementation
 2022-06-04 01:46:56 +03:00
Nikolai Kochetov
3f67abc3b0 Fix some tests. 2022-06-03 18:49:12 +00:00
mergify[bot]
ddf7210ecc
Merge branch 'master' into remove-useless-code-2 2022-06-03 13:58:45 +00:00
Maksim Kita
8469e0d10d
ExecutableLambdaAdapter forward job and callback (#37830) 2022-06-03 15:54:47 +02:00
alesapin
ac03e3af3e
Merge branch 'master' into metadata_storage 2022-06-03 14:42:21 +02:00
Anton Popov
5cf2c67410
Merge branch 'master' into fix-mutations-again 2022-06-03 14:17:05 +02:00
Nikolai Kochetov
8991f39412 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 17:00:08 +00:00
alesapin
46bccae078 Trying to create metadata layer 2022-06-02 18:09:40 +02:00
Nikolai Kochetov
25c8a437ea Fixing build. 2022-06-02 15:00:07 +00:00
HeenaBansal2009
e3080f2a97 Merge remote-tracking branch 'origin' into Fix-all-CheckTriviallyCopyableMove-Errors 2022-06-02 07:30:08 -07:00
mergify[bot]
fbfaaa9c74
Merge branch 'master' into fix-mutations-again 2022-06-02 11:57:19 +00:00
Alexander Tokmakov
45657d203a
Merge pull request #37763 from ClickHouse/fix_replace_range_again 
Fix flaky 01154_move_partition_long
 2022-06-02 14:15:16 +03:00
Alexander Gololobov
0b004b907a Added AVX2 implementation of numZerosInTail 2022-06-02 13:01:13 +02:00
Nikolai Kochetov
fc726c236f
Merge pull request #37661 from ClickHouse/no_alias_actions 
Trying to remove prewhere_info.alias_actions as they seem to always be included in prewhere_actions
 2022-06-02 12:31:04 +02:00
Nikolai Kochetov
89c5855d20 Merge branch 'master' into refactor-something-in-part-volumes 2022-06-02 12:19:07 +02:00
Nikita Mikhaylov
d34e051c69
Support for simultaneous read from local and remote parallel replica (#37204) 2022-06-02 11:46:33 +02:00
Nikolai Kochetov
edac3d6714 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-06-02 09:36:20 +00:00
Alexander Gololobov
d8e4dcf9e2 Use multitarget framework for numZerosInTail implementation 2022-06-02 11:23:44 +02:00
mergify[bot]
88bed9de20
Merge branch 'master' into fix_replace_range_again 2022-06-02 07:05:44 +00:00
Alexander Gololobov
a0cf902d49
Merge pull request #37588 from yaqi-zhao/avx512_tail_zero 
add avx512 support for mergetree reader
 2022-06-02 08:51:52 +03:00
lthaooo
6632616733
Fix TTL merge scheduling bug (#36387) 2022-06-01 21:09:53 +02:00
Alexander Tokmakov
06f80770b8 fix stuck REPALCE_RANGE 2022-06-01 20:11:53 +02:00
Sema Checherinda
2626a49616 FR: Expose what triggered the merge in system.part_log #26255 2022-06-01 16:58:06 +02:00
Anton Popov
cae2478b3f Revert "Merge pull request #37355 from ClickHouse/revert-37266-fix-mutations-with-object" 
This reverts commit a53cfa9fca, reversing
changes made to 9acb42fcdb.
 2022-06-01 10:57:20 +00:00
Alexander Gololobov
26609a1875 Style fixes 2022-05-31 21:41:10 +02:00
Nikolai Kochetov
86fbb74703 Merge branch 'master' into refactor-read-metrics-and-callbacks 2022-05-31 18:07:47 +00:00