Anton Popov
6863cd152f
Merge pull request #42181 from CurtizJ/optimize-loading-parts
...
Do not load inactive parts at startup
2023-01-16 14:38:50 +01:00
serxa
693489a8ad
review fixes
2023-01-12 15:51:04 +00:00
Sergei Trifonov
12d8543578
Merge branch 'master' into cancellable-mutex-integration
2023-01-12 16:03:49 +01:00
Igor Nikonov
edbcb24da0
Merge pull request #44954 from ClickHouse/fix-insert-delay
...
Fix: insert delay calculation
2023-01-12 09:43:00 +01:00
Nikolai Kochetov
4673b3fe1d
Revert "Revert "Custom reading for mutation""
2023-01-10 16:31:01 +01:00
Alexander Tokmakov
c8ec130be4
Revert "Custom reading for mutation"
2023-01-10 17:51:30 +03:00
Anton Popov
b9bf92fa5d
check for network errors while loading parts
2023-01-10 11:55:08 +00:00
Nikolai Kochetov
3e430b1b6e
Add comments. Remove unused flag from Context.
2023-01-09 19:52:11 +00:00
serxa
15bb127b01
replace every std::shared_mutex with DB::FastSharedMutex
2023-01-06 23:35:38 +00:00
Igor Nikonov
a67afdff6a
Fix: insert delay calculation
2023-01-05 22:43:41 +00:00
Anton Popov
8d2788445b
fix style check
2022-12-28 19:29:56 +00:00
Anton Popov
2adf0e9db3
do not wait loading parts at shutdown
2022-12-28 16:07:14 +00:00
Anton Popov
0722020cae
Merge remote-tracking branch 'upstream/master' into HEAD
2022-12-23 18:15:55 +00:00
alesapin
fa467b3398
Minor code polishing
2022-12-22 14:31:42 +01:00
Anton Popov
b7e76e3539
do not hold data parts during insert
2022-12-15 23:57:10 +00:00
Alexander Tokmakov
f4c8281ccc
avoid outdating part twice
2022-12-13 21:24:26 +01:00
Anton Popov
a2fee110b8
add query for waiting of loading parts and fix test_transactions
2022-12-12 14:33:42 +00:00
Anton Popov
df6d2f2163
Merge remote-tracking branch 'upstream/master' into HEAD
2022-11-29 23:21:23 +00:00
Alexander Tokmakov
5cc99312d7
provide hint for loading uncommitted blocks in merge predicate
2022-11-25 16:41:20 +01:00
Sema Checherinda
a29437e179
delete verbose logging about cleaning old parts
2022-11-23 15:16:09 +00:00
Sema Checherinda
3a8024002d
fix race in read full path from deleting part, add more logs in grabOldParts
2022-11-23 15:16:09 +00:00
Sema Checherinda
4f592bc208
fix isolation breach, droping outdated part should't be ignored
2022-11-23 15:16:09 +00:00
Sema Checherinda
6b4cb25e25
fixed 01130_in_memory_parts_partitons, remove flushed tmp part from in memory part
2022-11-23 15:16:09 +00:00
Sema Checherinda
c976b28104
less pedantic checks in operations, rely on existed tx functionality
2022-11-23 15:16:09 +00:00
Sema Checherinda
c1e6b560f0
no groupping parts for cover with empty
2022-11-23 15:16:08 +00:00
Sema Checherinda
db86cd0f0a
fix several critical races, fix tests
2022-11-23 15:16:08 +00:00
Sema Checherinda
15012c7070
address the review comments
2022-11-23 15:16:08 +00:00
Sema Checherinda
fb86019db9
Delete parts over replacing them with empty parts
2022-11-23 15:16:08 +00:00
Anton Popov
75c807609b
add some comments
2022-11-22 16:11:01 +00:00
Anton Popov
e3db29aebf
better waiting for outdated data parts
2022-11-22 15:15:20 +00:00
Anton Popov
92ac9385b9
wait loading of outdated parts for partition commands
2022-11-21 17:01:46 +00:00
Anton Popov
d96ed50610
fix race
2022-11-16 13:21:07 +00:00
Anton Popov
f94182384a
Merge remote-tracking branch 'upstream/master' into HEAD
2022-11-16 03:30:29 +00:00
Anton Popov
2ae3cfa9e0
Merge branch 'master' into dynamic-columns-14
2022-10-31 16:15:19 +01:00
Alexander Tokmakov
599ccb9939
Update MergeTreeData.h
2022-10-31 16:18:17 +03:00
Alexander Tokmakov
b2b9479afa
Update src/Storages/MergeTree/MergeTreeData.h
...
Co-authored-by: Sergei Trifonov <sergei@clickhouse.com>
2022-10-31 16:15:35 +03:00
Alexander Tokmakov
f53df7870c
fix race between drop and failed insert
2022-10-30 17:59:47 +01:00
Alexander Tokmakov
c68ab231f9
fix accessing part in Deleting state
2022-10-30 17:30:51 +01:00
Anton Popov
b40d9200d2
better semantic of constsness of DataPartStorage
2022-10-23 15:24:20 +00:00
Anton Popov
56e5daba0c
remove DataPartStorageBuilder
2022-10-23 00:23:15 +00:00
Anton Popov
95fdb0a877
remove outdated parts asynchronously
2022-10-14 14:52:57 +00:00
Anton Popov
71a223c1c9
Merge remote-tracking branch 'upstream/master' into optimize-loading-parts
2022-10-10 11:31:47 +00:00
Anton Popov
286da7ee19
fix loading of parts with old syntax
2022-10-09 15:39:20 +00:00
Anton Popov
9f1104c2ad
do not load inactive parts
2022-10-08 01:39:46 +00:00
Alexey Milovidov
169c505603
Relax "too many parts" thresholds
2022-10-08 00:24:51 +02:00
Anton Popov
6e61cf92f5
Merge remote-tracking branch 'upstream/master' into HEAD
2022-10-03 13:16:57 +00:00
Alexander Tokmakov
7b5b7e4963
Merge pull request #41945 from ClickHouse/enable_more_s3_tests
...
Enable some disabled S3 tests
2022-10-03 13:25:50 +03:00
Vitaly Baranov
7be9744bab
Remove confusing warning when inserting with perform_ttl_move_on_insert=false.
2022-09-30 20:51:48 +02:00
alesapin
350ff96628
Fix potential race condition with broken parts
2022-09-30 19:01:06 +02:00
alesapin
695133cc5b
Better mutations
2022-09-27 15:23:02 +02:00
alesapin
8ba74d8655
Fix bugs found by CI
2022-09-20 14:00:57 +02:00
Alexander Tokmakov
8301f5758b
Update IMergeTreeDataPart.cpp
2022-09-15 21:32:57 +02:00
Anton Popov
ee54451741
fix adding of columns of type Object
2022-09-13 22:43:59 +00:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze
...
Fix SYSTEM UNFREEZE for ordinary database
2022-09-13 14:55:11 +03:00
Anton Popov
d1fa2148de
Merge branch 'master' into dynamic-columns-14
2022-09-09 19:32:07 +02:00
Alexander Tokmakov
48927ba0ac
Merge branch 'master' into no-hardlinks-while-making-backup-of-mergetree-in-atomic-db
2022-09-09 14:24:44 +03:00
Anton Popov
ba41239ecd
Merge remote-tracking branch 'upstream/master' into HEAD
2022-09-08 15:20:29 +00:00
Alexander Tokmakov
507ad0c4d9
Merge pull request #40691 from azat/mergetree/transaction-wal-fix
...
Move committing InMemory parts to WAL out of NOEXCEPT_SCOPE()
2022-09-08 15:55:30 +03:00
Vitaly Baranov
122009a2bd
Use table lock if database is ordinary and zero-copy-replication is enabled.
2022-09-08 13:54:59 +02:00
Vitaly Baranov
9c847ceec9
No hardlinks while making backup of MergeTree in atomic database.
2022-09-07 11:44:50 +02:00
Anton Popov
f0a404e2c8
Merge remote-tracking branch 'upstream/master' into HEAD
2022-09-06 15:51:16 +00:00
Kseniia Sumarokova
3558361a05
Merge branch 'master' into refactor-merge-tree-read
2022-09-06 16:00:43 +02:00
Alexey Milovidov
940a53e519
Merge pull request #40984 from Lucky-Chang/typo_fix
...
Fix some typos and clang-tidy warnings
2022-09-06 02:37:50 +03:00
kssenii
83514fa2ef
Refactor
2022-09-05 20:08:22 +02:00
ianton-ru
39e1fc7a0f
Merge branch 'master' into fix-ordinary-system-unfreeze
2022-09-05 17:10:59 +03:00
Luck-Chang
1ac8e739c9
fix some typos and clang-tidy warnings
2022-09-05 09:50:24 +08:00
Azat Khuzhin
2e85f9f0ad
Remove completely processed WAL files
...
Previously all WAL files had been stored, though with the time of use
this can take too much space on disk, and also the startup time will be
increased.
But it is pretty easy to prune old WAL files (the one parts from which
had been completely written to disk already).
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-04 14:18:14 +02:00
Azat Khuzhin
3c478da003
Move committing InMemory parts to WAL out of NOEXCEPT_SCOPE()
...
Since this commit can definitelly throw (i.e. due to ENOSPC).
Note, that it should be safe, since rollback() will call dropPart() for
those parts.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-26 23:37:59 +02:00
alesapin
4398a357c8
Merge pull request #40151 from ClickHouse/fix_cannot_quickly_remove_directory
...
Fix "Cannot quickly remove directory"
2022-08-17 12:11:18 +02:00
Alexander Tokmakov
e691888267
fix build
2022-08-12 13:03:57 +02:00
Alexander Tokmakov
fe572104aa
fix old tmp dirs cleanup
2022-08-09 18:44:51 +02:00
Alexander Gololobov
a64aa00869
Merge pull request #37893 from zhangjmruc/feature/sql-standard-delete
...
Support SQL standard "delete from ... where ..." syntax and lightweight implementation on merge tree tables
2022-07-26 23:39:07 +02:00
Kruglov Pavel
c683cb252f
Merge pull request #39227 from amosbird/rename-log1
...
Rename log when rename merge tree tables
2022-07-26 17:12:44 +02:00
Alexander Gololobov
25deba2c1b
Merge branch 'master' into feature/sql-standard-delete
2022-07-25 22:13:20 +02:00
Alexander Gololobov
460950ecdc
Merge branch 'master' into feature/sql-standard-delete
2022-07-24 21:27:22 +02:00
Alexander Gololobov
c8b3c574a4
Disable lightweight delete if table has projections
2022-07-24 12:21:47 +02:00
mergify[bot]
c01ff2d38a
Merge branch 'master' into improve_replicated_merge_logging
2022-07-23 19:10:10 +00:00
Nikolai Kochetov
91043351aa
Fixing build.
2022-07-20 20:30:16 +00:00
Alexander Tokmakov
d82f378a9d
do not enqueue uneeded parts for check
2022-07-18 23:37:07 +02:00
Raúl Marín
aea045f297
Improve logging around replicated merges
2022-07-15 15:48:35 +02:00
jianmei zhang
b4a37e1e22
Disable optimizations for count() when lightweight delete exists, add hasLightweightDelete() function in IMergeTreeDataPart
2022-07-15 12:32:41 +08:00
Amos Bird
96bb6e0cd2
Rename log when rename merge tree tables
2022-07-14 21:22:46 +08:00
Vitaly Baranov
ed27987646
Merge pull request #38861 from vitlibar/backup-improvements-9
...
Backup Improvements 9
2022-07-07 02:24:47 +02:00
Vitaly Baranov
7f84cf3968
Fix style.
2022-07-06 16:36:59 +02:00
Vitaly Baranov
43d35eec1b
Write unfinished mutations to backup.
2022-07-05 14:51:09 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder
2022-07-02 15:39:44 +02:00
alesapin
eb5046ab26
Simplify everything
2022-06-30 22:51:27 +02:00
alesapin
cb90aca2ef
Some changes in merge tree
2022-06-30 14:12:45 +02:00
Vitaly Baranov
031ca28fdc
Add test for partition clause. More checks for data compatibility on restore.
2022-06-30 08:37:18 +02:00
alesapin
0a3fab1cb6
Some sad changes
2022-06-28 12:51:49 +02:00
alesapin
d88cdcd5c1
AAA
2022-06-27 21:41:29 +02:00
Vadim Volodin
3296ba2532
Fix SYSTEM UNFREEZE for ordinary database
2022-06-24 20:46:16 +03:00
alesapin
d963f262f8
Fix style
2022-06-24 17:43:18 +02:00
alesapin
f685cf2268
Fix comment
2022-06-24 17:33:43 +02:00
alesapin
011d58d7a0
Simplify more
2022-06-24 17:19:59 +02:00
alesapin
612c4571d5
Split method into smaller
2022-06-24 15:41:09 +02:00
alesapin
7517e1f4d5
Remove some complexity
2022-06-24 15:24:02 +02:00
alesapin
37310dc9df
Simpler
2022-06-24 14:10:15 +02:00
alesapin
36b810b076
Remove unused parameter
2022-06-24 13:42:36 +02:00
alesapin
af1a9d18ab
Remove transaction argument
2022-06-24 13:34:00 +02:00
alesapin
9910395823
Simplify method signature
2022-06-24 13:19:29 +02:00
Nikolai Kochetov
cc6fdfe0eb
Merge pull request #36555 from ClickHouse/refactor-something-in-part-volumes
...
Separate data storage abstraction for MergeTree
2022-06-22 11:13:36 +02:00
Robert Schulze
0d80874d40
Merge pull request #38068 from ClickHouse/clang-tsa
...
Support for Clang Thread Safety Analysis (TSA)
2022-06-21 20:19:33 +02:00
Nikolai Kochetov
1e8c9ecd4c
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-21 12:37:21 +02:00
mergify[bot]
9bdd9e14a6
Merge branch 'master' into fix_flaky_tests_with_transactions
2022-06-20 18:11:30 +00:00
Robert Schulze
55b39e709d
Merge remote-tracking branch 'origin/master' into clang-tsa
2022-06-20 16:39:32 +02:00
Robert Schulze
5a4f21c50f
Support for Clang Thread Safety Analysis (TSA)
...
- TSA is a static analyzer build by Google which finds race conditions
and deadlocks at compile time.
- It works by associating a shared member variable with a
synchronization primitive that protects it. The compiler can then
check at each access if proper locking happened before. A good
introduction are [0] and [1].
- TSA requires some help by the programmer via annotations. Luckily,
LLVM's libcxx already has annotations for std::mutex, std::lock_guard,
std::shared_mutex and std::scoped_lock. This commit enables them
(--> contrib/libcxx-cmake/CMakeLists.txt).
- Further, this commit adds convenience macros for the low-level
annotations for use in ClickHouse (--> base/defines.h). For
demonstration, they are leveraged in a few places.
- As we compile with "-Wall -Wextra -Weverything", the required compiler
flag "-Wthread-safety-analysis" was already enabled. Negative checks
are an experimental feature of TSA and disabled
(--> cmake/warnings.cmake). Compile times did not increase noticeably.
- TSA is used in a few places with simple locking. I tried TSA also
where locking is more complex. The problem was usually that it is
unclear which data is protected by which lock :-(. But there was
definitely some weird code where locking looked broken. So there is
some potential to find bugs.
*** Limitations of TSA besides the ones listed in [1]:
- The programmer needs to know which lock protects which piece of shared
data. This is not always easy for large classes.
- Two synchronization primitives used in ClickHouse are not annotated in
libcxx:
(1) std::unique_lock: A releaseable lock handle often together with
std::condition_variable, e.g. in solve producer-consumer problems.
(2) std::recursive_mutex: A re-entrant mutex variant. Its usage can be
considered a design flaw + typically it is slower than a standard
mutex. In this commit, one std::recursive_mutex was converted to
std::mutex and annotated with TSA.
- For free-standing functions (e.g. helper functions) which are passed
shared data members, it can be tricky to specify the associated lock.
This is because the annotations use the normal C++ rules for symbol
resolution.
[0] https://clang.llvm.org/docs/ThreadSafetyAnalysis.html
[1] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42958.pdf
2022-06-20 16:13:25 +02:00
Nikolai Kochetov
7452d04e3a
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-20 15:31:02 +02:00
alesapin
50801e41c5
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-19 14:05:46 +02:00
Alexander Tokmakov
83adf56383
fix race
2022-06-17 18:13:57 +02:00
Alexander Tokmakov
39c0219c11
fixes
2022-06-16 19:41:32 +02:00
Vitaly Baranov
21f3bed435
Simplify path calculations in backup.
2022-06-15 20:32:34 +02:00
Vitaly Baranov
592f568f83
Move backup/restore code to storages and databases - part 2.
2022-06-15 20:32:31 +02:00
Vitaly Baranov
724bc4dc57
Move backup/restore code to storages and databases - part 1.
2022-06-15 20:28:43 +02:00
Vitaly Baranov
73b1894a21
Rework collecting replicated parts.
2022-06-15 20:28:42 +02:00
Vitaly Baranov
5cabdbd982
Restore parts of MergeTree in correct order.
2022-06-15 20:28:40 +02:00
Alexander Tokmakov
2ac72319bd
Merge pull request #37185 from amosbird/projection-fix-three
...
Fix possible heap-use-after-free error when reading system.projection_parts and system.projection_parts_columns
2022-06-15 19:00:10 +03:00
Nikolai Kochetov
2a9a63ac7b
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-15 15:35:26 +02:00
alesapin
af1cd745e1
Merge pull request #37975 from kssenii/clean-up-broken-detached
...
Clean up broken detached parts after timeout
2022-06-14 20:53:31 +02:00
Amos Bird
d5a7a5be8e
Fix use-after-free in system.projection_parts
2022-06-14 23:41:42 +08:00
kssenii
7a2676c7ab
Clean up broken detached parts with timeout
2022-06-10 12:27:57 +02:00
Vadim Volodin
637d293fbd
Add SYSTEM UNFREEZE query
2022-06-08 15:21:14 +03:00
Nikolai Kochetov
5bc9b32025
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-08 11:10:06 +00:00
alesapin
e32d36d790
Proper fix
2022-06-07 17:58:32 +02:00
Nikolai Kochetov
678d978acf
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-07 15:23:00 +00:00
Anton Popov
ef6f5a6500
Merge pull request #37570 from azat/column-ttl-expired-fix
...
Do not write expired columns by TTL after subsequent merges
2022-06-07 13:05:03 +02:00
Nikolai Kochetov
89c5855d20
Merge branch 'master' into refactor-something-in-part-volumes
2022-06-02 12:19:07 +02:00
Azat Khuzhin
0de1a64436
Log empty parts in IMergedBlockOutputStream::removeEmptyColumnsFromPart()
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-05-26 20:14:09 +03:00
Anton Popov
16e839ac71
add profile events for introspection of part types
2022-05-25 14:54:49 +00:00
Anton Popov
aec30c4076
Merge pull request #37053 from CurtizJ/remove-streams-comments
...
Remove last mentions of data streams
2022-05-10 13:38:13 +02:00
Anton Popov
e911900054
remove last mentions of data streams
2022-05-09 19:15:24 +00:00
alesapin
28e492bc17
Followup fix
2022-05-09 15:21:33 +02:00
alesapin
018ed10684
Add test
2022-05-09 15:21:21 +02:00
alesapin
46712f1d98
Fix forgotten parts in cleanup thread
2022-05-08 00:53:55 +02:00
Anton Popov
515f68eead
Merge remote-tracking branch 'upstream/master' into dynamic-columns-14
2022-05-06 16:10:51 +00:00
Anton Popov
566c08086a
support Object type inside other types
2022-05-06 14:44:00 +00:00
Alexander Tokmakov
af9b4a5b9c
fix intersecting parts
2022-05-04 16:22:06 +02:00
Nikolai Kochetov
35095191eb
Merge branch 'master' into refactor-something-in-part-volumes
2022-05-03 17:51:47 +02:00
Nikolai Kochetov
e44af67fee
Merge branch 'master' into refactor-something-in-part-volumes
2022-04-26 21:08:00 +02:00
Memo
c38a4b4255
Update src/Storages/MergeTree/MergeTreeData.h
...
Co-authored-by: alesapin <alesapin@gmail.com>
2022-04-26 10:26:18 +08:00
mergify[bot]
a5aab53b70
Merge branch 'master' into add_query_level_settingss
2022-04-25 21:41:36 +00:00
Nikolai Kochetov
8c00692844
Part 8
2022-04-22 16:58:09 +00:00
alesapin
5465415751
Fix replace/move partition with zero copy replication
2022-04-21 14:39:12 +02:00
alesapin
c14e2e0b96
Fix more
2022-04-20 21:08:26 +02:00
alesapin
40c15222f8
Merge branch 'master' into fix_trash
2022-04-20 12:45:49 +02:00
Nikolai Kochetov
bcbab2ead8
Part 6.
2022-04-19 19:34:41 +00:00
alesapin
7cb7c120cc
Less ugly
2022-04-19 15:53:10 +02:00
alesapin
bd7b3847c1
Some code
2022-04-19 01:09:09 +02:00
Memo
b3adf150b5
add_query_level_settings
2022-04-18 12:15:41 +08:00
alesapin
1706ae9e15
Some trash implementation
2022-04-15 18:36:23 +02:00
