Commit Graph

1458 Commits

Author SHA1 Message Date
Alexander Tokmakov
75ffc8451b fix 2023-03-31 16:09:00 +02:00
Alexander Tokmakov
b5fe1ada0b more options for sync replica 2023-03-28 01:39:36 +02:00
Kruglov Pavel
3ee12e21fb
Merge branch 'master' into non-blocking-connect 2023-03-23 20:53:44 +01:00
Alexander Tokmakov
0989dd837a
Merge pull request #47802 from ClickHouse/fix_some_tests
Fix 993 and two other tests
2023-03-21 22:20:06 +03:00
avogar
38e44861ae Fix possible race conditions 2023-03-21 16:01:54 +00:00
Alexander Tokmakov
d5a561a2ef fix 993 and two other tests 2023-03-20 22:51:32 +01:00
flynn
cac37c8a3c
Merge branch 'master' into remove 2023-03-20 18:43:48 +08:00
flynn
8a6e4daefa remove unused code 2023-03-20 08:31:38 +00:00
Alexander Tokmakov
d11aee2b0c
Merge branch 'master' into fix_993 2023-03-17 21:28:23 +03:00
alesapin
1022c47421 Review fixes 2023-03-17 17:45:02 +01:00
alesapin
e44fca8bb9 Add test 2023-03-17 12:56:31 +01:00
alesapin
4a80197f16 Merge branch 'master' into fix_zero_copy_lock_check 2023-03-17 11:18:51 +01:00
alesapin
3731567b6a Bad code 2023-03-16 21:48:47 +01:00
Alexander Tokmakov
d621b2c4ad fix intersecting parts, better fault injections 2023-03-16 21:28:07 +01:00
Azat Khuzhin
8495deb7e3 Remove unused removePartFromZooKeeper()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit dbdb46e906)
2023-03-16 16:43:23 +01:00
Alexander Tokmakov
8e36f883d3 add 'strict' mode for sync replica, fix 00993 2023-03-16 16:04:42 +01:00
alesapin
ef7508469c Fix wait for zero-copy replication move during manual move 2023-03-15 22:29:24 +01:00
Alexander Tokmakov
ba6c6255a3 hide too noisy log messages, fix some tests 2023-03-14 00:23:36 +01:00
Dmitry Novik
4f24a7bd11 Merge remote-tracking branch 'origin/master' into background-memory-tracker 2023-03-10 18:57:45 +00:00
alesapin
588cc0f159
Merge branch 'master' into revert-46909-revert-45911-mutations_rename_hang 2023-03-09 14:54:10 +01:00
Alexander Tokmakov
38f9f08e1a
Merge pull request #47268 from myrrc/improvement/more-to-base-3
use std::lerp, constexpr hex.h
2023-03-08 19:07:34 +03:00
alesapin
4423be8332
Merge branch 'master' into revert-46909-revert-45911-mutations_rename_hang 2023-03-08 12:59:07 +01:00
Mike Kot
9920a52c51 use std::lerp, constexpr hex.h 2023-03-07 22:50:17 +00:00
alesapin
c10cb436f4 Fix build 2023-03-07 12:42:42 +01:00
alesapin
1db6b9414e
Update src/Storages/StorageReplicatedMergeTree.cpp
Co-authored-by: Ilya Yatsishin <2159081+qoega@users.noreply.github.com>
2023-03-07 12:39:17 +01:00
alesapin
b5dffe7417 Fix bug in zero copy replica which can lead to dataloss 2023-03-06 19:57:09 +01:00
Alexander Tokmakov
a97e15e36f Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-03-01 14:26:41 +01:00
alesapin
e3961c118a
Revert "Revert "Fix several RENAME COLUMN bugs."" 2023-02-27 12:27:57 +01:00
Alexey Milovidov
8f40bbd5b4
Revert "Fix several RENAME COLUMN bugs." 2023-02-26 05:47:05 +03:00
Alexey Milovidov
eb89ab471f
Merge pull request #45911 from ClickHouse/mutations_rename_hang
Fix several `RENAME COLUMN` bugs.
2023-02-26 01:56:40 +03:00
Alexey Milovidov
656af55831
Merge pull request #46384 from ClickHouse/fixes_for_993
Fixes for 993
2023-02-26 01:55:20 +03:00
Alexander Tokmakov
fe92fd8a61 Merge branch 'master' into fix_insert_cancellation_in_native_protocol 2023-02-23 22:35:31 +01:00
Alexander Tokmakov
f5c232707d cancel insert queries correctly 2023-02-21 22:04:39 +01:00
Vladimir C
a791250a4a
Merge pull request #38614 from bharatnc/ncb/profile-events-partlog 2023-02-21 11:06:44 +01:00
Alexey Milovidov
d8cda3dbb8 Remove PVS-Studio 2023-02-19 23:30:05 +01:00
Alexander Tokmakov
9a664efeb6
Merge branch 'master' into fixes_for_993 2023-02-17 14:17:08 +03:00
Dmitry Novik
0336f0d8cc
Merge branch 'master' into background-memory-tracker 2023-02-16 20:57:44 +01:00
youennL-cs
6526c2a8ab
[RFC] Replacing merge tree new engine (#41005)
* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add new engine to ReplacingMergeTree corresponding to the ReplacingCollapsingMergeTree

* Add new test for the new ReplacingMergeTree engine

* Limit sign value to -1/1

* Replace sign column(Int8) by is_deleted(UInt8)

* Add keyword 'CLEANUP' when OPTIMIZE

* Cleanup uniquely when it's a replacingMergeTree

* Propagate CLEANUP information and change from 'with_cleanup' to 'cleanup'

* Cleanup data flagged as 'is_deleted'

* Fix merge when optimize and add a test

* Fix OPTIMIZE and INSERT + add tests

* New fix for cleanup at the merge

* Cleanup debug logs

* Add the SETTINGS option 'clean_deleted_rows' that can be 'never' or 'always'

* Fix regression bug; Now REplicatedMergeTree can be called as before without 'is_deleted'

* Add Replicated tests

* Disable tag 'long' for our test and cleanup some white spaces

* Update tests

* Fix tests and remove additional useless whitespace

* Fix replica test

* Style clean && add condition check for is_deleted values

* clean_deleted_rows settings is nom an enum

* Add valid default value to the clean_deleted_rows settings

* Update cleanup checkers to use the enum and fix typos in the test

* Fix submodule contrib/AMQP-CPP pointer

* Add missing messages in test reference and remove a print with non derterministic order

* fix replica test reference

* Fix edge case

* Fix a typo for the spell checker

* Fix reference

* Fix a condition to raise an error if is_deleted differ from 0/1 and cleanup

* Change tests file name and update number

* This should fix the ReplacingMergeTree parameter set

* Fix replicated parameters

* Disable allow_deprecated_syntax_for_merge_tree for our new column

* Fix a test

* Remove non deterministic order print in the test

* Test on replicas

* Remove a condition, when checking optional parameters, that should not be sueful since we disabled the deprected_syntaxe

* Revert "Remove a condition, when checking optional parameters, that should not be useful since we disabled the deprected_syntaxe"

This reverts commit b65d64c05e.

* Fix replica management and limit the number of argument to two maximum, due to the possiblity of deprecated table create/attach failing otherwise

* Test a fix for replicated log information error

* Try to add sync to have consistent results

* Change path of replicas that should cause one issue and add few prints in case it's not that

* Get cleanup info on replicas only if information found

* Fix style issues

* Try to avoid replication error 'cannot select parts...' and and replica read/write field order

* Cleanup according to PR reviews
 and add tests on error raised.

* Update src/Storages/MergeTree/registerStorageMergeTree.cpp

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>

* Select ... FINAL don't show rows with is_deleted = true

* Update and fix SELECT ... FINAL merge parameter

* Remove is_deleted rows only on the version inserted when merge

* Fix (master) updates issues

* Revert changes that should not be commited

* Add changes according to review

* Revert changes that should not be commited - part 2

---------

Co-authored-by: Alexander Tokmakov <tavplubix@gmail.com>
2023-02-16 16:03:16 +03:00
Dmitry Novik
e7a94a28ee
Merge branch 'master' into background-memory-tracker 2023-02-15 19:39:28 +01:00
Alexander Tokmakov
885f14311c fix 2023-02-15 16:37:44 +01:00
Alexander Tokmakov
8ddbc85919
Merge branch 'master' into followup_revert_39737 2023-02-15 01:15:17 +03:00
vdimir
4c1aeb7972
Use overload of PartLog::addNewPart with profile_events 2023-02-14 14:11:12 +00:00
vdimir
994dcb93f9
Rename ScopedProfileEvents -> ProfileEventsScope 2023-02-14 14:11:10 +00:00
vdimir
a228f7f419
[wip] ProfileCounters for each part 2023-02-14 14:11:09 +00:00
Alexey Milovidov
f554ff9d99
Remove trash 2023-02-14 14:11:08 +00:00
John Skopis
f7604cc686
[feat] Add ProfileEvents map to PartLog
closes #10316
2023-02-14 14:11:02 +00:00
Alexander Tokmakov
e712fbecb2 fix race between drop and create 2023-02-14 00:45:28 +01:00
Antonio Andelic
95db6fb1a6
Merge pull request #45659 from ClickHouse/add-support-for-settings-alias
Add support for settings aliases/Rename `replication_alter_partitions_sync` to `alter_sync`
2023-02-13 15:31:12 +01:00
Alexander Tokmakov
69f579b8ec remove an unused argument 2023-02-13 14:46:46 +01:00
Antonio Andelic
8d16fe5793
Merge branch 'master' into add-support-for-settings-alias 2023-02-13 08:46:00 +01:00
Alexey Milovidov
e6b44756dc Fixed build 2023-02-12 07:45:28 +01:00
Alexey Milovidov
e2079b78a3 [RFC] Revert #39737 2023-02-12 07:24:42 +01:00
alesapin
5c83ddabf9 Merge branch 'master' into mutations_rename_hang 2023-02-11 12:18:58 +01:00
robot-ch-test-poll2
4e9d1275c7
Merge pull request #46278 from CheSema/race-check-table
fix data race between check table request and background checker
2023-02-11 09:50:48 +01:00
Sema Checherinda
ef133ee30d fix data race between check table query and background part checker thread 2023-02-10 16:22:35 +01:00
alesapin
808a939ad2 Small rehacktoing 2023-02-10 14:25:19 +01:00
Robert Schulze
566fb5b9da
Merge pull request #46006 from ClickHouse/rs/fix-fragile-linking
Fix terribly broken, fragile and potentially cyclic linking
2023-02-09 18:19:44 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias 2023-02-09 16:07:45 +01:00
alesapin
9b23068d5e
Merge pull request #45821 from ClickHouse/explicit_drop_part_entry_type
Make separate DROP_PART log entry type
2023-02-09 15:11:13 +01:00
Robert Schulze
10af0b3e49
Reduce redundancies 2023-02-07 12:27:23 +00:00
Smita Kulkarni
fb765696b6 Removed state_mutex lock from addSubscriber - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 12:19:58 +01:00
Smita Kulkarni
135615cda6 Removed unwanted check from before callback - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 10:27:45 +01:00
Smita Kulkarni
ef29b61786 Updated callback to return log entry ids - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 10:02:38 +01:00
Smita Kulkarni
f34ef86332 Moved background task trigger before getting log entries to wait - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 08:49:41 +01:00
alesapin
dd3a98e88d Fixes 2023-02-06 22:30:31 +01:00
Dmitry Novik
1e065b32f3 Add MemoryTracker for the background tasks 2023-02-06 17:25:58 +00:00
Alexander Tokmakov
86feaae253 Merge branch 'master' into explicit_drop_part_entry_type 2023-02-06 14:59:59 +01:00
alesapin
b94f9f8006
Merge branch 'master' into 45508_Update_strategy_for_system_sync_replica 2023-02-06 12:38:39 +01:00
alesapin
b265300a19 Merge branch 'master' into mutations_rename_hang 2023-02-06 12:25:29 +01:00
Robert Schulze
84b9ff450f
Fix terribly broken, fragile and potentially cyclic linking
Sorry for the clickbaity title. This is about static method
ConnectionTimeouts::getHTTPTimeouts(). It was be declared in header
IO/ConnectionTimeouts.h, and defined in header
IO/ConnectionTimeoutsContext.h (!). This is weird and caused issues with
linking on s390x (##45520). There was an attempt to fix some
inconsistencies (#45848) but neither did @Algunenano nor me at first
really understand why the definition is in the header.

Turns out that ConnectionTimeoutsContext.h is only #include'd from
source files which are part of the normal server build BUT NOT part of
the keeper standalone build (which must be enabled via CMake
-DBUILD_STANDALONE_KEEPER=1). This dependency was not documented and as
a result, some misguided workarounds were introduced earlier, e.g.
0341c6c54b

The deeper cause was that getHTTPTimeouts() is passed a "Context". This
class is part of the "dbms" libary which is deliberately not linked by
the standalone build of clickhouse-keeper. The context is only used to
read the settings and the "Settings" class is part of the
clickhouse_common library which is linked by clickhouse-keeper already.

To resolve this mess, this PR

- creates source file IO/ConnectionTimeouts.cpp and moves all
  ConnectionTimeouts definitions into it, including getHTTPTimeouts().

- breaks the wrong dependency by passing "Settings" instead of "Context"
  into getHTTPTimeouts().

- resolves the previous hacks
2023-02-05 20:49:34 +00:00
alesapin
51b6154d68 Fix stupid bug 2023-02-03 20:56:09 +01:00
alesapin
f91c10d09d Initial support for version 2023-02-03 20:49:07 +01:00
alesapin
7a0d0f0c53 Some code which doesn't work 2023-02-03 19:14:49 +01:00
Antonio Andelic
8929838fcc Merge branch 'master' into add-support-for-settings-alias 2023-02-03 17:47:00 +00:00
Alexander Tokmakov
3f11948bb0
Merge branch 'master' into stack_trace_in_part_log 2023-02-03 20:05:00 +03:00
Nikita Mikhaylov
33877b5e00
Parallel replicas. Part [2] (#43772) 2023-02-03 14:34:18 +01:00
Smita Kulkarni
123f3cfe1f Used unordered_set for log entries and fixed the callback function - Updated checking of SYSTEM SYNC REPLICA 2023-02-03 08:48:16 +01:00
Frank Chen
d38adfab30 Merge two overridden functions as one 2023-02-03 15:27:45 +08:00
Frank Chen
d3a05a11da Merge remote-tracking branch 'remotes/github/master' into stack_trace_in_part_log 2023-02-03 11:39:43 +08:00
alesapin
bdc530dead Fix style 2023-02-02 17:30:38 +01:00
Antonio Andelic
f613bfd8d2 Merge branch 'master' into add-support-for-settings-alias 2023-02-02 11:15:39 +00:00
Sema Checherinda
41a1dddc7a
Merge branch 'master' into better_postpone_reason 2023-02-01 15:16:34 +01:00
Alexander Gololobov
e705f11ad1
Merge pull request #45786 from ClickHouse/delete_on_cluster
Support DELETE ON CLUSTER
2023-02-01 10:52:01 +01:00
Antonio Andelic
ec04cf6876 Merge branch 'master' into add-support-for-settings-alias 2023-02-01 09:01:01 +00:00
Alexander Tokmakov
7862990721 Merge branch 'master' into explicit_drop_part_entry_type 2023-01-31 13:49:04 +01:00
Alexander Tokmakov
2e8100b6e4 make separate DROP_PART log entry type 2023-01-31 13:37:56 +01:00
Alexander Tokmakov
e42cd3163f fix race condition on a part check cancellation 2023-01-30 18:48:20 +01:00
Alexander Gololobov
863548114d force_wait flag is no longer needed 2023-01-30 18:38:28 +01:00
Smita Kulkarni
6cf2498609 Fixed clang-tidy build for comparing strings and updated to notify after adding subscriber if queue is already empty- Updated checking of SYSTEM SYNC REPLICA 2023-01-30 08:18:20 +01:00
SmitaRKulkarni
b22fd40d92
Merge branch 'master' into 45508_Update_strategy_for_system_sync_replica 2023-01-27 15:00:47 +01:00
Alexander Tokmakov
f375b1246f
Merge pull request #45363 from ClickHouse/fix_hang_during_drop_in_zero_copy_replication
Fix no shared id during drop for the fourth time
2023-01-27 15:59:27 +03:00
Smita Kulkarni
39966457d9 Updated to remove waiting ids till the removed log entry - Updated checking of SYSTEM SYNC REPLICA 2023-01-27 13:36:44 +01:00
alesapin
716d25c33f
Merge pull request #45619 from CurtizJ/refactoring-merge-tree-parts
Refactoring of code near merge tree parts
2023-01-27 13:36:33 +01:00
Smita Kulkarni
a95609e0bc Updated to validate & remove each element instead of just last element - Updated checking of SYSTEM SYNC REPLICA 2023-01-27 10:35:12 +01:00
Antonio Andelic
714fad1529 Add support for settings alias 2023-01-26 14:06:46 +00:00
alesapin
84af5eb58b Fxi 2023-01-26 11:42:53 +01:00
alesapin
d5036f2ebc Merge branch 'fix_hang_during_drop_in_zero_copy_replication' of github.com:ClickHouse/ClickHouse into fix_hang_during_drop_in_zero_copy_replication 2023-01-26 11:42:30 +01:00
alesapin
1f61abce46 One useful logging line 2023-01-26 11:41:43 +01:00
Smita Kulkarni
8c885ffeae Updated checking of SYSTEM SYNC REPLICA
Implementation:
* Updated to wait for current last entry to be processed (after pulling shared log) instead of queue size becoming 0.
* Updated Subscriber to notify both queue size and removed log_entry_id.
2023-01-26 11:40:09 +01:00
Alexander Tokmakov
a584ad0eb1 forbid runtime strings 2023-01-26 10:52:47 +01:00