Keeper support manually assigning leader

docs/en/operations/clickhouse-keeper.md

@@ -57,7 +57,7 @@ Internal coordination settings are located in the `<keeper_server>.<coordination
 -    `auto_forwarding` — Allow to forward write requests from followers to the leader (default: true).
 -    `shutdown_timeout` — Wait to finish internal connections and shutdown (ms) (default: 5000).
 -    `startup_timeout` — If the server doesn't connect to other quorum participants in the specified timeout it will terminate (ms) (default: 30000).
--    `four_letter_word_white_list` — White list of 4lw commands (default: `conf,cons,crst,envi,ruok,srst,srvr,stat,wchc,wchs,dirs,mntr,isro`).
+-    `four_letter_word_white_list` — White list of 4lw commands (default: `conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rcvr,apiv,csnp,lgif,rqld`).
 
 Quorum configuration is located in the `<keeper_server>.<raft_configuration>` section and contain servers description.
 
@@ -126,7 +126,7 @@ clickhouse keeper --config /etc/your_path_to_config/config.xml
 
 ClickHouse Keeper also provides 4lw commands which are almost the same with Zookeeper. Each command is composed of four letters such as `mntr`, `stat` etc. There are some more interesting commands: `stat` gives some general information about the server and connected clients, while `srvr` and `cons` give extended details on server and connections respectively.
 
-The 4lw commands has a white list configuration `four_letter_word_white_list` which has default value `conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rcvr,apiv,csnp,lgif`.
+The 4lw commands has a white list configuration `four_letter_word_white_list` which has default value `conf,cons,crst,envi,ruok,srst,srvr,stat,wchs,dirs,mntr,isro,rcvr,apiv,csnp,lgif,rqld`.
 
 You can issue the commands to ClickHouse Keeper via telnet or nc, at the client port.
 
@@ -328,6 +328,12 @@ target_committed_log_idx    101
 last_snapshot_idx   50
 ```
 
+- `rqld`: Request to become new leader. Return `Sent leadership request to leader.` if request sent or `Failed to send leadership request to leader.` if request not sent. Note that if node is already leader it will not send the request. 
+
+```
+Sent leadership request to leader.
+```
+
 ## Migration from ZooKeeper {#migration-from-zookeeper}
 
 Seamlessly migration from ZooKeeper to ClickHouse Keeper is impossible you have to stop your ZooKeeper cluster, convert data and start ClickHouse Keeper. `clickhouse-keeper-converter` tool allows converting ZooKeeper logs and snapshots to ClickHouse Keeper snapshot. It works only with ZooKeeper > 3.4. Steps for migration:

src/Coordination/FourLetterCommand.cpp

@@ -142,6 +142,9 @@ void FourLetterCommandFactory::registerCommands(KeeperDispatcher & keeper_dispat
         FourLetterCommandPtr log_info_command = std::make_shared<LogInfoCommand>(keeper_dispatcher);
         factory.registerCommand(log_info_command);
 
+        FourLetterCommandPtr request_leader_command = std::make_shared<RequestLeaderCommand>(keeper_dispatcher);
+        factory.registerCommand(request_leader_command);
+
         factory.initializeAllowList(keeper_dispatcher);
         factory.setInitialize(true);
     }
@@ -507,4 +510,9 @@ String LogInfoCommand::run()
     return ret.str();
 }
 
+String RequestLeaderCommand::run()
+{
+    return keeper_dispatcher.requestLeader() ? "Sent leadership request to leader." : "Failed to send leadership request to leader.";
+}
+
 }

src/Coordination/FourLetterCommand.h

@@ -364,4 +364,17 @@ struct LogInfoCommand : public IFourLetterCommand
     ~LogInfoCommand() override = default;
 };
 
+/// Request to be leader.
+struct RequestLeaderCommand : public IFourLetterCommand
+{
+    explicit RequestLeaderCommand(KeeperDispatcher & keeper_dispatcher_)
+        : IFourLetterCommand(keeper_dispatcher_)
+    {
+    }
+
+    String name() override { return "rqld"; }
+    String run() override;
+    ~RequestLeaderCommand() override = default;
+};
+
 }

src/Coordination/KeeperDispatcher.h

@@ -215,6 +215,12 @@ public:
     {
         return server->getKeeperLogInfo();
     }
+
+    /// Request to be leader.
+    bool requestLeader()
+    {
+        return server->requestLeader();
+    }
 };
 
 }

src/Coordination/KeeperServer.cpp

@@ -932,4 +932,9 @@ KeeperLogInfo KeeperServer::getKeeperLogInfo()
     return log_info;
 }
 
+bool KeeperServer::requestLeader()
+{
+    return raft_instance->request_leadership();
+}
+
 }

src/Coordination/KeeperServer.h

@@ -135,6 +135,8 @@ public:
     uint64_t createSnapshot();
 
     KeeperLogInfo getKeeperLogInfo();
+
+    bool requestLeader();
 };
 
 }

tests/integration/test_keeper_four_word_command/test.py

@@ -641,3 +641,22 @@ def test_cmd_lgif(started_cluster):
         assert int(result["last_snapshot_idx"]) >= 1
     finally:
         destroy_zk_client(zk)
+
+
+def test_cmd_rqld(started_cluster):
+    for node in [node1, node2, node3]:
+        stat = keeper_utils.send_4lw_cmd(cluster, node, cmd="stat")
+        stats = [n for n in stat.split("\n") if "=" not in n]
+        reader = csv.reader(stats, delimiter=":")
+        result = {}
+
+        for row in reader:
+            if len(row) != 0:
+                result[row[0].strip()] = row[1].strip()
+
+        data = keeper_utils.send_4lw_cmd(cluster, node, cmd="rqld")
+
+        if result["Mode"] == "leader":
+            assert data == "Failed to send leadership request to leader."
+        else:
+            assert data == "Sent leadership request to leader."