SpringCloud Alibaba系列——3Nacos数据一致性协议

In the following diagram, one of the Acceptors in the Quorum fails, so the Quorum size becomes 2. In this case, 
the Basic Paxos protocol still succeeds.
Client   Proposer      Acceptor     Learner
   |         |          |  |  |       |  |
   X-------->|          |  |  |       |  |  Request
   |         X--------->|->|->|       |  |  Prepare(1)
   |         |          |  |  !       |  |  !! FAIL !!
   |         |<---------X--X          |  |  Promise(1,{Va, Vb, null})
   |         X--------->|->|          |  |  Accept!(1,V)
   |         |<---------X--X--------->|->|  Accepted(1,V)
   |<---------------------------------X--X  Response
   |         |          |  |          |  | ​
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

The most complex case is when multiple Proposers believe themselves to be Leaders. For instance, the current leader may fail and later recover, 
but the other Proposers have already re-selected a new leader. The recovered leader has not learned this yet and attempts to begin one round in
conflict with the current leader. In the diagram below, 4 unsuccessful rounds are shown, but there could be more (as suggested at the bottom of
the diagram). 
Client   Leader         Acceptor     Learner
|      |             |  |  |       |  |
X----->|             |  |  |       |  |  Request
|      X------------>|->|->|       |  |  Prepare(1)
|      |<------------X--X--X       |  |  Promise(1,{null,null,null})
|      !             |  |  |       |  |  !! LEADER FAILS
|         |          |  |  |       |  |  !! NEW LEADER (knows last number was 1)
|         X--------->|->|->|       |  |  Prepare(2)
|         |<---------X--X--X       |  |  Promise(2,{null,null,null})
|      |  |          |  |  |       |  |  !! OLD LEADER recovers
|      |  |          |  |  |       |  |  !! OLD LEADER tries 2, denied
|      X------------>|->|->|       |  |  Prepare(2)
|      |<------------X--X--X       |  |  Nack(2)
|      |  |          |  |  |       |  |  !! OLD LEADER tries 3
|      X------------>|->|->|       |  |  Prepare(3)
|      |<------------X--X--X       |  |  Promise(3,{null,null,null})
|      |  |          |  |  |       |  |  !! NEW LEADER proposes, denied
|      |  X--------->|->|->|       |  |  Accept!(2,Va)
|      |  |<---------X--X--X       |  |  Nack(3)
|      |  |          |  |  |       |  |  !! NEW LEADER tries 4
|      |  X--------->|->|->|       |  |  Prepare(4)
|      |  |<---------X--X--X       |  |  Promise(4,{null,null,null})
|      |  |          |  |  |       |  |  !! OLD LEADER proposes, denied
|      X------------>|->|->|       |  |  Accept!(3,Vb)
|      |<------------X--X--X       |  |  Nack(4)
|      |  |          |  |  |       |  |  ... and so on ...
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.

1、从Basic Paxos Protocol的角色关系出发:  
In the following diagram, only one instance (or "execution") of the basic Paxos protocol, with an initial Leader (a Proposer), 
is shown. Note that a Multi-Paxos consists of several instances of the basic Paxos protocol.
Client   Proposer      Acceptor     Learner
|         |          |  |  |       |  | --- First Request ---
X-------->|          |  |  |       |  |  Request
|         X--------->|->|->|       |  |  Prepare(N)
|         |<---------X--X--X       |  |  Promise(N,I,{Va,Vb,Vc})
|         X--------->|->|->|       |  |  Accept!(N,I,V)
|         |<---------X--X--X------>|->|  Accepted(N,I,V)
|<---------------------------------X--X  Response
|         |          |  |  |       |  |
where V = last of (Va, Vb, Vc).
2、从Multi Paxos Protocol角色关系出发：
A common deployment of the Multi-Paxos consists in collapsing the role of the Proposers, Acceptors and Learners to "Servers". 
So, in the end, there are only "Clients" and "Servers".
Client      Servers
|         |  |  | --- First Request ---
X-------->|  |  |  Request
|         X->|->|  Prepare(N)
|         |<-X--X  Promise(N, I, {Va, Vb})
|         X->|->|  Accept!(N, I, Vn)
|         X<>X<>X  Accepted(N, I)
|<--------X  |  |  Response
|         |  |  |
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.

1、从Basic Paxos Protocol的角色关系出发: 
In this case, subsequence instances of the basic Paxos protocol (represented by I+1) use the same leader, so the phase 1 (of these subsequent
instances of the basic Paxos protocol), which consist in the Prepare and Promise sub-phases, is skipped. Note that the Leader should be stable,
i.e. it should not crash or change.
The following diagram represents the first "instance" of a basic Paxos protocol, when the roles of the Proposer, Acceptor and Learner are collapsed to a single role, called the "Server".
Client   Proposer       Acceptor     Learner
|         |          |  |  |       |  |  --- Following Requests ---
X-------->|          |  |  |       |  |  Request
|         X--------->|->|->|       |  |  Accept!(N,I+1,W)
|         |<---------X--X--X------>|->|  Accepted(N,I+1,W)
|<---------------------------------X--X  Response
|         |          |  |  |       |  |
2、从Multi Paxos Protocol角色关系出发：
In the subsequent instances of the basic Paxos protocol, with the same leader as in the previous instances of the basic Paxos protocol, 
the phase 1 can be skipped.
Client      Servers
X-------->|  |  |  Request
|         X->|->|  Accept!(N,I+1,W)
|         X<>X<>X  Accepted(N,I+1)
|<--------X  |  |  Response
|         |  |  |
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.

@DependsOn("ProtocolManager")
@Component
public class RaftCore {    
    //构建一个单线程池
    private final ScheduledExecutorService executor = ExecutorFactory.Managed
            .newSingleScheduledExecutorService(ClassUtils.getCanonicalName(NamingApp.class),
                    new NameThreadFactory("com.alibaba.nacos.naming.raft.notifier"));
    @PostConstruct
    public void init() throws Exception {
        Loggers.RAFT.info("initializing Raft sub-system");
		//开启一个notifier监听，这个线程中会遍历listeners，根据ApplyAction执行相应的逻辑     
        executor.submit(notifier);
        final long start = System.currentTimeMillis();
        //启动的时候先加载本地日志
        //遍历/nacos/data/naming/data/文件件,也就是从磁盘中加载Datum到内存，用来做数据恢复。(数据同步采用2pc协议，leader收到请求会写写入到磁盘日志，然后再进行数据同步)
        //Datum：kv对
        //datums：ConcurrentMap<String, Datum>内存数据存储
        raftStore.loadDatums(notifier, datums);
        //设置term值,从/nacos/data/naming/meta.properties本地磁盘中读取term的值,如果为null,默认为0
        setTerm(NumberUtils.toLong(raftStore.loadMeta().getProperty("term"), 0L));
        Loggers.RAFT.info("cache loaded, datum count: {}, current term: {}", datums.size(), peers.getTerm());
        while (true) {
            if (notifier.tasks.size() <= 0) {
                break;
            }
            Thread.sleep(1000L);
        }
        initialized = true;
        Loggers.RAFT.info("finish to load data from disk, cost: {} ms.", (System.currentTimeMillis() - start));
		//开启定时任务，每500ms执行一次，用来判断是否需要发起leader选举
        GlobalExecutor.registerMasterElection(new MasterElection());
        //每500ms发起一次心跳
        GlobalExecutor.registerHeartbeat(new HeartBeat());

        Loggers.RAFT.info("timer started: leader timeout ms: {}, heart-beat timeout ms: {}",
                          GlobalExecutor.LEADER_TIMEOUT_MS, GlobalExecutor.HEARTBEAT_INTERVAL_MS);
    }

}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.

public class MasterElection implements Runnable {

    @Override
    public void run() {
        try {
            //如果还没有初始化完成
            if (!peers.isReady()) {
                return;
            }
			//获取当前机器上跑的这个peer节点信息
            RaftPeer local = peers.local();
            //leader选举触发间隔时间，第一次进来，会生成（0~15000毫秒）之间的一个随机数-500.
			//后面由于500ms调度一次，所以每次该线程被调起，会将该leaderDueMs减去TICK_PERIOD_MS(500ms)，直到小于0的时候会触发选举
			//后面每次收到一次leader的心跳就会重置leaderDueMs = 15s+（随机0-5s）
            local.leaderDueMs -= GlobalExecutor.TICK_PERIOD_MS;
			//当间隔时间>0，直接返回，等到下一次500ms后再调用
            if (local.leaderDueMs > 0) {
                return;
            }

            // reset timeout
            //重置选举间隔时间
            local.resetLeaderDue();
            //重置心跳间隔时间
            local.resetHeartbeatDue();
			//将本地选举投票通过http发送其他几台服务器
            sendVote();
        } catch (Exception e) {
            Loggers.RAFT.warn("[RAFT] error while master election {}", e);
        }

    }

    private void sendVote() {
		//获取本机的节点信息
        RaftPeer local = peers.get(NetUtils.localServer());
        Loggers.RAFT.info("leader timeout, start voting,leader: {}, term: {}", JacksonUtils.toJson(getLeader()),
                          local.term);
		//重置peers，各个peer的voteFor与leader设为null
        peers.reset();
		//每一次投票，都累加一次term，表示当前投票的轮数,选举计数器，记录本地发起的是第几轮选举
        local.term.incrementAndGet();
        //投票选自己，此时peers中有一个votefor就是自己
        local.voteFor = local.ip;
        //本地server状态设置为CANDIDATE竞选状态
        local.state = RaftPeer.State.CANDIDATE;

        Map<String, String> params = new HashMap<>(1);
        params.put("vote", JacksonUtils.toJson(local));//设置本机请求参数
        //遍历除了本机ip之外的其他节点，把自己的票据发送给所有节点,将选自己的投票发送给其他servers,获取其他机器的选票信息
        for (final String server : peers.allServersWithoutMySelf()) {
            //API_VOTE:  /raft/vote
            final String url = buildUrl(server, API_VOTE);
            try {
                //发起投票
                HttpClient.asyncHttpPost(url, null, params, new AsyncCompletionHandler<Integer>() {
                    @Override
                    public Integer onCompleted(Response response) throws Exception {
                        if (response.getStatusCode() != HttpURLConnection.HTTP_OK) {
                            Loggers.RAFT
                                .error("NACOS-RAFT vote failed: {}, url: {}", response.getResponseBody(), url);
                            return 1;
                        }
						//获取其他server的响应
                        RaftPeer peer = JacksonUtils.toObj(response.getResponseBody(), RaftPeer.class);

                        Loggers.RAFT.info("received approve from peer: {}", JacksonUtils.toJson(peer));
						//计算leader
                        peers.decideLeader(peer);

                        return 0;
                    }
                });
            } catch (Exception e) {
                Loggers.RAFT.warn("error while sending vote to server: {}", server);
            }
        }
    }
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.

@PostMapping("/vote")
public JsonNode vote(HttpServletRequest request, HttpServletResponse response) throws Exception {
    RaftPeer peer = raftCore.receivedVote(JacksonUtils.toObj(WebUtils.required(request, "vote"), RaftPeer.class));
    return JacksonUtils.transferToJsonNode(peer);
}
1.
2.
3.
4.
5.

public RaftPeer decideLeader(RaftPeer candidate) {
    peers.put(candidate.ip, candidate);

    SortedBag ips = new TreeBag();
    //选票最多的票数
    int maxApproveCount = 0;
    //选票最多的ip
    String maxApprovePeer = null;
    /**
	* 假设3个节点：A,B,C
	* local节点为A，假设A,B,C第一轮同时发起选举请求
	* 第一轮投票结果：
    * 第一次for循环是A自己的投票(投票给自己)：maxApproveCount = 1，maxApprovePeer = A
	* 第二次for循环是B服务器返回的投票，该投票投向B：
	* 此时 if (ips.getCount(peer.voteFor) > maxApproveCount) 条件不成立，maxApproveCount = 1，maxApprovePeer = A
	* 第三次for循环是C服务器返回的投票，该投票投向C
	* 此时 if (ips.getCount(peer.voteFor) > maxApproveCount) 条件不成立，maxApproveCount = 1，maxApprovePeer = A
	* 第二轮投票结果：
    * 第一次for循环是A自己的投票(投票给自己)：maxApproveCount = 1，maxApprovePeer = A
	* 第二次for循环是B服务器返回的投票，该投票投向A：
	* 此时 if (ips.getCount(peer.voteFor) > maxApproveCount) 条件成立，maxApproveCount = 2，maxApprovePeer = A
	* 第三次for循环是C服务器返回的投票，该投票投向C
	* 此时 if (ips.getCount(peer.voteFor) > maxApproveCount) 条件不成立，maxApproveCount = 1，maxApprovePeer = A
	* 
	*/
    for (RaftPeer peer : peers.values()) {
        if (StringUtils.isEmpty(peer.voteFor)) {
            continue;
        }
		//收集选票
        ips.add(peer.voteFor);
        if (ips.getCount(peer.voteFor) > maxApproveCount) {
            maxApproveCount = ips.getCount(peer.voteFor);
            maxApprovePeer = peer.voteFor;
        }
    }
	//majorityCount()过半节点数：2（假设3个节点）
	//第一轮：maxApproveCount = 1 if条件不成立，返回leader，此时leader为null，没有选举成功
	//第二轮：maxApproveCount = 2 if条件成立，返回leader，此时leader为A，没有选举成功
    if (maxApproveCount >= majorityCount()) {
        RaftPeer peer = peers.get(maxApprovePeer);
        peer.state = RaftPeer.State.LEADER;//成为Leader
        if (!Objects.equals(leader, peer)) {
            leader = peer;
            // 如果当前leader和选举出来的leader不是同一个，那么将选举的leader重置并且发布一个leader选举完成的事件
            ApplicationUtils.publishEvent(new LeaderElectFinishedEvent(this, leader, local()));
            Loggers.RAFT.info("{} has become the LEADER", leader.ip);
        }
    }
	//返回Leader
    return leader;
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.

public void addInstance(String namespaceId, String serviceName, boolean ephemeral, Instance... ips)
    throws NacosException {
    String key = KeyBuilder.buildInstanceListKey(namespaceId, serviceName, ephemeral);
    Service service = getService(namespaceId, serviceName);
    synchronized (service) {
        List<Instance> instanceList = addIpAddresses(service, ephemeral, ips);
        Instances instances = new Instances();
        instances.setInstanceList(instanceList);
		//数据同步
        consistencyService.put(key, instances);
    }
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

@Override
public void put(String key, Record value) throws NacosException {
    try {
        raftCore.signalPublish(key, value);
    } catch (Exception e) {
        Loggers.RAFT.error("Raft put failed.", e);
        throw new NacosException(NacosException.SERVER_ERROR, "Raft put failed, key:" + key + ", value:" + value,
                                 e);
    }
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

public static final Lock OPERATE_LOCK = new ReentrantLock();

public void signalPublish(String key, Record value) throws Exception {
	//如果接受的节点不是Leader节点
    if (!isLeader()) {
        ObjectNode params = JacksonUtils.createEmptyJsonNode();
        params.put("key", key);
        params.replace("value", JacksonUtils.transferToJsonNode(value));
        Map<String, String> parameters = new HashMap<>(1);
        parameters.put("key", key);
		//获取Leader节点
        final RaftPeer leader = getLeader();
        //转发到Leader节点
        raftProxy.proxyPostLarge(leader.ip, API_PUB, params.toString(), parameters);
        return;
    }

    //如果自己是leader，则向所有节点发送onPublish请求。这个所有节点包含自己
    try {
        //加锁
        OPERATE_LOCK.lock();
        final long start = System.currentTimeMillis();
        final Datum datum = new Datum();
        datum.key = key;
        datum.value = value;
        if (getDatum(key) == null) {
            datum.timestamp.set(1L);
        } else {
            datum.timestamp.set(getDatum(key).timestamp.incrementAndGet());
        }

        ObjectNode json = JacksonUtils.createEmptyJsonNode();
        json.replace("datum", JacksonUtils.transferToJsonNode(datum));
        json.replace("source", JacksonUtils.transferToJsonNode(peers.local()));
		//onPublish可以当做是一次心跳了，更新选举检查时间，然后一个重点就是term增加100了。
		//当然还是就是更新内容了，先写文件，再更新内存缓存。(也就是先记录本地日志)
        onPublish(datum, peers.local()); //发送数据到所有节点

        final String content = json.toString();
		 //CountDownLatch 用于控制过半提交
        final CountDownLatch latch = new CountDownLatch(peers.majorityCount());
        //遍历所有节点，发送事务提交请求，把记录在本地日志中的数据进行提交
        for (final String server : peers.allServersIncludeMyself()) {
            if (isLeader(server)) {
                latch.countDown();
                continue;
            }
            //API_ON_PUB: /raft/datum/commit  采用的是二阶段提交
            final String url = buildUrl(server, API_ON_PUB);
            HttpClient.asyncHttpPostLarge(url, Arrays.asList("key=" + key), content,
                                          new AsyncCompletionHandler<Integer>() {
                                              @Override
                                              public Integer onCompleted(Response response) throws Exception {
                                                  if (response.getStatusCode() != HttpURLConnection.HTTP_OK) {
                                                      Loggers.RAFT
                                                          .warn("[RAFT] failed to publish data to peer, datumId={}, peer={}, http code={}",
                                                                datum.key, server, response.getStatusCode());
                                                      return 1;
                                                  }
                                                  latch.countDown();
                                                  return 0;
                                              }
                                              @Override
                                              public STATE onContentWriteCompleted() {
                                                  return STATE.CONTINUE;
                                              }
                                          });
        }
        if (!latch.await(UtilsAndCommons.RAFT_PUBLISH_TIMEOUT, TimeUnit.MILLISECONDS)) {
            // only majority servers return success can we consider this update success
            Loggers.RAFT.error("data publish failed, caused failed to notify majority, key={}", key);
            throw new IllegalStateException("data publish failed, caused failed to notify majority, key=" + key);
        }
        long end = System.currentTimeMillis();
        Loggers.RAFT.info("signalPublish cost {} ms, key: {}", (end - start), key);
    } finally {
        OPERATE_LOCK.unlock();
    }
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.

private volatile ConcurrentMap<String, Datum> datums = new ConcurrentHashMap<>();

public void onPublish(Datum datum, RaftPeer source) throws Exception {
    RaftPeer local = peers.local();
    if (datum.value == null) {
        Loggers.RAFT.warn("received empty datum");
        throw new IllegalStateException("received empty datum");
    }

    if (!peers.isLeader(source.ip)) {
        Loggers.RAFT
            .warn("peer {} tried to publish data but wasn't leader, leader: {}", JacksonUtils.toJson(source),
                  JacksonUtils.toJson(getLeader()));
        throw new IllegalStateException("peer(" + source.ip + ") tried to publish " + "data but wasn't leader");
    }

    if (source.term.get() < local.term.get()) {
        Loggers.RAFT.warn("out of date publish, pub-term: {}, cur-term: {}", JacksonUtils.toJson(source),
                          JacksonUtils.toJson(local));
        throw new IllegalStateException(
            "out of date publish, pub-term:" + source.term.get() + ", cur-term: " + local.term.get());
    }
	//重置选举间隔时间
    local.resetLeaderDue();

    // if data should be persisted, usually this is true:
    if (KeyBuilder.matchPersistentKey(datum.key)) {
        //存储到本地磁盘中
        raftStore.write(datum);
    }
	//并且存储到内存中
    datums.put(datum.key, datum);
	//如果是leader,term增加100
    if (isLeader()) {
        local.term.addAndGet(PUBLISH_TERM_INCREASE_COUNT);
    } else {
        if (local.term.get() + PUBLISH_TERM_INCREASE_COUNT > source.term.get()) {
            //set leader term:
            getLeader().term.set(source.term.get());
            local.term.set(getLeader().term.get());
        } else {
            local.term.addAndGet(PUBLISH_TERM_INCREASE_COUNT);
        }
    }
    //更新本地磁盘文件meta.properties下的term值
    raftStore.updateTerm(local.term.get());
    
    notifier.addTask(datum.key, ApplyAction.CHANGE);

    Loggers.RAFT.info("data added/updated, key={}, term={}", datum.key, local.term);
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.

@PostMapping("/datum/commit")
public String onPublish(HttpServletRequest request, HttpServletResponse response) throws Exception {

    response.setHeader("Content-Type", "application/json; charset=" + getAcceptEncoding(request));
    response.setHeader("Cache-Control", "no-cache");
    response.setHeader("Content-Encode", "gzip");

    String entity = IoUtils.toString(request.getInputStream(), "UTF-8");
    String value = URLDecoder.decode(entity, "UTF-8");

    JsonNode jsonObject = JacksonUtils.toObj(value);
    String key = "key";

    RaftPeer source = JacksonUtils.toObj(jsonObject.get("source").toString(), RaftPeer.class);
    JsonNode datumJson = jsonObject.get("datum");
    Datum datum = null;
    //根据不同数据类型进行处理
    if (KeyBuilder.matchInstanceListKey(datumJson.get(key).asText())) {
        datum = JacksonUtils.toObj(jsonObject.get("datum").toString(), new TypeReference<Datum<Instances>>() {
        });
    } else if (KeyBuilder.matchSwitchKey(datumJson.get(key).asText())) {
        datum = JacksonUtils.toObj(jsonObject.get("datum").toString(), new TypeReference<Datum<SwitchDomain>>() {
        });
    } else if (KeyBuilder.matchServiceMetaKey(datumJson.get(key).asText())) {
        datum = JacksonUtils.toObj(jsonObject.get("datum").toString(), new TypeReference<Datum<Service>>() {
        });
    }

    raftConsistencyService.onPut(datum, source);
    return "ok";
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.

public void onPut(Datum datum, RaftPeer source) throws NacosException {
    try {
        //在本地写入数据
        raftCore.onPublish(datum, source);
    } catch (Exception e) {
        Loggers.RAFT.error("Raft onPut failed.", e);
        throw new NacosException(NacosException.SERVER_ERROR,
                                 "Raft onPut failed, datum:" + datum + ", source: " + source, e);
    }
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.