ZOOKEEPER-3823: Add a benchmark tool for testing watch feature performance by maoling · Pull Request #1406 · apache/zookeeper

maoling · 2020-07-20T13:25:30Z

Here is a detailed design document. Copying its abstract here.

This watch benchmark report is mainly organized as followed: Section 1 and 2: we introduce the resource, environment and software settings of our benchmark server. Section 3: we talk about the implementations of our watch benchmark tool. Section 4: we evaluate the experimental effects using our watch benchmark tool on the standard cloud machine from four dimensions: server mode, znode count, client threads, watch mode. Section 6: we look forward to our future work

Link to PR-1011
A test evidence in Windows OS is included in JIRA
More details in ZOOKEEPER-3823

nkalmar

Thanks @maoling , nice work, I left a few comments. I will also run the tool to test.

nkalmar · 2020-08-06T17:10:55Z

+- Usages:
+
+
+        nohup ./zkWatchBenchmark.sh -root_path /bench-watch -threads 100 -znode_count 1000 -force -connect_string \n


I think nohup is unnecessary to include in the example command (while useful here I agree, there are other options, and everyone can decide how to handle the long runtime - at least this is how I see it, not a blocker though)

nkalmar · 2020-08-06T17:13:06Z

+
+### zkWatchBenchmark.sh
+- A benchmark tool that benchmarks the watch throughput and latency, which supports multiple clients threads, multiple watch modes
+- See **ZOOKEEPER-3823** for the design document


this jira does not contain a design document. I don't think we need to link jira here.

nkalmar · 2020-08-06T17:16:21Z

+
+/**
+ * A benchmark tool that benchmarks the watch throughput and latency.
+ * See ZOOKEEPER-3823 for the design document


again, I think adding the jira number is unnecessary, as it does not contain a design document, or any further information.

nkalmar · 2020-08-06T17:21:23Z

+        // wait for deleting all nodes
+        long deleteAwaitStart = System.currentTimeMillis();
+        deleteNodeCountDownLatch.await();
+        if (isDebug) {


For readability, I would name this "isVerbose" or something similar.

nkalmar · 2020-08-06T17:23:44Z

+    private static void shutDownThreadPool(ExecutorService executorService) {
+        long shutDownStart = System.currentTimeMillis();
+        executorService.shutdown();
+        while (true) {


executorService.awaitTermination(); isntead?

### 1. Brief Introduction At first, I want to implement this benchmark tool based on the JMH. However, its has the following drawbacks: - For this situation：I want to simulate 1000000 requests with 100 concurrency client threads,What the parameters/options of JMH(mode,fork,threads,measurementIterations,measurementBatchSize) should I set? - Based on JMH, it also needs or use great effort to write a satisfactory script/CLI. Then, I want to implement the tool from scratch. Have to admit that: it's difficult to implement a tool as powerful as YCSB did. Finally, the YCSB is our choice. Here are its introductions: - [Home Page](https://github.com/brianfrankcooper/YCSB/blob/master/README.md) - [YCSB Paper](https://www2.cs.duke.edu/courses/fall13/cps296.4/838-CloudPapers/ycsb.pdf) is one of the best materials(A Great Reading) YCSB has the following pros: - It supports requests number and concurrency threads number that user appoints - It supports various workloads for different read/write/insert/scan request ratio - It supports different request distributions(uniform, zipfian, latest) - It can output the benchmark results to different report formats(e.g.: `hdrhistogram`) - It has abundant metrics: throughout, avg/min/max/percentile latency, etc. - It has a detailed benchmark usage documentation ### 2. Benchmark Report Here is a [WIP Benchmark Report](https://docs.google.com/document/d/1_kGwnx5y7SYLSLtEPbkGJkrWbnsSftwNTvi8RCC9L7Y/edit?usp=sharing) which tested on the standard cloud machines with release-3.6.1(still a very messy draft and lots of works to do in the future). Two conspicuous performance issues: #### 2.1. Tail latency The tail latency is sometimes up to 200 ms. we can observer long-fat tail. Look at [figure-1](https://issues.apache.org/jira/secure/attachment/13017005/bm-read-histogram.png) and [figure-2](https://issues.apache.org/jira/secure/attachment/13017004/bm-update-histogram.png). We demand a smoother latency curve. The potential causes: - Java GC - The commit algorithm which schedules read/write requests in the `CommitProcessor`. (Most likely place where significantly improves the performance) #### 2.2. Connection numbers When connection numbers >= 1000, the throughput and latency extremely degrade. Look at [figure-3](https://issues.apache.org/jira/secure/attachment/13017006/bm-clients.png). Using the TCP connection can guarantee the message/packet order, but TCP connections that one Linux VM could support/hold is limited. Perhaps based/focused on Netty, we can improve this issue further. ### 3. Related Works 3.1 [YCSB-PR-1327](brianfrankcooper/YCSB#1327) 3.2 [ZooKeeper Watch Benchmark Tool](#1406) Author: maoling <maoling199210191@sina.com> Reviewers: Mate Szalay-Beko <symat@apache.org>, Damien Diederen <ddiederen@apache.org> Closes #1558 from maoling/ZOOKEEPER-3264

### 1. Brief Introduction At first, I want to implement this benchmark tool based on the JMH. However, its has the following drawbacks: - For this situation：I want to simulate 1000000 requests with 100 concurrency client threads,What the parameters/options of JMH(mode,fork,threads,measurementIterations,measurementBatchSize) should I set? - Based on JMH, it also needs or use great effort to write a satisfactory script/CLI. Then, I want to implement the tool from scratch. Have to admit that: it's difficult to implement a tool as powerful as YCSB did. Finally, the YCSB is our choice. Here are its introductions: - [Home Page](https://github.com/brianfrankcooper/YCSB/blob/master/README.md) - [YCSB Paper](https://www2.cs.duke.edu/courses/fall13/cps296.4/838-CloudPapers/ycsb.pdf) is one of the best materials(A Great Reading) YCSB has the following pros: - It supports requests number and concurrency threads number that user appoints - It supports various workloads for different read/write/insert/scan request ratio - It supports different request distributions(uniform, zipfian, latest) - It can output the benchmark results to different report formats(e.g.: `hdrhistogram`) - It has abundant metrics: throughout, avg/min/max/percentile latency, etc. - It has a detailed benchmark usage documentation ### 2. Benchmark Report Here is a [WIP Benchmark Report](https://docs.google.com/document/d/1_kGwnx5y7SYLSLtEPbkGJkrWbnsSftwNTvi8RCC9L7Y/edit?usp=sharing) which tested on the standard cloud machines with release-3.6.1(still a very messy draft and lots of works to do in the future). Two conspicuous performance issues: #### 2.1. Tail latency The tail latency is sometimes up to 200 ms. we can observer long-fat tail. Look at [figure-1](https://issues.apache.org/jira/secure/attachment/13017005/bm-read-histogram.png) and [figure-2](https://issues.apache.org/jira/secure/attachment/13017004/bm-update-histogram.png). We demand a smoother latency curve. The potential causes: - Java GC - The commit algorithm which schedules read/write requests in the `CommitProcessor`. (Most likely place where significantly improves the performance) #### 2.2. Connection numbers When connection numbers >= 1000, the throughput and latency extremely degrade. Look at [figure-3](https://issues.apache.org/jira/secure/attachment/13017006/bm-clients.png). Using the TCP connection can guarantee the message/packet order, but TCP connections that one Linux VM could support/hold is limited. Perhaps based/focused on Netty, we can improve this issue further. ### 3. Related Works 3.1 [YCSB-PR-1327](brianfrankcooper/YCSB#1327) 3.2 [ZooKeeper Watch Benchmark Tool](apache#1406) Author: maoling <maoling199210191@sina.com> Reviewers: Mate Szalay-Beko <symat@apache.org>, Damien Diederen <ddiederen@apache.org> Closes apache#1558 from maoling/ZOOKEEPER-3264

sonatype-lift · 2022-09-02T12:13:28Z

+        TRADITION("traditional_watch", "t"),
+        PERSISTENT("persistent_watch", "p");
+        private String name;
+        private String abbreviation;


ImmutableEnumChecker: enums should be immutable: 'WatchMode' has non-final field 'abbreviation'

Suggested change

private String abbreviation;

private final String abbreviation;

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:29Z

+        // the traditional one-off watch
+        TRADITION("traditional_watch", "t"),
+        PERSISTENT("persistent_watch", "p");
+        private String name;


ImmutableEnumChecker: enums should be immutable: 'WatchMode' has non-final field 'name'

Suggested change

private String name;

private final String name;

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:31Z

+
+        // receive, loss notifications count and ratio summary
+        double receivedRatio = (double) copyLatencyList.size() / (double) (watchMultiple * clientThreads * znodeCount);
+        long lossCount = watchMultiple * clientThreads * znodeCount - copyLatencyList.size();


IntLongMath: Expression of type int may overflow before being assigned to a long

Suggested change

long lossCount = watchMultiple * clientThreads * znodeCount - copyLatencyList.size();

long lossCount = (long) watchMultiple * clientThreads * znodeCount - copyLatencyList.size();

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:32Z

+    private static final Logger LOG = LoggerFactory.getLogger(WatchBenchmarkTool.class);
+
+    private static final ConcurrentHashMap<Integer, Long> watchTriggerTimeMap = new ConcurrentHashMap<>();
+    private static final List<Long> latencyList = new Vector<>();


JdkObsolete: Vector performs synchronization that is usually unnecessary; prefer ArrayList.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:33Z

+         * A deep copy of the latencyList, to avoid this situation when we statistics latencyList
+         * at the same time, watch events in flight are added to latencyList concurrently.
+         */
+        List<Long> copyLatencyList = new LinkedList<>();


JdkObsolete: It is very rare for LinkedList to out-perform ArrayList or ArrayDeque. Avoid it unless you're willing to invest a lot of time into benchmarking. Caveat: LinkedList supports null elements, but ArrayDeque does not.

Suggested change

List<Long> copyLatencyList = new LinkedList<>();

List<Long> copyLatencyList = new ArrayList<>();

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:35Z

+                }
+            } else {
+                // help user to create the znode: rootPath
+                String[] paths = rootPath.split("/");


StringSplitter: String.split(String) has surprising behavior

Suggested change

String[] paths = rootPath.split("/");

List<String> paths = Splitter.on('/').splitToList(rootPath);

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:40Z

+if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then
+  . "$ZOOBINDIR"/../libexec/zkEnv.sh
+else
+  . "$ZOOBINDIR"/zkEnv.sh


SC1091: Not following: ./zkEnv.sh was not specified as input (see shellcheck -x).

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:42Z

+ZOOBINDIR="$(cd "${ZOOBIN}"; pwd)"
+
+if [ -e "$ZOOBIN/../libexec/zkEnv.sh" ]; then
+  . "$ZOOBINDIR"/../libexec/zkEnv.sh


SC1091: Not following: ./../libexec/zkEnv.sh was not specified as input (see shellcheck -x).

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:43Z

+  . "$ZOOBINDIR"/zkEnv.sh
+fi
+
+"$JAVA" -cp "$CLASSPATH" $JVMFLAGS \


SC2086: Double quote to prevent globbing and word splitting.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:45Z

+# use POSIX interface, symlink is followed automatically
+ZOOBIN="${BASH_SOURCE-$0}"
+ZOOBIN="$(dirname "${ZOOBIN}")"
+ZOOBINDIR="$(cd "${ZOOBIN}"; pwd)"


SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:46Z

+                        zk.addWatch(rootPath + "/" + i, simpleWatcher, PERSISTENT);
+                    }
+                } catch (KeeperException | InterruptedException e) {
+                    e.printStackTrace();


💬 7 similar findings have been found in this PR

CatchAndPrintStackTrace: Logging or rethrowing exceptions should usually be preferred to catching and calling printStackTrace

🔎 Expand here to view all instances of this finding

File Path Line Number

zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java 497

zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java 524

zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java 201

zookeeper-jute/target/generated-sources/java/org/apache/zookeeper/server/quorum/QuorumPacket.java 126

zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java 480

zookeeper-jute/target/generated-sources/java/org/apache/zookeeper/proto/GetMaxChildrenRequest.java 61

zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java 544

Visit the Lift Web Console to find more details in your report.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

sonatype-lift · 2022-09-02T12:13:48Z

+        try {
+            ZooKeeper zk = initZKClient();
+            if (zk.exists(rootPath, null) != null) {
+                if (!cmd.hasOption("force")) {


💬 6 similar findings have been found in this PR

RESOURCE_LEAK: resource of type org.apache.zookeeper.ZooKeeper acquired by call to initZKClient() at line 267 is not released after line 269.

🔎 Expand here to view all instances of this finding

File Path Line Number

zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/SnapStream.java 132

zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/SnapStream.java 132

zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java 1366

zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java 1335

zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java 276

zookeeper-server/src/main/java/org/apache/zookeeper/server/persistence/SnapStream.java 107

Visit the Lift Web Console to find more details in your report.

ℹ️ Learn about @sonatype-lift commands

You can reply with the following commands. For example, reply with @sonatype-lift ignoreall to leave out all findings.

Command Usage

@sonatype-lift ignore Leave out the above finding from this PR

@sonatype-lift ignoreall Leave out all the existing findings from this PR

@sonatype-lift exclude <file|issue|path|tool> Exclude specified file|issue|path|tool from Lift findings by updating your config.toml file

Note: When talking to LiftBot, you need to refresh the page to see its response.
_{Click here to add LiftBot to another repo.}

Was this a good recommendation?
[ 🙁 Not relevant ] - [ 😕 Won't fix ] - [ 😑 Not critical, will fix ] - [ 🙂 Critical, will fix ] - [ 😊 Critical, fixing now ]

### 1. Brief Introduction At first, I want to implement this benchmark tool based on the JMH. However, its has the following drawbacks: - For this situation：I want to simulate 1000000 requests with 100 concurrency client threads,What the parameters/options of JMH(mode,fork,threads,measurementIterations,measurementBatchSize) should I set? - Based on JMH, it also needs or use great effort to write a satisfactory script/CLI. Then, I want to implement the tool from scratch. Have to admit that: it's difficult to implement a tool as powerful as YCSB did. Finally, the YCSB is our choice. Here are its introductions: - [Home Page](https://github.com/brianfrankcooper/YCSB/blob/master/README.md) - [YCSB Paper](https://www2.cs.duke.edu/courses/fall13/cps296.4/838-CloudPapers/ycsb.pdf) is one of the best materials(A Great Reading) YCSB has the following pros: - It supports requests number and concurrency threads number that user appoints - It supports various workloads for different read/write/insert/scan request ratio - It supports different request distributions(uniform, zipfian, latest) - It can output the benchmark results to different report formats(e.g.: `hdrhistogram`) - It has abundant metrics: throughout, avg/min/max/percentile latency, etc. - It has a detailed benchmark usage documentation ### 2. Benchmark Report Here is a [WIP Benchmark Report](https://docs.google.com/document/d/1_kGwnx5y7SYLSLtEPbkGJkrWbnsSftwNTvi8RCC9L7Y/edit?usp=sharing) which tested on the standard cloud machines with release-3.6.1(still a very messy draft and lots of works to do in the future). Two conspicuous performance issues: #### 2.1. Tail latency The tail latency is sometimes up to 200 ms. we can observer long-fat tail. Look at [figure-1](https://issues.apache.org/jira/secure/attachment/13017005/bm-read-histogram.png) and [figure-2](https://issues.apache.org/jira/secure/attachment/13017004/bm-update-histogram.png). We demand a smoother latency curve. The potential causes: - Java GC - The commit algorithm which schedules read/write requests in the `CommitProcessor`. (Most likely place where significantly improves the performance) #### 2.2. Connection numbers When connection numbers >= 1000, the throughput and latency extremely degrade. Look at [figure-3](https://issues.apache.org/jira/secure/attachment/13017006/bm-clients.png). Using the TCP connection can guarantee the message/packet order, but TCP connections that one Linux VM could support/hold is limited. Perhaps based/focused on Netty, we can improve this issue further. ### 3. Related Works 3.1 [YCSB-PR-1327](brianfrankcooper/YCSB#1327) 3.2 [ZooKeeper Watch Benchmark Tool](apache#1406) Author: maoling <maoling199210191@sina.com> Reviewers: Mate Szalay-Beko <symat@apache.org>, Damien Diederen <ddiederen@apache.org> Closes apache#1558 from maoling/ZOOKEEPER-3264

maoling · 2025-10-12T13:05:59Z

Updates at 2025-10-12

process review comments and rebase with master
use AvgMinMaxPercentileCounter for memory efficiency

…mance

nkalmar requested changes Aug 6, 2020

View reviewed changes

maoling mentioned this pull request Dec 14, 2020

ZOOKEEPER-3264: [YCSB-binding] Add a benchmark tool for zookeeper #1558

Closed

maoling force-pushed the ZOOKEEPER-3823 branch from a785d18 to 3710c60 Compare September 2, 2022 11:53

sonatype-lift Bot reviewed Sep 2, 2022

View reviewed changes

ztzg force-pushed the master branch from 1c60545 to e2070be Compare October 3, 2023 12:57

maoling force-pushed the ZOOKEEPER-3823 branch from 3710c60 to 720eb26 Compare October 12, 2025 13:03

maoling force-pushed the ZOOKEEPER-3823 branch from 720eb26 to 9b41e6d Compare October 12, 2025 13:16

ZOOKEEPER-3823: Add a benchmark tool for testing watch feature perfor…

5455d86

…mance

maoling force-pushed the ZOOKEEPER-3823 branch from 9b41e6d to 5455d86 Compare October 12, 2025 13:25

		- Usages:


		nohup ./zkWatchBenchmark.sh -root_path /bench-watch -threads 100 -znode_count 1000 -force -connect_string \n

	private String abbreviation;
	private final String abbreviation;

Command	Usage
`@sonatype-lift ignore`	Leave out the above finding from this PR
`@sonatype-lift ignoreall`	Leave out all the existing findings from this PR
`@sonatype-lift exclude <file\|issue\|path\|tool>`	Exclude specified `file\|issue\|path\|tool` from Lift findings by updating your config.toml file

	long lossCount = watchMultiple * clientThreads * znodeCount - copyLatencyList.size();
	long lossCount = (long) watchMultiple * clientThreads * znodeCount - copyLatencyList.size();

	List<Long> copyLatencyList = new LinkedList<>();
	List<Long> copyLatencyList = new ArrayList<>();

	String[] paths = rootPath.split("/");
	List<String> paths = Splitter.on('/').splitToList(rootPath);

File Path	Line Number
zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java	497
zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java	524
zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java	201
zookeeper-jute/target/generated-sources/java/org/apache/zookeeper/server/quorum/QuorumPacket.java	126
zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java	480
zookeeper-jute/target/generated-sources/java/org/apache/zookeeper/proto/GetMaxChildrenRequest.java	61
zookeeper-server/src/main/java/org/apache/zookeeper/server/watch/WatchBenchmarkTool.java	544

Conversation

maoling commented Jul 20, 2020

Uh oh!

nkalmar left a comment

Choose a reason for hiding this comment

Uh oh!

nkalmar Aug 6, 2020

Choose a reason for hiding this comment

Uh oh!

nkalmar Aug 6, 2020

Choose a reason for hiding this comment

Uh oh!

nkalmar Aug 6, 2020

Choose a reason for hiding this comment

Uh oh!

nkalmar Aug 6, 2020

Choose a reason for hiding this comment

Uh oh!

nkalmar Aug 6, 2020

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

sonatype-lift Bot Sep 2, 2022

Choose a reason for hiding this comment

Uh oh!

maoling commented Oct 12, 2025

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants