Java Microbenchmark Harness

Micro-benchmarking in Java means checking how fast a small piece of code runs. To do this correctly, we use a tool called JMH (Java Microbenchmark Harness), which helps give accurate results. It uses special tags like @Benchmark and runs the code multiple times to measure the real performance.

In this blog, we will discuss the micro benchmarking fundamentals in detail.

Table of Contents:

• What is a Micro-Benchmark?
• Features of Java Microbenchmark Harness (JMH)
• Common Annotations in Benchmarks
• @State in JMH
• Java Benchmarking Techniques
  • Benchmarking Using Start/End Time
  • Benchmarking Using Java Microbenchmark Harness (JMH)
• Challenges in Micro-benchmarking
  • JVM Warm-Up Overhead
  • JIT Optimizations
  • Garbage Collection Interference
  • External Interference
  • Loop Optimization
  • Dead code elimination
  • Constant folding
• Best Practices in Micro-benchmarking
• Conclusion

What is a Micro-Benchmark?

Micro-Benchmark is an approach where small units of code are tested on the basis of their storage and execution time. Unlike considering the whole application at once, it focuses on the small code and gives the results on it. Due to this, the user can know the area where the optimization has to be performed.

It is used mainly for the following reasons:

1. For comparing different approaches that have the same logic in the code.
2. To identify any bottleneck (problems) in the code.

Features of Java Microbenchmark Harness (JMH)

1. Handles JVM Tricks (Optimizations)

As the JVM knows the tricks, it does not read the repeated code or combine the loops. The JMH makes sure to measure the real performance without considering the tricks of the JVM.

2. Different Ways to Measure Speed

JMH allows you to measure the speed in different ways. Some of these are:

• How many times does it run per second (Throughput)
• How long does each run take (AverageTime)
• Random time samples (SampleTime)
• One-time execution (SingleShotTime)

3. Tests with Multiple Threads

JMH works on multithreads, i.e., it can also check the performance of multithreaded applications.

4. Try with Different Inputs

You can give your benchmark of different inputs using @Param. This helps you see how your code performs with small, medium, or large data. It also helps to see if there are any problems present with any input or not.

5. Works with Performance Monitoring Tools

JMH works with tools that can show what your CPU is doing, how much memory your code is using, and where the time is being spent.

Master Java Today - Accelerate Your Future

Enroll Now and Transform Your Future

Explore Program

Common Annotations in Benchmarks

• @Benchmark is used to annotate a benchmark method.
• @BenchmarkMode is used to set the benchmark mode, i.e., Throughput, AverageTime, SampleTime, SingleShotTime, or all.
• @OutputTimeUnit is used to set the unit of time like Nano, Micro, Milliseconds, Seconds, Minutes, Hours, or Days.
• @State allows you to define the state of fields per thread. The thread scope can be useful when you are running multi-threaded benchmarks.
• @Param is used to mark the parameter. Its fields should be non-final.
• @Fork finds how many separate JVM processes should be launched to run your benchmark.
• @Warmup runs the code a few times first so the JVM gets ready before measuring the performance.
• @Measurement tells JMH how many times to run the code to measure its performance.

@State in JMH

The @State annotation in JMH is used to define the scope of the lifecycle of data shared between different benchmarks. There are Different scopes to control how often the state object is created and shared. Some of them are:

1. Scope.Thread: A new state instance is created per thread. It is used when the state is not shared between threads. It is the most common for thread-safe benchmarking.

2. Scope.Benchmark: A single state instance is shared across all threads for the entire benchmark. It is useful for shared resources like caches, maps, etc.

3. Scope.Group: One state object per group of threads is shared. Used when you are testing thread interactions within a defined group. It is used less.

There are some requirements for the state class. Some of them are:

1. The class must be public.
2. The fields annotated with @Param should be non-final.
3. Annotated with @State(Scope.X), where X can be Thread, Benchmark, or Group.

Java Benchmarking Techniques

There are mainly 2 most used techniques for benchmarking. These are as follows:

Benchmarking Using Start/End Time

This is the easiest method used to know how much time a piece of code takes to execute. It is good for quick checks, but does not give accurate results.

Example:

Output:

Explanation: The above code first checks how long it is taking to add the numbers from 0 to 999,999.
It records the time before and after the loop runs. Then, it finds how much time the loop takes by subtracting two times. Finally, it prints out the time taken in nanoseconds.

Note:  This method isn’t reliable because of JVM warm-up, garbage collection, and other optimizations. Use JMH for accurate results.

Benchmarking Using Java Microbenchmark Harness (JMH)

It is the professional toolkit used for creating and running microbenchmarks in Java. The simple annotation is used here to get the output of a method or a class. It also handles the JIT compilation.

The JMH can be added in 2 ways: Maven and Gradle.

Step 1: Make the project.

Step 2: For Maven, add the following dependencies in the pom.xml file.

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>${jmh.version}</version>
</dependency>

<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>${jmh.version}</version>
    <scope>provided</scope>
</dependency>

Add the Maven compiler plugin for annotation processing.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    ...
    <annotationProcessorPaths>
        ...
    </annotationProcessorPaths>
</plugin>

Also, add the Maven shaded plugin for a runnable jar file.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    ...

    <mainClass>org.openjdk.jmh.Main</mainClass>
</plugin>

For Gradle, add the following dependencies to the build.gradle file.

dependencies {
   implementation 'org.openjdk.jmh:jmh-core:1.37'
   annotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.37'
}

Step 3: For Maven, run the Maven clean and package command from the top right corner.

It will install all the dependencies and will create a jar file.

For Gradle, refresh the project from the top right corner.

It will install all the dependencies and will create a JAR file.

Step 4: Write the code

Now, create the MyBenchmark.java as follows.

Now, modify your main class as follows.

Step 5: Run your Main.java class

Output:

Explanation: In the above code, the JMH benchmark is comparing two ways of summing an array, a loop, and Java streams. It initializes a small array of 1,000 integers. The @Warmup and @Measurement annotations make sure that the benchmark runs quickly. @BenchmarkMode(Mode.AverageTime) measures the average execution time, per operation, in milliseconds. @Fork(1) ensures that the benchmark runs once in a single JVM instance for faster feedback.

Unlock Your Future in Java

Start Your Java Journey for Free Today

Explore Program

Challenges in Micro-benchmarking

Benchmarking is a good technique for measuring the execution time of a process.  Despite its functionality, challenges persist. They are listed below:

1. JVM Warm-Up Overhead

When a Java program starts running, the JVM does not run at its full speed in the beginning. It takes some time to warm up and then applies optimizations like Just-In-Time (JIT) compilation. If benchmarking starts too early, it might get a warm-up time in its results.

2. JIT Optimizations

The JVM uses JIT to make the code faster by compiling it to machine code. However, this happens while the code is running hence, the performance can vary during the test, if not calculated properly.

3. Garbage Collection Interference

Java cleans up the memory by itself, but when it runs out of memory, the JVM stops for some time to clean it up. If this process happens at the time of the speed test, it can change the results.

4. External Interference

Other processes running on your machine can come in between your benchmark, which can lead to changed results.

5.  Loop Optimization

It is a technique where, compiler improves the performance of the loop, like some variables are used inside the loop and do not take any part in the calculation, then these variables can be removed.

6. Dead code elimination

It is a process done by the JVM that removes the line of code that does not take part in the computation process. It reduces CPU workload by removing unnecessary operations and saves CPU cycles.

A dead code is identified by the JVM when it has no side effects on the running program, if it has been removed, or if it has not been used in the output anywhere.

7. Constant folding

It is a process in which the JVM computes the constants at compile or runtime itself to reduce the computation time. It is a good thing, but it creates a problem in the calculation of benchmarking. The constant expressions are evaluated at compile-time, which can skip the actual runtime measurement.

Best Practices in Micro-benchmarking

1. Before measuring the performance, a warm-up of the code is a necessity as the JVM performs many operations like JIT compilation that can change the results. Always use the @Warmup annotations in the benchmark code to exclude the timing of the JVM warmup.
2. You should run your benchmark multiple times, as the results of a single run can have errors in them and can give unstable results.
3. Do not let your compiler optimize the code, as it will give you incorrect results.
4. Use annotations like @Benchmark, @Setup, and @State correctly. This will help JMH run your benchmarks in the right way and give accurate results.
5. Close the unused applications that can impact your application performance. Use environments like Docker as they do not impact your benchmark.
6. Blackhole is a JMH tool that keeps the code removed by the compiler during benchmarking. Use a blackhole to avoid dead code elimination.

Example:

@Benchmark
public void deadCode() {
    int result = 1 + 2; // JVM will eliminate this
}

You can fix the above code by returning the result like this,

@Benchmark
public int safeCode() {
    return 1 + 2;
}

Conclusion

To write a correct micro-benchmark in Java, you can use the Start/end time method and JMH. Start/End is simple to use, but not accurate. JMH can handle JVM optimizations like warm-up and JIT and is accurate. Always warm up your code, remove unused code, and run tests multiple times to get accurate results. This will help you measure the real performance.

If you want to learn more about Java, you can refer to our Java Course.

Java Microbenchmark Harness – FAQs