Explore Courses Blog Tutorials Interview Questions
0 votes
in Blockchain by (16.8k points)

During trying to achieve the performance with Hyperledger Fabric which IBM team reported in their article Hyperledger Fabric: A Distributed Operating System for Permissioned Blockchains, I faced some problems and errors. I collected all useful information and want to share it with the HF community. Also, I have a couple of questions to the Fabric developers about its performance.

Target description

Hyperledger Fabric v1.1.0 network deployed using Cello on four c5.9xlarge (36vCPU) aws instances:


    fabric001: {

      cas: [],

      peers: ["[email protected]"],

      orderers: ["orderer1st.orderer"],

      zookeepers: ["zookeeper1st"],

      kafkas: ["kafka1st"]


    fabric002: {

      cas: [],

      peers: ["[email protected]"],

      orderers: ["orderer2nd.orderer"],

      zookeepers: ["zookeeper2nd"],

      kafkas: ["kafka2nd"]


    fabric003: {

      cas: [],

      peers: ["[email protected]"],

      orderers: ["orderer3rd.orderer"],

      zookeepers: ["zookeeper3rd"],

      kafkas: ["kafka3rd"]


    fabric004: {

      cas: ["ca1st.main"],

      peers: [],

      orderers: ["orderer4th.orderer"],

      zookeepers: ["zookeeper4th"],

      kafkas: ["kafka4th"]



TLS is disabled.

Fabric channel configuration (all others parameters are the default):

BatchTimeout: 1s


    MaxMessageCount: 500

    AbsoluteMaxBytes: 200 MB

    PreferredMaxBytes: 50 MB

I performed tests for both CouchDB and LevelDB as a state database. I use official Fabcar chaincode (Golang implementation) for my tests. I created simple nodejs app which interacts with the Fabric network using SDK and exposes HTTP API for load tests. This app is stateless and can be easily scaled. For load testing, I'm using tool YandexTank. I've performed two kinds of tests with high load: query (requests via peer001 to the Fabric state when blockchain is empty) and insert (transactions within the blockchain).


CouchDB as a state database

  • Query results: At ~1100 rps latency starts to increase. But Fabric instance is not loaded and have a lot of free resources. On the figure below you can see CPU and Memory usage by the Fabric network containers on the instance fabric001 during the test. 100% CPU usage means one full vCPU load. 

fabric001 container instances (couchdb, query)

fabric001 container instances (couchdb, insert)

Based on this I can conclude that Fabric Peer has problems with the CouchDB connection under the load.

My questions: Does Fabric comminity know about this bug? Do you have plans how to solve it?

LevelDB as a state database

fabric001 container instances (leveldb, query)

  • There are no any errors from the blockchain, I just see latency degradation.
  • Insert results: CPU and Memory usage of the fabric001 containers on the figure below: 

fabric001 container instances (leveldb, insert)

  • Aggressive latency degradation starts at ~850 rps. No errors from the blockchain.

My questions: What is the cause of this latency degradation? Why I can't achieve 3500 rps performance that IBM report in their article? What plans does Fabric community have on improving the performance?


1 Answer

0 votes
by (14.4k points)
edited by

Being a queueing system, the waiting time of Hyperledger Fabric increases exponentially as load increases. Therefore transaction latency is quite low. Howsoever, when we leverage golevelDB we should get at least 2000 tps, no matter the latency.

What I can figure out from the CPU utilization plot, there are 36 vCPUs and only 16 vCPUs which are completely utilized fully. You can set the value for validatorPoolSize in core.yaml for each peer as equal or lesser than the block size, and henceforth check for an increase in throughput.

The performance, however, is dependent on some parameters and would, therefore, differ based on them. These parameters are:  

  1. workload (fab car versus fabcoin)

  2. disk (hdd vs ssd, local versus network (attached))

  3. load generator (CLI versus SDK)

  4. Load generation method (open versus closed versus distributions) 

  5. Network bandwidth (should be at least 1.6 Gbps for 2700 tps)

Also, before looking for results you need to make sure that the load generator is not becoming a bottleneck. Therefore you can divide latency into endorsement latency, ordering latency, and commit latency. Then, you collect other resource utilization values including network and disk. This will help you identify the bottleneck (if any exists) easily.

Want to make your career in Blockchain? Enroll in Blockchain Course to acquire the essential skills.

Browse Categories