Due to the scalability dilemma, performance has always been the major factor to restraint the development of blockchain. Thus, working on the breakthrough of TPS is the main trend as well as solution for the arms race between public blockchains. Under such circumstance, many public blockchains advertise high TPS without the uniform and clear metrics and standard test methods of the test data.
In order to execute the performance tests with standard, complete, rigorous and feasible assessment methods, PlatON defined a suit of standard performance assessment model and criteria test for executing the standard performance test of blockchian system.
We applied the assessment model in every performance test. We have completed all kinds of functional tests, inception tests, security tests and stress tests with PlatON community. So far, PlatON has gone through 4 stress tests in both simulated environments and testing environments, and we will share the structure and tasks of standard performance assessment model according to the test data.
Complete Test Structure Model
Below is the technical structure of blockchain system criteria test, which injects the workload into the to-be-tested inner system through the outermost testing tools or the test plug-ins in the nodes to simulate the transaction scenarios, and monitor and collect the index data.
1. To-be-tested System
In this article, the to-be-tested system refers to the configuration specifications including hardware, software, network, etc. Client API, external linked libraries, and the external cache or databases that are used for optimizing the read operation are excluded.
2. Testing Tool
To evaluate the blockchain systems overall, we need to do it from different dimensions, and different dimensions require different testing methods. Here, PlatON designed two kinds of testing tools, testing plug-ins and test clients, and will apply different testing tools to test the blockchain system from different levels.
Multidimensional Test Indexes
There are five kinds of test indexes of blockchain system, namely, performance index, resource index, stability index, scalability index and reliability index .
1. Performance Index
We defines several key indexes to evaluate the system performance of blockchain, namely, transaction response time, system processing capacity and transaction success rate.
Analysis methods of index are shown as follows:
-
Each kind of index should analyze the average, highest, lowest and standard differences according to the time sequence.
-
Collect the data where the system run stably for 3 secs, and calculate the average, highest and lowest value of the event index in the last 3 secs every 3 secs.
2. Resource Index
Resource index refers to the system resources when system is under loading including CPU, memory, disk storage and network traffic.
Every server can join PlatON by running a node, and different node has different specifications. If there are too much resource consumption caused, some nodes may crash or are less efficient in its processing performance as the weakness of the whole network.
The resource index also evaluates the cost that nodes participate in the whole network.
3. Stability Index
The system will run for a while under the 80% of the maximum capacity or the standard load (the expected stress of system).
Blockchain systems require non-stop 7*24 work, thus it is necessary that the system need to run at least 24 hours. If not, the performance will decrease and even crash along with the increase of transaction volume and long period of running when the system is launched.
4. Scalability Index
The system will run according to the standard load (the expected stress of system). Then increase the number of nodes gradually, and check the relationship between the number of added nodes and the added processing capability.
Generally speaking, with the increase of hardware resources, the processing capability of distributed system will improve linearly or also. However, when blockchain system run with P2P network, all messages including the consensus will be broadcast in the P2P way. The communication complexity will increase linearly or exponentially along with the increase of node number, and the processing capability will reduce or even stop accordingly.
5. Reliability Index
The reliability index evaluates the recovery capability of system when it’s crashed, and the influence when double spending or selsh mining attack happened, namely the network or node failures, including node crash, network delay, damage of random message, the malicious fork attempts from malicious nodes.
There shouldn’t be block production suspension or forks in the fault-tolerant range of system.
Set the Real Testing Environment
The operating environment of blockchain, especially decentralized blockchain is not in controlled, and the node distribution and configuration vary widely. Thus, the node configuration should be set according to the actual operating environment as much as possible, to evaluate the operation performance of blockchain system in the real environment.
1. Node Distribution
Nodes should be located worldwide, covering at last Europe, North-America, South-America, Asia-Pacific and the Middle East.
Or, you can set the network delay and packet loss rate in local area network to simulate the network in different regions around the world. You can refer to 300 ms delayed average network in different AWS areas across the Pacific, and 1% of the network packet loss rate in different Verizon areas across the Pacific.
2. Node Configuration
According to the node operation performance of mainstream blockchain systems, there are three levels of node configuration, namely, Low, Medium
Apart from that, PlatON applied the test load with the combination of basic data, transaction type and request distribution to evaluate the specific points of the performance spaces. Plus, PlatON divides the basic test into 5 layers, and execute the comprehensive evaluation based on performance, scalability, availability, security and stability to ensure the correctness of evaluation results.