Standard deviation and its importance in performance testing

What is Standard Deviation?

Mathematical Definition : It is the measure of how spread out the numbers are around the mean.
Performance Definition : It is the measure of how response time is spread out around the mean.

It is denoted by Sigma (σ).


Formula: 



Where

N = Count of the response time
x = Individual response time value
μ = Mean or Average response time

Importance of Standard Deviation in Performance Testing

• Standard Deviation tells whether a transaction is consistent throughout the test or not.
• A small standard deviation means that the response time of all the iterations of the same transaction are close to each other. 
• So smaller the value more close the transaction response times are and more consistent the transaction is.

Example:

Lets calculate the Std. Deviation for the above transactions.

Login Transaction:

Step 1 - Average Response Time or Mean value =  5

Step 2 - 

(5-5)²  = 0

(5-5)²  = 0

(6-5)²  = 1

(5-5)²  = 0

(4-5)²  = 1

Step 3 - (0+0+1+0+1)/5 = 0.4

Step 4  - √ 0.4 = 0.632

Inference :

In the Login transaction we can observe that response time of each iteration are close enough so the Std. Deviation is small. It means Login transaction is more consistent throughout the test.

Logout Transaction:

Step 1 - Average Response Time or Mean value =  5

Step 2 - 

(5-5)²  = 0

(4-5)²  = 1

(12-5)²  = 49

(3-5)²  = 4

(1-5)²  = 16

Step 3 - (0+1+49+4+16)/5 = 14

Step 4  - √ 14 = 3.741

Inference :

In the Logout transaction we can observe that response time of each iteration are having large deviations so the Std. Deviation is high. It means Logout transaction is less consistent throughout the test.