This is a DRAFT version of the testing plan of Holmes.
Overview
Comments
Load Testing
Alarm sending rate will be increasing from 50 alarms/s to 1000 alarms/s, depending on whether the usage of CPU or memory reaches 100% or the number of alarms lining up to be processed keeps increasing stably. This is intended for indentifying the limit of Holmes' processing capability.
When Holmes reaches the peak of its processing capability, the corresponding footprint will be recorded.
Performance Testing
Because the performance of Holmes varies when the number of deployed rules or the complexity of the rules changes, three sets of testing will be carried out to try to find out a pattern of the impacts resulted in by the number or the complexity of the rules.
Soak Testing
The bottleneck of Holmes mainly resides in the engine manamgement module and the workflow of alarm processing. So the soak testing will focus on telling the stability of this module.