...
- stability test: vFw (then vFWCL) run continuously during 72h (https://docs.onap.org/en/elalto/submodules/integration.git/docs/integration-s3p.html?highlight=stability)
- resilency test: test when we destroy some pods and retest that the use case vFw is still OK (only up to El Alto)
In frankfurt we also consider the stability of the installation through teh Dialy chainsthe Daily chains (https://docs.onap.org/projects/onap-integration/en/frankfurt/integration-s3p.html#integration-s3p)
Guilin
The stability tests considered for the release were:
...
what do we want to test, which figures? Nb of onboardings / instantiations? test duration//
In a first step, we estimate our needs to < to be discussed/commented/challenged/questioned/...>:
- 10 parallel service onboarding
- 50 parallel intsantiation
- ....
- - 10 simultaneous module upload in ONAP
- 50 parallel instantiation - 50 simultaneous service creations of service declared in ONAP
We could imagine additional KPIs
- number of simultaneous loop creation/instantiation
- number of dmaap messages
- number of event messages
The question has been raised at the community level especially among the service provider operating ONAP in production.
Testcase 1: Parallel onboarding tests
Description
The goal of this test is to create in parallel several services in the SDC.
...
Environment
Tests executed from 07/01/2021 to 13/01/2021 on a Guilin lab. Reusing the basic_vm with different service names (it means that we recreate all the SDC objects VSP, VF Services).
2 series run several times:
- 5 simutalneous simultaneous onboarding
- 10 simultaneous onboarding
The main component used for this test is the SDC (+AAI).
the reporting page can be described as follows:
The name of the service is basic_onboard_<Random string>, the random string is needed to ensure we reuse the onboarding mechanism (with the same name pythonsdk will retrieved the service already onboarded)
During the test we monitor the ONAP cluster resources through a prometheus/grafana
<graph grafana memory & CPU générale>
...
:
Common Cassandra resource consumption:
Results
Data format is MM:SS,00.
5 parallel onboarding (10 series)
| criteria \ Serie | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | AverageGlobal | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Success rate (%) | 100 | 100 | 100 | 100 | 100 | 80100 | 100 | 100 | 100 | 100 | 98100 | |
| Min duration | 04:46,0027:39,00 | 10:23,0024 | 10:14,0015 | 10:26,00 | 11:18,00 | 07:41,0042 | 07:53,0054 | 08:05,00 | 08:35 | 08:34,00204 | 07:46,0042 | |
| Max duration | 04:53,0027:43,00 | 10:36,00 | 10:17,00 | 10:26,0027 | 11:20,0022 | 07:53,0007 | 08:59,00 | 08:18,0019 | 08:42,00 | 08:44 | 27:43,00 | |
| Average duration | 0427:50,0041 | 27:40,80 | 10:27,4016 | 10:15,602710 | 11:26,002011 | 07:18,7548 | 07:47,805807 | 08:57,2012 | 08:11,2039 | 08:38,40:42 | 11:09 | |
| Median duration | 04:50,0027:41,00 | 10:26,00 | 10:16,00 | 10:26,0027 | 11:18,5019 | 07:48,0049 | 07:59,00 | 08:13 | 08:12,0040 | 08:39,0038 | 09:30 | |
| Comments/Errrors | / | / | / | / | / | ERROR : maximum recursion depth exceeded in a python object | / | / | / | / | / | / |
<graph min/max/mean/average = f(serie)Evolution of the average duration in seconds over time for series of 5.
10 parallel onboarding (5 series)
| criteria \ Serie | 1 | 2 | 3 | 4 | 5 | AverageGlobal | ||
|---|---|---|---|---|---|---|---|---|
| Success rate (%) | 100 | 100 | 100 | 100 | 90100 | 98100 | ||
| Min duration | 16:03,00 | 16:03,00 | 15:23,0004 | 15:24 | 16:32,00 | 19:40 | 19:39,0007 | 15:23,0024 |
| Max duration | 16:22,00 | 16:22,00 | 17:10,00 | 17:36,00 | 20:00,00:01 | 19:50 | 20:00,0001 | |
| Average duration | 16:14,4015 | 16:14,405116 | 17:50,402317 | 19:22,2052 | 19:50,46 | 18:00 | ||
| Median duration | 16:18,502016 | 17:18,5008 | 17:08,003317 | 19:31,5053 | 19:52,00:38 | 17:33 | ||
| Comments/Errrors | / | / | / | / | ERROR : Resource Category with "Generic" name does not exist |
<graph min/max/mean/average = f(serie)
...
/ | / |
Evolution of the average duration in seconds over time for series of 10.
Evolution of test durations over the campaign for series of 5 (red/first circle) and 10 (green/second circle).
Conclusions
ONAP Guilin is able to support 10 parallel onboarding, which is what we do expect.
We may also observe that:
- The number of previous onboarded services has no impact on the onboarding duration. The creation of resources is linear. It means that on serie 10, 9 services have been already created. We could have expected a linear increase of the onboarding duration because the client used for test list several times the services.So the more services in SDC, the bigger the list is. So globally the SDC resources increases continuously because we cannot delete them but it has no direct impact on the onboarding duration. The duration evolution is not linear and the duration may depend mainly on the cluster status.
- The more // processing we have, the slower the onboarding this. duration = f(nb parallel onboarding) seems almost linear.