The Platform Maturity requirements for Beijing Casablanca are on this page:
Platform Maturity Requirements (aka Carrier Grade)Casablanca Release Requirements
The ambition for the ONPA ONAP Policy Framework is to reach level 1 2 in performance in Beijing, see JIRA POLICY-392Casablanca, see
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
For Performance Level 1, we have had to define and measure our performance metrics:
Level 1: baseline performance criteria identified and measured (such as response time, transaction/message rate, latency, footprint, etc. to be defined on per componentI don't think we have to state what the metrics should be, it looks like we just define the metrics and measure them. For
Measurements were taken in the Beijing release for the Single-Threaded case. Additional measurements are to be taken in the Casablanca release, for the Multi-Threaded case (i.e., multiple threads, each generating their own ONSET events).
Depending on the results, for Level 2, an improvement plan must be created and implemented.
Although there should be measurements for design time and deployment time performance, here we continue to focus on the run time performance metrics for policy execution in the PDPs, the most critical run time metrics for Policy. We propose to measure these metrics in a simulated environment and in a full ONAP deployment
Measurements for the "Single Threaded" case were made in Beijing using the Stability test, which uses timers to determine when to inject responses via REST. It also uses the REST API to look for "FINAL" states, thus it is limited to two threads. On the other hand, the measurements for the "Multi Threaded" case are made using the Performance test, which uses DMaaP notifications to determine when to inject responses via REST. It also uses DMaaP notifications to look for "FINAL" states, thus it is only limited by how many simultaneous consumers and connections are supported by DMaaP.
| No. | Metric | Description | |
|---|---|---|---|
| 1 | Single Threaded Response Time | Measure the execution time for onset and abatement in each use case with only a single policy executing | |
| 2 | Multi-Threaded Response Time | Measure the execution time for onset and abatement in Single Threaded CPU Usage | CPU Usage for each use case when the maximum number of threads are executing alone |
| 3 | Single Threaded CPU Memory Usage | CPU Usage Memory Usage for each use case when executing alone | |
| 4 | Maximum Simultaneous Executions | Measure the maximum number of simultaneous policy executions that can be achieved whilst maintaining system stability and resource utilization | |
| 5 | Multi Threaded CPU UsageCPU Usage for Response Time | Measure the execution time for onset and abatement in each use case when executing in multiple threads are injecting ONSET events simultaneously | |
| 56 | Single Multi Threaded Memory CPU Usage | Memory Usage CPU Usage for each use case when executing alone6multiple threads are injecting ONSET events simultaneously | |
| 7 | Multi Threaded Memory Usage | Memory Usage for each use | case when executing incase when multiple threads |
| 7 | Maximum Simultaneous Executions | Measure the maximum number of simultaneous policy executions that can be achieved whilst maintaining system stability and resource utilization | |
| are injecting ONSET events simultaneously | |||
| 8 | Installation Footprint | The requirements of the ONAP Policy Framework installation in terms of VM resources including the amount of disk space required for the database and for logs. |
Based on the results of the tests, performance of the PDP-D is satisfactory. Should the load increase to the point where performance becomes an issue, then additional PDP-D's can be added; this is already supported and does not require any code changes. As a result, PDP-D satisfies both level 2 and level 3 performance requirements. Additional testing must be done to determine level 0-3 conformance for the other policy components (e.g., PDP-X).