Integration (5/11/2017)

Testing Principles

Testing Roles and Responsibilities

End-to-End Test Cases

• VoLTE

- VoLTEservice design and creation: VNFs, network resource, workflow, alarms, DGs, DCAE template are onboarded. VoLTE e2e network service, including the connectivity between data centers, are designed and distributed successfully

- Close loop design: DCAE threshold crossing rules, Holmes correlation rules, and operation policy are designed and distributedby CLAMP successfully

- Instantiation: VNFs are deployed in edge and core data centers successfully, underlay and overlay WAN network cross the two data centers is setup correctly

- Auto-healing/auto-scaling: With event data from VIM or VNF, certain auto-healing/auto-scaling policy is triggered and proper actions are taken to remedy the situation

- Service termination: After user terminates the VoLTE service, VNFs and related resource for the service are properly removed, and WAN connectivity between the two data center is also removed successfully.          

• Residential vCPE use case

- Design time: All the VPP-based VNFs are successfully onboarded.

- Design time: Infrastructure and per-customer service are created. Associated workflows, policies, DGs, and data analytics created, validated, and properly distributed

- Instantiation: Infrastructure service is instantiated and properly configured. This is performed only once.

- Customer order: Per-customer service is instantiated and properly configured. This is performed on-demand.

- Data plane: Once a customer service is up and running, packets can be exchanged between the customer BRG emulator and the webserver.

- Auto-healing: Inject a packet loss event to invoke threshold crossing event, which then causes APPC to restart the vG_MUX VM. Service is back to normal once restart is complete.

Testing Terminology

• Unit Testing (UT) – Unit testing focuses on individual software components or modules. Typically, UT is done by the programmers and not by testers, as it requires detailed knowledge of the internal program design and code; UT may require developing test driver modules or test harnesses. Code coverage is also one of the objectives of UT.
• Feature/Functional Testing – Feature/Functional testing, unlike unit testing, focuses on the output as defined per requirement (user story). This type of testing is black-box type testing, geared towards functional requirements on an application basis.
• Integration/Pair-Wise Testing – Integration/Pair-wise testing integratesall of the modules of the solution in a single environment to verify combined functionality after integration. It ensures everything comes together and there is end-to-end communications between all the integrated elements.
• End-to-End Testing – End-to-End testing involves testing of a complete application environment in scenarios that closelymimic real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems, if appropriate..
• Regression Testing – Testing the application as a whole for the modification in any module or functionality. Typically, automation tools are used for regression testing since it is difficult to cover all aspects of the system in a manual fashion.
• Performance Testing – Performance testing tests the solution to see what throughput levels can be achieved based on the given platform. Performance testing term isoften used interchangeably with ‘stress’ testing (pushes the system beyond its specifications to check how and when it fails) and ‘load’ testing (which checks the system behavior under steady load.). An objective ofthis testing isto verify that the system meetsperformance requirements. Different performance and load tools  are used for this type of testing. [Note: Capacity testing and performance testingareclosely aligned. Capacity testing tests against product requirements to ensure the system meets requirements. Performance testing pushes the system to highest numbers it can achieve before it becomes unstable or success rate is unacceptable.]
• Acceptance Testing – Normally this type of testing is done to verify if system meets the customer specified requirements. User or customer do this testing to determine whether to accept application.
• Usability Testing – Usability testing is focused on testing of user interfaces (UI’s). It checks user-friendliness along with application flows; e.g., can new users understand the application easily, is proper help documented whenever users get stuck at any point, etc…. Basicallysystemnavigation is checked in this testing.
• Install/Uninstall Testing – This category of testing validates full or partial install/uninstall, and upgrade and rollback processes on different operating systems under different hardware, software environment including backup/restore mechanisms.
• Recovery Testing – This category of testing validates how well a system recovers from crashes, hardware failures, or other catastrophic problems in various configurations such as HA and Geo-Redundancy.
• Security Testing – Security testing looks for vulnerabilities in the software that would make the system susceptible to malicious users.  It tests how well the system is protected againstunauthorized internal or external access. It checks if system and databases are safe from external attacks.
• Stability Testing – This type of testing validates that the system can run in steady state for an extended period of time without any system downtime or crashes. Typical stability test is run for 72hrs. If any problems are encountered during the stability test, such as application crash, high failure rates, the test is stopped and issue is investigated. The stability test is restarted after a fix is identified.
• Scalability Testing – This type of testing is an extension of performance testing. The purpose of scalability testing is to validate that the system can scale efficiently by identifying major workloads and mitigate bottlenecks that can impede the scalability of the application.
• Application Testing – Tests the platform in the context of  a particular application.