Integration (5/11/2017)

Project Name:

Proposed name for the project: Integration
Proposed name for the repository: integration

Project description:

Integration is responsible for ONAP cross-project system integration, CI/CD, and all related end-to-end release use cases testing with VNFs necessary for the successful delivery and industry adaption of the ONAP project as a whole.

Scope:

It provides all the cross-project infrastructure framework and DevOps toolchain (Continuous Integration, etc.), code and scripts, best practice guidance, benchmark and testing reports and white papers related to:

Cross-project Continuous System Integration Testing (CSIT)
End-to-End (ETE) release use cases testing with VNFs with repeatability
Service design for end-to-end release use cases
(obsolete)Open Lab: building and maintenance of community integration labs
Continuous Distribution (CD) to ONAP community integration labs
Reference VNFs that can be used to show how the ONAP platform handles
- VNF installation and
- VNF life cycle management
- VNF Requirement compliance

	Category	Primary Contact / Participants	Sub-Category	Description	Problem Being Solved
1	Test	Daniel Rose ChrisC Anaël Closson	Test	Automated testing infrastructure and tools CSIT: testing of individual ONAP microservices and small collections of ONAP microservices supported by mocked services as necessary End-to-End (ETE) test flows using a full ONAP deployment Code and tools for automatic system testing and continuous integration test flows across ONAP projects Common guidelines, templates, generic tools, infrastructure, and best practices to help project developers to write unit and system test code Test requirement from developer point of view.	Automate the building artifacts/binaries to minimize human errors and reduce engineering costs Ensure that changes in one project will not break the functionality of other projects Assure that the entire ONAP project/product functions correctly in the case of continual change in subprojects Ensure consistency in unit and system testing methodology across all the ONAP projects Capture security issues
1	Test	Daniel Rose ChrisC Anaël Closson	S3P	Test cases for performance, scalability, resilience/stress testing, longevity Benchmarking and performance whitepapers	Define standard S3P testing metrics Provide and publish benchmarking results
2	Release	Gary Wu ChrisC Anaël Closson	CI Management (ci-management repo)	Scripts and definitions for build and CI jobs in Jenkins includes any docker build jobs for mock/simulated services excludes docker build jobs for ONAP components (assumed to be handled by the ONAP Operations Manager project)	Required to support the executing of CI jobs (e.g. for Jenkins)
			Autorelease	Define community-wide artifact versioning and release strategy Scripts and Jenkins job definitions to build the artifacts/binaries (e.g. zip/tar.gz files) that are used in the release candidates and final release	Detect/resolve cross-project compilation dependency issues Generate release candidates and final release artifacts
			Distribution (Refer to ONAP Operations Manager project)	Current decision is to go with docker images as the primary distribution method Docker builds and images are assumed to be handled by the ONAP Operations Manager project.	N/A
			Packaging (Refer to ONAP Operations Manager project)	Current decision is to forgo any deb or RPM packages, and go with docker images as the primary distribution method Docker builds and images are assumed to be handled by the ONAP Operations Manager project.	N/A
3	Bootstrap		Bootstrap	A framework to automatically install and test a set of base infrastructure components for new developer	Reduce the barrier of entry to allow new ONAP developers to ramp up onto active development quickly Reduce the cost to the community in responding to simple environment setup questions faced by new developers
3	Bootstrap		Infrastructure Specification	Develop the specifications for the “ONAP compliant” deployment and test environment	Assist the planning and procurement of the necessary hardware and infrastructure for setting up ONAP environments
4	Developer Lab	Yang Xu	End-to-end release use cases testing with VNFs with repeatability	Create automatic test cases and script for VF testing Perform VF compliant testing and verification using tools provided by ONAP Delivery the testing reports and whitepaper	Assist define the testing metrics Reduce adoption risks for end-users
5	E2E Integration Lab		End to end deployment in "real" env using Open Lab	Scripts and definitions for setting up a POC sample deployment of use cases in lab settings Provisioning, installation, and setup of all the telco equipment such as switches, routers, and gateways to enable end to end testing Allow remote access to the lab environment for interoperability testing Automatic updates of code in lab environment from future releases	Support the needs of consistent, reproducible lab setup for demo and POC purposes Promote easy interoperability testing with different hardware devices, SDN controllers, etc. Automate the process of keeping the lab code up to date with the latest changes
6	Reference VNFs Project		Reference VNFs Project (now part of Integration project)	Two basic VNFs, namely a virtual firewall and a virtual load balancer (with virtual DNSs), have been provided. The objectives of the project are to improve, extend and maintain the vFirewall and vLoadBalancer VNFs: Allow ONAP to change vFirewall rules during execution Platform independence (Rackspace, vanilla Openstack, Azure, ...) Visualization tools that allow users to monitor the behavior of the reference VNFs as well as the effect of ONAP closed-loop operations against the VNFs Tools that allow users to interact with the reference VNFs (e.g. alter the behavior of a VNF so as to violate predefined policies, in order to trigger ONAP closed-loop operations)	The goal is to build reference VNFs that can be used to show how the ONAP platform manages VNFs installation and lifetime management. Reference VNFs can also be used as a means to test the platform itself, e.g. verify whether VNFs on-boarding, deployment, and ONAP closed-loop operations work. Reference VNFS should also demonstrate and document VNF Requirement compliance
7	O-Parent	Gary Wu	O-Parent	ONAP Parent provides common default settings for all the projects participating in simultaneous release.	Isolate all the common external dependencies, default version, dependency management, plugin management, etc. Avoid duplicate/conflicting settings for each project Each project sets its parent to inherit the defaults from ONAP Parent Project level external dependencies and versions can be overridden if necessary

Testing Principles

We expect test automation for all testing in scope for release 1.0

Regression, Unit and Feature/Function testing should be triggered by built process
All testing must be able to execute on the selected ONAP environments
Unit Testing for any project should have at least 30% code coverage
Any new feature should be delivered with its associated unit tests/feature tests

Jenkins Testing Flow

Testing Roles and Responsibilities

Types of Testing	Dev. Team	CSIT Team	E2E Team	S3P Team
Unit Testing	x
Feature/Functional Testing	x
Integration/Pair-Wise Testing		x
End-to-End Testing			x
Regression Testing	x	x	x	x
Performance Testing				x
Acceptance Testing		x	x
Usability Testing	x
Install/Uninstall Testing	x
Recovering Testing			x	x
Security Testing				x
Stability Testing				x
Scalability Testing				x
Application Testing	x

Testing Terminology

Unit Testing (UT) – Unit testing focuses on individual software components or modules. Typically, UT is done by the programmers and not by testers, as it requires detailed knowledge of the internal program design and code; UT may require developing test driver modules or test harnesses. Code coverage is also one of the objectives of UT.
Feature/Functional Testing – Feature/Functional testing, unlike unit testing, focuses on the output as defined per requirement (user story). This type of testing is black-box type testing, geared towards functional requirements on an application basis.
Integration/Pair-Wise Testing – Integration/Pair-wise testing integrates all of the modules of the solution in a single environment to verify combined functionality after integration. It ensures everything comes together and there is end-to-end communications between all the integrated elements.
End-to-End Testing – End-to-End testing involves testing of a complete application environment in scenarios that closely mimic real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems, if appropriate..
Regression Testing – Testing the application as a whole for the modification in any module or functionality. Typically, automation tools are used for regression testing since it is difficult to cover all aspects of the system in a manual fashion.
Performance Testing – Performance testing tests the solution to see what throughput levels can be achieved based on the given platform. Performance testing term is often used interchangeably with ‘stress’ testing (pushes the system beyond its specifications to check how and when it fails) and ‘load’ testing (which checks the system behavior under steady load.). An objective of this testing is to verify that the system meets performance requirements. Different performance and load tools are used for this type of testing. [Note: Capacity testing and performance testing are closely aligned. Capacity testing tests against product requirements to ensure the system meets requirements. Performance testing pushes the system to highest numbers it can achieve before it becomes unstable or success rate is unacceptable.]
Acceptance Testing – Normally this type of testing is done to verify if system meets the customer specified requirements. User or customer do this testing to determine whether to accept application.
Usability Testing – Usability testing is focused on testing of user interfaces (UI’s). It checks user-friendliness along with application flows; e.g., can new users understand the application easily, is proper help documented whenever users get stuck at any point, etc…. Basically system navigation is checked in this testing.
Install/Uninstall Testing – This category of testing validates full or partial install/uninstall, and upgrade and rollback processes on different operating systems under different hardware, software environment including backup/restore mechanisms.
Recovery Testing – This category of testing validates how well a system recovers from crashes, hardware failures, or other catastrophic problems in various configurations such as HA and Geo-Redundancy.
Security Testing – Security testing looks for vulnerabilities in the software that would make the system susceptible to malicious users. It tests how well the system is protected against unauthorized internal or external access. It checks if system and databases are safe from external attacks.
Stability Testing – This type of testing validates that the system can run in steady state for an extended period of time without any system downtime or crashes. Typical stability test is run for 72hrs. If any problems are encountered during the stability test, such as application crash, high failure rates, the test is stopped and issue is investigated. The stability test is restarted after a fix is identified.
Scalability Testing – This type of testing is an extension of performance testing. The purpose of scalability testing is to validate that the system can scale efficiently by identifying major workloads and mitigate bottlenecks that can impede the scalability of the application.
Application Testing – Tests the platform in the context of a particular application.

Architecture Alignment:

How does this project fit into the rest of the ONAP Architecture?
- What other ONAP projects does this project depend on?
  - All ONAP projects
How does this align with external standards/specifications?
Are there dependencies with other open source projects?
- Robot
- Jenkins
- OpenStack
- Docker

Other Information:

Preliminary V1 Plan
- Integration V1 Release Plan
ONAP Community Labs Specification
- Lab Resource
Link to seed code (if applicable)
ECOMP existing repos:
- testsuite
- testsuite/heatbridge
- testsuite/properties
- testsuite/python-testing-utils
- demo
- ci-management
OPEN-O existing repos:
- integration
- ci-management
- oparent
Vendor Neutral
- This project is vendor neutral

Meets Board policy (including IPR)
- Yes

Use the above information to create a key project facts section on your project page

Key Project Facts

Project Name: Integration

JIRA project name: integration

JIRA project prefix: integration

Repo name:

- integration
- demo
- testsuite
- testsuite/heatbridge
- testsuite/properties
- testsuite/python-testing-utils
- ci-management
- oparent

Lifecycle State: incubation
Primary Contact: Helen Chen helen.chen@huawei.com
Project Lead:
mailing list tag [integration]
Committers:

See above

*Link to TSC approval:
Link to approval of additional submitters: