Introduction

The ONAP Operations Manager (OOM) is responsible for life-cycle management of the ONAP platform itself; components such as MSO, SDNC, etc. It is not responsible for the management of services, VNFs or infrastructure instantiated by ONAP or used by ONAP to host such services or VNFs. OOM uses the open-source Kubernetes container management system as a means to manage the Docker containers that compose ONAP where the containers are hosted either directly on bare-metal servers or on VMs hosted by a 3rd party management system. OOM ensures that ONAP is easily deployable and maintainable throughout its life cycle while using hardware resources efficiently. There are two deployment options for OOM:

A minimal deployment where single instances of the ONAP components are instantiated with no resource reservations, and
A production deployment where ONAP components are deployed with redundancy and anti-affinity rules such that single faults do not interrupt ONAP operation.
When deployed as containers directly on bare-metal, the minimal deployment option requires a single host (32GB memory with 12 vCPUs) however further optimization should allow this deployment to target a laptop computer. Production deployments will require more resources as determined by anti-affinity and geo-redundancy requirements.

OOM deployments of ONAP provide many benefits:

Life-cycle Management Kubernetes is a comprehensive system for managing the life-cycle of containerized applications. Its use as a platform manager will ease the deployment of ONAP, provide fault tolerance and horizontal scalability, and enable seamless upgrades.
Hardware Efficiency ONAP can be deployed on a single host using less than 32GB of memory. As opposed to VMs that require a guest operating system be deployed along with the application, containers provide similar application encapsulation with neither the computing, memory and storage overhead nor the associated long term support costs of those guest operating systems. An informal goal of the project is to be able to create a development deployment of ONAP that can be hosted on a laptop.
Rapid Deployment With locally cached images ONAP can be deployed from scratch in 7 minutes. Eliminating the guest operating system results in containers coming into service much faster than a VM equivalent. This advantage can be particularly useful for ONAP where rapid reaction to inevitable failures will be critical in production environments.
Portability OOM takes advantage of Kubernetes' ability to be hosted on multiple hosted cloud solutions like Google Compute Engine, AWS EC2, Microsoft Azure, CenturyLink Cloud, IBM Bluemix and more.
Minimal Impact As ONAP is already deployed with Docker containers minimal changes are required to the components themselves when deployed with OOM.

Features of OOM:

Platform Deployment Automated deployment/un-deployment of ONAP instance(s) / Automated deployment/un-deployment of individual platform components using docker containers & kubernetes
Platform Monitoring & healing Monitor platform state, Platform health checks, fault tolerance and self-healing using docker containers & kubernetes
Platform Scaling Platform horizontal scalability through using docker containers & kubernetes
Platform Upgrades Platform upgrades using docker containers & kubernetes
Platform Configurations Manage overall platform components configurations using docker containers & kubernetes
Platform migrations Manage migration of platform components using docker containers & kubernetes
Please note that the ONAP Operations Manager does not provide support for containerization of services or VNFs that are managed by ONAP; the OOM orchestrates the life-cycle of the ONAP platform components themselves.

Linux containers allow for an application and all of its operating system dependencies to be packaged and deployed as a single unit without including a guest operating system as done with virtual machines. The most popular container solution is Docker which provides tools for container management like the Docker Host (dockerd) which can create, run, stop, move, or delete a container. Docker has a very popular registry of containers images that can be used by any Docker system; however, in the ONAP context, Docker images are built by the standard CI/CD flow and stored in Nexus repositories. OOM uses the "standard" ONAP docker containers and three new ones specifically created for OOM.

Containers are isolated from each other primarily via name spaces within the Linux kernel without the need for multiple guest operating systems. As such, multiple containers can be deployed with little overhead such as all of ONAP can be deployed on a single host. With some optimization of the ONAP components (e.g. elimination of redundant database instances) it may be possible to deploy ONAP on a single laptop computer.

Life Cycle Management via Kubernetes

As with the VNFs deployed by ONAP, the components of ONAP have their own life-cycle where the components are created, run, healed, scaled, stopped and deleted. These life-cycle operations are managed by the Kubernetes container management system which maintains the desired state of the container system as described by one or more deployment descriptors - similar in concept to OpenStack HEAT Orchestration Templates. The following sections describe the fundamental objects managed by Kubernetes, the network these components use to communicate with each other and other entities outside of ONAP and the templates that describe the configuration and desired state of the ONAP components.

ONAP Components to Kubernetes Object Relationships

Kubernetes deployments consist of multiple objects:

nodes - a worker machine - either physical or virtual - that hosts multiple containers managed by kubernetes.
services - an abstraction of a logical set of pods that provide a micro-service.
pods - one or more (but typically one) container(s) that provide specific application functionality.
persistent volumes - One or more permanent volumes need to be established to hold non-ephemeral configuration and state data.

The relationship between these objects is shown in the following figure:

Gliffy


name	kubernetes objects

OOM uses these kubernetes objects as described in the following sections.

Nodes

OOM works with both physical and virtual worker machines.

Virtual Machine Deployments - If ONAP is to be deployed onto a set of virtual machines, the creation of the VMs is outside of the scope of OOM and could be done in many ways, such as:
- manually, for example by a user using the OpenStack Horizon dashboard or AWS EC2, or
- automatically, for example with the use of a OpenStack Heat Orchestration Template which builds an ONAP stack, or
- orchestrated, for example with Cloudify creating the VMs from a TOSCA template and controlling their life cycle for the life of the ONAP deployment.
Physical Machine Deployments - If ONAP is to be deployed onto physical machines there are several options but the recommendation is to use Rancher along with Helm to associate hosts with a kubernetes cluster.

Pods

A group of containers with shared storage and networking can be grouped together into a kubernetes pod. All of the containers within a pod are co-located and co-scheduled so they operate as a single unit. Within ONAP Amsterdam release, pods are mapped one-to-one to docker containers although this may change in the future. As explained in the Services section below the use of Pods within each ONAP component is abstracted from other ONAP components.

Services

OOM uses the kubernetes service abstraction to provide a consistent access point for each of the ONAP components independent of the pod or container architecture of that component. For example, the SDNC component may introduce OpenDaylight clustering as some point and change the number of pods in this component to three or more but this change will be isolated from the other ONAP components by the service abstraction. A service can include a load balancer on its ingress to distribute traffic between the pods and even react to dynamic changes in the number of pods if they are part of a replica set (see the MSO example below for a brief explanation of replica sets).

Persistent Volumes

As pods and containers are ephemeral, any data that must be persisted across pod restart events needs to be stored outside of the pod in a persistent volume(s). Kubernetes supports a wide variety of types of persistent volumes such as: Fibre Channel, NFS, iSCSI, CephFS, and GlusterFS (for a full list look here) so there are many options as to how storage is configured when deploying ONAP via OOM.

OOM Networking with Kubernetes

DNS
Ports - Flattening the containers also expose port conflicts between the containers which need to be resolved.

Name Spaces

Within the namespaces are kubernete's services that provide external connectivity to pods that host Docker containers. The following is a list of the namespaces and the services within:

onap-aai
- aai-service
- hbase
- model-loader-service
- aai-dmaap
- aai-kafka
- aai-zookeeper
- aai-resources
- aai-traversal
- data-router
- elasticsearch
- gremlin
- search-data-service
- sparky-be
onap-appc
- dbhost
- dgbuilder
- sdnctldb01
- sdnctldb02
- sdnhost
onap-dcae
- cdap0
- cdap1
- cdap2
- dcae-collector-common-event
- dcae-collector-dmaapbc
- dcae-controller
- dcae-pgaas
- dmaap
- kafka
- zookeeper
onap-message-router
- dmaap
- global-kafka
- zookeeper
onap-mso
- mariadb
- mso
onap-policy
- brmsgw
- drools
- mariadb
- nexus
- pap
- pdp
- pypdp

onap-portal
- portalapps
- portaldb
- vnc-portal
onap-robot
- robot
onap-sdc
- sdc-be
- sdc-cs
- sdc-es
- sdc-fe
- sdc-kb
onap-sdnc
- dbhost
- sdnc-dgbuilder
- sdnc-portal
- sdnctldb01
- sdnctldb02
- sdnhost
onap-vid
- vid-mariadb
- vid-server

Note that services listed in italics are local to the namespace itself and not accessible from outside of the namespace.

Kubernetes Deployment Specifications for ONAP

Each of the ONAP components are deployed as described in a deployment specification. This specification documents key parameters and dependencies between the pods of an ONAP components such that kubernetes is able to repeatably startup the component. The components artifacts are stored here in the oom/kubernetes repo in ONAP gerrit. The mso project is a relatively simple example, so let's start there.

MSO Example

Within the oom/kubernetes/mso repo, one will find three file in yaml format:

The db-deployment.yaml file describes deployment of the database component of mso. Here is the contents:

Code Block

title	db-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mariadb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
      name: mariadb
    spec:
      hostname: mariadb
      containers:
      - args:
        image: nexus3.onap.org:10001/mariadb:10.1.11
        name: "mariadb"
        env:
          - name: MYSQL_ROOT_PASSWORD
            value: password
          - name: MARIADB_MAJOR
            value: "10.1"
          - name: MARIADB_VERSION
            value: "10.1.11+maria-1~jessie"
        volumeMounts:
        - mountPath: /etc/mysql/conf.d
          name: mso-mariadb-conf
        - mountPath: /docker-entrypoint-initdb.d
          name: mso-mariadb-docker-entrypoint-initdb
        ports:
        - containerPort: 3306
          name: mariadb
        readinessProbe:
          tcpSocket:
            port: 3306
          initialDelaySeconds: 5
          periodSeconds: 10
      volumes:
        - name: mso-mariadb-conf
          hostPath:
            path: /dockerdata-nfs/onapdemo/mso/mariadb/conf.d
        - name: mso-mariadb-docker-entrypoint-initdb
          hostPath:
            path: /dockerdata-nfs/onapdemo/mso/mariadb/docker-entrypoint-initdb.d
      imagePullSecrets:
      - name: onap-docker-registry-key

The first part of the yaml file simply states that this is a deployment specification for a mariadb pod.

The spec section starts off with 'replicas: 1' which states that only 1 'replica' will be use here. If one was to change the number of replicas to 3 for example, kubernetes would attempt to ensure that three replicas of this pod are operational at all times. One can see that in a clustered environment the number of replicas should probably be more than 1 but for simple deployments 1 is sufficient.

The selector label is a grouping primitive of kubernetes but this simple example doesn't exercise it's full capabilities.

The template/spec section is where the key information required to start this pod is found.

image: is a reference to the location of the docker image in nexus3
name: is the name of the docker image
env is a section supports the creation of operating system environment variables within the container and are specified as a set of key/value pairs. For example, MYSQL_ROOT_PASSWORD is set to "password".
volumeMounts: allow for the creation of custom mount points
ports: define the networking ports that will be opened on the container. Note that further in the all-services.yaml file ports that are defined here can be exposed outside of ONAP component's name space by creating a 'nodePort' - a mechanism used to resolve port duplication.
readinessProbe: is the mechanism kubernetes uses to determine the state of the container.
volumes: a location to define volumes required by the container, in this case configuration and initialization information.
imagePullSecrets: an key to access the nexus3 repo when pulling docker containers.

As one might image, the mso-deployment.yaml file describes the deployment artifacts of the mso application. Here are the contents:

Code Block

title	mso-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mso
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mso
  template:
    metadata:
      labels:
        app: mso
      name: mso
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
              "args": [
                  "--container-name",
                  "mariadb"
              ],
              "command": [
                  "/root/ready.py"
              ],
              "env": [
                  {
                      "name": "NAMESPACE",
                      "valueFrom": {
                          "fieldRef": {
                              "apiVersion": "v1",
                              "fieldPath": "metadata.namespace"
                          }
                      }
                  }
              ],
              "image": "oomk8s/readiness-check:1.0.0",
              "imagePullPolicy": "Always",
              "name": "mso-readiness"
          }
          ]'
    spec:
      containers:
      - command:
        - /docker-files/scripts/start-jboss-server.sh
        image: nexus3.onap.org:10001/openecomp/mso:1.0-STAGING-latest
        name: mso
        volumeMounts:
        - mountPath: /shared
          name: mso
        - mountPath: /docker-files
          name: mso-docker-files
        env:
        - name: JBOSS_DEBUG
          value: "false"
        ports:
        - containerPort: 3904
        - containerPort: 3905
        - containerPort: 8080
        - containerPort: 9990
        - containerPort: 8787
        readinessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
      volumes:
        - name: mso
          hostPath:
            path: /dockerdata-nfs/onapdemo/mso/mso
        - name: mso-docker-files
          hostPath:
            path: /dockerdata-nfs/onapdemo/mso/docker-files
      imagePullSecrets:
      - name: onap-docker-registry-key

Much like the db deployment specification the first and last part of this yaml file describe meta-data, replicas, images, volumes, etc. The template section has an important new functionality though, a deployment specification for a new "initialization" container . The entire purpose of the init-container is to allow dependencies to be resolved in an orderly manner such that the entire ONAP system comes up every time. Once the dependencies are met and the init-containers job is complete, this container will terminate. Therefore, when OOM starts up ONAP one is able to see a number of init-containers start and then disappear as the system stabilizes. Note that more than one init-container may be specified, each completing before starting the next, if complex startup relationships need to be specified.

In this particular init-container, the command '/root/ready.py' will be executed to determine when mariadb is ready, but this could be a simple bash script. The image/name section describes where and how to get the docker image from the init-container.

The last of the three files is the all-services.yaml file which defines the kubernetes service(s) that will be exposed in this name space. Here is the contents of the file:

Code Block

title	all-services.yaml

apiVersion: v1
kind: Service
metadata:
  name: mariadb
  labels:
    app: mariadb
spec:
  ports:
    - port: 3306
      nodePort: 30252
  selector:
    app: mariadb
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  name: mso
  labels:
    app: mso
spec:
  selector:
    app: mso
  ports:
    - name: mso1
      port: 8080
      nodePort: 30223
    - name: mso2
      port: 3904
      nodePort: 30225
    - name: mso3
      port: 3905
      nodePort: 30224
    - name: mso4
      port: 9990
      nodePort: 30222
    - name: mso5
      port: 8787
      nodePort: 30250
  type: NodePort

First of all, note that this file is really two service specification in a single file: the mariadb service and the mso service. In some circumstances it may be possible to hide some of the complexity of the containers/pods by hiding them behind a single service.

The mariadb service specification is quite simple; other than the name the only section of interest is the nodePort specification. When containers require exposing ports to the world outside of a kubernetes namespace, there is a potential for port conflict. To resolve this potential port conflict kubernetes uses the concept of a nodePort that is mapped one-to-one with a port within the namespace. In this case the port 3306 (which was defined in the db-deployment.yaml file) is mapped to 30252 externally thus avoiding the conflict that would have arisen from deployment multiple mariadb containers.

The mso service definition is largely the same as the mariadb service with the exception that the ports are named.

Customizing Deployment Specifications

For each ONAP component deployed by OOM, a set of deployment specifications are required. Use fortunately there are many examples to use as references such that the previous 'mso' example, as well as: aai, appc, message-router, policy, portal, robot, sdc, sdnc and vid. If your components isn't already deployed by OOM, you can create your own set of deployment specifications that can be easily added to OOM.

Development Deployments

The deployment specifications currently (as of 21 Aug 2017 ) represent a simple simplex deployment of ONAP that may not have the robustness typically required of a full operational deployment. Follow on releases will enhance these deployment specifications as follows:

Load Balancers - kubernets has built in support for user defined or simple 'ingress' load balances at the service layer to hide the complexity of multi-pod deployments from other components.
Horizontal Scaling - replica sets can be used to dynamically scale the number of pods behind a service to that of the offered load.
Stateless Pods - using concepts such as DBaaS (database as a service) database technologies could be removed (where appropriate) from the services thus moving to the 'cattle' model so common in cloud deployments.

Kubernetes Under-Cloud Deployments

The automated ONAP deployment depends on a fully functional kubernetes environment being available prior to ONAP installation. Fortunately, kubenetes is supported on a wide variety of systems such as Google Compute Engine, AWS EC2, Microsoft Azure, CenturyLink Cloud, IBM Bluemix and more. If you're setting up your own kubernetes environment, please refer to ONAP on Kubernetes for a walk through of how to set this environment up on several platforms.

ONAP 'OneClick' Deployment Walk-though

Once a kubernetes environment is available and the deployment artifacts have been customized for your location, ONAP is ready to be installed.

The first step is to setup the /oom/kubernetes/config/onap-parameters.yaml file with key-value pairs specific to your OpenStack environment. There is a sample that may help you out or even be usable directly if you don't intend to actually use OpenStack resources. Here is the contents of this file:

Code Block

title	onap-parameters.yaml
collapse	true

OPENSTACK_UBUNTU_14_IMAGE: "Ubuntu_14.04.5_LTS"
OPENSTACK_PUBLIC_NET_ID: "e8f51956-00dd-4425-af36-045716781ffc"
OPENSTACK_OAM_NETWORK_ID: "d4769dfb-c9e4-4f72-b3d6-1d18f4ac4ee6"
OPENSTACK_OAM_SUBNET_ID: "191f7580-acf6-4c2b-8ec0-ba7d99b3bc4e"
OPENSTACK_OAM_NETWORK_CIDR: "192.168.30.0/24"
OPENSTACK_USERNAME: "vnf_user"
OPENSTACK_API_KEY: "vnf_password"
OPENSTACK_TENANT_NAME: "vnfs"
OPENSTACK_REGION: "RegionOne"
OPENSTACK_KEYSTONE_URL: "http://1.2.3.4:5000"
OPENSTACK_FLAVOUR_MEDIUM: "m1.medium"
OPENSTACK_SERVICE_TENANT_NAME: "services"
DMAAP_TOPIC: "AUTO"
DEMO_ARTIFACTS_VERSION: "1.1.0-SNAPSHOT"

Note that these values are required or the following steps will fail.

In-order to be able to support multiple ONAP instances within a single kubernetes environment a configuration set is required. The createConfig.sh script is used to do this.

Code Block

title	createConfig.sh

> ./createConfig.sh -n onapTrial

The bash script createAll.bash is used to create an ONAP deployment with kubernetes. It has two primary functions:

Creating the namespaces used to encapsulate the ONAP components, and
Creating the services, pods and containers within each of these namespaces that provide the core functionality of ONAP.

Code Block

title	createAll.bash

> ./createAll.bash -n onapTrial

Namespaces provide isolation between ONAP components as ONAP release 1.0 contains duplicate application (e.g. mariadb) and port usage. As such createAll.bash requires the user to enter a namespace prefix string that can be used to separate multiple deployments of onap. The result will be set of 10 namespaces (e.g. onapTrial-sdc, onapTrial-aai, onapTrial-mso, onapTrial-message-router, onapTrial-robot, onapTrial-vid, onapTrial-sdnc, onapTrial-portal, onapTrial-policy, onapTrial-appc) being created within the kubernetes environment. A prerequisite pod config-init (pod-config-init.yaml) may editing to match you environment and deployment into the default namespace before running createAll.bash.

Integration with MSB

The Microservices Bus Project provides facilities to integrate micro-services into ONAP and therefore needs to integrate into OOM - primarily through Consul which is the backend of MSB service discovery. The following is a brief description of how this integration will be done (thanks Huabing):

A registrator to push the service endpoint info to MSB service discovery.

The needed service endpoint info is put into the kubernetes yaml file as annotation, including service name, Protocol,version, visual range,LB method, IP, Port,etc.
OOM deploy/start/restart/scale in/scale out/upgrade ONAP components
Registrator watch the kubernetes event
When an ONAP component instance has been started/destroyed by OOM, Registrator get the notification from kubernetes
Registrator parse the service endpoint info from annotation and register/update/unregister it to MSB service discovery
MSB API Gateway uses the service endpoint info for service routing and load balancing.

Details of the registration service API can be found at Microservice Bus API Documentation.

How to define the service endpoints using annotation ONAP Services List#OOMIntegration

A preliminary view of the OOM-MSB integration is as follows:

A message sequence chart of the registration process:

MSB Usage Instructions

Code Block

language	bash
title	Pull and run MSB docker containers
collapse	true

Login the ONAP docker registry first: docker login -u docker -p docker nexus3.onap.org:10001


sudo docker run -d --net=host --name msb_consul consul agent -dev
sudo docker run -d --net=host --name msb_discovery nexus3.onap.org:10001/onap/msb/msb_discovery
sudo docker run -d --net=host -e "ROUTE_LABELS=visualRange:1" --name msb_internal_apigateway nexus3.onap.org:10001/onap/msb/msb_apigateway

Code Block

language	bash
title	Register a REST service to MSB via curl
collapse	true

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"serviceName": "aai", "version": "v8", "url": "/aai/v8/","protocol": "REST", "path": "/aai/v8", "nodes": [ {"ip": "10.74.215.65","port": "8443"}]}' \
  "http://127.0.0.1:10081/api/microservices/v1/services”

Code Block

language	bash
title	Test the REST Service via the internal API gateway
collapse	true

curl http://127.0.0.1/aai/v8/cloud-infrastructure/cloud-regions

FAQ (Frequently Asked Questions)

Does OOM enable the deployment of VNFs on containers?

No. OOM provides a mechanism to instantiate and manage the ONAP components themselves with containers but does not provide a Multi-VIM capability such that VNFs can be deployed into containers. The Multi VIM/Cloud Project may provide this functionality at some point.

DCAE has its own controller - how is this managed with OOM?

The DCAE controller will merge with OOM during the Amsterdam release as described in the Data Collection Analytics & Events Project. In the short term the DCAE controller is problematic in a container environment as it directly interfaces to OpenStack and request multiple VMs (e.g. CDAP, etc). The short term proposal is to containerize the DCAE components and statically create them as part of the larger ONAP deployment. Advanced DCAE controller features like hierarchical and geographically diverse deployments need further investigation.

Related Tools

Current Limitations and Feature Requests

DCAE - The DCAE component not only is not containerized but also includes its own VM orchestration system. A possible solution is to not use the DCAE Controller but port this controller's policies to Kubenetes directly, such as scaling CDAP nodes to match offered capacity.
Single Name Space
Deployment Parameter Optimization

Configuration Parameters

Currently ONAP configuration parameters are stored in multiple files; a solution to coordinate these configuration parameters is required. Kubernetes Config Maps may provide a solution or at least partial solution to this problem.

Centralized Parameter Control
Component Rationalization
Duplicate containers - The VM structure of ONAP hides internal container structure from each of the components including the existence of duplicate containers such as Maria DB.

Presentations and Demos

ONAP OOM K8S Deployment v1.1-final.pptx

Jira Stories

References

Docker container list - source of truth: https://git.onap.org/integration/tree/packaging/docker/docker-images.csv
Docker - http://docker.com
Kubernetes - http://kubernetes.io

Versions Compared

Old Version 38

New Version 39

Key

Introduction

Table of Contents

Container Background

Life Cycle Management via Kubernetes

ONAP Components to Kubernetes Object Relationships

Nodes

Pods

Services

Persistent Volumes

OOM Networking with Kubernetes

Name Spaces

Kubernetes Deployment Specifications for ONAP

MSO Example

Customizing Deployment Specifications

Development Deployments

Kubernetes Under-Cloud Deployments

ONAP 'OneClick' Deployment Walk-though

Integration with MSB

MSB Usage Instructions

FAQ (Frequently Asked Questions)

Related Tools

Current Limitations and Feature Requests

Configuration Parameters

Presentations and Demos

Jira Stories

References

Page Comparison

Versions Compared

Old Version 38

New Version 39

Key

Introduction

Table of Contents

Container Background

Life Cycle Management via Kubernetes

ONAP Components to Kubernetes Object Relationships

Nodes

Pods

Services

Persistent Volumes

OOM Networking with Kubernetes

Name Spaces

Kubernetes Deployment Specifications for ONAP

MSO Example

Customizing Deployment Specifications

Development Deployments

Kubernetes Under-Cloud Deployments

ONAP 'OneClick' Deployment Walk-though

Integration with MSB

MSB Usage Instructions

FAQ (Frequently Asked Questions)

Related Tools

Current Limitations and Feature Requests

Configuration Parameters

Presentations and Demos

Jira Stories

References