Page Comparison

Kubernetes clusters using ONAP Multicloud K8s Plugin Project for R4/R5 do not report hardware features to A&AI.
Consequently during the CNF/VNF life cycle CNFs/VNFs that require or recommend specific hardware during instantiation cannot dynamically reach the correct cluster and node that provides the needed hardware capabilities.

This Frankfurt Release epic (

Jira Legacy

server	System Jira
serverId	4733707d-2057-3a0f-ae5e-4fd8aff50176
key	MULTICLOUD-729

) adds the ability to discover, report and use hardware features during CNF/VNF life cycle to accelerate use cases using ONAP.

Overall Flow Diagram for HPA for K8s

...

Image Added

Discovery and Labeling of Node Features

The below figure shows a K8s cluster registering with the K8s plugin, wherein capabilities are transmitted

Image Added

Discovery of HPA features will be done using the node feature discovery [NFD] addon (https://github.com/kubernetes-sigs/node-feature-discovery) for kubernetes. NFD will be deployed as a daemonSet, with pods on each node and will discover and label hardware features on each kubernetes node with features labels (https://github.com/kubernetes-sigs/node-feature-discovery#feature-labels).

For SR-IOV Add Network Controllers Class "0200" to NFD-Worker Config File on deploy to label network cards on nodes

For HPA Features not labeled by NFD, investigate creation of Feature Detector Hooks (User Specific Features)

Registering of K8s Clusters

The GREEN FLOW is Register HPA features to AAI.

Image Added

Registration of K8s Cluster will happen at the ESR UI (see: https://onap.readthedocs.io/en/latest/guides/onap-user/cloud_site/openstack/index.html - use k8s for cloud-type). ESR will then send a POST request to

http://{{MSB_IP}}:{{MSB_PORT}}/api/multicloud-k8s/v1/{{cloud-owner}}/{{cloud-region-id}}/registry

...

The registrationHandler will then query the k8s cluster under registration using the Node App or Node Plugin code to get a list of nodes and the labels for each node.

registrationHandler - onap/multicloud-k8s/src/k8splugin/api/registrationhandler.go
Registers K8s cloud to K8s plugin using onap/multicloud-k8s/src/k8splugin/internal/connection/connection.go – check into if we need kubeconfig or ESR change
Requires update of onap/multicloud-k8s/src/k8splugin/api/api.go
Invokes AAI Client to K8s AAI Module to update AAI with Cloud/Tenant/Flavor information for the cluster

Node App or Node Plugin

Node App - onap/multicloud-k8s/src/k8splugin/internal/app/node or onap/multicloud-k8s/src/k8splugin/internal/plugin/node.go
Requests labels from nodes in K8s cluster
~~Requires update of onap/multicloud-k8s/src/k8splugin/internal/app/client.go extend to node Resource (helm.KubernetesResource does not have resource type)~~

Reporting of Cluster Info to AA&I

Reporting of HPA features to AA&I is done by the registrationHandler. Data will be munged to cloud-region/tenant/flavors schema for that cluster and reported to AA&I. registrationHandler is planning on using AA&I module under development for Frankfurt to report to AA&I. In this case we will send a request to the AA&I module instructing it to update the cloud-region, create a tenant and create flavors that match the node feature labels in said cloud-region.

Convert node feature label to flavor properties

below table is example how to convert node feature label to flavor properties

node feature label	Openstack flavor properties
feature.node.kubernetes.io/network-sriov.capable=True feature.node.kubernetes.io/network-sriov.configured=True feature.node.kubernetes.io/pci-1200_8086.present=True Comments: The other parameter we can set default because node feature label can not provide.	Image Added

We can get the latest mapping table in HPA Policies and Mappings + K8s

Update of CNF Artifacts to Include nodeSelector

...

Code Block

language	yml

# Hardware Feature Labels:
feature.node.kubernetes.io/cpu-cpuid.AESNI=true,
feature.node.kubernetes.io/cpu-cpuid.AVX2=true,
feature.node.kubernetes.io/cpu-cpuid.AVX=true,
feature.node.kubernetes.io/cpu-cpuid.FMA3=true,
feature.node.kubernetes.io/cpu-cpuid.IBPB=true,
feature.node.kubernetes.io/cpu-cpuid.STIBP=true,
feature.node.kubernetes.io/cpu-hardware_multithreading=true,
feature.node.kubernetes.io/cpu-pstate.turbo=true,
feature.node.kubernetes.io/cpu-rdt.RDTCMT=true,
feature.node.kubernetes.io/cpu-rdt.RDTMON=true,
feature.node.kubernetes.io/memory-numa=true,
feature.node.kubernetes.io/network-sriov.capable=true,
feature.node.kubernetes.io/pci-0300_102b.present=true, # GPU Present
feature.node.kubernetes.io/pci-0200_8086.present=true, # Network Card Present
feature.node.kubernetes.io/storage-nonrotationaldisk=true

# Software Feature Labels:
feature.node.kubernetes.io/kernel-config.NO_HZ=true,
feature.node.kubernetes.io/kernel-config.NO_HZ_FULL=true,
feature.node.kubernetes.io/kernel-version.full=3.10.0-957.el7.x86_64,
feature.node.kubernetes.io/kernel-version.major=3,
feature.node.kubernetes.io/kernel-version.minor=10,
feature.node.kubernetes.io/kernel-version.revision=0,
feature.node.kubernetes.io/system-os_release.ID=centos,
feature.node.kubernetes.io/system-os_release.VERSION_ID.major=7,
feature.node.kubernetes.io/system-os_release.VERSION_ID.minor=,
feature.node.kubernetes.io/system-os_release.VERSION_ID=7

Where needed add resource limits and request to ResourceBundle charts. I.E.:

  resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Charts to Update

https://github.com/onap/multicloud-k8s/tree/master/kud/demo/firewall

https://github.com/onap/multicloud-k8s/tree/master/kud/tests/vnfs/edgex/helm/edgex

For CPU Pinning etc - investigate CMK for kubernetes (see links at bottom of page) and specifying nodeSelector of "cmk.intel.com/cmk-node": "true"

Instantiation of K8s HPA

See Deploying vFw and EdgeXFoundry Services on Kubernets Cluster with ONAP for instantiation flow for K8s. See https://docs.onap.org/en/latest/submodules/integration.git/docs/docs_vfwHPA.html?highlight=hpa VFW deployment with HPA for HPA use case.

...

Ensure NFD (Node feature discovery - K8S CNCF project) deploy to Multicloud + K8s Plugin is working and labeling Kubernetes nodes
Jira Legacy
server System Jira
serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176
key MULTICLOUD-741
Add registrationHandler to K8s Plugin and expand K8s Plugin API to work with ESR VIM registration of K8s Clusters
Jira Legacy
server System Jira
serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176
key MULTICLOUD-740
- Use POST/DELETE/GET of http://{{MSB_IP}}:{{MSB_PORT}}/api/multicloud-k8s/v1/{{cloud-owner}}/{{cloud-region-id}}/registry
- Ensure registrationHandler can get K8s feature labels - (labels discovered and placed by NFD representing hardware features)
- Ensure registrationHandler can populate A&AI cloud-region/tenant/compute flavors based on cluster under registration
  Jira Legacy
  server System Jira
  serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176
  key MULTICLOUD-739
- Register K8s cluster internally with the K8s Plugin
Add Node App in K8s Plugin to retrieve node labels from a cluster
Identify and Address any gaps in policy and OOF to do match making of CNFs/VNFs using K8s Plugin
Update CNF packages (Resource Bundles, CSAR) to include nodeSelector and resource requests and limits
Update docs for HPA and K8s and Multicloud and ESR to reflect any change

Jira Legacy
server System Jira
serverId 4733707d-2057-3a0f-ae5e-4fd8aff50176
key MULTICLOUD-742