Status and Monitoring

Pre-requisite in each cluster where resources are being scheduled "Monitor" controller is running:

https://github.com/onap/multicloud-k8s/tree/master/src/monitor

To monitor the resources that are scheduled by orchestrator following steps are required required.

During instantiation of rsync instantiation call:

Add Label to each resource at the time of instantiation including pods.
The label is of the format emco/deployment-id: project-name+composite-app-name+deployment-intent-group+app-name
Create CR with the format for each app in each cluster. This should be added in the appcontext of the application:

Monitoring CR

apiVersion: k8splugin.io/v1alpha1
kind: ResourceBundleState
metadata:
  name: example-resourcebundlestate
  labels:
    "emco/deployment-id": "project-name+composite-app-name+deployment-intent-group+app-name"
spec:
  selector:
    matchLabels:
      "emco/deployment-id": "project-name+composite-app-name+deployment-intent-group+app-name"
status:
  ready: false

Read all clusters in the instantiation path for the composite app. Start watcher routine for the cluster for which watcher is not running. The watcher watches for changes to the ResourceBundleState CR's.
Continue with normal instantiation flow.

Watcher Flow:

Watcher is triggered every time there is a change in any resource of the app. On receiving a callback rsync reads the CR obtained from the callback. The ResourceBundleState CR will have the status of the resources populated.

Monitoring CR

apiVersion: k8splugin.io/v1alpha1
kind: ResourceBundleState
metadata:
  name: example-resourcebundlestate
  labels:
    "emco/deployment-id": "project-name+composite-app-name+version+deployment-intent-group+app-name"
spec:
  selector:
    matchLabels:
      "emco/deployment-id": "project-name+composite-app-name+version+deployment-intent-group+app-name"
status:
  configMapStatuses: []
  daemonSetStatuses: []
  deploymentStatuses: []
  ingressStatuses: []
  jobStatuses: []
  podStatuses:
  - metadata:
      annotations:
      labels:
        app: busybox1
        emco/deployment-id: 
      name: busybox1
      namespace: default
      resourceVersion: "1370717"
      selfLink: /api/v1/namespaces/default/pods/busybox1
      uid: 39a4ba77-8f0b-46d4-ab09-8bc4387252c4
   ...........
   ............
  secretStatuses: []
  serviceStatuses:
  - metadata:
      annotations:
      labels:
        app: busybox1
        emco/deployment-id:
      name: busybox1
  ..........
  ..........
  statefulSetStatuses: []

Get the appcontext id from MongoDB based on the key project, composite app, version, and deployment-intent-group
Store the above information in the appcontext(etcd) for the app based on the cluster and app name (also available in label)

Status API:

On getting the Status API call in the orchestrator

/v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/deployment-intent-groups/{deployment-intent-group-name}/status

read status from all the apps and clusters from the appcontext(etcd). Interpret the status of each resource in each app and cluster and come up with the status for each resource (Ready/Terminated/Fail/..) for all apps in all clusters. Also, decide the final status for the composite application (Ready/Terminated/Fail/...) and return all this data to the caller.

Also support Queries like:

URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/deployment-intent-groups/{deployment-intent-group-name}/status?output=<output-type>&app=<app-name>&cluster=<cluster-name>&resource=<resource-name>

Output-type can be small or full.
The query can be a combination of the app, cluster, and resources.