Pre-requisite in each cluster where resources are being scheduled "Monitor" controller is running:
https://github.com/onap/multicloud-k8s/tree/master/src/monitor
To monitor the resources that are scheduled by orchestrator following steps are required required.
During instantiation of rsync instantiation call:
- Add Label to each resource at the time of instantiation including pods.
- The label is of the format emco/deployment-id: project-name+composite-app-name+deployment-intent-group+app-name
- Create CR with the format for each app in each cluster. This should be added in the appcontext of the application:
apiVersion: k8splugin.io/v1alpha1 kind: ResourceBundleState metadata: name: example-resourcebundlestate labels: "emco/deployment-id": "project-name+composite-app-name+deployment-intent-group+app-name" spec: selector: matchLabels: "emco/deployment-id": "project-name+composite-app-name+deployment-intent-group+app-name" status: ready: false
- Read all clusters in the instantiation path for the composite app. Start watcher routine for the cluster for which watcher is not running. The watcher watches for changes to the ResourceBundleState CR's.
- Continue with normal instantiation flow.
Watcher Flow:
- Watcher is triggered every time there is a change in any resource of the app. No receiving a callback read the CR. The ResourceBundleState CR will have the status of the resources populated.
apiVersion: k8splugin.io/v1alpha1 kind: ResourceBundleState metadata: name: example-resourcebundlestate labels: "emco/deployment-id": "project-name+composite-app-name+version+deployment-intent-group+app-name" spec: selector: matchLabels: "emco/deployment-id": "project-name+composite-app-name+version+deployment-intent-group+app-name" status: configMapStatuses: [] daemonSetStatuses: [] deploymentStatuses: [] ingressStatuses: [] jobStatuses: [] podStatuses: - metadata: annotations: labels: app: busybox1 emco/deployment-id: name: busybox1 namespace: default resourceVersion: "1370717" selfLink: /api/v1/namespaces/default/pods/busybox1 uid: 39a4ba77-8f0b-46d4-ab09-8bc4387252c4 ........... ............ secretStatuses: [] serviceStatuses: - metadata: annotations: labels: app: busybox1 emco/deployment-id: name: busybox1 .......... .......... statefulSetStatuses: []
- Get the appcontext id from MongoDB based on the key project, composite app, version, and deployment-intent-group
- Store the above information in the appcontext(etcd) for the app based on the cluster and app name (also available in label)
Status API:
On getting the Status API call in the orchestrator
/v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/deployment-intent-groups/{deployment-intent-group-name}/status
read status from all the apps and clusters from the appcontext(etcd). Interpret the status of each resource in each app and cluster and come up with the status for each resource (Ready/Terminated/Fail/..) for all apps in all clusters. Also, decide the final status for the composite application (Ready/Terminated/Fail/...) and return all this data to the caller.
Also support Queries like:
URL: /v2/projects/{project-name}/composite-apps/{composite-app-name}/{version}/deployment-intent-groups/{deployment-intent-group-name}/status?output=<output-type>&app=<app-name>&cluster=<cluster-name>&resource=<resource-name>
Output-type can be small or full.
The query can be a combination of the app, cluster, and resources.