Heartbeat MicroService

Owned by Vijay Kumar
Last updated: May 01, 2019
12 min read

Overview

Missing Heartbeat MS is one among the higher-level DCAE services responsible for tracking VES heartbeat events generated by the VNF's posted into DMAAP by VESCollector. Each VNF when configured to be monitored will be checked periodically by service if Heartbeat event was received. If the Heartbeat events from specific VNF is missed 'x' times (configured through policy), then Heartbeat Service will generate a control-loop event output.  Once the VNF starts sending Heartbeat events, the service will automatically clear the original Onset created.



Following diagram explains the event flow involved.





Blueprint/image



Βlueprint (deployment artifact) : https://git.onap.org/dcaegen2/services/heartbeat/tree/dpo/k8s-heartbeat.yaml

Input file (deployment input)    :  https://git.onap.org/dcaegen2/services/heartbeat/tree/dpo/k8s-heartbeat.yaml

Docker image                            : nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.heartbeat:2.1.0



Deployment Prerequisite/dependencies

Heartbeat service has following dependencies

Dmaap  : Usedfor sub/pub for VES heartbeat events; this should be deployed using OOM helm chart

ConfigBindingService :  Used for fetching the latest configuration from Consul; this should be deployed using DCAE OOM charts.

Postgres : Used for Event data store, leader management (scale), heartbeat computation. Blueprint includes postgres plugin and postgres node, which will be used for setting up the PG pod.



Deployment Steps



Deployment of Heartbeat Service can be done using Dashboard UI or CloudifyUI or via CLI. Below steps are based on CLI.



  • Transfer blueprint component file in DCAE bootstrap POD under /blueprints directory

  • Transfer blueprint component inputs file in DCAE bootstrap POD under / directory

  • Log-in to the DCAE bootstrap POD's main container

  • Validate blueprint

    Validate Blueprint

    cfy blueprints validate /blueprints/k8s-heartbeat.yaml



  • Verify Plugin versions in target Cloudify instance match to blueprint imports

    Verify Plugin version

    cfy plugins list

    If the version of plugin used are different, update the blueprint import to match.

  • Deploy Service

    Upload and deploy blueprint

    cfy install -b heartbeat -d heartbeat -i /k8s-heartbeat-inputs.yaml /blueprints/k8s-heartbeat.yaml



To un-deploy

  • Uninstall running component and delete deployment

    Uninstall component



  • Delete blueprint

    Delete blueprint



Initial Validation

After deployment, verify if Heartbeat POD and PG pod are running correctly

Verify Heartbeat is running



And then check the logs to see if it can connect to DMaaP, polling for events.

Verify Logs for Dmaap poll



Functional tests

Following default configuration is loaded into Heartbeat (set in blueprint configuration)



Configuration



To simulate the event flow and trigger missing heartbeat event, we can simulate a VES event into Heartbeat subscription topic (using curl).

Generate Heartbeat CL Onset

Send a triggering event to DMaaP topic unauthenticated.SEC_HEARTBEAT_OUTPUT

Before sending, validate following

  • Correct DMaaP address

  • "eventName" field in VES input matches eventName configuration set in Heartbeat Service

  • "lastEpochMicrosec" in reflecting current time stamp approximately



Trigger a Heartbeat



Logs showing VES heartbeat event being processed



We can check that a new DCAE_CL_OUTPUT event has been published (make sure you target the correct DMaaP IP address)

Fetch events from DCAE_CL_OUTPUT topic



Published event



Generate Heartbeat CL Abatement Event



Send a triggering event to DMaaP topic unauthenticated.SEC_HEARTBEAT_OUTPUT

Before sending, validate following

  • Correct DMaaP address

  • "eventName" field in VES input matches eventName configuration set in Heartbeat Service

  • "lastEpochMicrosec" in reflecting current time stamp approximately

  •  Ensure the event is on the same "sourceName" as original event sent



Resend heartbeat with new timestamp



Logs showing CL abatement event generation



We can check that a new DCAE_CL_OUTPUT event has been published (make sure you target the correct DMaaP IP address)

Fetch events from DCAE_CL_OUTPUT topic



CL Abatement Published event





Dynamic Configuration Update

As the Heartbeat service periodically polls Consul KV using configbindingService api's - the run time configuration of Heartbeat service can be updated dynamically without having to redeploy/restart the service. The updates to configuration can be triggered either from Policy (or CLAMP) or made directly in Consul.



Locate the servicename by executing into Heartbeat Service pod and getting env HOSTNAME value

ServiceName of Heartbeat Service



Change the configuration for Service in KV-store through UI



Consul URL