Heartbeat MicroService

Overview

Missing Heartbeat MS is one among the higher-level DCAE services responsible for tracking VES heartbeat events generated by the VNF's posted into DMAAP by VESCollector. Each VNF when configured to be monitored will be checked periodically by service if Heartbeat event was received. If the Heartbeat events from specific VNF is missed 'x' times (configured through policy), then Heartbeat Service will generate a control-loop event output. Once the VNF starts sending Heartbeat events, the service will automatically clear the original Onset created.

Following diagram explains the event flow involved.

Blueprint/image

Βlueprint (deployment artifact) : https://git.onap.org/dcaegen2/services/heartbeat/tree/dpo/k8s-heartbeat.yaml

Input file (deployment input) : https://git.onap.org/dcaegen2/services/heartbeat/tree/dpo/k8s-heartbeat.yaml

Docker image : nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.heartbeat:2.1.0

Deployment Prerequisite/dependencies

Heartbeat service has following dependencies

Dmaap : Usedfor sub/pub for VES heartbeat events; this should be deployed using OOM helm chart

ConfigBindingService : Used for fetching the latest configuration from Consul; this should be deployed using DCAE OOM charts.

Postgres : Used for Event data store, leader management (scale), heartbeat computation. Blueprint includes postgres plugin and postgres node, which will be used for setting up the PG pod.

Deployment Steps

Deployment of Heartbeat Service can be done using Dashboard UI or CloudifyUI or via CLI. Below steps are based on CLI.

Transfer blueprint component file in DCAE bootstrap POD under /blueprints directory
Transfer blueprint component inputs file in DCAE bootstrap POD under / directory
Log-in to the DCAE bootstrap POD's main container
Validate blueprint
Validate Blueprint
cfy blueprints validate /blueprints/k8s-heartbeat.yaml
Verify Plugin versions in target Cloudify instance match to blueprint imports
Verify Plugin version
cfy plugins list
If the version of plugin used are different, update the blueprint import to match.
Deploy Service
Upload and deploy blueprint
cfy install -b heartbeat -d heartbeat -i /k8s-heartbeat-inputs.yaml /blueprints/k8s-heartbeat.yaml

To un-deploy

Uninstall running component and delete deployment
Uninstall component
cfy uninstall heartbeat
Delete blueprint
Delete blueprint
cfy blueprints delete heartbeat

Initial Validation

After deployment, verify if Heartbeat POD and PG pod are running correctly

Verify Heartbeat is running

root@k8s-rancher:~# kubectl get pods -n onap | egrep "heartbeat|postgres"
dep-dcae-heartbeat-service-5ff7558fd-4nt6z                2/2       Running            2          4h
dep-hbpostgres-write-854dc899b6-nb22c                     1/1       Running            0          4h

And then check the logs to see if it can connect to DMaaP, polling for events.

Verify Logs for Dmaap poll

kubectl logs -f -n onap dep-dcae-heartbeat-service-5ff7558fd-4nt6z  dcae-heartbeat-service
.
.
2019-05-02 03:10:15,697 | urllib3.connectionpool | connectionpool | _make_request | 396 |  DEBUG | http://message-router.onap.svc.cluster.local:3904 "GET /events/unauthenticated.SEC_HEARTBEAT_OUTPUT/hbgrpID/1?timeout=15000 HTTP/1.1" 200 2
2019-05-02 03:10:15,697 | __main__ | htbtworker | process_msg | 93 |  INFO | ('HBT:', '[]')
2019-05-02 03:10:18,952 | __main__ | misshtbtd | main | 364 |  INFO | ('MSHBT: hb_common values ', 7, 'RUNNING', 'dcae-heartbeat-service-', 1556766594)
2019-05-02 03:10:18,952 | __main__ | misshtbtd | main | 368 |  INFO | ('MSHBD:pid,srcName,state,time,ctime,timeDiff is', 7, 'dcae-heartbeat-service-', 'RUNNING', 1556766594, 1556766619, 25)
2019-05-02 03:10:18,953 | __main__ | misshtbtd | main | 384 |  INFO | ('MSHBD:config status is', 'RUNNING')
2019-05-02 03:10:18,962 | __main__ | misshtbtd | create_update_hb_common | 143 |  INFO | MSHBT:Updated  hb_common DB with new values

Functional tests

Following default configuration is loaded into Heartbeat (set in blueprint configuration)

Configuration

{
 "vnfs": [{
  "eventName": "Heartbeat_vDNS",
  "heartbeatcountmissed": 3,
  "heartbeatinterval": 60,
  "closedLoopControlName": "ControlLoopEvent1",
  "policyVersion": "1.0.0.5",
  "policyName": "vFireWall",
  "policyScope": "resource=sampleResource,type=sampletype,CLName=sampleCLName",
  "target_type": "VNF",
  "target": "genVnfName",
  "version": "1.0"
 }, {
  "eventName": "Heartbeat_vFW",
  "heartbeatcountmissed": 3,
  "heartbeatinterval": 60,
  "closedLoopControlName": "ControlLoopEvent1",
  "policyVersion": "1.0.0.5",
  "policyName": "vFireWall",
  "policyScope": "resource=sampleResource,type=sampletype,CLName=sampleCLName",
  "target_type": "VNF",
  "target": "genVnfName",
  "version": "1.0"
 }, {
  "eventName": "Heartbeat_xx",
  "heartbeatcountmissed": 3,
  "heartbeatinterval": 60,
  "closedLoopControlName": "ControlLoopEvent1",
  "policyVersion": "1.0.0.5",
  "policyName": "vFireWall",
  "policyScope": "resource=sampleResource,type=sampletype,CLName=sampleCLName",
  "target_type": "VNF",
  "target": "genVnfName",
  "version": "1.0"
 }]
}

To simulate the event flow and trigger missing heartbeat event, we can simulate a VES event into Heartbeat subscription topic (using curl).

Generate Heartbeat CL Onset

Send a triggering event to DMaaP topic unauthenticated.SEC_HEARTBEAT_OUTPUT

Before sending, validate following

Correct DMaaP address
"eventName" field in VES input matches eventName configuration set in Heartbeat Service
"lastEpochMicrosec" in reflecting current time stamp approximately

Trigger a Heartbeat

curl -X POST http://10.12.5.8:30227/events/unauthenticated.SEC_HEARTBEAT_OUTPUT -H 'Content-Type: application/json' -d '{
        "event": {
                "commonEventHeader": {
                        "startEpochMicrosec": 1556753402000,
                        "sourceId": "79e90d76-513a-4f79-886d-470a0037c5cf",
                        "eventId": "Heartbeat_vDNS_10.0.0.1",
                        "nfcNamingCode": "DNS",
                        "reportingEntityId": "79e90d76-513a-4f79-886d-470a0037c5cf",
                        "eventType": "applicationVnf",
                        "priority": "Normal",
                        "version": 3,
                        "reportingEntityName": "testcmd001",
                        "sequence": 36312,
                        "domain": "heartbeat",
                        "lastEpochMicrosec": 1556753402000,
                        "eventName": "Heartbeat_vDNS",
                        "sourceName": "testnode001",
                        "nfNamingCode": "MDNS"
                }
        }
}'

Logs showing VES heartbeat event being processed

2019-05-02 03:35:08,638 | __main__ | htbtworker | process_msg | 78 |  INFO | HBT:Getting :http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_HEARTBEAT_OUTPUT/hbgrpID/1?timeout=15000
2019-05-02 03:35:08,649 | urllib3.connectionpool | connectionpool | _new_conn | 208 |  DEBUG | Starting new HTTP connection (1): message-router.onap.svc.cluster.local
2019-05-02 03:35:18,986 | urllib3.connectionpool | connectionpool | _make_request | 396 |  DEBUG | http://message-router.onap.svc.cluster.local:3904 "GET /events/unauthenticated.SEC_HEARTBEAT_OUTPUT/hbgrpID/1?timeout=15000 HTTP/1.1" 200 541
2019-05-02 03:35:18,987 | __main__ | htbtworker | process_msg | 93 |  INFO | ('HBT:', '["{\\"event\\":{\\"commonEventHeader\\":{\\"startEpochMicrosec\\":1556753402000,\\"sourceId\\":\\"79e90d76-513a-4f79-886d-470a0037c5cf\\",\\"eventId\\":\\"Heartbeat_vDNS_10.0.0.1\\",\\"nfcNamingCode\\":\\"DNS\\",\\"reportingEntityId\\":\\"79e90d76-513a-4f79-886d-470a0037c5cf\\",\\"eventType\\":\\"applicationVnf\\",\\"priority\\":\\"Normal\\",\\"version\\":3,\\"reportingEntityName\\":\\"testcmd001\\",\\"sequence\\":36312,\\"domain\\":\\"heartbeat\\",\\"lastEpochMicrosec\\":1556753402000,\\"eventName\\":\\"Heartbeat_vDNS\\",\\"sourceName\\":\\"testnode001\\",\\"nfNamingCode\\":\\"MDNS\\"}}}"]')
2019-05-02 03:35:18,987 | __main__ | htbtworker | process_msg | 128 |  INFO | ('HBT:Newly received HB event values ::', 'Heartbeat_vDNS', 1556753402000, 'testnode001')
2019-05-02 03:35:18,999 | __main__ | htbtworker | process_msg | 135 |  INFO | HBT:vnf_table_2 is already there
2019-05-02 03:35:18,999 | __main__ | htbtworker | process_msg | 139 |  INFO | ('HBT:', "Select source_name_count from vnf_table_1 where event_name='Heartbeat_vDNS'")
2019-05-02 03:35:19,002 | __main__ | htbtworker | process_msg | 156 |  INFO | ('HBT:event name, source_name & source_name_count are', 'Heartbeat_vDNS', 'testnode001', 2)
2019-05-02 03:35:19,002 | __main__ | htbtworker | process_msg | 160 |  INFO | ('HBT:eppc query is', "Select source_name from vnf_table_2 where event_name= 'Heartbeat_vDNS' and source_name_key=1")
2019-05-02 03:35:19,005 | __main__ | htbtworker | process_msg | 160 |  INFO | ('HBT:eppc query is', "Select source_name from vnf_table_2 where event_name= 'Heartbeat_vDNS' and source_name_key=2")
2019-05-02 03:35:19,006 | __main__ | htbtworker | process_msg | 176 |  INFO | ('HBT: The source_name_key and source_name_count are ', 1, 2)
2019-05-02 03:35:19,006 | __main__ | htbtworker | process_msg | 180 |  INFO | ('HBT: Insert entry in table_2 : ', [('testcmd001',)])
2019-05-02 03:35:24,045 | __main__ | misshtbtd | main | 364 |  INFO | ('MSHBT: hb_common values ', 7, 'RUNNING', 'dcae-heartbeat-service-', 1556768099)
2019-05-02 03:35:24,045 | __main__ | misshtbtd | main | 368 |  INFO | ('MSHBD:pid,srcName,state,time,ctime,timeDiff is', 7, 'dcae-heartbeat-service-', 'RUNNING', 1556768099, 1556768124, 25)
2019-05-02 03:35:24,045 | __main__ | misshtbtd | main | 384 |  INFO | ('MSHBD:config status is', 'RUNNING')
2019-05-02 03:35:24,059 | __main__ | misshtbtd | create_update_hb_common | 143 |  INFO | MSHBT:Updated  hb_common DB with new values
2019-05-02 03:35:24,411 | __main__ | db_monitoring | db_monitoring | 158 |  INFO | DBM: Active DB Monitoring Instance
2019-05-02 03:35:24,421 | __main__ | db_monitoring | sendControlLoopEvent | 39 |  INFO | ('DBM:Time to raise Control Loop Event for Control loop typ /target type - ', 'ONSET', 'VNF')
2019-05-02 03:35:24,421 | __main__ | db_monitoring | sendControlLoopEvent | 41 |  INFO | DBM:Heartbeat not received, raising alarm event
2019-05-02 03:35:24,421 | __main__ | db_monitoring | sendControlLoopEvent | 117 |  INFO | ('DBM: CL Json object is', '{"closedLoopEventClient": "DCAE_Heartbeat_MS", "policyVersion": "1.0.0.5", "policyName": "vFireWall", "policyScope": "resource=sampleResource,type=sampletype,CLName=sampleCLName", "target_type": "VNF", "AAI": {"generic-vnf.vnf-name": "testnode001"}, "closedLoopAlarmStart": 1556768124419, "closedLoopEventStatus": "ONSET", "closedLoopControlName": "ControlLoopEvent1", "version": "1.0", "target": "genVnfName", "requestID": "8c1b8bd8-06f7-493f-8ed7-daaa4cc481bc", "from": "DCAE"}')
2019-05-02 03:35:24,421 | __main__ | db_monitoring | sendControlLoopEvent | 121 |  INFO | ('DBM:', 'http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.DCAE_CL_OUTPUT')
2019-05-02 03:35:24,433 | urllib3.connectionpool | connectionpool | _new_conn | 208 |  DEBUG | Starting new HTTP connection (1): message-router.onap.svc.cluster.local
2019-05-02 03:35:24,539 | urllib3.connectionpool | connectionpool | _make_request | 396 |  DEBUG | http://message-router.onap.svc.cluster.local:3904 "POST /events/unauthenticated.DCAE_CL_OUTPUT HTTP/1.1" 200 41
2019-05-02 03:35:24,540 | __main__ | db_monitoring | sendControlLoopEvent | 126 |  INFO | ('DBM:', 200, 'OK')

We can check that a new DCAE_CL_OUTPUT event has been published (make sure you target the correct DMaaP IP address)

Fetch events from DCAE_CL_OUTPUT topic

curl http://10.12.5.8:30227/events/unauthenticated.DCAE_CL_OUTPUT/vv/1

Published event


["{\"closedLoopEventClient\": \"DCAE_Heartbeat_MS\", \"policyVersion\": \"1.0.0.5\", \"policyName\": \"vFireWall\", \"policyScope\": \"resource=sampleResource,type=sampletype,CLName=sampleCLName\", \"target_type\": \"VNF\", \"AAI\": {\"generic-vnf.vnf-name\": \"testnode001\"}, \"closedLoopAlarmStart\": 1556768124419, \"closedLoopEventStatus\": \"ONSET\", \"closedLoopControlName\": \"ControlLoopEvent1\", \"version\": \"1.0\", \"target\": \"genVnfName\", \"requestID\": \"8c1b8bd8-06f7-493f-8ed7-daaa4cc481bc\", \"from\": \"DCAE\"}"]

Generate Heartbeat CL Abatement Event

Send a triggering event to DMaaP topic unauthenticated.SEC_HEARTBEAT_OUTPUT

Before sending, validate following

Correct DMaaP address
"eventName" field in VES input matches eventName configuration set in Heartbeat Service
"lastEpochMicrosec" in reflecting current time stamp approximately
Ensure the event is on the same "sourceName" as original event sent

Resend heartbeat with new timestamp

curl -X POST http://10.12.5.8:30227/events/unauthenticated.SEC_HEARTBEAT_OUTPUT -H 'Content-Type: application/json' -d '{
        "event": {
                "commonEventHeader": {
                        "startEpochMicrosec": 1556768319000,
                        "sourceId": "79e90d76-513a-4f79-886d-470a0037c5cf",
                        "eventId": "Heartbeat_vDNS_10.0.0.1",
                        "nfcNamingCode": "DNS",
                        "reportingEntityId": "79e90d76-513a-4f79-886d-470a0037c5cf",
                        "eventType": "applicationVnf",
                        "priority": "Normal",
                        "version": 3,
                        "reportingEntityName": "testcmd001",
                        "sequence": 36312,
                        "domain": "heartbeat",
                        "lastEpochMicrosec": 1556768319000,
                        "eventName": "Heartbeat_vDNS",
                        "sourceName": "testnode001",
                        "nfNamingCode": "MDNS"
                }
        }
}'

Logs showing CL abatement event generation

2019-05-02 03:39:28,249 | __main__ | htbtworker | process_msg | 78 |  INFO | HBT:Getting :http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_HEARTBEAT_OUTPUT/hbgrpID/1?timeout=15000
2019-05-02 03:39:28,257 | urllib3.connectionpool | connectionpool | _new_conn | 208 |  DEBUG | Starting new HTTP connection (1): message-router.onap.svc.cluster.local
2019-05-02 03:39:28,929 | urllib3.connectionpool | connectionpool | _make_request | 396 |  DEBUG | http://message-router.onap.svc.cluster.local:3904 "GET /events/unauthenticated.SEC_HEARTBEAT_OUTPUT/hbgrpID/1?timeout=15000 HTTP/1.1" 200 541
2019-05-02 03:39:28,930 | __main__ | htbtworker | process_msg | 93 |  INFO | ('HBT:', '["{\\"event\\":{\\"commonEventHeader\\":{\\"startEpochMicrosec\\":1556768319000,\\"sourceId\\":\\"79e90d76-513a-4f79-886d-470a0037c5cf\\",\\"eventId\\":\\"Heartbeat_vDNS_10.0.0.1\\",\\"nfcNamingCode\\":\\"DNS\\",\\"reportingEntityId\\":\\"79e90d76-513a-4f79-886d-470a0037c5cf\\",\\"eventType\\":\\"applicationVnf\\",\\"priority\\":\\"Normal\\",\\"version\\":3,\\"reportingEntityName\\":\\"testcmd001\\",\\"sequence\\":36312,\\"domain\\":\\"heartbeat\\",\\"lastEpochMicrosec\\":1556768319000,\\"eventName\\":\\"Heartbeat_vDNS\\",\\"sourceName\\":\\"testnode001\\",\\"nfNamingCode\\":\\"MDNS\\"}}}"]')
2019-05-02 03:39:28,930 | __main__ | htbtworker | process_msg | 128 |  INFO | ('HBT:Newly received HB event values ::', 'Heartbeat_vDNS', 1556768319000, 'testnode001')
2019-05-02 03:39:28,936 | __main__ | htbtworker | process_msg | 135 |  INFO | HBT:vnf_table_2 is already there
2019-05-02 03:39:28,937 | __main__ | htbtworker | process_msg | 139 |  INFO | ('HBT:', "Select source_name_count from vnf_table_1 where event_name='Heartbeat_vDNS'")
2019-05-02 03:39:28,940 | __main__ | htbtworker | process_msg | 156 |  INFO | ('HBT:event name, source_name & source_name_count are', 'Heartbeat_vDNS', 'testnode001', 3)
2019-05-02 03:39:28,940 | __main__ | htbtworker | process_msg | 160 |  INFO | ('HBT:eppc query is', "Select source_name from vnf_table_2 where event_name= 'Heartbeat_vDNS' and source_name_key=1")
2019-05-02 03:39:28,944 | __main__ | htbtworker | process_msg | 160 |  INFO | ('HBT:eppc query is', "Select source_name from vnf_table_2 where event_name= 'Heartbeat_vDNS' and source_name_key=2")
2019-05-02 03:39:28,945 | __main__ | htbtworker | process_msg | 160 |  INFO | ('HBT:eppc query is', "Select source_name from vnf_table_2 where event_name= 'Heartbeat_vDNS' and source_name_key=3")
2019-05-02 03:39:28,946 | __main__ | htbtworker | process_msg | 168 |  INFO | ('HBT: Update vnf_table_2 : ', 2, [('testnode001',)])
2019-05-02 03:39:28,946 | __main__ | htbtworker | process_msg | 176 |  INFO | ('HBT: The source_name_key and source_name_count are ', 3, 3)
2019-05-02 03:39:34,820 | __main__ | misshtbtd | main | 364 |  INFO | ('MSHBT: hb_common values ', 7, 'RUNNING', 'dcae-heartbeat-service-', 1556768350)
2019-05-02 03:39:34,820 | __main__ | misshtbtd | main | 368 |  INFO | ('MSHBD:pid,srcName,state,time,ctime,timeDiff is', 7, 'dcae-heartbeat-service-', 'RUNNING', 1556768350, 1556768375, 25)
2019-05-02 03:39:34,820 | __main__ | misshtbtd | main | 384 |  INFO | ('MSHBD:config status is', 'RUNNING')
2019-05-02 03:39:34,841 | __main__ | misshtbtd | create_update_hb_common | 143 |  INFO | MSHBT:Updated  hb_common DB with new values
2019-05-02 03:39:46,086 | __main__ | db_monitoring | db_monitoring | 158 |  INFO | DBM: Active DB Monitoring Instance
2019-05-02 03:39:46,208 | __main__ | db_monitoring | sendControlLoopEvent | 39 |  INFO | ('DBM:Time to raise Control Loop Event for Control loop typ /target type - ', 'ABATED', 'VNF')
2019-05-02 03:39:46,209 | __main__ | db_monitoring | sendControlLoopEvent | 77 |  INFO | DBM:Heartbeat received, clearing alarm event
2019-05-02 03:39:46,209 | __main__ | db_monitoring | sendControlLoopEvent | 117 |  INFO | ('DBM: CL Json object is', '{"closedLoopEventClient": "DCAE_Heartbeat_MS", "policyVersion": "1.0.0.5", "policyName": "vFireWall", "policyScope": "resource=sampleResource,type=sampletype,CLName=sampleCLName", "target_type": "VNF", "AAI": {"generic-vnf.vnf-name": "testnode001"}, "closedLoopAlarmStart": 1556768386209, "closedLoopEventStatus": "ABATED", "closedLoopControlName": "ControlLoopEvent1", "version": "1.0", "target": "genVnfName", "requestID": "8c1b8bd8-06f7-493f-8ed7-daaa4cc481bc", "from": "DCAE"}')
2019-05-02 03:39:46,209 | __main__ | db_monitoring | sendControlLoopEvent | 121 |  INFO | ('DBM:', 'http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.DCAE_CL_OUTPUT')
2019-05-02 03:39:46,217 | urllib3.connectionpool | connectionpool | _new_conn | 208 |  DEBUG | Starting new HTTP connection (1): message-router.onap.svc.cluster.local
2019-05-02 03:39:46,260 | urllib3.connectionpool | connectionpool | _make_request | 396 |  DEBUG | http://message-router.onap.svc.cluster.local:3904 "POST /events/unauthenticated.DCAE_CL_OUTPUT HTTP/1.1" 200 41
2019-05-02 03:39:46,262 | __main__ | db_monitoring | sendControlLoopEvent | 126 |  INFO | ('DBM:', 200, 'OK')
2019-05-02 03:39:46,263 | __main__ | db_monitoring | sendControlLoopEvent | 129 |  INFO | ('DBM:Status code for sending the control loop event is', 200)
2019-05-02 03:39:49,086 | __main__ | htbtworker | process_msg | 72 |  INFO | ('\n\nHBT:eventnameList values ', ['Heartbeat_vDNS', 'Heartbeat_vFW', 'Heartbeat_xx'])
2019-05-02 03:39:49,086 | __main__ | htbtworker | process_msg | 78 |  INFO | HBT:Getting :http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_HEARTBEAT_OUTPUT/hbgrpID/1?timeout=15000
2019-05-02 03:39:49,108 | urllib3.connectionpool | connectionpool | _new_conn | 208 |  DEBUG | Starting new HTTP connection (1): message-router.onap.svc.cluster.local

We can check that a new DCAE_CL_OUTPUT event has been published (make sure you target the correct DMaaP IP address)

Fetch events from DCAE_CL_OUTPUT topic

curl http://10.12.5.8:30227/events/unauthenticated.DCAE_CL_OUTPUT/vv/1

CL Abatement Published event

["{\"closedLoopEventClient\": \"DCAE_Heartbeat_MS\", \"policyVersion\": \"1.0.0.5\", \"policyName\": \"vFireWall\", \"policyScope\": \"resource=sampleResource,type=sampletype,CLName=sampleCLName\", \"target_type\": \"VNF\", \"AAI\": {\"generic-vnf.vnf-name\": \"testnode001\"}, \"closedLoopAlarmStart\": 1556768386209, \"closedLoopEventStatus\": \"ABATED\", \"closedLoopControlName\": \"ControlLoopEvent1\", \"version\": \"1.0\", \"target\": \"genVnfName\", \"requestID\": \"8c1b8bd8-06f7-493f-8ed7-daaa4cc481bc\", \"from\": \"DCAE\"}"]

Dynamic Configuration Update

As the Heartbeat service periodically polls Consul KV using configbindingService api's - the run time configuration of Heartbeat service can be updated dynamically without having to redeploy/restart the service. The updates to configuration can be triggered either from Policy (or CLAMP) or made directly in Consul.

Locate the servicename by executing into Heartbeat Service pod and getting env HOSTNAME value

ServiceName of Heartbeat Service


root@k8s-rancher:~# kubectl exec -it -n onap dep-s78f36f2daf0843518f2e25184769eb8b-dcae-heartbeat-servithzx2 /bin/bash
Defaulting container name to s78f36f2daf0843518f2e25184769eb8b-dcae-heartbeat-service.
Use 'kubectl describe pod/dep-s78f36f2daf0843518f2e25184769eb8b-dcae-heartbeat-servithzx2 -n onap' to see all of the containers in this pod.

misshtbt@s78f36f2daf0843518f2e25184769eb8b-dcae-heartbeat-service:~/bin$ env | grep HOSTNAME
HOSTNAME=s78f36f2daf0843518f2e25184769eb8b-dcae-heartbeat-service

Change the configuration for Service in KV-store through UI

Consul URL

http://<k8snodeip>:30270/ui/#/dc1/kv/