Installation of Holmes (for Testing Purpose)
- Guangrong Fu
- Yang Xu
- Wiktor Garbarek
Prerequisites
MSB must be installed and started and the user knows the IP address of the MSB API gateway service. Regarding how to install MSB, please go to MSB Test Environment Setup.
PostgreSQL(v9.5) must be installed and started. For the guidance on how to run a PostgreSQL, please refer to Offical Repository of PostgreSQL.
While setting up PostgreSQL, a database and a user named ‘holmes’ must be created. The corresponding password should be set to ‘holmespwd’. Otherwise, Holmes could not be started up successfully.
Installation Steps
As a DCAE Analytics Application
When integrated with DCAE, there's not too much to do from Holmes' perspective. DCAE spins up Holmes, deploys the default rules and provides the configuration infomation to Holmes so that Holmes could complete self-configuration using those information.
Anyway, if you want to test whether Holmes is woking as expected. Please follow the steps below.
Before we start, please make sure that DCAE is deployed correctly and Holmes has been started.
Check whether the containers of Holmes are running.
curl http://${MSB_IP}/api/holmes-rule-mgmt/v1/healthcheck curl http://${MSB_IP}/api/holmes-engine-mgmt/v1/healthcheckIf the returned value is true and the status code is 200, it means that Holmes is ready. Also, this could be done by checking the status shown on the DCAE Consul page.
Check whether the rule of Holmes has been deployed.
curl http://${MSB_IP}/api/holmes-rule-mgmt/v1/ruleIf the rule is deployed, the API should return an array with a rule in it. Otherwise, an empty array is returned. If the rule is not deployed successfully, you have to check whether there are any exceptions inside the holmes-rule-management docker container.
Alternatively, you could deploy the rule manually by calling "curl -X PUT -H 'Content-Type: application/json' -d @holmes-rule.txt http://${MSB_IP}/api/holmes-rule-mgmt/v1/rule". Please notice that this is not a normal way in which the rule is supposed to be deployed when Holmes is integrated with DCAE. But for testing purpose, it's OK for us to do so.- Mock some data into A&AI to support alarm correlation analysis.
PUT the model info into A&AI. The API to be used is: https://${AAI_IP}:${AAI_PORT}/aai/v11/service-design-and-creation/models/model/example-model-invariant-id-val-16059
PUT the VNF info into A&AI. The API to be used is: https://${AAI_IP}:${AAI_PORT}/aai/v11/network/generic-vnfs/generic-vnf/example-vnf-id-val-92494
PUT the VM into into A&AI. The API to be used is: https://${AAI_IP}:${AAI_PORT}/aai/v11/cloud-infrastructure/cloud-regions/cloud-region/example-cloud-owner-val-45051/example-cloud-region-id-val-56689
PUT the link info into A&AI. The API to be used is: https://${AAI_IP}:${AAI_PORT}/aai/v11/network/generic-vnfs/generic-vnf/example-vnf-id-val-92494/relationship-list/relationship
Send alarms to the unauthenticated.SEC_FAULT_OUTPUT topic of DMaaP.
Note that the value of the sourceId and the sourceName field in the parent alarm should be identical with that in the VM info in A&AI. Similarly, the values of the specific fields of the child alarm should be the same with the corresponding info in VNF.
After the root cause (parent alarm) is sent, there should be a corresponding control loop event on the unauthenticated.DCAE_CL_OUTPUT topic of DMaaP. Also, you could find the corresponding message with a leading "policyMsg" in the log file in the holmes-engine-management docker container. If both the parent alarm and the child alarm exist in Holmes, there should be some log items saying "Correlation Identified" as well.
As an Independent Application
Installation
Log in to the ONAP docker repo:
sudo docker login -u docker -p docker nexus3.onap.org:10001
Start Holmes using the commands below:
# DB_IP - the IP address of Postgres. Do not specify the port. The default port(5432) is adopted. # MSB_IP - the IP address of the MSB API gateway service. # HOST_IP - the IP address of the host (not the docker daemon) on which the docker container is running. # If the 'TESTING' variable is set to '1', you're telling Holmes not to query the IP address of the Holmes containers from the DCAE config binding service but to get it directly from the environment variable 'HOST_IP'. sudo docker run --name holmes-rule-management -p 9101:9101 -d -e URL_JDBC=$DB_IP -e MSB_ADDR=$MSB_IP -e TESTING=1 -e HOST_IP=$HOST_IP nexus3.onap.org:10001/onap/holmes/rule-management sudo docker run --name holmes-engine-management -p 9102:9102 -d -e URL_JDBC=$DB_IP -e MSB_ADDR=$MSB_IP -e TESTING=1 -e HOST_IP=$HOST_IP nexus3.onap.org:10001/onap/holmes/engine-management
Call the health check API to check whether Holmes has been started up successfully.
# the rule management component curl http://$MSB_IP/api/holmes-rule-mgmt/v1/healthcheck # the engine management component curl http://$MSB_IP/api/holmes-engine-mgmt/v1/healthcheck
If the component returns "true", it is spun up successfully. Otherwise you have to use the 'sudo docker ps' and 'sudo docker logs <container_name>' to check what happened during the service instantiation.
Configurations
When Holmes is run in the standalone mode, it has to be configured manually in order to perform the sub/pub actions on DMaaP.
Subscribing
curl -X PUT -H 'Content-Type: application/json' \ -d "{\"name\":\"unauthenticated.SEC_FAULT_OUTPUT\", \"url\":\"http://$DMAAP_IP:$DMAAP_PORT/events/unauthenticated.SEC_FAULT_OUTPUT\"}" \ http://$MSB_IP/api/holmes-engine-mgmt/v1/dmaap/subAfter setting the subscribing url, a timer task will be started using the default query interval (15s).
Publishing
curl -X PUT -H 'Content-Type: application/json' \ -d "{\"name\":\"unauthenticated.DCAE_CL_OUTPUT\", \"url\":\"http://$DMAAP_IP:$DMAAP_PORT/events/unauthenticated.DCAE_CL_OUTPUT\"}" \ http://$MSB_IP/api/holmes-engine-mgmt/v1/dmaap/pub
Then you have to deploy the rule into Holmes by calling:
curl -X PUT -H 'Content-Type: application/json' -d @holmesrule http://$MSB_IP/api/holmes-rule-mgmt/v1/rule
The rule is saved into a file named "holmesrule" and it is attached here.
If there are modifications performed by yourself, please update the contents of the holmesrule file first.
If you want to update an existing rule, you have to add a field named "ruleid" into the holmesrule file using the real id returned by the API called above and then call:
curl -X POST -H 'Content-Type: application/json' -d @holmesrule http://$MSB_IP/api/holmes-rule-mgmt/v1/rule
For more information, please refer to Rule Management - Beijing.
Theoretically, after the configurations are finished, when you push the fault data on to the topic named "unauthenticated.SEC_FAULT_OUTPUT" of DMaaP, the data will be fetched and processed by Holmes and then push back to the topic named "unauthenticated.DCAE_CL_OUTPUT" of DMaaP. To support the correlation analysis, A&AI has to be ready with certain resource data in it and registered to MSB.
Sample Data
DMaaP Message for Alarms from the EMS Driver
["{\"VESversion\":\"v5\",\"event\":{\"commonEventHeader\":{\"sourceId\":\"VNF_194.15.13.138\",\"startEpochMicrosec\":1501518702,\"eventId\":\"1501489595451\",\"nfcNamingCode\":\"\",\"reportingEntityId\":\"0000ZTHX1\",\"internalHeaderFields\":{\"collectorTimeStamp\":\"Fri, 10 20 2017 01:49:27 GMT\"},\"eventType\":\"applicationVnf\",\"priority\":\"High\",\"version\":3,\"reportingEntityName\":\"0000ZTHX1\",\"sequence\":960,\"domain\":\"fault\",\"lastEpochMicrosec\":1501518702,\"eventName\":\"Slave MPU is offline\",\"sourceName\":\"VNF_194.15.13.138\",\"nfNamingCode\":\"\"},\"faultFields\":{\"eventSeverity\":\"CRITICAL\",\"alarmCondition\":\"The slave MPU board is offline or abnormal\",\"faultFieldsVersion\":2,\"eventCategory\":\"equipmentAlarm\",\"specificProblem\":\"The slave MPU board is offline or abnormal\",\"alarmInterfaceA\":\"VNF_194.15.13.138\",\"alarmAdditionalInformation\":[{\"name\":\"specificProblemID\",\"value\":\"315\"},{\"name\":\"objectUID\",\"value\":\"0000ZTHX1PGWGJI6V1\"},{\"name\":\"locationInfo\",\"value\":\"MPU_22_20_0\"},{\"name\":\"addInfo\",\"value\":\"Aid:17;AlarmCode:110010;AlarmReasonCode:110010;Remark:\\\"DeployUnit=22,Node=21,SubNode=0\\\";\"}],\"eventSourceType\":\"PgwFunction\",\"vfStatus\":\"Active\"}}}"]
Above is the literals returned by DMaaP. Hence some symbols are escaped. If you want to push the data onto DMaaP, please use the data contained in VES Component.
DMaaP Message for Control Loop Events from Holmes
{
"target": "vserver.vserver-id",
"target_type": "VM",
"AAI": {
"vserver.in-maint": true,
"vserver.is-closed-loop-disabled": true,
"vserver.prov-status": "example-prov-status-val-2",
"vserver.resource-version": "1504912891060",
"vserver.vserver-id": "example-vserver-id-val-2",
"vserver.vserver-name": "example-vserver-name-val-2",
"vserver.vserver-name2": "example-vserver-name2-val-2",
"vserver.vserver-selflink": "example-vserver-selflink-val-2",
"generic-vnf.vnf-id": "5e49ca06-2972-4532-9ed4-6d071588d792",
"service-instance.service-instance-id": "USUCP0PCOIL0110UJZZ01"
},
"closedLoopAlarmStart": 1484855291527925,
"closedLoopEventStatus": "ONSET",
"closedLoopControlName": "VoLTE",
"version": "1.0.2",
"requestID": "8c1b8bd8-06f7-493f-8ed7-daaa4cc481bc",
"from": "DCAE"
}
{
"target": "vserver.vserver-id",
"target_type": "VM",
"AAI": {
"vserver.vserver-id": "example-vserver-id-val-2",
"vserver.vserver-name": "example-vserver-name-val-2",
"vserver.vserver-name2": "example-vserver-name2-val-2",
"vserver.vserver-selflink": "example-vserver-selflink-val-2"
},
"closedLoopAlarmStart": 1484855291527925,
"closedLoopAlarmEnd": 1484855278392541,
"closedLoopEventStatus": "ABATED",
"closedLoopControlName": "VoLTE",
"version": "1.0.2",
"requestID": "8c1b8bd8-06f7-493f-8ed7-daaa4cc481bc",
"from": "DCAE"
}
A&AI Info That Needs to Be Registered to MSB
# Please replace $AAI_IP, $AAI_PORT and $MSB_IP with the real ones.
#aai-cloudInfrastructure
curl -X POST \
-H "Content-Type: application/json" \
-d '{"serviceName": "aai-cloudInfrastructure", "version": "v11", "url": "/aai/v11/cloud-infrastructure", "protocol": "REST", "enable_ssl":"true", "visualRange":"0|1", "nodes": [ {"ip": "$AAI_IP","port": "$AAI_PORT"}]}' \
http://$MSB_IP/api/microservices/v1/services
#aai-search
curl -X POST \
-H "Content-Type: application/json" \
-d '{"serviceName": "aai-search", "version": "v11", "url": "/aai/v11/search", "protocol": "REST", "enable_ssl":"true", "visualRange":"0|1", "nodes": [ {"ip": "$AAI_IP","port": "$AAI_PORT"}]}' \
http://$MSB_IP/api/microservices/v1/services
#aai-network
curl -X POST \
-H "Content-Type: application/json" \
-d '{"serviceName": "aai-network", "version": "v11", "url": "/aai/v11/network", "protocol": "REST", "enable_ssl":"true", "visualRange":"0|1", "nodes": [ {"ip": "$AAI_IP","port": "$AAI_PORT"}]}' \
http://$MSB_IP/api/microservices/v1/services