Verifying your ONAP Deployment
- Michael O'Brien
- Keong Lim
- Yang Xu
Warning: Draft Content
This wiki is under construction
Motivation: Having issues with MSO provisioning on Rackspace - need all REST API endpoints
Use or combine with Overall Deployment Architecture to determine what a fully functional set of VMs, processes and containers should look like for 1.0.0 and 1.1.0 - so we can be sure that the ONAP deployment is sound as much as possible before provisioning VFs.
http://onap.readthedocs.io/en/latest/index.html
ONAP GUIs
see https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-services/
APP | Name | URL | Docker port | Kubernetes port | Heat port |
|---|---|---|---|---|---|
AAI | http://146.20.65.5:9517/services/aai/webapp/index.html#/viewInspect | 9517 | |||
Grafana | Grafana | Part of the Kubernetes distribution - expose via last lines of https://git.onap.org/logging-analytics/tree/deploy/rancher/oom_rancher_setup.sh kubectl expose -n kube-system deployment monitoring-grafana --type=LoadBalancer --name monitoring-grafana-client |
Monitoring
watch for DCAE collector traffic -
tcpdump -i eth0 port not 22 | grep 3904
We have monitoring currently in DCAE Project Proposal (5/11/17). Proposed monitoring functionality is in Holmes (5/11/17) and ONAP Operations Manager / ONAP on Containers. For the purposes of this demo, we would like to see some overall system/jvm/rest level monitoring while we exercise the demo.
One option is using New Relic agents.
Run the following (use your own account/token) on each VM (Note: JVM processes in docker containers will be visible to the host - so currently - until I run into issues - we don't need to expose extra ports on the containers)
echo deb http://apt.newrelic.com/debian/ newrelic non-free >> /etc/apt/sources.list.d/newrelic.list wget -O- https://download.newrelic.com/548CIEEE16BF.gpg | apt-key add - apt-get update apt-get install newrelic-sysmond nrsysmond-config --set license_key=<akey> /etc/init.d/newrelic-sysmond start |
|---|
Postman/Curl REST calls
Passwords in /testsuite/properties/integration_robot_properties.py
Remember to load each server URL in chrome to accept the cert (save it for curls)
Note AAI V11 is a flat model (no child nodes) - V8 is deep - IE the region contains the tenant
VM | Name | Req | Res | |
|---|---|---|---|---|
AAI | https://{{aai_ip}}:8443/aai/v8/business/customers/customer/Demonstration/service-subscriptions/service-subscription/vFW/service-instances/ | {"service-instance": [{ | ||
AAI | CURL (will require the aai certificate (export it from firefox) root@ip-172-31-82-46:~# curl -X GET https://127.0.0.1:30233/aai/v11/cloud-infrastructure/cloud-regions/ -H "authorization: Basic TW9kZWxMb2FkZXI6TW9kZWxMb2FkZXI=" -H "X-TransactionId:jimmy-postman" -H "X-FromAppId:AAI" -H "Content-Type:application/json" -H "Accept:application/json" --cacert aaiapisimpledemoopenecomporg_20171003.crt -k {"requestError":{"serviceException":{"messageId":"SVC3001","text":"Resource not found for %1 using id %2 (msg=%3) (ec=%4)","variables":["GET","cloud-infrastructure/cloud-regions/","Node Not Found:No Node of type cloud-region found at: cloud-infrastructure/cloud-regions/","ERR.5.4.6114"]}}}root@ip-172-31-82-46:~#  update after | |||
AAI | cloud-region put - to fix above - and before we run init PUT /aai/v11/cloud-infrastructure/cloud-regions/cloud-region/CloudOwner/RegionOne HTTP/1.1 { on the CD server curl -X PUT https://127.0.0.1:30233/aai/v11/cloud-infrastructure/cloud-regions/cloud-region/CloudOwner/RegionOne --data "@aai-cloud-region-put.json" -H "authorization: Basic TW9kZWxMb2FkZXI6TW9kZWxMb2FkZXI=" -H "X-TransactionId:jimmy-postman" -H "X-FromAppId:AAI" -H "Content-Type:application/json" -H "Accept:application/json" --cacert aaiapisimpledemoopenecomporg_20171003.crt -k | |||
AAI | customer post demo.sh init https://{{aai_ip}}:8443/aai/v8/business/customers auth: AAI:AAI or Basic QUFJOkFBSQ== rootTarget.request().header("X-FromAppId", "AAI").get(String.class) | { "customer": [ { | ||
AAI | Same as above - but different customer endpoint https://{{aai_ip}}:8443/aai/v8/business/customers/customer | {"global-customer-id": "Demonstration","subscriber-name": "Demonstration","subscriber-type": "INFRA", | ||
aai | GET https://{{aai_ip}}:{{aai_port}}/aai/v8/cloud-infrastructure/cloud-regions/cloud-region/IAD/RegionOne | { | ||
appc | restconf http://{{appc_ip}}:30230/restconf/config/network-topology:network-topology/topology/topology-netconf user:pass ? | |||
DCAE | ||||
DCAE DMaaP | http://{{collector_ip}}:3904/events/unauthenticated.TCA_EVENT_OUTPUT/group3/sub1?timeout=30000 | |||
DCAE DMaaP | http://{{collector_ip}}:3904/events/unauthenticated.SEC_MEASUREMENT_OUTPUT/group3/sub1?timeout=3000 | [ "{\"event\":{\"measurementsForVfScalingFields\":{\"measurementInterval\":10,\"measurementsForVfScalingVersion\":1.1,\"vNicUsageArray\":[{\"multicastPacketsIn\":0,\"bytesIn\":10,\"unicastPacketsIn\":0,\"multicastPacketsOut\":0,\"broadcastPacketsOut\":0,\"packetsOut\":0,\"bytesOut\":0,\"packetsIn\":500,\"broadcastPacketsIn\":0,\"vNicIdentifier\":\"eth1\",\"unicastPacketsOut\":0}]},\"commonEventHeader\":{\"reportingEntityName\":\"mux1-vnf\",\"startEpochMicrosec\":1486118565570584,\"lastEpochMicrosec\":1486118575570584,\"eventId\":\"1\",\"sourceName\":\"mux_key_gIr3\",\"sequence\":1,\"priority\":\"Normal\",\"functionalRole\":\"vFirewall\",\"domain\":\"measurementsForVfScaling\",\"reportingEntityId\":\"No UUID available\",\"version\":1.1,\"sourceId\":\"b49a2e0e-ee40-48c0-8f9e-842712bea52a\"}}}"] | ||
dmaap | Get the list of topics with this: curl -X GET http://dmaap-hostname:3904/topics Create a missing topic with this: curl -X POST -H "content-type: application/json" --data @bogus-empty-event.json http://dmaap-hostname:3904/events/POA-AUDIT-INIT | |||
Holmes | ||||
MSO | API history for service instance http://{{mso_ip}}:8080/ecomp/mso/infra/orchestrationRequests/v2/?filter=serviceInstanceId%3AEQUALS%3Ac54316d8-464e-4967-bece-8c2b2f458b66 auth: InfraPortalClient:password1$ or Basic SW5mcmFQb3J0YWxDbGllbnQ6cGFzc3dvcmQxJA== | ... | ||
Policy | auth: testrest:3c0mpU#h01@N1c3 or Basic dGVzdHJlc3Q6M2MwbXBVI2gwMUBOMWMz | |||
SDC | auth: sdcclient:password or Basic c2RjY2xpZW50OnBhc3N3b3Jk Example: A GET query sent from VID to SDC to retrieve a service metadata URL: http://{sdc_ip}:8080/sdc/v1/catalog/services/1eec58c0-d5e2-45c5-be9c-c873a1749541/metadata Headers: Authorization:Basic dmlkOktwOGJKNFNYc3pNMFdYbGhhazNlSGxjc2UyZ0F3ODR2YW9HR21KdlV5MlU= verify curl -X GET http://{{your-server}}:8080/sdc2/rest/v1/catalog/services/serviceName/vepc/serviceVersion/1.0 -H 'user_id: cs0008' verify $ curl -X GET http://sdc:8080/sdc/v1/catalog/resources -H "authorization: Basic YWFpOktwOGJKNFNYc3pNMFdYbGhhazNlSGxjc2UyZ0F3ODR2YW9HR21KdlV5MlU=" -H "x-ecomp-instanceid:AAI" | { | ||
SDNC | auth: admin:Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U or Basic YWRtaW46S3A4Yko0U1hzek0wV1hsaGFrM2VIbGNzZTJnQXc4NHZhb0dHbUp2VXkyVQ== | |||
VID | 8080/vid auth <null>:<null> or Basic Og== |
Additional sources:
just found consolidated auth info in the oom repo
root@obriensystemskub0:~/oom/kubernetes/config# vi ../config/docker/init/src/config/robot/eteshare/config/integration_robot_properties.py
Username / Password / Authorization Token
An aside: (I didn't realize or I forgot an aspect of base64 - dthat we could reverse engineer the password/username combination from the encoded <alphanumeric_token> in the header key:value = Authorization:Basic <alphanumeric_token>. Again thanks Yves - use a public site like https://www.base64decode.org/ For example for MSO we take the token SW5mcmFQb3J0YWxDbGllbnQ6cGFzc3dvcmQxJA== and get InfraPortalClient:password1$ which is defined throughout the ONAP codebase in for example the VID DockerFile.
Prerequisites
If running postman https endpoints (some of our VM's run SSL like AAI - but MSO for example will still run straight HTTP) - trust the server certificate in Postman (thank you Yves): You won't be able to run an https endpoint until the certificate is trusted in Chrome - paste an https request into the browser - one time - to enable the trusted certificate in postman.
For example: postman will work against AAI after launching the following (your IP) in chrome
https://{{aai_ip}}:8443/aai/v8/cloud-infrastructure/cloud-regions/
For programmatic JAX-RS 2.0 clients add the following
// fix java.security.cert.CertificateException: No subject alternative names present |
|---|
MSO VM
WIP: work in progress
An example get on a specific vFW VF from the demo. In this case we use the cs0001 user to get the Vf Module ID from the edit page of a service instance's VF in VID
Postman Request | Query API History for VF Module GET /ecomp/mso/infra/orchestrationRequests/v2/?filter=vfModuleInstanceId%3AEQUALS%3A90e7cbda-053c-46fb-9df7-c8559c29299c HTTP/1.1 |
|---|---|
Postman Response | {"requestList": [ {"request": { |
curl | obrienbiometrics:onap michaelobrien$ curl -X GET -H "Authorization: Basic SW5mcmFQb3J0YWxDbGllbnQ6cGFzc3dvcmQxJA==" -H "Content-Type: application/json" -aH "Cache-Control" -d '' http://104.130.169.999:8080/ecomp/mso/infra/orchestrationRequests/v2/?filter=vfModuleInstanceId%3AEQUALS%3A90e7cbda-053c-46fb-9df7-c8559c29299c {"requestList":[{"request":{"requestId":"8230aa5f-cbcf-492d-817a-37243475b46f","startTime":"Mon, 15 May 2017 12:25:25 GMT","requestScope":"vfModule","requestType":"createInstance","requestDetails":{"modelInfo":{"modelCustomizationName":null,"modelInvariantId":"ce3e0e4e-3189-4798-b4b2-f60f3d69e378","modelType":"vfModule","modelNameVersionId":"d55da365-52e2-47ee-8d48-011891909f4f","m...... |
AAI
Add Sample data set to AAI for Sparky
GET /aai/v8/service-design-and-creation/services HTTP/1.1 | {"service": [{ |
|---|---|
Workarounds for Distribution and Deployment
SDC Distribution failure
20170914: SDC service distribution is failing - a partial workaround is to run ./demo.sh distribute - after manually adding the "Demonstration" customer and associating with a new cloud region.
PUT /aai/v8/cloud-infrastructure/cloud-regions/cloud-region/IAD/IAD HTTP/1.1
Host: 146.20.65.5:8443
Accept: application/json
Content-Type: application/json
X-FromAppId: AAI
X-TransactionId: get_aai_subscr
Authorization: Basic QUFJOkFBSQ==
Cache-Control: no-cache
Postman-Token: db2cb2b2-5ecc-1acf-3d9b-a08cdf53102c
{
"cloud-owner": "IAD",
"cloud-region-id": "IAD",
"cloud-region-version": "v1",
"cloud-type": "SharedNode",
"cloud-zone": "CloudZone",
"owner-defined-type": "OwnerType",
"tenants": {
"tenant": [{
"tenant-id": "1035021",
"tenant-name": "ecomp-dev"
}]
}
}
GET /aai/v8/cloud-infrastructure/cloud-regions/ HTTP/1.1
Host: 146.20.65.5:8443
Accept: application/json
Content-Type: application/json
X-FromAppId: AAI
X-TransactionId: jimmy-postman
Authorization: Basic TW9kZWxMb2FkZXI6TW9kZWxMb2FkZXI=
Cache-Control: no-cache
Postman-Token: 48cc0bab-4a34-af53-ea77-48fa4a536813
{
"cloud-region": [
{
"cloud-owner": "IAD",
"cloud-region-id": "IAD",
"cloud-type": "SharedNode",
"owner-defined-type": "OwnerType",
"cloud-region-version": "v1",
"cloud-zone": "CloudZone",
"resource-version": "1505416531254",
"tenants": {
"tenant": [
{
"tenant-id": "1035021",
"tenant-name": "ecomp-dev",
"resource-version": "1505416532060"
}
]
}
}
]
}
GET /aai/v8/cloud-infrastructure/cloud-regions/cloud-region/IAD/IAD/tenants/tenant/1035021 HTTP/1.1
Host: 146.20.65.5:8443
Accept: application/json
Content-Type: application/json
X-FromAppId: AAI
X-TransactionId: get_aai_subscr
Authorization: Basic QUFJOkFBSQ==
Cache-Control: no-cache
Postman-Token: bf5b90a0-38f3-1767-d3d0-c97ee1dfe2f0
{
"tenant-id": "1035021",
"tenant-name": "ecomp-dev",
"resource-version": "1505416532060"
}
PUT /aai/v10/business/customers/customer/Demonstration HTTP/1.1
Host: 146.20.65.5:8443
Accept: application/json
Content-Type: application/json
X-FromAppId: AAI
X-TransactionId: get_aai_subscr
Authorization: Basic QUFJOkFBSQ==
Cache-Control: no-cache
Postman-Token: 2e9eb0a4-ae81-6f07-80a4-4d4939e133b6
{ "global-customer-id": "Demonstration",
"service-subscriptions": {
"service-subscription": [{
"relationship-list": {
"relationship": [{
"related-to": "tenant",
"relationship-data": [{
"relationship-key": "cloud-region.cloud-owner",
"relationship-value": "IAD"
}, {
"relationship-key": "cloud-region.cloud-region-id",
"relationship-value": "IAD"
}, {
"relationship-key": "tenant.tenant-id",
"relationship-value": "1035021"
}] }]},
"service-type": "vFW"
}] },
"subscriber-name": "Demonstration",
"subscriber-type": "INFRA"}
GET /aai/v8/business/customers HTTP/1.1
Host: 146.20.65.5:8443
Accept: application/json
Content-Type: application/json
X-FromAppId: AAI
X-TransactionId: get_aai_subscr
Authorization: Basic QUFJOkFBSQ==
Cache-Control: no-cache
Postman-Token: 6a887711-f76e-4529-a42f-30bcb4e2d127
{
"customer": [
{
"global-customer-id": "Demonstration",
"subscriber-name": "Demonstration",
"subscriber-type": "INFRA",
"resource-version": "1505417454006",
"service-subscriptions": {
"service-subscription": [
{
"service-type": "vFW",
"resource-version": "1505417454761",
"relationship-list": {
"relationship": [
{
"related-to": "tenant",
"related-link": "https://146.20.65.5:8443/aai/v8/cloud-infrastructure/cloud-regions/cloud-region/IAD/IAD/tenants/tenant/1035021",
"relationship-data": [
{
"relationship-key": "cloud-region.cloud-owner",
"relationship-value": "IAD"
},
{
"relationship-key": "cloud-region.cloud-region-id",
"relationship-value": "IAD"
},
{
"relationship-key": "tenant.tenant-id",
"relationship-value": "1035021"
}
],
"related-to-property": [
{
"property-key": "tenant.tenant-name",
"property-value": "ecomp-dev"
}
]
}
]
}
}
]
}
}
]
}
PUT /aai/v8/service-design-and-creation/services/service/demoVFW HTTP/1.1
Host: 146.20.65.5:8443
Accept: application/json
Content-Type: application/json
X-FromAppId: AAI
X-TransactionId: get_aai_subscr
Authorization: Basic QUFJOkFBSQ==
Cache-Control: no-cache
Postman-Token: 0436aabb-197c-3c58-3c65-3387aebab2bb
{
"service-description": "demoVFW",
"service-id": "demoVFW"
}
then
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #ffffff; background-color: #2b66c9}
span.s1 {font-variant-ligatures: no-common-ligatures}
root@vm1-robot:/opt# ./demo.sh distribute
Starting Xvfb on display :89 with res 1280x1024x24
Executing robot tests at log level TRACE
==============================================================================
OpenECOMP ETE
==============================================================================
OpenECOMP ETE.Robot
==============================================================================
OpenECOMP ETE.Robot.Testsuites
==============================================================================
OpenECOMP ETE.Robot.Testsuites.Demo :: Executes the VNF Orchestration Test ...
==============================================================================
Initialize Models | PASS |
------------------------------------------------------------------------------
OpenECOMP ETE.Robot.Testsuites.Demo :: Executes the VNF Orchestrat... | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
OpenECOMP ETE.Robot.Testsuites | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
OpenECOMP ETE.Robot | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
OpenECOMP ETE | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output: /share/logs/demo/InitDistribution/output.xml
Log: /share/logs/demo/InitDistribution/log.html
Report: /share/logs/demo/InitDistribution/report.htmlSwagger API Endpoints
TODO: get swagger docs for all servers
This is an expanding list of API endpoints to verify your ONAP deployment - ideally we would post a postman config and environment file.
VM | Container | ext port | URL | user:pass | gerrit source | Generating Artifacts |
AAI | ||||||
Policy | pdp | 8081 | <service>:8081/pdp/swagger-ui.html oom http://10.12.5.81:30220/pdp/swagger-ui.html#/policy-engine-services | |||
SDNC | sdnc_controller_container | 8282 | admin:Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U |
Robot Healthcheck
There actually is a set of scripts already on the robot vm inside its docker container - currently attempting to get these to pass (won't check your VFs though)
root@16e8d4997655:/var/opt/OpenECOMP_ETE# ./runTags.sh -i health h -d ./html -V /share/config/robot_properties_ete.py |
|---|
ID | VM | Container | Process | Healthcheck |
|---|---|---|---|---|
AAI | ||||
MSO | ||||
Policy | drools docker exec -it -u 0 drools su - policy | note: escape any special chars like the exclamation point in the password curl http://healthcheck:zb\!XztG34@policy:6969/healthcheck {"healthy":true,"details":[{"name":"PDP-D","url":"self","healthy":true,"code":200,"message":"alive"},{"name":"PAP","url":"http://pap:9091/pap/test","healthy":true,"code":200,"message":""},{"name":"PDP","url":"http://pdp:8081/pdp/test","healthy":true,"code":200,"message":""}]} for postman: Basic aGVhbHRoY2hlY2s6emIhWHp0RzM0 { "healthy": true, |
Portals
Portal
SDC - http://sdnc:8843/user/listUsers
Demo VMs
ssh keys
The private key for the 3 vFW demo VM's is in /testsuite/robot/assets/keys/robot_ssh_private_key.pvt
obrienbiometrics:onap michaelobrien$ ssh -i robot_ssh_private_key.pvt root@172.99.67.148 root@demofwl01pgn:~# history |
|---|
Artifacts Required
log files for each VM and set of docker containers - or how to aggregate them - not necessarily at the SumoLogic level
VM/container or JVM health checks - ideally something like New Relic agents on the box
Would be nice to have something we can run on vm1-robot that would automate a healthcheck on all the containers
Get the list of topics with this:
curl -X GET http://dmaap-hostname:3904/topics
Create a missing topic with this:
curl -X POST -H "content-type: application/json" --data @bogus-empty-event.json http://dmaap-hostname:3904/events/POA-AUDIT-INIT
Integration Sanity Tests
Using the Robot Framework in AAI R4 Integration Sanity Test Plans for more than just the basic healthcheck network calls (actual create, update, delete of data in AAI)