Problems/Issues encountered or steps required when using ONAP Dublin after a fresh installation

1 Portal HTTP 500
2 DMaaP message router pods in ContainerConfigError
3 Policy PODs in CrashLoopBackOff
4 Most of SO PODs in CrashLoopBackOff
5 Remove duplicate objects from A&AI

Portal HTTP 500

PORTAL-571: Can't access Portal UIClosed

DMaaP message router pods in ContainerConfigError

Restart kubelet container in affected hosts

Policy PODs in CrashLoopBackOff

POLICY-1640: Three Policy pods are in CrashLoopBackOff after ONAP deploymentClosed

Most of SO PODs in CrashLoopBackOff

See SO-1674: SO pods failed to come up in ONAP deploymentClosed and https://lists.onap.org/g/onap-discuss/message/16509

workaround

1. Connect to MariaDB container
2. mysql -uroot
3. use mysql;
4. SET PASSWORD FOR 'root'@'localhost' = PASSWORD('password');

5. cd docker-entrypoint-initdb.d
6. Run scripts 01-… to 05-…


Wait for SO PODs to become in Running state

Remove duplicate objects from A&AI

Context of issue is described here

For instance, we had many duplicate Logical-links and we could not fetch/delete them.

So, we have to first get their Vertex IDs so that we can manually delete them from Graph database.

Fetching Vertex IDs

curl -X GET \
'https://172.30.0.90:30233/aai/v16/network/logical-links?format=id' \
-H 'Accept: application/json' \
-H 'Authorization: Basic QUFJOkFBSQ==' \
-H 'Content-Type: application/json' \
-H 'Postman-Token: 5f14e474-f704-4f48-859e-ac1977e7243c' \
-H 'X-FromAppId: Postman Application' \
-H 'X-TransactionId: Postman REST Transaction' \
-H 'cache-control: no-cache'

An example of an output is shown the official documentation

Upon fetching all the vertex IDs that need to be manually removed, for each one of them we execute the following command (replace the graph-admin POD and VERTEX_ID placeholders as per your environment)

Manually Remove Objects from A&AI

kubectl exec -it <graph-admin-pod> -- bash -c 'gosu aaiadmin ./scripts/forceDeleteTool.sh -action DELETE_NODE -userId testId1 -vertexId <VERTEX_ID>'

For example, here is one of the executions to delete a duplicate entry of a logical-link

Command Output

root@onap-rancher-daily:/home/ubuntu# kubectl exec -it dev-aai-aai-graphadmin-c8b9c58c5-kh7t7 -- bash -c 'gosu aaiadmin ./scripts/forceDeleteTool.sh -action DELETE_NODE -userId testId1 -vertexId 749696'

Defaulting container name to aai-graphadmin.
Use 'kubectl describe pod/dev-aai-aai-graphadmin-c8b9c58c5-kh7t7 -n onap' to see all of the containers in this pod.

Wed Jul 3 08:20:16 UTC 2019 Starting ./scripts/forceDeleteTool.sh
NOTE - if you are deleting data, please run the dataSnapshot.sh script first or
at least make a note the details of the node that you are deleting.
---- NOTE --- about to open graph (takes a little while)--------

ForceDelete called by: userId [testId1] with these params: [ -action DELETE_NODE -userId testId1 -vertexId 749696]
>>> Found Vertex with VertexId = 749696, properties:
[resource-version|1560082563941]
[in-maint|false]
[last-mod-source-of-truth|prh]
[aai-created-ts|1560082563941]
[aai-last-mod-ts|1560082563941]
[source-of-truth|prh]
[aai-uri|/network/logical-links/logical-link/167772160-3-0]
[aai-uuid|5686dadd-d40d-454f-9018-5a2afb8656ba]
[link-name|167772160-3-0]
[link-type|attachment-point]
[aai-node-type|logical-link]
No OUT edges were found for this vertex.
No IN edges were found for this vertex.
Found 0 descendant nodes. Note - forceDelete does not cascade to child nodes, but they may become unreachable after the delete.

Found total of 0 edges incident on this node.


Are you sure you want to do this delete? (y/n): y

User [testId1] has confirmed this delete request.
>>>>>>>>>> Removed node with vertexId = 749696
Failed to run the tool ./scripts/forceDeleteTool.sh successfully
command terminated with exit code 1

ONAP Wiki

ONAP Dublin Troubleshooting