PTL 2018-10-29
- Gildas Lanilis Gmail
- Michael O'Brien
- Pamela Dragosh
Duration 60 minutes
| Duration | Agenda Item | Requested by | Notes / Links |
|---|---|---|---|
| START RECORDING | |||
| Release Management | I won't be able to attend Monday Oct 29 PTL Meeting. Pamela Dragosh is my delegate. Main focus aera are:
| ||
Notes
Yang Xu - Release image cannot pass distribution - fail A&AI model distribution. Mismatch with pulling from OOM master branch with release images.
- Brian Freeman - may need OOM to branch. May be stability issues on release branch until teams can provide stable released images.
- James Forsyth - there is a workaround for manual fix for model distribution. There is a ticket for this for Yang to verify -Â AAI-1759 - Getting issue details... STATUS
- Per Yang Xu - thinks it is a different issue and will create issue.
Marco Platania - wasn't able to finish vFW. Policy images where expiring. vDNS - unable to instantiate VNF's. Problem with SO - will create a JIRA. Need a clean environment to test.
Dan Timoney - cannot reproduce SDNC-482 - Getting issue details... STATUS . Michael O'Brien will try to create environment so Dan can look at the logs.
Update:Â LOG-789 - Getting issue details... STATUS
SDNC is mostly up the single node db and app clusters and the ueb container have issues - restarted the db container
ubuntu@ip-172-31-14-89:~$ kubectl get pods --all-namespaces -o wide | grep sdnc
onap onap-sdnc-controller-blueprints-848fbddb78-6p885 1/1 Running 0 31m 10.42.138.162 ip-172-31-6-214.us-east-2.compute.internal
onap onap-sdnc-controller-blueprints-db-0 1/1 Running 0 31m 10.42.183.47 ip-172-31-12-220.us-east-2.compute.internal
onap onap-sdnc-nengdb-0 1/1 Running 0 31m 10.42.34.54 ip-172-31-5-233.us-east-2.compute.internal
onap onap-sdnc-network-name-gen-5f4879488b-4kqm4 1/1 Running 0 31m 10.42.233.50 ip-172-31-6-214.us-east-2.compute.internal
onap onap-sdnc-sdnc-0 2/2 Running 0 31m 10.42.141.158 ip-172-31-4-211.us-east-2.compute.internal
onap onap-sdnc-sdnc-ansible-server-fc756bc5c-d626t 1/1 Running 0 31m 10.42.147.208 ip-172-31-12-220.us-east-2.compute.internal
onap onap-sdnc-sdnc-db-0 1/2 Running 1 52s 10.42.6.29 ip-172-31-4-211.us-east-2.compute.internal
onap onap-sdnc-sdnc-dgbuilder-5859759949-k6nmd 1/1 Running 0 31m 10.42.211.47 ip-172-31-5-233.us-east-2.compute.internal
onap onap-sdnc-sdnc-dmaap-listener-5fddc9c948-f9529 1/1 Running 0 31m 10.42.116.136 ip-172-31-4-211.us-east-2.compute.internal
onap onap-sdnc-sdnc-portal-658cb796c-7qd6d 1/1 Running 0 31m 10.42.126.68 ip-172-31-5-233.us-east-2.compute.internal
onap onap-sdnc-sdnc-ueb-listener-d86768bf4-gkscm 0/1 Init:0/1 0 31m 10.42.9.210 ip-172-31-4-211.us-east-2.compute.internal
onap onap-so-so-sdnc-adapter-7785b77b75-j75tv 1/1 Running 0 31m 10.42.124.56 ip-172-31-12-220.us-east-2.compute.internal
onap onap-vfc-vfc-zte-sdnc-driver-c5d9868fd-5jczk 1/1 Running 0 30m 10.42.249.65 ip-172-31-4-211.us-east-2.compute.internal
dcae will be up at the 70 min mark
OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp components are...
==============================================================================
Basic A&AI Health Check | PASS |
------------------------------------------------------------------------------
Basic AAF Health Check | PASS |
------------------------------------------------------------------------------
Basic AAF SMS Health Check | PASS |
------------------------------------------------------------------------------
Basic APPC Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic CLI Health Check | PASS |
------------------------------------------------------------------------------
Basic CLAMP Health Check | PASS |
-----------------------------------------------------------------------------
Basic DCAE Health Check | FAIL |
500 != 200
------------------------------------------------------------------------------
Basic DMAAP Data Router Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ac81e8650>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
[ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ac887e1d0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
[ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ac89abd10>: Failed to establish a new connection: [Errno 111] Connection refused',)': /internal/fetchProv
| FAIL |
ConnectionError: HTTPConnectionPool(host='dmaap-dr-node.onap', port=8080): Max retries exceeded with url: /internal/fetchProv (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ac89b54d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
------------------------------------------------------------------------------
Basic DMAAP Message Router Health Check | PASS |
------------------------------------------------------------------------------
Basic External API NBI Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Elasticsearch Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Kibana Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Logstash Health Check | PASS |
------------------------------------------------------------------------------
Basic Microservice Bus Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-ocata API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-pike API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-titanium_cloud API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-vio API Health Check | PASS |
------------------------------------------------------------------------------
Basic OOF-Homing Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic OOF-SNIRO Health Check | PASS |
------------------------------------------------------------------------------
Basic Policy Health Check | PASS |
------------------------------------------------------------------------------
Basic Pomba AAI-context-builder Health Check | PASS |
------------------------------------------------------------------------------
Basic Pomba SDC-context-builder Health Check | PASS |
------------------------------------------------------------------------------
Basic Pomba Network-discovery-context-builder Health Check | PASS |
------------------------------------------------------------------------------
Basic Portal Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic SDC Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic SDNC Health Check | FAIL |
Resolving variable '${resp.json()['output']['response-code']}' failed: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
------------------------------------------------------------------------------
Basic SO Health Check [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ac8a88950>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health
[ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ac8a88a90>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health
[ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ac8a88a50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /manage/health
FAIL |
ConnectionError: HTTPConnectionPool(host='so.onap', port=8080): Max retries exceeded with url: /manage/health (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6ac8a88e10>: Failed to establish a new connection: [Errno 111] Connection refused',))
------------------------------------------------------------------------------
Basic UseCaseUI API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC catalog API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC emsdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC gvnfmdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC huaweivnfmdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC jujuvnfmdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC multivimproxy API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC nokiavnfmdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC nokiav2driver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC nslcm API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC resmgr API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC vnflcm API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC vnfmgr API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC vnfres API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC workflow API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC ztesdncdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC ztevnfmdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VID Health Check | PASS |
------------------------------------------------------------------------------
Basic VNFSDK Health Check | PASS |
------------------------------------------------------------------------------
Basic Holmes Rule Management API Health Check | FAIL |
502 != 200
------------------------------------------------------------------------------
Basic Holmes Engine Management API Health Check | FAIL |
502 != 200
------------------------------------------------------------------------------
OpenECOMP ETE.Robot.Testsuites.Health-Check :: Testing ecomp compo... | FAIL |
50 critical tests, 40 passed, 10 failed
50 tests total, 40 passed, 10 failed
==============================================================================
OpenECOMP ETE.Robot.Testsuites | FAIL |
50 critical tests, 40 passed, 10 failed
50 tests total, 40 passed, 10 failed
==============================================================================
OpenECOMP ETE.Robot | FAIL |
50 critical tests, 40 passed, 10 failed
50 tests total, 40 passed, 10 failed
==============================================================================
OpenECOMP ETE | FAIL |
50 critical tests, 40 passed, 10 failed
50 tests total, 40 passed, 10 failed
Yang Xu - can close SO-1166 - Getting issue details... STATUS  - other JIRA ticket fixed it per Brian Freeman. Duplicate.
Per cl664y@att.com - USECASEUI-174 - Getting issue details... STATUS  with Andreas Geißler is testing as Tao gave a fix.
No update on USECASEUI-173 - Getting issue details... STATUS  - in progress
External Labs:
- https://jenkins.onap.org/view/External%20Labs/job/lab-windriver-oom-staging-daily/125/robot/
- OOF and Holmes are known issues
- OOF changes merged last night -Â Shankaranarayanan Puzhavakath Narayanan will check to see if their fix caused today's failure.
- Yang Xu - create a JIRA for Guangrong Fu Holmes failure. There is an open task HOLMES-176 - Getting issue details... STATUS  - is it the same??
- Pamela Dragosh will follow up on the Policy failure.
- OOF and Holmes are known issues
cl664y@att.com - Sonar Javascript code coverage was turned on for POC with CLAMP project and security team for Dublin code coverage requirements.
- Gervais-Martial Ngueko - provide instructions to disable javascript in a project pom.xml codebase. PTL's are responsible for creating JIRA's to re-enable it post-Casablanca to meet Dublin code coverage requirements.
Chat Text
09:11:42 From Jimmy Forsyth (AT&T) : Workaround is documented here: https://jira.onap.org/browse/AAI-1759
09:12:19 From Jimmy Forsyth (AT&T) : kubectl exec -it $(kubectl get pods -n onap | grep 'traversal' | awk '{ print $1; }') -n onap gosu aaiadmin /opt/app/aai-traversal/bin/install/updateQueryData.sh
09:14:20 From Yang Xu : Thanks Jimmy
09:17:33 From Catherine Lefevre : I am not sure any SO team member online, SO-1156 is a showstopper for VID/SO pairwise testing. Code submitted by Marcus but we need SO committers to review and merge the code - thank you
09:23:32 From Marco Platania (AT&T) : SO-1170 created
09:25:11 From Brian : OOF-Homing and Holmes (2) failing
09:25:48 From Brian : and Policy 403
09:26:09 From Brian : Integration-OOM-staging-daily
09:32:50 From Catherine Lefevre : another comment JS Sonar has been activated so your test coverage could be impacted if your code is not only java code base
09:33:51 From Catherine Lefevre : Casablanca criteria did not change the focus of test coverage is NOT in JS so I have asked Gildas to review in case of any team has a important drop due to recent JS sonar activation
09:38:55 From Catherine Lefevre : nov 29th
09:44:18 From Catherine Lefevre : martial - please provide instructions
09:44:56 From Yang Xu : @Jimmy, I think the problem in AAI distribution is different from AAI-1759, we see 403 error in model loader log
09:45:19 From Yang Xu : 2018-10-29T07:15:01.565Z|46abaf85-53ab-4ba2-8428-4ddf14d10c2a|pool-2-thread-4|ModelLoader|Event-Bus|AAIRESTClient||INFO|AC0004I|AC0004I request at url = https://aai.onap:8443/aai/v14/service-design-and-creation/models/model/5f0ff5e3-5641-4a8e-872e-1832f208b008 resulted in http response: 403 Forbidden|
09:46:11 From Brian : @Yang which environment ?
09:46:11 From Yang Xu : I will create a JIRA to track it, this is tested on OOM-Daily
Action items
- Â