Work in Progress
Lesson Learned
Retrospective
Identification of issues - earlier than later
• Pair-wise testing performed by the project teams
• 72h stability test run
Need for continuous integration testing:
- Some experience is based on HPA testing on vCPE use case. vCPE use case, even though, it was tested on Amsterdam successfully, it took a while to make vCPE work on Beijing components. It could be that there was some regression. Since HPA depends on vCPE use case, it could not be tested till the end. This can be avoided if previously tested use cases are verified continuously. Automation of use case testing & continuous integration verifications are key to add new functionalities in future releases.
Lesson Learned Areas
• Communication – Wiki, Mailing Lists, IRC
- When you are a mailing list moderator do not approve subscription requests from accounts with obviously pornographic domain names. (yes, this really happened. recently.)
- Lack of transparency on LF IT ticket raised by the community. The LF RT tools has poor capability to email properly the whole set of information. The ONAP community has no visibility on what LF IT is currently working on.
- Feedback compiled by Kenny Paulfrom ONS-NA: (See TSC 2018-04-05 Agenda slides)
Onboarding an engineer is very hard; The wiki isn’t laid out well and is confusing, needs to be cleaned up
Responsiveness from existing Community members answering new contributor questions can be a challenge
- All approved projects should be required to post meeting minutes to the wiki
The volume of untagged emails makes filtering almost impossible
- Need absolutely clear and unambiguous validation that RocketChat can be used via https from China and most companies without requiring a VPN, or the use of an alternative device/network -kenny:. Must be secured if setup by LF (simply b/c of well known pwd).
- Would like the Community to decide whether the use of IRC as the "official" minute mechanism is a practice worth continuing or not; very few people are using it, many can't use it and unlike other communities, virtually no one contributes to the minutes. Compare that with the fact that everyone can use zoom chat and there is always a great deal of contribution being performed there. -kenny
- WIKI Help: things are hard to find unless you type the right keywords or have the proper URL. Matter of organization, not matter of quantity of information. Recommendation to use ReadTheDocs to start versus the wiki?
- WIKI: Emboarding new community members is the challenge.
- Is there a good enough flow of information-communication between sub-committees and projects? Overall: yes. Sub-committee feedback of to the TSC is not systematic(as we wish).
- Expectation from sub-sub-committee back to the team and then driving the execution. Use-case owner is missing. Use-case team to fill a checklist? We may defined what that role is.
• Labs & Test Environment
- After M4 code freeze, the community is spending a lot of time to get the Health Check sanity test to pass. HealthCheck is the kind of automated test that MUST pass everyday all along the release lifecycle.
- Despite the effort made by some ONAP partners in providing Labs, we have reached the limit on current labs infrastructure. This is preventing further needed testing (like test job during the verify phase).
- Need to develop a full Agile CI-CD pipeline. Full Chain to run automatically the sanity HC, CSIT, E2E. Everything running continuously.
- How to make better usage of XCI-CD environment (OPNFV,...)
- CSIT tests under integration repo must branch early to support the other component branches and their test automation (at least must branch on code freeze deadline to provide enough time to fix the CSIT tests in the release branches).
- Supporting HEAT and OOM based deployments is getting harder with duplicate maintenance of the config values (it may help if we can somehow abstract such duplication of config values).
More ONS-NA Feedback: The amount of time to setup the dev environment is a barrier to engagement - Developers don’t want to invest a lot of time on that.
• Release Management
- Progress check between M2 and M4 of intermediate development by release manager might help to improve the quality of the final product (for example scheduling the demos of each component to show the current progress at M3 deadline might be right choice too). This avoids any misunderstanding of the features or requirements by the dev team and can receive feedback from the usecase or architecture teams to improve them when there is still time until code freeze.
- Lack of formality to froze model and scope.
• JIRA Management
- Often lack of updated information in "Status" field, that prevent to know if someone is working on a defect. This issue varies depending on project and people.
- Adopt a singe JIRA workflow. Currently 2 are in place (1 from openecomp, 1 simple from openo).
- PTLs are owner of the scope of iteration.
- Review the 2 JIRA workflows and define 1 workflow for ALL.
• Code Review
- Requires active committers and more active code reviewers (right now only few committers or reviewers are always reviewing and merging the code).
• Development Infrastructure
- The LF toolchain that is currently in place, do allow to merge in master code that has not been thoroughly tested. This leads to massive disruption in the testing.
- Nexus 3 slowness. This has been impacting Integration Team tremendously as it tooks 3-4 more times just to download Docker images.
- Current 1/3 party scan security tool (Nexus IQ) do not work for Python and JavaScript packages and thus prevent the community to have an holistic view on vulnerabilities.
- For the record, it was decided that the community would not account for JavaScript in using Sonar (code coverage) in Beijing Release.
- Slowness of the full IT chain (jenkins, nexus, wiki,..)
- Local testing: how easy to setup an env for a developer to perform its own testing before submitting code. Need reference VNF, AAI, ... (too much time spend to setup environment). Idea of lab reference to be used as a model for configuration. (currently SB-07 serves that purpose).
- Pair wise testing for a great value added in Beijing release.
- 72 hours helps to identify defects. Docker Upgrades went fine.
- Backup and restore capacity for SB 04-07 in Windriver? Have we ever asked Windriver?
- Feature parity on LABS (do not over taxe Windriver).
• PTL Role
- More ONS-NA feedback:
“Either you commit or you forfeit. There really isn’t any middle ground here."
If you can’t attend a meeting, assign a delegate. If you send a delegate more often than you attend, you shouldn’t be a PTL.
PTL-cross PTL discussion: feeling of not involvement in some decision. Having a place for PTLs to speak up.
• Architecture
- Need for simplifying ONAP Micro-Service architecture by leveraging best practices from Industry :
- Certificate enrollment (for Mutual TLS communication among services) in Beijing was manual, time consuming and error prone. Need automation of certificate enrollment is needed.
- Scale-out of ONAP is removed from the scope in Beijing release. Services that attempt do it has some learnings. Scale-Out/Load balancing/Circuit-breaking/DB-sync related functionality should be removed from actual micro services to sidecars/side-kicks to avoid each developer spending time and getting it right.
• Use cases:
- Increase the scope of vCPE use case in deploying multiple virtual Gateways (Multi-Customer support)
• Open Source Values
- One critical aspect of Open Source is transparency (but opposition to Silos). We still see some corporation submitting huge pile of code days prior to major milestones. This approach is preventing the whole community to collaborate effectively.
- More ONS-NA feedback:
- internal company staff meetings/decisions != project team meetings/decisions
• Others
- Commit process: code review. Dialog necessary.
- Use Case: more details functional requirements. more focus on cross system integration (architecture). Lack of defined type, flow and content of messages (need additional details).