Scaling Meeting Minutes
October 23, 2018
Agenda
Casablanca Testing
Casablanca Building Blocks in SO
Dublin Planning
Attendees
Scott Blandford, Yuriy Malakov, Jacques Bolduc, Lathish, Prabhu, Ramja Balaji, Joss Armstrong, Brittany Plummer, Lauren Lewis, Prema Bhatt, Lusheng Ji
Discussion
Casablanca Testing
Completed testing of Design Time flow in K8s environment
Run Time testing will begin Wednesday
Casablanca Building Blocks
Added Building blocks to Casablanca Work Flow
Need to add building blocks layer to Dublin Work Flow
Dublin Planning
Met with APPC Team to review Scale In Requirements
Action Items
October 16, 2018
Agenda
Casablanca Testing
Dublin Workflow
Dublin Requirements
Attendees
Scott Blandford, Lauren Lewis, Marco Platania, Lathish, Prabhu, Daurban, Joss Armstrong, Prema Bhatt
Discussion
Casablanca Testing
No Progress in the past week
K8s environment still not working (Hopefully, it will be functioning within the next couple days)
Marco has been busy with other projects
Marco will continue testing tomorrow assuming the test environment is ready to go
Dublin Planning
Building Block meeting with SO team
SO is building workflows so that they can be easily transformed into Building blocks for Work Flow Designer
Scott will try to organize the work flow diagrams to depict the Building Blocks
Manual A&AI entries for Policy
Need to have consult with Pam (Policy) to determine if Policy can implement the new SO API so that no manual AAI entries will be required.
Action Items
October 9, 2018
Agenda
Casablanca Testing
Dublin Workflow
Dublin Requirements
Attendees
Scott Blandford, Lathish, Taka Cho, Marco Platania, Sebastien Determe, Martial Ngueko
Recording
Discussion
Testing only being done in SB07 at this time since OOM is not fully working
We should merge ConfigScaleIn and ConfigScaleOut
Maybe merge to ConfigModify but at least to ConfigScaling
We will need to support an Ansible port on vDNS
Action Items
October 2, 2018
Agenda
Casablanca Testing Status
Dublin Workflow
Dublin Requirements
Attendees
Scott Blandford, Wojciech Sliwka, Lukasz Grech, Taka Cho, Marco Platania, Lauren Lewis, Yuriy Malakov, Prema Bhatt, Prabhu
Recording
Discussion
Testing (Marco)
Early stages. Still trying to get platform stable
Tried pushing policies from CLAMP to Policy
Guard Policies did not work. A fix is being developed
Marco will send more updates to Scott
Scale In VM Decision
Decided to use LIFO for Dublin
OOF is the right strategy for making the decision
Should Policy, OOF, or SO determine the last VM added to the pool?
Pre Action
We will send the VNF a Pre-action signal for Scale In to allow the VNF to take care of VNF specific Tasks before the scale in action is done.
SO will wait for a response from the VNF before moving on to the next action in the workflow
Moving Traffic
Distribute_traffic will be used to move new traffic away from target VM
Quiesce_traffic will be used to signal when all existing traffic has been drained from target VM
Both Distribute_Traffic and Quiesce_Traffic will need to be modified to work at the VF_Module/VNFC level
Configuration
Not much changes here except for incorporating the use of Auto Configuration. This will need to be done for Scale Out also
CDS should be ready in Dublin timeframe
We will need a new controller API for "ConfigScaleIn"
Cleanup
Need to do a STOP VM before we terminate the VM so we do not corrupt any resources
All resources will be released and the next scale out will likely get a different set of resources.
Action Items
September 25, 2018
Agenda
Casablanca Developer Issues
Casablanca Testing (Marco)
Dublin planning
Scaling Prep and Post phases
Attendees
Scott Blandford, Latish, Marco Platania, Lauren Lewis, Jacques Bolduc, Joss Armstrong, David Ho, Taka Cho, Piotr Krysiak, Sebastien Determe, Brittany Plummer, Milind Phadke, Vivian Ye
Recording
Discussion
Due to extended holiday in Israel, the VID code developed by Nokia has not been reviewed and merged. Marco is following up with Catherine and others to figure out a work around
Testing
Helen has asked for more detailed test cases
Marco will document tests he is running and then send them to Scott to add to the wiki page
Prep and Post phases for Scale In
Scott will get back to Liu (Nokia) to explain what should be done by ONAP vs what should be done by the VNF. once this is established we will need to determine if the VNF needs to be signaled for each phase and if a response is needed.
Action Items
September 18, 2018
Agenda
Casablanca Status
M4 update
Round table
Dublin Planning
Preparation Phase
Attendees
Scott Blandford, Marco Platania, Brittany Plummer, Jacques Bolduc, Prabhu, Wochiech Sliwka, Lauren Lewis, Sebastien Determe, jarmstrong001, Taka Cho, Yuriy Malakov, Lukasz Rajewski, Prema Bhatt, Pam Dragosh
Recording
Discussion
Casablanca M4 Status
CLAMP - Martial - On Track
Policy - Marco working with Policy - everything should be submitted. Pam - everything should be fine
SDNC - Marco touched base with Ruchira, everything is on track
SO - Brittany - all code is submitted
Discussion - We may need a delay between ConfigScaleOut and the final Healthcheck
VID - Wochiech Sliwka - Problems with Testing Environment (Marco suggested they use Windriver), All code will be pused prior to M4
Marco will run testing for Scaling Use Case. Communication will be via email, Q, and possibly RocketChat
Integration Testing will start after M4
Integration Testing will run in SB07 and SB04 test environments
Dublin (Prep phase for scaling actions)
So far everyone seems ok with this
Action Items
September 11, 2018
Agenda
Casablanca Status
M4 Code Freeze (9/20)
Testing Status
Dublin Planning
Go over current Scale In Flow
Redistribute_Traffic Discussion
Attendees
Scott Blandford, Brittany Plummer, David Ho, Jacques Bolduc, jarmstrong001, Lukasz Rajewski (O-PL), Michael Z (AT&T), Shankar Narayanan, Taka Cho, Marc Fiedler (DT), Milind Phadke, Randa Maher (AT&T), Prema Bhatt (AT&T), Lauren Lewis, Ajay Mahimkar, Martial Ngueko
Recording
Discussion
Action Items
August 28, 2018
Agenda
Discuss Policy concerns
Round Table on R3 status
Daniel Urban (Nokia) wants to help
China Mobile updates to roadmap
Dublin Scale In Discussion
Attendees
Scott Blandford, Prabhu, Shankar Narayanan, Daniel Urban, Brittany Plummer, Martial Ngueko, Taka Cho, David Ho, Jacques Bolduc, Pam Dragosh, Lauren Lewis
Recording
Discussion
Pam will work with Jim Hahn to determine if he is still having issue with the policy requirements
All projects are currently on target to implement the R3 requirements for Scale Out
Pam will engage Daniel Urban on Control Loops
Scale In Discussion:
We discussed the Scale in Work Flow described on this wiki page
Pam will work with Policy team to determine if Policy should make the decision about which instance to remove
Scott should work with Brian Freeman and Architecture comm to determine which component should make the decision on which instance
Pam will look into monitoring the target instance to determine when it is inactive and sending the signal to SO
Action Items
August 21, 2018
Agenda
M3 Milestone (API Freeze)
Casablanca Status
Scale In work flow for Dublin
Attendees
Scott Blandford, Lauren Lewis, Martial Ngueko, Brittany Plummer, Lathish, Randa Maher, Taka Cho, Xdy Shang, laura Gugliotta, Prabhu
Discussion
CLAMP has some concerns about being able to implement guard policies in Casablanca time frame
May want to use quiesce (gracefully drain traffic) only at a VNF Level
Controllers do not have the capability to help in optimization decision
Action Items
August 14, 2018
Agenda
SO Scaleout API (what is mandatory vs optional)
Round Table
Partial Recording
Attendees
Scott Blandford, Marco Platania, Brittany Plummer, Pamela Dragosh, Prabhu, Lathish, Sebastien Determe, Michael Zinnikas, Prema Bhatt, Ofir Sonsino, Randa Maher
Discussion
SO Scaleout API
Which API Parameters are required vs which ones are optional?
Policy currently uses a named query to get the required parameters for the request from AAI
The real fix will come in Dublin when the LCM actions are modeled in SDC
Until then we will continue to use the solution that was first used in R1 to get the required parameters
Generic Resource API
It was pointed out that during the Aug 13 Use Case Subcommittee meeting the availability of the Generic Resource API was questioned
Scott will follow up with Seshu and Steve to be sure we are all in sync
Action Items
August 7, 2018
Agenda
Review Roadmap
Review VID API - still needs the API Definition
Review Beijing Shortcuts
Development Tracking
Round table
Recording
Attendees
Scott Blandford, Marco Platania, Yuri Malakov, Steve Kozakiewicz, Jacques Bolduc, David Ho, Prema Bhatt, Lauren Lewis, Ruchira Agarwal, Taka Cho, Randa Maher, Vivian Ye, Brittany Plummer, Laura Gugliotta, Prabhu
Action Items
Add platform level features to roadmap
add 8 sub tasks to SO Epic in the development tracking page
Send note to Seshu to have them update the Casablanca Risks page
July 31, 2018
Agenda:
Detailed Review of Workflow
Need to define the VID API to SO
Attendees
Scott Blandford, Marco Platania, Steve Smokowski, Lthish, Seshu, Ruchira Agarwal, Prittany lummer, Jacques Bolduc, Prabhu, Shankar, Narayanan, Steve Kozakiewicz, jarmstrong001, Prema Bhatt, Lauren Lewis, Taka Cho, Pamela Dragosh, David Ho, Jayant Ragde, Martial Ngueko, Ofir Sonsino, Vivien Ye. Discussion
Action Items
Update Workflow with information from Marco's Document and Lauren's Write up
Create Roadmap and features for future releases
Set up a meeting with VID, SO, Policy and CLAMP to finalize SO API for VID and Policy
July 17, 2018
Agenda
Review new Workflow diagram (Scott)
Review Tracking page (Scott)
Generic Configuration (Marco)
Workflow detail (Lauren)
Recording
Attendees
Scott Blandford, Lathish, Lauren Lewis, Marco Platania, Pamela Dragosh, Ruchira Agarwal, Taka Cho, Yuriy Malakov, Prabhu Somasundaram, Jayant Ragde, Prema Bhatt, jarmstrong001, Martial Ngueko, Laura Gugliotta, Brittany Plummer, Steve Kozakiewicz, Ofir Sonsino
Discussion
Need to policy-enable VID so it can get configuration parameters from Policy when Manual Scale Out is called. This needs to be explored for future releases
Option 1 from Marco's document (included below) has been decided on for Casablanca (Adding all configuration parameters to VID/CLAMP to be passed through SO to the controllers)
Need to determine all the impacts to projects and the workflow
Option 2 still needs to be discussed more to determine if it is the right strategy for future releases
VF Module Scale Out Use Case for ONAP Casablanca Release.docx
Action Items
Set up a meeting to flesh out requirements for option 1
Set up a meeting to flesh out requirement for Healthcheck
July 10, 2018
Agenda
Review the July 9 Use Case Subcommittee Meeting
Overview of Marco's work on work for generic configuration parameters
Go through work flow and add more detail (Lauren)
Recording
Attendees
Scott Blandford, David Ho, Lauren Lewis, Steve Smokowski, Adam Krysiak, jarmstrong001, Lathish, Mark Ho, Michael Z, Ruchira, Ruchira Agarwal, Taka Cho, Marco Platania
Discussion
the Beijing Tracking page is being used as the template for all Use Case tracking with a few modifications
Gliffy should be used by all Use Cases for Sequence Diagrams
Use Cases should include both detailed and summary Sequence diagrams using Gliffy
All Scaling Checks (ie. Prov_Status, Orchestration_status, Min/Max instances) should be done by SO, but may also be done by other components in addition to SO
Need to add a requirement for SO to check against Min and Max number of instances before initiating scaling action
Need to define the event name an even structure between Policy and SO to pass the required parameters
Need to define and implement Rainy Day handling Requirements (This is a stretch goal for Casablanca and may be pushed to a future release)
Need to determine if we are using pre-loaded configuration data and if so, then how is it used and implemented
Action Items
Set up another meeting to continue detailing the workflows for scaling
July 3, 2018
Agenda
M1 Status
PTLs begin entering EPICs and User Stories
Schedule EPIC reviews for next week with PTLs
Review Controller Type meeting
We will begin sending emails to track exact APIs and parameters
Go over workflow to get API and Parameter Details
Recording:
Attendees:
Scott Blandford, Marco Platania, Steve Smokowski, Anatoly (Nokia), Lauren Lewis, Adam Krysiak, David Ho, Randa Maher, Hector Anapan, Michael Z (AT&T), Ofir Sonsino (AT&T), Jayant (AT&T), Rahul G, Taka Cho
Action Items:
On Casablanca Requirements wiki change "Beijing Fixes" feature to "Beijing Improvements"
Set up a meeting with Marco, SDNC, SO, AAI and APPC to ensure that IP Addresses are written to AAI by SDNC/SO during resource assignment and instantiation
Add meeting minutes to wiki
June 5, 2018
Agenda
Beijing Status
Beijing Learnings
Casablanca Goals with Beijing fixes
New sequence diagram
Recording:
Slides
Attendees:
Scott Blandford, Prabhu, Matti Hiltunen, Marco Plantania, Shankar Narayanan, Lauren Lewis, Jayant, Scott Seabolt, Randa Maher, Ruchira Agarwal, Ajay.
Discussion
Lauren going through email of beijing issues
Min/max comparison in VID. The values were added to the vDNS model; Values need to be added for all VNFs that want to be scaled
Missing records in A&AI
Prov Status
This is manual step that we fixed by using Robot
Update User Guide to ensure that provider knows to update Prov_Status
IP addresses - affected HealthCheck and ConfigScaleOut
IP Address should be updated in A&AI when SDNC spins up new instance
They may be in the wrong location of A&AI (Where is the correct location?)
Very difficult to get the information.
VID does not have memory of the number of VF Modules that it scales
Issue being addressed with VID
VF_Module name appends the instance number
Need to address this solution for Scale In issues
Missing payload in APP-C request from SO - Requirements Gap
Is Config data passed to controller or does controller retrieve it?
Controller type not being sent correctly from VID to SO
Controller_type should be in VNF Model (talk to SDC)
There may be issues with SDC team
In Beijing SO ignored value from VID and just hardcoded APPC
Messages not being posted to DMaaP
Not sure if this a larger issue or if this is just something we stumbled on
DMaaP says that the topic needs to be created before the message is sent
Create a Jira item that describes issue we had.
SDNC delivered partial Scale Out Requirements
SDNC needs to inherit APPC functionality for Casablanca
Action Items
May 29, 2018
Agenda
Beijing Status
Recording:
Slides
Attendees:
Marco Platania, Scott Seabolt, Anatoly (Nokia), Michael Z (AT&T), Lauren Lewis
Discussion
Manual Scale Out Use Case is currently being tracked as a possible working Beijing Use Case
Much progress has been made over the past week.
Catherine Lefevre has been leading the charge to get the UC included in Beijing
All VID issues appear to be fixed
A Work Around is being developed for getting the IP Addresses to APPC for configuration of the VNF
The IP Addresses are not in AAI. (SDNC will need to add these addresses in Casablanca)
SDNC will send the IP Addrs directly to SO after assignment
SO will include the addresses in its ConfigScaleOut() call to APPC/SDNC via DMaaP
There is still some work to be done:
SO still needs to construct the ConfigScaleOut() Request and send it to DMaaP
APPC/SDNC needs to test this request
Healthcheck still needs to be tested.
Current status can be found on the Testing Status Page
Action Items
May 22, 2018
Agenda
Beijing Status
Recording:
Slides
Attendees:
Discussion
Who sets prov-status? Should it be SDNC? Yes! Prabhu is looking into this
VID has disabled the filter
Forward JIRA ticket to Prabhu
APPC config data
APPC is expecting configuration file to be passed from VID through SO to APPC
Not enough time available to add new features
APPC does not have enough time to find config data in AAI (Preferred fix). Will need to push to Casablanca
SO/VID does not have enough time to create pass-through of manual configuration data
Team to continue to test as much as possible within Beijing and report status on Wiki Page
Action Items
May 15, 2018
Agenda
Beijing Status
Recording:
Slides
Attendees:
Discussion
Creating APPC VNF Template
Passed a few blockers with VID
VID Not showing some required options
APPC should have the VMs needed to do testing
Need to ensure that VID and SO and SDNC pieces are working
APPC
Used CDT tool to create a good template
Template works and pushes items to APPC
Netconf is opened, bu there is a failure soon after it is called
Could be some issue with Netconf XML
Is IPADDR include din Payload of ConfigScaleOut request? APPC is modeling it as if it is.
SO is not including the ip_addrs in the payload. Not part of initial requirements
Action Items
May 8, 2018
Agenda
Beijing
Development
Testing
Formalize testing???
Casablanca
SO
SO - OOF
Slides
Scaling Use Case weekly MTG 180508.pptx
Attendees:
Ajay, David Ho, Lauren Lewis, Marco, Ofir, Steve Smokowski, Michael Z, Pam Dragosh, Sanchita Pathak, Shankar Narayanan, Anatoly (Nokia)
Discussion
Many of the Beijing blockers have been fixed. We should be able to move ahead on Scaling tests today.
Some current problems with APPC
When does this need to work? RC2 5/16
Need to formalize testing tracking
See what Helen is doing for tracking
Ofir (vid blocked for over 2 weeks) -
Can we instantiate a VF_Module?
Marco - Yes, we should be able to do it
Casablanca
InstanceName: Needs to be passed or generated by a controller
Does Cloud Region need to be passed? No
ModelCoustomizationId, OptionalCloudConfiguration, VolumeInstanceID, InstanceID
OOF
Need to build SDNC interface
Capacity checks done through Multi-Cloud VERY Basic
Need to add multicloud to call flow for capacity checks. Not A&AI
WorkFlow Designer - Not signing up for in Casablanca. SO Code will not be ready for real Use Cases