Scaling Meeting Minutes

October 23, 2018

Agenda

  1. Casablanca Testing

  2. Casablanca Building Blocks in SO

  3. Dublin Planning

Attendees

Scott Blandford, Yuriy Malakov, Jacques Bolduc, Lathish, Prabhu, Ramja Balaji, Joss Armstrong, Brittany Plummer, Lauren Lewis, Prema Bhatt, Lusheng Ji 

Discussion

  • Casablanca Testing

    • Completed testing of Design Time flow in K8s environment

    • Run Time testing will begin Wednesday

  • Casablanca Building Blocks

    • Added Building blocks to Casablanca Work Flow

    •  Need to add building blocks layer to Dublin Work Flow

  • Dublin Planning

    • Met with APPC Team to review Scale In Requirements 

Action Items

Scott to add building blocks layer to Dublin workflow 

October 16, 2018

Agenda

  1. Casablanca Testing

  2. Dublin Workflow

  3. Dublin Requirements 

Attendees

Scott Blandford, Lauren Lewis, Marco Platania, Lathish, Prabhu, Daurban, Joss Armstrong, Prema Bhatt 

Discussion

  • Casablanca Testing

    • No Progress in the past week

      • K8s environment still not working (Hopefully, it will be functioning within the next couple days)

      • Marco has been busy with other projects

      • Marco will continue testing tomorrow assuming the test environment is ready to go

  • Dublin Planning

    • Building Block meeting with SO team

      • SO is building workflows so that they can be easily transformed into  Building blocks for Work Flow Designer

      • Scott will try to organize the work flow diagrams to depict the Building Blocks

    • Manual A&AI entries for Policy

      • Need to have consult with Pam (Policy) to determine if Policy can implement the new SO API so that no manual AAI entries will be required.

Action Items

Scott to add Building Blocks to Flow diagrams
Marco to check with Pam about use of new SO API for Dublin  



October 9, 2018

Agenda

  1. Casablanca Testing

  2. Dublin Workflow

  3. Dublin Requirements 

Attendees

Scott Blandford, Lathish, Taka Cho, Marco Platania, Sebastien Determe, Martial Ngueko 

Recording

ONAP Scale Out 181009.mp4

Discussion

  • Testing only being done in SB07 at this time since OOM is not fully working

  • We should merge ConfigScaleIn and ConfigScaleOut

    • Maybe merge to ConfigModify but at least to ConfigScaling

  • We will need to support an Ansible port on vDNS 

Action Items

October 2, 2018

Agenda

  1. Casablanca Testing Status

  2. Dublin Workflow

  3. Dublin Requirements 

Attendees

Scott Blandford, Wojciech Sliwka, Lukasz Grech, Taka Cho, Marco Platania, Lauren Lewis, Yuriy Malakov, Prema Bhatt, Prabhu 

Recording

ONAP Scale Out 181002.mp4

Discussion

  • Testing (Marco)

    • Early stages.  Still trying to get platform stable 

    • Tried pushing policies from CLAMP to Policy

      • Guard Policies did not work.  A fix is being developed

    • Marco will send more updates to Scott

  • Dublin Work Flow

  •  

    •  Scale In VM Decision

      • Decided to use LIFO for Dublin

      • OOF is the right strategy for making the decision

      • Should Policy, OOF, or SO determine the last VM added to the pool?

    • Pre Action

      • We will send the VNF a Pre-action signal for Scale In to allow the VNF to take care of VNF specific Tasks before the scale in action is done.

      • SO will wait for a response from the VNF before moving on to the next action in the workflow

    • Moving Traffic

      • Distribute_traffic will be used to move new traffic away from target VM

      • Quiesce_traffic will be used to signal when all existing traffic has been drained from target VM

      • Both Distribute_Traffic and Quiesce_Traffic will need to be modified to work at the VF_Module/VNFC level

    • Configuration

      • Not much changes here except for incorporating the use of Auto Configuration.  This will need to be done for Scale Out also

      • CDS should be ready in Dublin timeframe

      • We will need a new controller API for "ConfigScaleIn"

    •  Cleanup

    •  

      • Need to do a STOP VM before we terminate the VM so we do not corrupt any resources

      • All resources will be released and the next scale out will likely get a different set of resources.

  • Dublin project level requirements

Action Items

Send Yuriy and Lauren the slides on Controller_Type selection and meeting logistics  
Set up meeting on Automating remaining Manual tasks with Pam, Yuriy, Steve S, Marco and Lauren.



September 25, 2018

Agenda

  1. Casablanca Developer Issues

  2. Casablanca Testing (Marco)

  3. Dublin planning

    1. Scaling Prep and Post phases 

Attendees



Scott Blandford, Latish, Marco Platania, Lauren Lewis, Jacques Bolduc, Joss Armstrong, David Ho, Taka Cho, Piotr Krysiak, Sebastien Determe, Brittany Plummer, Milind Phadke, Vivian Ye

Recording

ONAP Scale Out 180925.mp4

Discussion

  • Due to extended holiday in Israel, the VID code developed by Nokia has not been reviewed and merged.  Marco is following up with Catherine and others to figure out a work around

  • Testing

    • Helen has asked for more detailed test cases

    • Marco will document tests he is running and then send them to Scott to add to the wiki page

  • Prep and Post phases for Scale In

    • Scott will get back to Liu (Nokia) to explain what should be done by ONAP vs what should be done by the VNF.  once this is established we will need to determine if the VNF needs to be signaled for each phase and if a response is needed. 

Action Items

Update Test Status page with test cases that Marco runs
send email to Liu explaining what kinds of actions are run by ONAP vs those run by the VNF  
 

September 18, 2018

Agenda

  1. Casablanca Status

    1. M4 update

    2. Round table

  2. Dublin Planning

    1. Preparation Phase 

Attendees

Scott Blandford, Marco Platania, Brittany Plummer, Jacques Bolduc, Prabhu, Wochiech Sliwka, Lauren Lewis, Sebastien Determe, jarmstrong001, Taka Cho, Yuriy Malakov,  Lukasz Rajewski, Prema Bhatt, Pam Dragosh

Recording

ONAP Scale Out 180918.mp4

Discussion

  • Casablanca M4 Status

    • CLAMP - Martial - On Track 

    • Policy - Marco working with Policy - everything should be submitted.  Pam - everything should be fine

    • SDNC - Marco touched base with Ruchira, everything is on track

    • SO - Brittany - all code is submitted

      • Discussion - We may need a delay between ConfigScaleOut and the final Healthcheck

    • VID - Wochiech Sliwka - Problems with Testing Environment (Marco suggested they use Windriver), All code will be pused prior to M4

  • Marco will run testing for Scaling Use Case.  Communication will be via email, Q, and possibly RocketChat

  • Integration Testing will start after M4

  • Integration Testing will run in SB07 and SB04 test environments 

  • Dublin (Prep phase for scaling actions)

  •  

    • So far everyone seems ok with this

Action Items

Scott to set up a meeting with Marco and Lauren to discuss time delay between ConfigScaleOut and final Healthcheck  



September 11, 2018

Agenda

  1. Casablanca Status

    1. M4 Code Freeze (9/20)

    2. Testing Status 

  2. Dublin Planning

    1. Go over current Scale In Flow

    2. Redistribute_Traffic Discussion 

Attendees

Scott Blandford, Brittany Plummer, David Ho, Jacques Bolduc, jarmstrong001, Lukasz Rajewski (O-PL), Michael Z (AT&T), Shankar Narayanan, Taka Cho, Marc Fiedler (DT), Milind Phadke, Randa Maher (AT&T), Prema Bhatt (AT&T), Lauren Lewis, Ajay Mahimkar, Martial Ngueko

Recording

ONAP Scale Out 180911.mp4

Discussion

Action Items

Scott to update sequence diagram to change Scale Out to Scale In  

August 28, 2018

Agenda

  1. Discuss Policy concerns

  2. Round Table on R3 status

  3. Daniel Urban (Nokia) wants to help

  4. China Mobile updates to roadmap

  5. Dublin Scale In Discussion 

Attendees

Scott Blandford, Prabhu, Shankar Narayanan, Daniel Urban, Brittany Plummer, Martial Ngueko, Taka Cho, David Ho, Jacques Bolduc, Pam Dragosh, Lauren Lewis

Recording

ONAP Scale Out 180821.mp4

Discussion

  • Pam will work with Jim Hahn to determine if he is still having issue with the policy requirements

  • All projects are currently on target to implement the R3 requirements for Scale Out

  • Pam will engage Daniel Urban on Control Loops

  • Scale In Discussion:

    •  We discussed the Scale in Work Flow described on this wiki page

      • Pam will work with Policy team to determine if Policy should make the decision about which instance to remove

      • Scott should work with Brian Freeman and Architecture comm to determine which component should make the decision on which instance

      • Pam will look into monitoring the target instance to determine when it is inactive and sending the signal to SO

Action Items






August 21, 2018

Agenda

  1. M3 Milestone (API Freeze)

  2.  Casablanca Status

  3. Scale In work flow for Dublin

Attendees

Scott Blandford, Lauren Lewis, Martial Ngueko, Brittany Plummer, Lathish, Randa Maher, Taka Cho, Xdy Shang, laura Gugliotta, Prabhu

Discussion

  • CLAMP has some concerns about being able to implement guard policies in Casablanca time frame

  • May want to use quiesce (gracefully drain traffic) only at a VNF Level

  • Controllers do not have the capability to help in optimization decision 

Action Items

August 14, 2018

Agenda

  1. SO Scaleout API (what is mandatory vs optional)

  2. Round Table 

Partial Recording

ONAP Scale Out 180814.mp4

Attendees

Scott Blandford,  Marco Platania, Brittany Plummer, Pamela Dragosh, Prabhu, Lathish, Sebastien Determe, Michael Zinnikas, Prema Bhatt, Ofir Sonsino, Randa Maher

Discussion

  • SO Scaleout API

    • Which API Parameters are required vs which ones are optional?

    • Policy currently uses a named query to get the required parameters for the request from AAI

    • The real fix will come in Dublin when the LCM actions are modeled in SDC

    • Until then we will continue to use the solution that was first used in R1 to get the required parameters

  •  Generic Resource API

  •  

    • It was pointed out that during the Aug 13 Use Case Subcommittee meeting the availability of the Generic Resource API  was questioned

    • Scott will follow up with Seshu and Steve to be sure we are all in sync

Action Items




August 7, 2018

Agenda

  1. Review Roadmap

  2. Review VID API - still needs the API Definition

  3. Review Beijing Shortcuts

  4. Development Tracking

  5. Round table 

Recording

ONAP Scale Out 180807.mp4

Attendees

Scott Blandford, Marco Platania, Yuri Malakov, Steve Kozakiewicz, Jacques Bolduc, David Ho, Prema Bhatt, Lauren Lewis, Ruchira Agarwal, Taka Cho, Randa Maher, Vivian Ye, Brittany Plummer, Laura Gugliotta, Prabhu

Action Items

  1. Add platform level features to roadmap

  2. add 8 sub tasks to SO Epic in the development tracking page

  3. Send note to Seshu to have them update the Casablanca Risks page 

July 31, 2018

Agenda:

  1. Detailed Review of Workflow

  2. Need to define the VID API to SO

Attendees

Scott Blandford, Marco Platania, Steve Smokowski, Lthish, Seshu, Ruchira Agarwal, Prittany lummer, Jacques Bolduc, Prabhu, Shankar, Narayanan, Steve Kozakiewicz, jarmstrong001, Prema Bhatt, Lauren Lewis, Taka Cho, Pamela Dragosh, David Ho, Jayant Ragde, Martial Ngueko, Ofir Sonsino, Vivien Ye. Discussion

Action Items

  1. Update Workflow with information from Marco's Document and Lauren's Write up

  2. Create Roadmap and features for future releases

  3. Set up a meeting with VID, SO, Policy and CLAMP to finalize SO API for VID and Policy




July 17, 2018

Agenda

  1. Review new Workflow diagram (Scott)

  2. Review Tracking page (Scott)

  3. Generic Configuration (Marco)

  4. Workflow detail (Lauren)

Recording

ONAP Scale Out 180717.mp4

Attendees

Scott Blandford, Lathish, Lauren Lewis, Marco Platania, Pamela Dragosh, Ruchira Agarwal, Taka Cho, Yuriy Malakov, Prabhu Somasundaram, Jayant Ragde, Prema Bhatt, jarmstrong001, Martial Ngueko, Laura Gugliotta, Brittany Plummer, Steve Kozakiewicz, Ofir Sonsino

Discussion

  • Need to policy-enable VID so it can get configuration parameters from Policy when Manual Scale Out is called.  This needs to be explored for future releases

  •  Option 1 from Marco's document (included below) has been decided on for Casablanca (Adding all configuration parameters to VID/CLAMP to be passed through SO to the controllers) 

    • Need to determine all the impacts to projects and the workflow

  • Option 2 still needs to be discussed more to determine if it is the right strategy for future releases 

VF Module Scale Out Use Case for ONAP Casablanca Release.docx

Action Items

  1. Set up a meeting to flesh out requirements for option 1

  2. Set up a meeting to flesh out requirement for Healthcheck 

July 10, 2018

Agenda

  1. Review the July 9 Use Case Subcommittee Meeting

  2. Overview of Marco's work on work for generic configuration parameters

  3. Go through work flow and add more detail (Lauren)

Recording

ONAP Scale Out 180710.mp4

Attendees 

Scott Blandford, David Ho, Lauren Lewis, Steve Smokowski, Adam Krysiak, jarmstrong001, Lathish, Mark Ho, Michael Z, Ruchira, Ruchira Agarwal, Taka Cho, Marco Platania

Discussion

  • the Beijing Tracking page is being used as the template for all Use Case tracking with a few modifications

  • Gliffy should be used by all Use Cases for Sequence Diagrams

  • Use Cases should include both detailed and summary Sequence diagrams using Gliffy

  • All Scaling Checks (ie. Prov_Status, Orchestration_status, Min/Max instances) should be done by SO, but may also be done by other components in addition to SO

  • Need to add a requirement for SO to check against Min and Max number of instances before initiating scaling action

  • Need to define the event name an even structure between Policy and SO to pass the required parameters

  • Need to define and implement Rainy Day handling Requirements (This is a stretch goal for Casablanca and may be pushed to a future release)

  • Need to determine if we are using pre-loaded configuration data and if so, then how is it used and implemented

Action Items

  1. Set up another meeting to continue detailing the workflows for scaling 

July 3, 2018

Agenda

  1. M1 Status

  2. PTLs begin entering EPICs and User Stories

    1. Schedule EPIC reviews for next week with PTLs

  3. Review Controller Type meeting

  4. We will begin sending emails to track exact APIs and parameters

  5. Go over workflow to get API and Parameter Details

Recording:

ONAP Scale Out 180703.mp4

Attendees:

Scott Blandford, Marco Platania, Steve Smokowski, Anatoly (Nokia), Lauren Lewis, Adam Krysiak, David Ho, Randa Maher, Hector Anapan, Michael Z (AT&T), Ofir Sonsino (AT&T), Jayant (AT&T), Rahul G, Taka Cho

Action Items:

  1. On Casablanca Requirements wiki change "Beijing Fixes" feature to "Beijing Improvements"

  2. Set up a meeting with Marco, SDNC, SO, AAI and APPC to ensure that IP Addresses are written to AAI by SDNC/SO during resource assignment and instantiation

  3. Add meeting minutes to wiki

June 5, 2018

Agenda

  1. Beijing Status

  2. Beijing Learnings

  3. Casablanca Goals with Beijing     fixes

  4. New sequence diagram

Recording:

ONAP Scale Out 180605.mp4

Slides

Attendees:

Scott Blandford, Prabhu, Matti Hiltunen, Marco Plantania, Shankar Narayanan, Lauren Lewis, Jayant, Scott Seabolt, Randa Maher, Ruchira Agarwal, Ajay.

Discussion 

  • Lauren going through email of beijing issues

  •  

    1. Min/max comparison in VID.  The values were added to the vDNS model; Values need to be added for all VNFs that want to be scaled

    2. Missing records in A&AI 

    3.  

      1. Prov Status   

      2.  

        1. This is manual step that we fixed by using Robot 

        2. Update User Guide to ensure that provider knows to update Prov_Status

      3. IP addresses - affected HealthCheck and ConfigScaleOut   

      4.  

        1. IP Address  should be updated in A&AI when SDNC spins up new instance    

        2.  

          1. They may be in the wrong location of A&AI (Where is the correct location?)  

          2.  

            1. Very difficult to get the information.

    4. VID does not have memory of the number of VF Modules that it scales

    5.  

      1. Issue being addressed with VID

      2. VF_Module name appends the instance number

      3. Need to address this solution for Scale In issues

    6. Missing payload in APP-C request from SO - Requirements Gap

    7.  

      1. Is Config data passed to controller or does controller retrieve it?

    8. Controller type not being sent correctly from VID to SO

    9.  

      1. Controller_type should be in VNF Model (talk to SDC)

      2. There may be issues with SDC team

      3. In Beijing SO ignored value from VID and just hardcoded APPC

    10. Messages not being posted to DMaaP

    11.  

      1. Not sure if this a larger issue or if this is just something we stumbled on

      2. DMaaP says that the topic needs to be created before the message is sent

      3. Create a Jira item that describes issue we had.

    12. SDNC delivered partial Scale Out Requirements

    13.  

      1. SDNC needs to inherit APPC functionality for Casablanca

Action Items

May 29, 2018

Agenda

  1. Beijing Status

Recording:

May 29, 2018 Recording

Slides

Attendees:

Marco Platania, Scott Seabolt, Anatoly (Nokia), Michael Z (AT&T), Lauren Lewis 

Discussion 

  • Manual Scale Out Use Case is currently being tracked as a possible working Beijing Use Case

  • Much progress has been made over the past week.

  • Catherine Lefevre has been leading the charge to get the UC included in Beijing

  • All VID issues appear to be fixed

  • A Work Around is being developed for getting the IP Addresses to APPC for configuration of the VNF

    • The IP Addresses are not in AAI.  (SDNC will need to add these addresses in Casablanca)

    • SDNC will send the IP Addrs directly to SO after assignment

    • SO will include the addresses in its ConfigScaleOut() call to APPC/SDNC via DMaaP

  • There is still some work to be done:

    • SO still needs to construct the ConfigScaleOut() Request and send it to DMaaP

    • APPC/SDNC needs to test this request

    • Healthcheck still needs to be tested. 

  •  Current status can be found on the Testing Status Page

Action Items

May 22, 2018

Agenda

  1. Beijing Status

Recording:

May 22, 2018 Recording

Slides

Attendees:

Discussion 

  • Who sets prov-status?  Should it be SDNC?  Yes! Prabhu is looking into this

  •  

    • VID has disabled the filter

    • Forward JIRA ticket to Prabhu

  • APPC config data

  •  

    • APPC is expecting configuration file to be passed from VID through SO to APPC

  •  Not enough time available to add new features

    • APPC does not have enough time to find config data in AAI (Preferred fix).  Will need to push to Casablanca

    • SO/VID does not have enough time to create pass-through of manual configuration data 

  • Team to continue to test as much as possible within Beijing and report status on Wiki Page

  •  

Action Items



May 15, 2018

Agenda

  1. Beijing Status

Recording:

May 15, 2018 Recording

Slides

Attendees:

Discussion

  • Creating APPC VNF Template

  • Passed a few blockers with VID

  • VID Not showing some required options

  • APPC should have the VMs needed to do testing

  • Need to ensure that VID and SO and SDNC pieces are working

  • APPC

    • Used CDT tool to create a good template

    • Template works and pushes items to APPC

    • Netconf is opened, bu there is a failure soon after it is called

      • Could be some issue with Netconf XML

    • Is IPADDR include din Payload of ConfigScaleOut request? APPC is modeling it as if it is.

    • SO is not including the ip_addrs in the payload.  Not part of initial requirements

    •  

Action Items



May 8, 2018

Agenda

  1. Beijing

  2.  

    1. Development

    2. Testing

    3. Formalize testing???

  3.  Casablanca

  4.  

    1. SO

  5. SO - OOF

Slides

Scaling Use Case weekly MTG 180508.pptx 

Attendees:

Ajay, David Ho, Lauren Lewis, Marco, Ofir, Steve Smokowski, Michael Z, Pam Dragosh, Sanchita Pathak, Shankar Narayanan, Anatoly (Nokia)

Discussion

  • Many of the Beijing blockers have been fixed. We should be able to move ahead on Scaling tests today.

  • Some current problems with APPC

  • When does this need to work?  RC2 5/16

  • Need to formalize testing tracking

  •  

    • See what Helen is doing for tracking

  • Ofir (vid blocked for over 2 weeks) -

  •  

    • Can we instantiate a VF_Module?

    • Marco - Yes, we should be able to do it

  • Casablanca

  •  

    • InstanceName: Needs to be passed or generated by a controller

    • Does Cloud Region need to be passed? No

    • ModelCoustomizationId, OptionalCloudConfiguration, VolumeInstanceID, InstanceID

    • OOF

    •  

      • Need to build SDNC interface

      • Capacity checks done through Multi-Cloud VERY Basic

        • Need to add multicloud to call flow for capacity checks. Not A&AI

    • WorkFlow Designer - Not signing up for in Casablanca.  SO Code will not be ready for real Use Cases

Action Items