CPS-1733: Upgrade YANG Schema-Set for CM Handle Using a Module Set Tag

CPS-1733 - Getting issue details... STATUS

CPS-1734 - Getting issue details... STATUS

Study is required to clearly define the scope and impacts of updating YANG model schema sets for cm-handles. 

Requirements

Functional

Error Handling

Characteristics

Performance is posted here :  Performance test for updating YANG schema set API

Out-of-scope

• Upgrade of models for cached data. "ncmp-datastore:operational" is out-of-scope.  

 Note: Only pass-through i.e. non-cached data upgrade is in scope; "ncmp-datastore:passthrough-running" and "ncmp-datastore:passthrough-operational" is in scope.

Assumptions

Issues & Decisions

Solution Proposals 

Update Inventory REST Interface

Making the module upgrade request either over REST or Kafka Event with a supplied 'moduleSetTag' property will indicate to NCMP that a cmhandle has a new moduleSet (node has been upgraded).  The moduleSetTag is a unique identifier for the set of modules associated with a cmhandle. This event or rest request will trigger ncmp to either retrieve the new module set for this cmhandle from the dmi plugin or if NCMP is already aware of moduleSetTag as part of a previous retrieval for a set of modules for a cmhandle that supplied this moduleSetTag then it should not re-retrieve the modules from the dmi plugin as it already has them.  NCMP should use its stored moduleSetTag to get the set of modules.  It can then do a delta with the existing cmhandle moduleSet and update the moduleSet for the cmhandle.

The above proposal would require model updates to NCMP to store the association between a moduleSetTag supplied by a dmi-plugin (during and updateCmHandles request) and the modules.  Initially moduleSetTag is not known to NCMP.  However, after retrieval of the modules for the first time for the cmhandle, NCMP can store the association.  Any subsequent cmhandle that references the same moduleSetTag will not require NCMP to go back to the dmi plugin for the moduleSet.

Reuse the NCMP Inventory API (CPS-NCMP-I-01). URI : POST /v1/ch

{
    upgradedCmHandles : [      
      {
         "cmhandle" : "<cmhandle-id-1>",
         "moduleSetTag" : "ffsdfg55342"   #  new moduleSetTag ffsdfg55342 for the cmhandle        },       
      {
         "cmhandle" : "<cmhandle-id-2>",
         "moduleSetTag" : "ffsdfg55342"   #  new moduleSetTag ffsdfg55342 for the cmhandle        },     
      { 
         "cmhandle" : "<cmhandle-id-3>",    
         "moduleSetTag" : "ddger34324"    #  new moduleSetTag ddger34324 for the cmhandle         }
  ]
}

Proposed Inventory Model rev. 2023-08-23

Lines added: 11-14, 83-85

module dmi-registry {

  yang-version 1.1;

  namespace "org:onap:cps:ncmp";

  prefix dmi-reg;

  contact "toine.siebelink@est.tech";

  revision "2023-08-23" {
    description
    "Added ModuleSetTag";
  }
 
  revision "2022-05-10" {
    description
    "Added DataSyncEnabled, SyncState with State, LastSyncTime, DataStoreSyncState with Operational and Running syncstate";
  }

  revision "2022-02-10" {
    description
    "Added State, LockReason, LockReasonDetails to aid with cmHandle sync and timestamp to aid with retry/timeout scenarios";
  }

  revision "2021-12-13" {
    description
    "Added new list of public additional properties for a Cm-Handle which are exposed to clients of the NCMP interface";
  }

  revision "2021-10-20" {
    description
    "Added dmi-data-service-name & dmi-model-service-name to allow separate DMI instances for each responsibility";
  }

  revision "2021-05-20" {
    description
    "Initial Version";
  }

  grouping LockReason {
    leaf reason {
      type string;
    }
    leaf details {
      type string;
    }
  }

  grouping SyncState {
   leaf sync-state {
     type string;
   }
   leaf last-sync-time {
     type string;
   }
  }

  grouping Datastores {
    container operational {
      uses SyncState;
    }
    container running {
      uses SyncState;
    }
  }

  container dmi-registry {
    list cm-handles {
      key "id";
      leaf id {
        type string;
      }
      leaf dmi-service-name {
        type string;
      }
      leaf dmi-data-service-name {
        type string;
      }
      leaf dmi-model-service-name {
        type string;
      }
      leaf module-set-tag {
        type string;
      }

      list additional-properties {
        key "name";
        leaf name {
          type string;
        }
        leaf value {
          type string;
        }
      }

      list public-properties {
        key "name";
        leaf name {
          type string;
        }
        leaf value {
          type string;
        }
      }

      container state {
        leaf cm-handle-state {
          type string;
        }

        container lock-reason {
          uses LockReason;
        }

        leaf last-update-time {
          type string;
        }

        leaf data-sync-enabled {
          type boolean;
          default "false";
        }

        container datastores {
          uses Datastores;
        }
      }
    }
  }
}

Upgrade With Module Set Tag

1. Use new 'upgradedCmHandles' operation to upgrade CH-1
2. Find a cmHandle with given 'moduleSetTag' 
   (if not found use algorithm defined in next section)
3. Get all module references for cm handle (schemaset) with same tag (CH-2) 
4. Update module references for the anchor/schemaset CH-1
   (See open issue # 12:  should we create a NEW schema set and delete the old one ?)
5. Update inventory for CH-1 with given module set tag

Support for Upgrade Without Module Set Tag

Note. Same algorithm should be used when Module Set is not set (leaf not present), blank or no other cmHandle with same tag can be found

If the moduleSetTag json property is set to "" (empty string) or then it should also indicate that the moduleSet for a cmHandle has been updated but there is no associated moduleSetTag available for that cmHandle.  This approach will always result is a full request to the dmi plugin for the module set for the cmHandle.

Basically the same steps as during initial inventory can be followed. Except the creation of the cmHandle (anchor) as that already exist.
Refer to org.onap.cps.ncmp.api.inventory.sync.ModuleSyncTasks#performModuleSync for the relevant code

1. get all module references from the (upgraded) cm handle (via DMI)
2. find out which module references are NEW to CPS-Core
3. get the yang resources for the new modules from the (upgraded) cm handle (via DMI)
4. create/save a new (TBC, issue #12) schema set using the existing module reference and new yang resources
5. (new) update the schema set for the upgraded cm handle
6. (new) set the moduleSetTag in the ncmp inventory for the upgraded cm handle (mabey this needs to be done at the start?)
7. (new) delete the old schema set of the upgraded cm handle (depend on decision re issue #12)

Use ModuelSyncWatchDog (parallel processing)

Initial inventory is driven by (batches of) Cm Handle state changes on the main thread and then the org.onap.cps.ncmp.api.inventory.sync.ModuleSyncWatchdog processes those!

Examining this in detail lead to some additional design questions

1. should upgrade be dealt with in parallel too
2. should upgrade be done in batches (of 100) too?
3. need to use a (new?) state for UPGRADE that the watchdog can see an process
4. re-use existing methods and make creating aan anchor optional or create new upgrade specific methods and reuse/slightly duplicate initial inventory methods? Methods like
   1. org.onap.cps.ncmp.api.inventory.sync.ModuleSyncService#syncAndCreateSchemaSetAndAnchor
   2. org.onap.cps.ncmp.api.inventory.sync.ModuleSyncService#createSchemaSetAndAnchor

31 Aug 2023 team agreed to re-use ModuleSyncWatchDog  and batching with following considerations

1. Need to use a new (shared) Hazelcast map with ModuleSetTags as key (value list of module refs) that have been processed (but not saved yet) to be used both inside a batch and different instance to prevent unnecessary trips to DMI/Node
2. Watchdog needs to use Lock state AND lock reason to determine what node need to be upgraded
3. Initial inventory and upgrade is not likely to happen at the same time but watchdog can handle both, of course performance would be affected if that does occur
4. CACHE(s) needs to be cleared or updates as algorithm wil re-use existing schema set name
5. Create new upgrade specific methods and reuse/slightly duplicate initial inventory methods
6. Legacy (and new) checks for lock need to check the lock-reason too now! To differentiate between failed initial inventory and upgrade
7. Probably need more specific (new) failure reasons to differentiate between initial inventory and upgrade failures
8. as usual: small commits, early reviews to introduce all this functionality are advised, posisbel steps
   1. set lock state and reason upon request
   2. watch dog just list to be upgrade node (and does not mix them up with failed initial inventory)
   3. perform first upgrade for a new ModuleSetTag
   4. introduce and use new Hazelcast map
   5. perform upgrade of an node with a modueleSetTag that is already in cache
   6. perform upgrade of an node with a modueleSetTag that is already in DB (ie introduce DB query)
   7. handle failure of upgrade (re-use same retry mechanism as initial inventory but with different lock reason!)
   8. etc.

Use-Case Overview (Sync in watchDog)

Note. Error handling like invalid IDs are handled during the Synchronous part of registration and not part of this use-case overview. 
Upgrade to the same Tag should be captured in synchronous pre-processing.

re-usable methods

1. Gget modules (delta) from Node (DMI)
2. Create schemaset. Combination of exiting Refs and new yang resources
   New yang resources empty for known module sets (known tag)
3. Create anchor
4. Update schemaset. Combination of exiting Refs and new yang resources
   New yang resources empty for known module sets (known tag)

Changes agreed 15 Feb 2024 between Daniel Hanrahan , Sourabh Sourabh and Toine Siebelink 

void processCreate() {
   if (tagProvided && tag in DB) {
          allModuleRefences = referencesFromDb
          newYangResources = []
       else {
          delta = getModulesDelta()
          newYangResources = delta.newYangResources 
          allModuleRefences = delta.allModuleRefences 
    }
    creatSchemaSet(newYangResources,  allModuleRefences)
    createAnchor()
}       
     
void processUpgrade() {
    if (tagProvided && tag in DB) {
          allModuleRefences = referencesFromDb
          newYangResources = []
       else {
          delta = getModulesDelta()
          newYangResources = delta.newYangResources 
          allModuleRefences = delta.allModuleRefences 
    }
    udpateSchemaSet(newYangResources,  allModuleRefences)
}

Proposed JIRAs

Planning:

• Allow for 2 more user stories each may take 1 week. 
• Estimated date of completion is  15 Nov 2023 based on 1 person working. There is a possibility that 1 more person will join in between for implementation but there is no clear visibility on when the person can start contributing as of now. 
• These are estimates and should not be interpreted as commitments.