Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Related Jira(s)

Jira Legacy
serverONAP System Jira
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId425b2b0a4733707d-557c2057-3c0c3a0f-b515ae5e-579789cceedb4fd8aff50176
keyCPS-1415


Issues and Decisions

#IssueNotes Decision
1how fast should CPS (and DB) be able to process max heart beat failuresis 60K really realistic if ENM goes down we should get a notification for each node do we ?!

Description

  1. Define scenarios which cause a CM Handle to go stale
  2. Implement changes to support tracking of CM Handle Freshness/Staleness

...

  1. dmi plugin identifies that the device is no longer contactable
  2. dmi plugin identifies that an underlying device manager managing the device (node) is out of sync with the device itself. 

Requirements

Functional

2DMI-I-01A Rest endpoint to trigger DMI Plugin ReregistrationAsynchronous interaction to trigger DMI Plugin to hit endpoint in CPS-NCMP-I-01 with reregistration
#InterfaceRequirementAdditional Information
1CPS-NCMP-I-01
A Rest endpoint to allow DMI Plugin Reregistration, A kafka interface for DMI Plugin to provide trust level state changes for a CM HandleReregistration is to reregister all CMHandles managed by a CMHandle. Kafka interface schema allows for CMHandle as id and trust level as only value in data
The 'trustlevel' can be queried (is visible) on the methos as currently the 'cm handle state'
2CPS-NCMP-I-01Once a CM Handle is registered (TBD which state exactly?) the trust -level for that CM Handle should be  reported to be 'COMPLETE'
3CPS-NCMP-I-01Once DMI (plugin) is detected to be down the trust-level for all affected CM Handles should be reported to be 'NONE'It might not need to be persisted....
4CPS-NCMP-I-01
REST or ASYNC TBD
DMI plugin can report the current trustlevel of a single (or collection?) of cm handle (id)sie. the DMI can tell NCMP the trustlevel is 'NONE' when a  node heartbeat failure is detected and 'COMPLETE' once it is restored
5
Notification on trustlevel changes ?!

Error Handling

...

#Error ScenarioExpected behavior
1DMI Plugin goes downCMHandles managed by that DMI have NONE trust level, when the DMI comes back up, a reregistration process occurs, CMHandles are individually assessed for trust level then.
2Node goes downDMI Plugin informs NCMP of the trust level state change. DMI will update on changes to a cmhandles trust level change.

Capabilities

  1. re-registration, once a day, same requirement as first time registration
  2. single node heart beat failures 30,000 / minute per instance
    1. Kafka should be able handle 2 instances i.e. 60K notifications in one minute
    2. What is expected/realistic here from CPS-NCMP see open issue #1

Scope

NCMP restartOptions:
  1. Trustlevels should as they were before the restart? (might depend on how much time has elapsed)
  2. (preferred) Trustlevels should be 'NONE' and need to be restored using an audit-request (not in scope)
2

Characteristics

#ParameterExpectationNotes
1dmi-down detection speed60 seconds (TBD)

2maximum number of cm-handles down report by DMI in one request and/or per minute30,000 / minute This looks like an 'ENM down' not sure if that should be handled this way
3processing of all trustlevel time for DMI-Down and/or peak load by DMI 



Out-of-Scope

  1. This epic will only introduce trustlevels NONE and COMPLETE. PARTIAL and POOR may be added later as below.

Reregistration

  1. This process occurs when the DMI Plugin Availability is down and then comes back up.
  2. NCMP makes a synchronous call to the DMI Plugin (New Audit Endpoint) to trigger a reregistration
  3. DMI Plugin then reregisters its CMHandles with NCMP (new reregistaration Endpoint?)
  4. NCMP then compares the CMHandles which are being reregistered with the CMHandles which already exist.
  5. CMHandles which are in NCMP but not in DMI reregistration request are kept as trust level none
  6. What happens if there is conflict between the old and new properties of a CMHandle, just take the new properties?
  7. New CMHandles could be registeredRe-registration ie. resolve trutslevel degradation is not in scope of this epic


High Level Interactions

Drawio
bordertrue
diagramNameStaleness Freshness Overview
simpleViewerfalse
width
linksauto
tbstyletop
lboxtrue
diagramWidth939940
revision13

InterfaceNameTriggerDescriptionTypeEndpoint or TopicSchema
1HealthCheck30 second interval (configurable)NCMP is to perform a health check against each of the DMI PluginsRESThttp://'$1'/manage/health/readiness
2Reregistration requestDMI Plugin has gone down and comes back upNCMP makes a call to that DMI Plugin telling it to reregisterRESTTBD
3ReregistrationDMI Plugin received a reregistration requestDMI Plugin makes a call to NCMP to reregister its CM HandlesREST/v1/ch/reregistration
{
  "dmiPlugin": "my-dmi-plugin",
  "dmiModelPlugin": "my-dmi-model-plugin",
  "cmHandles": [
    {
      "cmHandle": "my-cm-handle",
      "publicCmHandleProperties": {
        "key": "my-property"
      },
      "cmHandleProperties": {
        "key": "my-property"
      }
    },
    {
      "cmHandle": "my-cm-handle",
      "publicCmHandleProperties": {
        "key": "my-property"
      },
      "cmHandleProperties": {
        "key": "my-property"
      }
    }
  ],
  "dmiDataPlugin": "my-dmi-data-plugin"
}
4CMHandle trust level changeA CMHandle managed by DMI Plugin's trust level has changed

data contains {trustLevel: ENUM} 

event id is cmhandle id

KafkaTBD

<cloudEvents-header>

  id : <cmhandleId>

  type : org.onap.cm.events.trustlevel-notification


  data : {

                                trustlevel : "COMPLETE"

  }

53TrustLevel RequestClient RequestTrustLevel is to be returned based on the values in above MapsRESTTBD

Managing TrustLevels

DMI Plugins

  1. NCMP is checking every DMI Plugin for health at interface 1 every 30 seconds using the DMI Trust Map
  2. IF a DMI Plugin goes down, that DMI Plugin's trust level is updated to NONE in the DMI Trust Map
  3. IF a DMI Plugin comes back up, NCMP requests that DMI Plugin to do a reregistration via interface 2
  4. That DMI reregisters itself using interface 3
  5. NCMP analyses the registration and compares the CMHandles it knows about to the CMHandles which have been reregistered, any CMHandles which did exist and are not reregistered are now added to the untrustworthy CMHandles Map. Existing CMHandles are updated and new ones are created.
  6. After the reregistration is complete, the DMI Plugin TrustMap is updated to set the trust level for that DMI to completeTrust level is set back to COMPLETE.

CMHandles HB

  1. It is the responsibility of the DMI Plugins to update NCMP about the HBs of CMHandles
  2. Through interface 4, DMI Plugins will provide a kafka event on the changing of trustworthiness state of a CMHandle.
    1. NCMP receives this event and updates the Untrustworthy CMHandles Map Set accordingly
  3. Needs to be able to handle a throughput of 3060,000 State changes per minute for 2 instances

Reading Trust Level

  1. Body of request to be discussed, Will the request provide a DMI or a list of CMHandles?
  2. Interface 53
  3. NCMP will first check DMI Trust Map for the CMHandle
    1. If that DMI which is managing the CMHandle is marked as untrustworthy then we return NONE without checking the Untrustworthy CMHandles Map
    2. If that DMI is trustworthy, we check the individual CMHandles Map, if the CMHandle is in the Map then return NONE.
  4. Logically IF (DMITrustMap.getDMIPlugin.getTrustLevel ==  NONE) Return NONE
    1. ELSE (IF UntrustworthyCMHandlesMap.getDMIPlugin.contains(CMHandle) RETURN NONE
    2. ELSE return COMPLETE

...