CPS-1964 - Support for Async datajobs Write

Owned by Toine Siebelink
Last updated: about 7 hours ago by Gliffy Diagram for Confluence
8 min read

References

Proposed JIRAS

key summary type created updated due assignee reporter priority status resolution
Refresh

Associated Studies

Priority

JIRA

Confluence 

1

CPS-2096 - Getting issue details... STATUS

CPS-2096: Create implementation proposal for NCMP to provide support for data producer identifier during registration.

2

CPS-2155 - Getting issue details... STATUS

CPS-2155 Java API: The Data Structure for DataJob Write Request

3

CPS-2141 - Getting issue details... STATUS

CPS-2141 Define DMI REST interface for Datajobs

Issues & Decisions


IssueNotesDecision
1CPS to deliver java-interface only

This affects our Wow in many aspects.

  1. Requirements on Java interface instead of REST
    1. REST first or Java first, parallel?
  2. How to Test & Demo in CPS Team

 

Csaba Kocsis  proposed that CPS Team will deliver Java impl. first and then E// will take our code and wrap and a REST interface around it.

Toine Siebelink CPS team will work out during Grooming how to demo withour REST interface, most like using Groovy (semi) Integartion Tests. Daniel Hanrahan his about to complete a test framework for test NCMP services.


2Validation in REST or Java Service

Balance between early validation and common code

Continue with current process. Review case by case Toine Siebelink  Csaba Kocsis  


3Why is NCMP forwarding the response, DMI could do this ?Involve extra performance cost from CPS 

  Implementation needed because of TBAC 


4Are Java doc expected to be publishedYes. Agreed to deliver as a separate artefact (Shall be versioned ... where should it go ?)

Versioning of the interface shall be the same as the service delivered by CPS. New v should not affect the interface, version numbers could change and shall be documented  


5Delivery artefacts

Required Artefacts

  1. JavaDoc for new interface
  2. Jar(s ?) for services support new interface

  AP to confirm with Andreas in team Lightening

 Csaba Kocsis to revert


6Characteristics

Need to discuss priority of this. Delivering a functioning MVP for EPIC is important. Discussion and agreement with Peter T needed

  Still needed, might not get response because the new team are yet to start any integration. AP Kolawole Adebisi-Adeolokun to get a decision from Peter/David 

 Csaba Kocsis  to revert


7Results - Why can't DMI send result after completion of job with be told to do soSimply less work if DMI can decide itself, no additional interface in other components needed to trigger this....

tbc with Kieran Mccarthy A / Rafael Rocha 


8Exact form of Alternate ID(s) FDNs

uri-FDN and  LDNs should contain "/" and not "," as delimiters at the start of the naming attribute

FDN example : 

Subnetwork=22,MeContext=Kista,ManagedElement=Kista,GNBDUFunction=1,UECC=1

URI FDN example : 

/Subnetwork=22/MeContext=Kista/ManagedElement=Kista/GNBDUFunction=1/UECC=1

Should be compliant with 3GPP, should start with '/ '  

https://github.com/jdegre/5GC_APIs/blob/Rel-18/TS28532_ProvMnS.yaml



Kieran Mccarthy A  Users are advised to use URI-FDNs for alternate IDs to minimize performance impacts of conversions etc.   Rafael Rocha  ALL  cps


9MVP (Minimum Viable Product) for DCM
  1. Expected Delivery to EIC  Q3(August) to EIC. (Fossing and Testing)

  2. Write Only Should be prioritized for target delivery (Q3). It should include
    - Batch write
    - NCMP should only check the cmhanlde of the first operation 
    - Should Databag be populated or not kieran mccarthy / Csaba Kocsis ?
  3. Also included is status and results endpoint
  4. NCMP should prioritize CPS-2009 Update remaining existing/legacy NCMP APIs to support alternate Id (FDN) - Developer Wiki - Confluence (onap.org) - over datajob Read. New Epic to be created for DataJob Read sperate this from 1964 - > CPS-2189
  1. EIC Expected Delivery  Q3(August)

10DataJob Read v Write prioritizationWrite should be prioritized over read


11Use of '/ ' causes some issuesNot at issue. FDN can support / and ,


12

Prioritize CPS-2009 Update remaining existing/legacy NCMP APIs to support alternate Id (FDN) - Developer Wiki - Confluence (onap.org) - over Read.

New Epic requested for DataJob Read sperate this from 1964.

Expected even earlier than q3

Agreed to move higher on R14 - #17


13

NCMP still need to convert Legacy Event to Cloud Event before we can Update the Legacy APIs to support AlternateId

Re-prioritise [CPS-1704] Separation of Headers from Event Payload for Legacy Events - ONAP JIRA over CPS-2009 Update remaining existing/legacy NCMP APIs to support alternate Id (FDN). Kolawole Adebisi-Adeolokun to get agreement with stakeholders

Currently ETH are only consuming the LCM event and we agreed shall remain Legacy event for MVP delivery.

Kolawole Adebisi-Adeolokun Peter Turcsanyi Csaba Kocsis


14

Decision to add status & result as part of this MVP


No new epic required for write, maintain current priority, wrap up this epic with;

  1. Status - (Finished/Ongoing)
  2. Result - (Success/Failure)


Kolawole Adebisi-Adeolokun 


15Is the requestId in the southbound request equal to the dataJobId coming in from the northound interfaces? (see writeJob for example)
Can the dataJobId used as the requestId?

Kolawole Adebisi-Adeolokun confirmed dataJobId can be used as the requestId


16

Define & agree on a Java interface (do we need a different method for write) The output should be defined

 We need a clarification on the response.

Kolawole Adebisi-Adeolokun // The output is currently being define by team Lighting AP @ko to follow-up


Requirements

E2E Requirement (for context)


Requirement

Additional Information

1

rApps shall be able to send async data requests and get back a job Id to track the execution/status of the job and at the end get the results of the job in a specified datastore. 

Implement new NCMP CRUDAQ data job / async interface aligned with 3GPP API - part of req #1

A client automation application shall be able to send async request to read, query and write network configuration. Current NCMP capabilities shall be extended to also support RANOAM CM data jobs.

  • NCMP as the Network CM proxy shall support the RANOAM CM data jobs. - re #1
  • RAPPS shall be able to send async data request - #1
  • RAPPS shall be able to receive Jobid to track status of job and translate result in a specified datastore- req #3

Functional


Interface

Requirement

Additional Information

Sign-off
1

 CPS-I-01

 NCMP to provide support for Data jobs qualifier during registration.

  • Can qualifier changed/updated ? → NCMP should allow upgrade from Blank (special values are allowed) to None-blank. This shall ONLY be supported as part of Initial Installation steps. Kieran Mccarthy A 
  • No validation is needed, any string is shall be accepted.
  • Qualifier are OPTIONAL during registration
  • Qualifier shall be EMS name/id
  • Qualifiers will be used with DMI plugin to allow NCMP splits requests with multiple cm handle into batches based on the DMI plugin.

 Kolawole Adeolokun

  Updated -> qualifier changes during II only Csaba Kocsis 

2

 CPS-E-11 

 NCMP to provide Java interface that can support Data jobs request

  1. NCMP shall splits and forward  the request to multiple DMIs/qualifier based on the alternate IDs (FDNs). See also requirement #5 below
  2. NCMP Shall report a list of sub jobs ids  (synchronously)
    1. CM Handles from DMIs that don't respond within http timeout shall be reported as failed asynchronous response (Java) 
  1. Order of FDNs and operations as per the Input-jobs should be maintained per qualifier
  2. Qualifier are not in the request.
  3. Datajob id will be provided by DCM.
  4.  How do we inform DCM all the result are not all reported ?
    1. NCMP to accept during the synchronous response and number of sub-jobs created.
    2. NCMP to report asynchronous the status of
      1. Sub-jobs that have started (Immediate response) 
        Include SubJob Id, DMI, Qualifier and subjob properties (as provided by the DMI)
      2. Sub-jobs that have not started with the DMI and the qualifier (after http timeout) See also Error handling below.

 Kieran Mccarthy A Csaba Kocsis  All cps

3

 CPS-E-12 (TBC)

NCMP shall provide Java interface which are expected to return EMS/DMIs Job and Sub Job Status

  • NCMP shall return the status of an EMS job id
  • NCMP shall define and specify DMI rest interface (Name ? - CPS to propose)
  1. Input parameters are DMI, jobId and sub-job id/property (Only 1 job).
  2. Output parameter : jobId, sub-job id, status(free string)

 Csaba Kocsis  all CPS CC Kieran Mccarthy A 

4CPS-E-11e
  1. NCMP to forward DMI responses to the correct Client Topic (as can be determined from message).
  1. Client topic will be communicated in all internal messages (as headers?) - NCMP to decide
  2. Headers should have request Id (correlationId), it's expected to be set by the REST Controller
  3. Topic names: The internal to DCM shall be proposed by CPS
  4. All events shall be Cloud event
  5. Topic names and messages details/Format of the event will be provided by E// Kieran Mccarthy A 

 All CPS, Kieran Mccarthy A  Csaba Kocsis

5

 CPS-E-11

NCMP is to be able to find the correct CM Handles based on the 'best' match for partial matches of the target FDN. The best match could be a Subnetwork (which are registered as separate cm handles)

Find best match, see solution proposal for details

  1. There should only be 1 or none target (or none)
  2. No impact on registration for sub-networks
  3. no complex pre-optimization!

 Csaba Kocsis  All CPS

cc Kieran Mccarthy A 

Error Handling (new error scenarios)


Functional Requirement

Error Scenario

Expected Behavior

Sign-off

1

#2

Non Responding DMI during new DataJob Request

NCM shall send asynchronous message containing:

  • Subjobs that have not started with the DMI and the qualifier (after http timeout), no subJobs Ids since they would have to be assigned by south bound system

Client is responsible for retry. 

Wait for implantation proposal 

 Csaba Kocsis , CPS

2

#2

Unmatched Alternate IDs 

Toine Siebelink  : Not discussed yet but  I would suggest similar behavior as legacy bulk interface (async response with failed FDNS) - If an unmatched alternated (1 or more), send a response back to client saying no jobs or 0 subjob

If request contains none-exited alternateid ? - Goes to client


 Csaba Kocsis, all cps 

3

#3

Non Responding DMI during new status update

Error handling just by HTTP timeout (java interface will throw an exception)

 Csaba Kocsis all cps 

4

#4 
General

Not existing Topics

NCMP is NOT responsible for checking or handling messages to non existing (client) topics
Also 'internal' topics are assumed to be set up correctly

 Csaba Kocsis  all cps

5

Misc.

Other Robustness 

  1. NCMP Down
  1. What happens to Sub-jobs that have not started with the DMI and the qualifier (after http timeout)
  2. What happens to if NCMP crashes immediately after responding to the request ?

 Agreed to revisit robustness once basic happy and standard error scenarios (DMI not responding) have been implemented 


AP Kolawole Adebisi-Adeolokun  To follow-up and check the possibility of merging this with the read & write or just keeping the error sci. separate as is;

High Level Solution



The 8 Steps

  1. DCM REST endpoint forwards request to NCMP for processing
  2. When processed subjobs are sent to relevent DMI
  3. DMI responds with acknowledgement of datajob received to NCMP
  4. NCMP informs DCM
  5. Job status sent to internal topic
  6. Topic published on Kafka channel
  7. Clients subscribed get status
  8. rApps receive status


Considerations

  1. Java interface to process a data job request
  2. Splitting the main job into multiple DMI sub-jobs
  3. Synchronous response with sub jobid
  4. Async mechanism to return list of jobs
  5. DMI or ENM responses ( Not CPS responsibility )
  6. Listen to the responses on internal topics 
  7. Forward responses to the relevant topics
  8. Error Handling
  9. Specify the REST interface spec between NCMP and DMI plugin


Suggested User Story Breakdown

  1. Define DMI REST APIs and implement
  2. Handle request break-down (steps 1 and 2 on the diagram)
  3. Handle synchronous data-job response (step 4) and synch sub-job responses (steps 3)
    • Agree on asynch data-job response with nm-engineering team
  4. Response schema for DMI response (Parallel to above steps 1, 2 and 3)
  5. Forward response (Parallel to above steps 1, 2 and 3)
  6. Delivery of product
  7. Error handling and more...

Solution Proposal

Sequence Diagram


Solution Aspects

  1. Deliver Java API only
    1. 3GPP (alternate Ids) only
  2. Asynchronous Impl.
    1. Job Id(s) One for each batch
  3. Qualifier (EMS Name, batch per Qualifier and/or DMI plugin)
  4. Update DMI
    1. DMI STub updates (already in CPS)
  5. publish job result to Kafka

Overlapping alternate Id (Sub-Networks)


Registered CM Handles & Alternate IDs

CH-1 = /SubNetwork=Europe
CH-2=  /SubNetwork=Erope/SubNetwork=Hungary
CH-3 = /SubNetwork=Europe/SubNetwork=Hungary/ManagedElement=XY
CH-4 = /SubNetwork=Europe/SubNetwork=Hungary/ManagedElement=Z
CH-5 = /SubNetwork=Europe/ManagedElement=AB
CH-6 = /ManagedElement=CD  (MeContext, RanInfraStructureSupport)

A) Batch Job Request FOR below FDN:
/SubNetwork=Europe/ManagedElement=AB/GNBDUFunction=1
Should match from the start CM HANDLE 5  (longest possible common string)

B) Batch JOb Request FOR below FDN:
/SubNetwork=Europe/ManagedElement=AB
Should match CM HANDLE 5

C) Batch JOb Request FOR below FDN:
/SubNetwork=Europe
Should match CM HANDLE 1


C) Batch JOb Request FOR below FDN:
/SubNetwork=Europe/SubNetwork=Hungary
Should match CM HANDLE 2


Out-of-Scope

  • Conflict management, DATAJOBS are BATCH OPERATION.


PO Comment

-CPS will need to leave epic open, Kolawole Adebisi-Adeolokun to organise a sync with team Lightning on their current progress

-dev wise, CPS has fulfilled all functional aspect of this epic but can't progress in further because the interface isn't ready yet from Team Lightning

-Should CPS consider demo ? - Kolawole Adebisi-Adeolokun  to Check with team Lightning when we can arrange a demo upon a working interface