CPS-2141 Define DMI REST interface for Datajobs

Issues & Decisions

#	Issue	Notes	Decision
1	Is the requestId in the southbound request equal to the dataJobId coming in from the northound interfaces? (see writeJob for example)	Can the dataJobId used as the requestId? Csaba Kocsis	01 Aug 2024 Csaba Kocsis and Toine Siebelink agreed that requestId should be renamed to jobId on all interfaces

NCMP → DMI

As part of the overall datajob solution there must be an interface for data flow between the NCMP-lib in DCM and the DMI Plugin.

This story involves the design and implementation of that interface for both the DMI request and response.

Define new REST interface :
- Method: POST
- Path: /dmi/v1/writeJob/{requestId}

Only writeJob required for MVP. Read, Status, Result will be completed in a later iteration.

Acceptance Criteria (pending stakeholder agreement):

New Interface DMI-I-02 with document and diagram updates. Both Wiki and RTD
Define swagger for request/response (additional info for forwarding data job request : moduleSetTag, privateProperties, dataProducerId, requestId)
Agree and implement yaml for this interface
Stakeholder acceptance of yaml

Proposed DMI REST Interface & Forwarded DMI Java Request Data Structure from NCMP

openapi: 3.0.3
info:
  title: NCMP Data Subjob API
  description: Support datajobs through one or more subjob for each DMI and Data Producer Identifier combination
  version: 1.0.0
servers:
  - url: /dmi
tags:
  - description: DMI plugin rest apis
    name: dmi-datajob
paths:
  /v1/readJob/{requestId}:
    post:

      description: Create a read request
      operationId: readDataJob
      parameters:
        - $ref: '#/components/parameters/requestIdInPath'
      requestBody:
        description: Operation body
        content:
          application/3gpp-json-patch+json:
            schema:
              $ref: '#/components/schemas/SubjobReadRequest'
      tags:
        - dmi-datajob
      responses:
        "200":
          description: Response for subjob
          content:
            application/json:
              schema:
                type: object
                properties:
                  dataProducerJobId:
                    type: string
                    description: The data job ID.
  /v1/writeJob/{requestId}:
    post:
      description: Create a write request
      operationId: writeDataJob
      parameters:
        - $ref: '#/components/parameters/requestIdInPath'
      requestBody:
        description: Operation body
        content:
          application/3gpp-json-patch+json:
            schema:
              $ref: '#/components/schemas/SubjobWriteRequest'
      tags:
        - dmi-datajob
      responses:
        "200":
          description: Response for subjob
          content:
            application/json:
              schema:
                type: object
                properties:
                  dataProducerJobId:
                    type: string
                    description: The data job ID.
components:
  parameters:
    requestIdInPath:
      description: Identifier for the overall Datajob
      in: path
      name: requestId
      required: true
      schema:
        example: some-identifier
        type: string
  schemas:
    SubjobReadRequest:
      type: object
      required:
        - dataProducerId
        - data
      properties:
        dataAcceptType:
          description: Defines the data response accept type
          example: application/vnd.3gpp.object-tree-hierarchical+json
          type: string
        dataContentType:
          description: Defines the data request content type
          example: application/3gpp-json-patch+json
          type: string
        dataProducerId:
          description: ID of the producer registered by DMI for the paths in the operations in this request
          example: my-data-producer-identifier
          type: string
        data:
          example:
            op: read
            operationId: 1
            path: SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003/GNBCUCPFunction=2
            attributes: userLabel
            scope:
              scopeTyp: BASE_ONLY
          type: array
          items:
            type: object
            required:
              - path
              - op
            properties:
              path:
                description: Defines the resource on which operation is executed
                example: SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003
                type: string
              op:
                description: Describes the operation to execute
                example: read
                type: string
              operationId:
                description: Unique identifier for the operation within the request
                example: 1
                type: string
              attributes:
                description: This parameter specifies the attributes of the scoped resources that are returned
                type: array
                items:
                  example: cellId
                  type: string
              fields:
                description: This parameter specifies the attribute fields of the scoped resources that are returned
                type: array
                items:
                  type: string
              filter:
                description: This parameter is used to filter the scoped Managed Objects. Only Managed Objects passing the filter criteria will be fetched
                example: NRCellDU/attributes/administrativeState==LOCKED
                type: string
              scopeType:
                description: ScopeType selects MOs depending on relationships with Base Managed Object
                example: BASE_ONLY
                type: string
              scopeLevel:
                description: Only used when the scope type is BASE_NTH_LEVEL to specify amount of levels to search
                example: 0
                type: integer
              moduleSetTag:
                description: Module set identifier
                example: my-module-set-tag
                type: string
              cmHandleProperties:
                description: Private properties of the cm handle for the given path
                $ref: '#/components/schemas/CmHandleProperties'
    SubjobWriteRequest:
      type: object
      required:
        - dataProducerId
        - data
      properties:
        dataAcceptType:
          description: Defines the data response accept type
          example: application/vnd.3gpp.object-tree-hierarchical+json
          type: string
        dataContentType:
          description: Defines the data request content type
          example: application/3gpp-json-patch+json
          type: string
        dataProducerId:
          description: ID of the producer registered by DMI for the paths in the operations in this request
          example: my-data-producer-identifier
          type: string
        data:
          example:
            op: add
            path: SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003/GNBCUCPFunction=1/EUtraNetwork=1/EUtranFrequency=12
            value:
              id: 12
              attributes:
                userLabel: label12
          type: array
          items:
            type: object
            required:
              - path
              - op
            properties:
              path:
                description: Defines the resource on which operation is executed
                example: SubNetwork=Europe/SubNetwork=Ireland/MeContext=NR03gNodeBRadio00003/ManagedElement=NR03gNodeBRadio00003
                type: string
              op:
                description: Describes the operation to execute
                example: add
                type: string
              operationId:
                description: Unique identifier for the operation within the request
                example: 1
                type: string
              moduleSetTag:
                description: Module set identifier
                example: my-module-set-tag
                type: string
              cmHandleProperties:
                description: Private properties of the cm handle for the given path
                $ref: '#/components/schemas/CmHandleProperties'
              value:
                description: Value dependent on the op specified. Resource for an add. Object for a replace. ActionParameters for an action.
                type: object
                oneOf:
                  - $ref: '#/components/schemas/Resource'
                  - $ref: '#/components/schemas/ActionParameters'
                  - $ref: '#/components/schemas/Object'
    CmHandleProperties:
      description: Private properties of the cm handle for the given path
      type: object
    Resource:
      type: object
      properties:
        id:
          description: Identifier of the resource object
          example: resource-identifier
          type: string
        attributes:
          description: Key value map representing the objects class attributes and values
          type: object
          additionalProperties:
            example: 'userLabel: label11'
            type: string
    ActionParameters:
      description: The input of the action in the form of key value pairs
      type: object
      additionalProperties:
        type: string
    Object:
      type: object

Data SubJob create (NCMP → DMI)

This is a mirror of the Datajob read/write request with the added details needed for the DMI plugins. Below you can see the changes for the READ but the same changes are applicable for the WRITE too.

Method: POST

Path: /dmi/v1/readJob

DMI Data SubJob READ request

Name	Description	Location	Type	Mandatory
destination	The destination of the results. ( e.g. S3 Bucket)	Query	string	N
dataaccepttype	Define the data response accept type. Passible values: · application/vnd.3gpp.object-tree-hierarchical+json (default) · application/vnd.3gpp.object-tree-flat+json	request body	enum	N
datacontenttype	Define the data request content type. Passible values: · application/3gpp-json-patch+json (default)	request body	enum	N
dataProducerId	ID of the producer registered by DMI for the alernateIDs in the operations in this request.	request body	String	Y
requestId	Identifier for the overall Datajob	request body	String	Y
data	List of operations to be executed.	request body	List of DMI3gppReadOperation	Y

DMI3gppReadOperation

Name	Description	Type	Mandatory
path	It is a unique identifier of a managed object (MO) on a network element. Defines the resource on which operation is executed. Typically could be Fully Distinguished Name (FDN).	String	Y
op	Describes the operation to execute. The value can be: "read"	String	Y
operationId	Unique identifier of the operation within the request	Integer	N
attributes	This parameter specifies the attributes of the scoped resources that are returned.	List of String	N
fields	This parameter specifies the attribute fields of the scoped resources that are returned. This should be used if an attribute is a struct and only a subset of its fields should be returned.	List of String	N
filter	The parameter is used to filter the scoped Managed Objects. Only Managed Objects passing the filter criteria will be fetched.	List of String	N
scopeType	ScopeType selects MOs depending on relationships with Base Managed Object.	enum[BASE_ONLY, BASE_ALL, BASE_NTH_LEVEL, BASE_SUBTREE]	N
scopeLevel	Only used when the scope type is BASE_NTH_LEVEL	Integer	N
moduleSetTag	Module set identifier	String	Y
privateProperties	CMHandle private properties for the alternateID in the path.	Map	Y

Response

Status code: 200 - OK

Name	Description	Location	Type	Mandatory
dataProducerJobId	The ID of the created job by the data producer.	response body	String	Y
dmiPluginName	The target DMI that the job was performed on	response body	String	Y
dataProducerJobID	ID of the job created by the data producer	response body	String	Y

DMI Data SubJob WRITE request

Method: POST

Path: /dmi/v1/writeJob

Name	Description	Location	Type	Mandatory
destination	The destination of the results. ( e.g. S3 Bucket)	Query	string	N
dataaccepttype	Define the data response accept type. Passible values: · application/vnd.3gpp.object-tree-hierarchical+json (default) · application/vnd.3gpp.object-tree-flat+json	request body	enum	N
datacontenttype	Define the data request content type. Passible values: · application/3gpp-json-patch+json (default)	request body	enum	N
dataProducerId	ID of the producer registered by DMI for the alernateIDs in the operations in this request.	request body	String	Y
requestId	Identifier for the overall Datajob	request body	String	Y
data	List of operations to be executed.	request body	List of DMI3gppWriteOperation	Y

DMI3gppWriteOperation

Name	Description	Type	Mandatory
path	It is a unique identifier of a managed object (MO) on a network element. Defines the resource on which operation is executed. Typically could be Fully Distinguished Name (FDN).	String	Y
op	Describes the operation to execute. The value can be: "add": creates a new MO with the id and attributes given in the value "replace": replaces the value of an attribute specified in the path (attribute is separated with a # from the MO path) with the object in the value "remove": removes the MO given in the path, value is not applicable "action": executes an MO action in the path (action is separated with a # from the MO path) with the input parameters in the value	String	Y
operationId	Unique identifier of the operation within the request	Integer	N
value	NA if op == remove Resource if op == add Object if op == replace ActionParameters if op == action	Object	N
moduleSetTag	Module set identifier	String	Y
privateProperties	CMHandle private properties for the alternateID in the path.	Map	Y

Response

Status code: 200 - OK

Name	Description	Location	Type	Mandatory
dataProducerJobId	The ID of the created job by the data producer.	response body	String	Y
dmiPluginName	The target DMI that the job was performed on	response body	String	Y
dataProducerJobId	ID of the job created by the data producer	response body	String	Y

Data subjob status check (NCMP → DMI)

Request:

Method: GET

Path: /dmi/v1/writeJob/dataProducer/{dataProducerd}/dataProducerJob/{dataProducerdJobId}/status

Name	Description	Location	Type	Mandatory
dataProducerId	ID of the producer registered by DMI for the alernateIDs in the operations in this request.	Path	String	Y
dataProducerJobId	Identifier of the job created by the data producer.	Path	String	Y

Response:

Status code: 200 - OK

Name	Description	Location	Type	Mandatory
status	enum that represents the status of the subjob. Possible values:	response body	enum: [ NOT_STARTED, RUNNING, FINSHED, FAILED, PARTIALLY_FAILED, CANCELLING, CANCELLED ]	Y

Data subjob retrieve result (NCMP → DMI)

Request:

Method: GET

Path: /dmi/v1/writeJob/dataProducer/{dataProducerd}/dataProducerJob/{dataProducerdJobId}/result?destination={destination}

Parameters:

Name	Description	Location	Type	Mandatory
dataProducerId	ID of the producer registered by DMI for the alternateIDs in the operations in this request.	Path	String	Y
dataProducerJobId	Identifier of the job created by the data producer. (not globally unique)	Path	String	Y
destination	The destination of the results: Kafka topic name or s3 bucket name. This shall be put into the Kafka message headers returned to NCMP	Query	String	Y

Response:

WRITE

Status Code: 200 - Ok

Data part of the message

READ

Status Code: 200 - Ok

Name	Description	Location	Type	Mandatory
resulturi	internal ncmp topic name	response body	String	Y