Versions Compared

Version Old Version 3 New Version Current
Changes made by

Lee Anjella Macabuhay

Lee Anjella Macabuhay

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Component

Responsibilities

NCMP

  1. Process the event consumed and identify subscriptions based on

    1. cm handle id/alternate id

    2. xPath

  2. Identify topics for given subscription

  3. Interact with Kafka through Consumer/Producer set up

  4. Handle errors

Kafka

Consumer

  1. Consume events with the following

    1. Consumer Group: ncmp-group

    2. Topic: app.ncmp.avc.cm-events-topic

  2. Deserializing of event

Producer

  1. Produce events with the following

    1. Topic: based on subscription

    2. Event key: same as consumed event

    3. Event: consumed event

  2. Use of right keys

  3. Handle failures/retries

Database

Subscription information

DME

  1. Life cycle of topic

  2. Creation/Deletion of subscriptions

Potential Challenges with Dependencies

Challenge

Event ordering

Service Layer Design

Mapping of events to correct subscriptions

Prerequisite:

  • consumed event

For each xPath collect subscription IDs by querying database , i.e.

Code Block
final Collection<DataNode> existingNodes =
        cpsQueryService.queryDataNodes(NCMP_DATASPACE_NAME, CM_SUBSCRIPTIONS_ANCHOR_NAME,
                isOngoingCmSubscriptionCpsPathQuery, OMIT_DESCENDANTS);
if (existingNodes.isEmpty()) {
    return Collections.emptyList();
}
return (List<String>) existingNodes.iterator().next().getLeaves().get("subscriptionIds")

Identifying topic for given subscriptions

Process each subscription IDs and get a set of topics

Prerequisite:

  • format of subscription ID

  • format of topic name

Assumption

  • simple regex for the subscription id to extract topic name

    • (.+)#(.*)

      • group 1 = subscription id

      • group 2 = topic name

Forwarding Notification based on subscription

...

Validating topic? (can this be done in creation of subscription?)

Prerequisite:

  • consumed event

  • topic name

Error Handling

Description

Dead-letter queues?

No write access to kafka topic

Log failed request

Kafka topic does not exist

Log failed request

No valid topic name in subscription ID

Reject*

move up top

Performance Considerations

NCMP

  • Caching

  • Deserializing**

  • regex

Kafka

  • Configuration on polling

Security Considerations

  • Data validation

    • Event consumed will be forwarded as is

  • Access control (write permissions) to topics is controlled by client

...

unit tests

integration tests

E2E

k6?

Scenario

Expected Result

1

Consume event from DMI

No error , Log trace level ?

2

Publish event

Published event with correct details (topic name, event key, event)

3

No subscription Ids returned for certain xpath (Notification maps to no subscription)

No forwarded event

Increase counter for not forwarded events

4

Notification maps to one subscription

Event forwarded to one topic

5

Notification maps to multiple subscription (no duplication on topics)

Event forwarded to more than one topic

6

Notification maps to two subscriptions having the same topic

Event forwarded to one topic ONLY once

7

No topic extracted from subscription ID

Reject event

8

No write access to topic

Log error

9

Kafka not available?

Log error

Monitoring and Logging

Metrics

...

  • log of errors? log key events? messaged produced, log consumed events?

  • Failed request when a client topic does not exist

...