CPS-2739: Map interface for CPS Data Service

1 Abstract
2 References
3 Issues & Decisions
4 Proposed Map Interface for CPS
5 Using the Map with a new Yang model
- 5.1 Define the Model
- 5.2 Load the Model
- 5.3 Expose the Model
- 5.4 Use the Model
6 Using the Map with any existing Yang list
7 Proposed Implementation
8 Efficiency issues in java.util.Map interface
9 Proof of Concept
10 Simplifying existing code with the Map interface
11 Extended interface for mapping complex types like NcmpServiceCmHandle
12 Conclusion

Abstract

This implementation proposal is about wrapping CPS Data Service in a Java Map interface. So instead of using methods like cpsDataService.getDataNodes you would instead use Map.get.

The original purpose of this was to provide a drop-in replacement of Hazelcast’s IMap, so that Hazelcast could be replaced by CPS with minimal code changes if needed. It was later discovered that using CPS through a Map interface could greatly simplify existing code.

References

https://lf-onap.atlassian.net/browse/CPS-2739

Issues & Decisions

	Issue	Notes	Decision

	Issue	Notes	Decision
1	What should CPS Map interface be called, and in what package?	Proposal: `org.onap.cps.api.model.LiveDataMap` LiveDataMap communicates clearly that data read or written is done live from CPS database.
2
3

Proposed Map Interface for CPS

The core concept of this proposal is to provide an implementation of java.util.Map interface (or even a custom interface modeled on Hazelcast’s IMap) that uses CPS under the hood. For example, calling map.get("key-1") would in turn call cpsDataService.getDataNodes("/some/xpath[@key='key-1']"), but it would hide away CPS-specific details.

Use case	Map method	Cps Data Service method

Use case	Map method	Cps Data Service method
Get an entry	map.get(key)	cpsDataService.getDataNodes(dataspace, anchor, xpath)
Delete an entry	map.remove(key)	cpsDataService.deleteDataNode(dataspace, anchor, xpath)
Insert or replace and entry	map.put(key)	cpsDataService.saveData or cpsDataService.updateNodeLeaves depending on circumstances.

It is proposed that CpsDataService have a new method that returns an interface implementing Map<K, V> which can map key and value leaves in a Yang list, given a dataspace, anchor, and xpath:

<K, V> LiveDataMap<K, V> getLiveDataMap(String dataspaceName, String anchorName, String listXpath,
                                        String keyLeaf, Class<K> keyClass,
                                        String valueLeaf, Class<V> valueClass);

Note that Hazelcast’s IMap provides additional methods that are not in standard Java Map interface for efficiency reasons. These include batch methods such as getAll(keys), removeAll(keys), replaceAll(map). That is why a new interface called LiveDataMap extending java.util.Map is proposed.

Using the Map with a new Yang model

Let’s say we want a new map to be able to efficiently resolve Alternate IDs to CM handle IDs. First we need a new Yang module to model the map.

Define the Model

We could define a new Yang module containing the mapping of alternate-id to cm-handle-id:

module alternate-id-map {
    yang-version 1.1;
    namespace "org:onap:cps";
    prefix cps;
    container map {
        list entry {
            key "alternate-id";
            leaf alternate-id { type string; }
            leaf cm-handle-id { type string; }
        }
    }
}

It is best practice to put a Yang list in a container. Even CPS has special optimizations for this case, e.g. see org.onap.cps.ri.repository.FragmentQueryBuilder#addAbsoluteParentXpathSearchCondition.

Load the Model

In InventoryModelLoader or similar, we would ensure that the model is loaded, and a schema-set and associated anchor exists:

    public static final String NCMP_ALTERNATE_ID_CACHE_ANCHOR_NAME = "alternate-id-to-cm-handle-id";
    private static final String NEW_MODEL_FILE_NAME = "alternate-id-map@2025-04-01.yang";
    private static final String NEW_SCHEMA_SET_NAME = "alternate-id-map-2025-04-01";
    private static final String TOP_LEVEL_DATANODE_NAME = "map";

    private void updateAlternateIdMapModel() {
        createDataspace(NCMP_DATASPACE_NAME);
        createSchemaSet(NCMP_DATASPACE_NAME, NEW_SCHEMA_SET_NAME, NEW_MODEL_FILE_NAME);
        createAnchor(NCMP_DATASPACE_NAME, NEW_SCHEMA_SET_NAME, NCMP_ALTERNATE_ID_CACHE_ANCHOR_NAME);
        createTopLevelDataNode(NCMP_DATASPACE_NAME, NCMP_ALTERNATE_ID_CACHE_ANCHOR_NAME, TOP_LEVEL_DATANODE_NAME);
    }

Expose the Model

Then a new configuration bean can be used to instantiate the map:

@Configuration
@RequiredArgsConstructor
public class AlternateIdMapConfig {
    private final CpsDataService cpsDataService;
    @Bean
    public LiveDataMap<String, String> cmHandleIdPerAlternateId() {
        return cpsDataService.getLiveDataMap(NCMP_DATASPACE_NAME, NCMP_ALTERNATE_ID_CACHE_ANCHOR_NAME,
                "alternate-id", String.class, "cm-handle-id", String.class);
    }
}

Use the Model

Once this is done, the map can be used like a normal Java map. For example, to insert an entry:

cmHandleIdPerAlternateId.put(alternateId, cmHandleId);

Using the Map with any existing Yang list

The map interface can be also used to create a Map ‘view’ of any Yang list.

Terminology: A Yang list is similar to a Java Map, as a Yang list always has a key. While a Yang leaf-list is more similar to a Java List.

In the below example, a map of CM-handle ID to Module Set Tag is created, using the existing CM-handle registry, mapping “id” leaf to “module-set-tag” leaf:

Map<String, String> moduleSetTagPerCmHandleId = cpsDataService.getLiveDataMap(
        NCMP_DATASPACE, NCMP_DMI_REGISTRY_ANCHOR, "/dmi-registry/cm-handles",
        "id", String.class, "module-set-tag", String.class);

This can be used to get or update a Module Set Tag for a given CM handle ID:

assert moduleSetTagPerCmHandleId.get('ch-1') == 'tagA';

Proposed Implementation

Under the hood, Map.get would use CpsDataService.getDataNodes, returning Null in case of DataNodeNotFoundException:

@Override
public V get(final Object key) {
    try {
        final DataNode dataNode = cpsDataService.getDataNodes(dataspaceName, anchorName, 
            getXpathForMapEntry(key), OMIT_DESCENDANTS).iterator().next();
        return valueClass.cast(dataNode.getLeaves().get(valueLeaf));
    } catch (final DataNodeNotFoundException dataNodeNotFoundException) {
        return null;
    }
}

Similarly, Map.remove would call CpsDataService.deleteDataNode, etc.

Special consideration is needed for writing entries, where Map.put allows inserting new entries or updating existing ones, while CpsDataService has separate methods for insert versus update. This could be resolved using a helper method catching AlreadyDefinedException:

void insertOrUpdateData(final String dataspaceName, final String anchorName, final String parentXpath, final String jsonData) {
    try {
        cpsDataService.saveData(dataspaceName, anchorName, parentXpath, jsonData, NO_TIMESTAMP, JSON);
    } catch (final AlreadyDefinedException alreadyDefinedException) {
        cpsDataService.updateNodeLeaves(dataspaceName, anchorName, parentXpath, jsonData, NO_TIMESTAMP, JSON);
    }
}

Efficiency issues in java.util.Map interface

Java’s Map interface was clearly defined with only in-memory implementations considered, such as HashMap and TreeMap.

Some Map methods like put(key) and remove(key) are expected to return the old value before updating/removing the entry:

V put(K key, V value);
V remove(Object key);

For a CPS-backed implementation, this would mean first looking up the old value from CPS database with getDataNodes! Indeed, for any database- or network-backed Map implementation, this would add overhead. Hazelcast addresses this by adding non-standard methods such as:

void set(K key, V value);                   // same as put, but don't return old value
void delete(Object key);                    // same as remove, but don't return old value
// Batch operations:
Map<K, V> getAll(Collection<String> keys);  // get a batch by keys
void removeAll(Collection<K> keys);         // remove a batch by keys
void putAll(Map<K, V> map);                 // insert/replace a batch of key-value pairs

It is proposed that CPS Map interface also define these more efficient methods.

Proof of Concept

Here is a complete implementation of a CPS-backed map, capable of mapping simple types like Map<String, String>.

23339: Add Java Map implementation backed by CPS | https://gerrit.nordix.org/c/onap/cps/+/23339

Simplifying existing code with the Map interface

See this prototype:

23343: POC Use new CPS Map interface in CM-handle update | https://gerrit.nordix.org/c/onap/cps/+/23343

Here is a before and after comparison of using the described Map interface to update CM handle properties:

Existing code using raw CPS Data Service	Proposed code using Map interface

Existing code using raw CPS Data Service

Proposed code using Map interface

    private void processUpdates(final DataNode existingCmHandleDataNode,
                                final NcmpServiceCmHandle updatedNcmpServiceCmHandle) {
        updateAlternateId(updatedNcmpServiceCmHandle);
        updateDataProducerIdentifier(existingCmHandleDataNode, updatedNcmpServiceCmHandle);
        if (!updatedNcmpServiceCmHandle.getPublicProperties().isEmpty()) {
            updateProperties(existingCmHandleDataNode, PUBLIC_PROPERTY,
                updatedNcmpServiceCmHandle.getPublicProperties());
        }
        if (!updatedNcmpServiceCmHandle.getDmiProperties().isEmpty()) {
            updateProperties(existingCmHandleDataNode, DMI_PROPERTY, updatedNcmpServiceCmHandle.getDmiProperties());
        }
    }
    
    private void updateProperties(final DataNode existingCmHandleDataNode, final PropertyType propertyType,
                                  final Map<String, String> updatedProperties) {
        final Collection<DataNode> replacementPropertyDataNodes =
                getReplacementDataNodes(existingCmHandleDataNode, propertyType, updatedProperties);
        replacementPropertyDataNodes.addAll(
                getUnchangedPropertyDataNodes(existingCmHandleDataNode, propertyType, updatedProperties));
        if (replacementPropertyDataNodes.isEmpty()) {
            removeAllProperties(existingCmHandleDataNode, propertyType);
        } else {
            inventoryPersistence.replaceListContent(existingCmHandleDataNode.getXpath(), replacementPropertyDataNodes);
        }
    }
    
    private void removeAllProperties(final DataNode existingCmHandleDataNode, final PropertyType propertyType) {
        existingCmHandleDataNode.getChildDataNodes().forEach(dataNode -> {
            final Matcher matcher = propertyType.propertyXpathPattern.matcher(dataNode.getXpath());
            if (matcher.find()) {
                log.info("Deleting dataNode with xpath : [{}]", dataNode.getXpath());
                inventoryPersistence.deleteDataNode(dataNode.getXpath());
            }
        });
    }
    
    private Collection<DataNode> getUnchangedPropertyDataNodes(final DataNode existingCmHandleDataNode,
                                                               final PropertyType propertyType,
                                                               final Map<String, String> updatedProperties) {
        final Collection<DataNode> unchangedPropertyDataNodes = new HashSet<>();
        for (final DataNode existingPropertyDataNode : existingCmHandleDataNode.getChildDataNodes()) {
            final Matcher matcher = propertyType.propertyXpathPattern.matcher(existingPropertyDataNode.getXpath());
            if (matcher.find()) {
                final String keyName = matcher.group(2);
                if (!updatedProperties.containsKey(keyName)) {
                    unchangedPropertyDataNodes.add(existingPropertyDataNode);
                }
            }
        }
        return unchangedPropertyDataNodes;
    }
    
    private Collection<DataNode> getReplacementDataNodes(final DataNode existingCmHandleDataNode,
                                                         final PropertyType propertyType,
                                                         final Map<String, String> updatedProperties) {
        final Collection<DataNode> replacementPropertyDataNodes = new HashSet<>();
        updatedProperties.forEach((updatedAttributeKey, updatedAttributeValue) -> {
            final String propertyXpath = getAttributeXpath(existingCmHandleDataNode, propertyType, updatedAttributeKey);
            if (updatedAttributeValue != null) {
                log.info("Creating a new DataNode with xpath {} , key : {} and value : {}", propertyXpath,
                        updatedAttributeKey, updatedAttributeValue);
                replacementPropertyDataNodes.add(
                        buildDataNode(propertyXpath, updatedAttributeKey, updatedAttributeValue));
            }
        });
        return replacementPropertyDataNodes;
    }
    
    private String getAttributeXpath(final DataNode cmHandle, final PropertyType propertyType,
                                     final String attributeKey) {
        return cmHandle.getXpath() + "/" + propertyType.xpathPrefix + String.format("[@name='%s']", attributeKey);
    }
    
    private DataNode buildDataNode(final String xpath, final String attributeKey, final String attributeValue) {
        final Map<String, String> updatedLeaves = new LinkedHashMap<>(1);
        updatedLeaves.put("name", attributeKey);
        updatedLeaves.put("value", attributeValue);
        log.debug("Building a new node with xpath {} with leaves (name : {} , value : {})", xpath, attributeKey,
                attributeValue);
        return new DataNodeBuilder().withXpath(xpath).withLeaves(ImmutableMap.copyOf(updatedLeaves)).build();
    }
    
    private void setAndUpdateCmHandleField(final String cmHandleIdToUpdate, final String fieldName,
                                           final String newFieldValue) {
        final Map<String, Map<String, String>> dmiRegistryData = new HashMap<>(1);
        final Map<String, String> cmHandleData = new HashMap<>(2);
        cmHandleData.put("id", cmHandleIdToUpdate);
        cmHandleData.put(fieldName, newFieldValue);
        dmiRegistryData.put("cm-handles", cmHandleData);
        cpsDataService.updateNodeLeaves(NCMP_DATASPACE_NAME, NCMP_DMI_REGISTRY_ANCHOR, NCMP_DMI_REGISTRY_PARENT,
                jsonObjectMapper.asJsonString(dmiRegistryData), OffsetDateTime.now(), ContentType.JSON);
        log.debug("Updating {} for cmHandle {} with value : {})", fieldName, cmHandleIdToUpdate, newFieldValue);
    }
    
    enum PropertyType {
        DMI_PROPERTY("additional-properties"), PUBLIC_PROPERTY("public-properties");
        private static final String LIST_INDEX_PATTERN = "\\[@(\\w+)[^\\/]'([^']+)']";
        final String xpathPrefix;
        final Pattern propertyXpathPattern;
        PropertyType(final String xpathPrefix) {
            this.xpathPrefix = xpathPrefix;
            this.propertyXpathPattern = Pattern.compile(xpathPrefix + LIST_INDEX_PATTERN);
        }
    }
}

    private void processUpdates(final NcmpServiceCmHandle updatedNcmpServiceCmHandle) {
        final String cmHandleId = updatedNcmpServiceCmHandle.getCmHandleId();
        updateAlternateId(cmHandleId, updatedNcmpServiceCmHandle.getAlternateId());
        updateDataProducerIdentifier(cmHandleId, updatedNcmpServiceCmHandle.getDataProducerIdentifier());
        updateProperties(cmHandleId, PropertyType.PUBLIC, updatedNcmpServiceCmHandle.getPublicProperties());
        updateProperties(cmHandleId, PropertyType.ADDITIONAL, updatedNcmpServiceCmHandle.getDmiProperties());
    }

    private void updateProperties(final String cmHandleId, final PropertyType propertyType,
                                  final Map<String, String> updatedProperties) {
        if (!updatedProperties.isEmpty()) {
            final String propertiesXpath = NCMP_DMI_REGISTRY_PARENT
                    + "/cm-handles[@id='" + cmHandleId + "]/" + propertyType.getYangContainerName();
            final LiveDataMap<String, String> propertiesSetter = cpsDataService.getLiveDataMap(NCMP_DATASPACE_NAME,
                    NCMP_DMI_REGISTRY_ANCHOR, propertiesXpath, "name", String.class, "value", String.class);
            updatedProperties.forEach((name, value) -> {
                if (value == null) {
                    propertiesSetter.remove(name);
                } else {
                    propertiesSetter.set(name, value);
                }
            });
        }
    }

Extended interface for mapping complex types like NcmpServiceCmHandle

Here is an implementation which can also handle complex types like Map<String, NcmpServiceCmHandle>:

23356: POC LiveDataMap | https://gerrit.nordix.org/c/onap/cps/+/23356

It provides a more generic interface to construct a map allowing complex types like NcmpServiceCmHandle by providing converters:

    <K, V> LiveDataMap<K, V> getLiveDataMap(String dataspaceName, String anchorName, String listXpath, 
                                            String keyLeaf, Class<K> keyClass,
                                            Function<DataNode, V> valueFromDataNode,
                                            Function<V, Object> valueToJsonEncodable);

If converting to or from NcmpServiceCmHandle, you can supply methods from the YangDataConverter utility class:

given: 'a map view of all CM handles'
    Map<String, NcmpServiceCmHandle> cmHandleMap = cpsDataService.getLiveDataMap(
        NCMP_DATASPACE, NCMP_DMI_REGISTRY_ANCHOR,
        '/dmi-registry/cm-handles', 'id', String.class,
        dataNode -> YangDataConverter.toNcmpServiceCmHandle(YangDataConverter.toYangModelCmHandle(dataNode)),
        ncmpServiceCmHandle -> YangDataConverter.toYangModelCmHandle((NcmpServiceCmHandle) ncmpServiceCmHandle))

when: 'a new CM handle is put in the map'
    cmHandleMap.put('ch-4', new NcmpServiceCmHandle(cmHandleId: 'ch-4', alternateId: 'alt-4'))

then: 'the new CM handle can be got from NCMP'
    def cmHandleFromNcmp = networkCmProxyInventoryFacade.getNcmpServiceCmHandle('ch-4')
    assert cmHandleFromNcmp.cmHandleId == 'ch-4'
    assert cmHandleFromNcmp.alternateId == 'alt-4'

Conclusion

Introduction of a Java Map interface for CPS could safeguard against future changes, such as replacing the externally-developed Hazelcast with an internal CPS-based solution. It could also simplify existing code.