CPS-43 Operating with Module Sets (groups of modules)

Owned by Ruslan Kashapov
Last updated: Nov 24, 2020 by Toine Siebelink
4 min read

AddressesĀ CPS-43: VSE: Introduce Module Sets (groups of modules)Closed

Problem description

Source files persistence requirement

On initial model set upload all the uploaded files are expected to be treated as single package, where all the dependenciesĀ 
(imports and augmentations) are fulfilled. This allows to parse all the incoming files all together.

However the module set partial update process became complicated because of necessity to restore initial set of resource files
in order to fulfill missing dependencies. It make the source files persistence a requirement. Original files recovery also requires
extra interfaces to persistence service and extra logic to determine which file in the set requires to be replaced (if requires)



Standalone module vs module set

While we have clear dependencies of yang described module on context (surrounding modules) it's possible there could
be several modules within same dataspace having same namespace+name+revisionĀ combination, so these parameters are no
longer can be used as unique module identifiers. The actual module set (context) the is now important, while namespace
and revision became obsolete.



Solution proposal

Due to the each module isn't used out of context (surrounding/neighbor modules) it makes sense to manage themĀ as an
indivisible immutable entity. On update all the modules are expected to be provided as input data and used to replace
all the existing modules (associated with). It make unnecessary to persist source files.

Due to there is no sense to use any standalone module out of context, it makes sense to store persist module as single unit
as well (API/SPI). It will be easier to extract whole context when data is validated vs model. It will require the anchor to
reference a particular module within a defined moduleset.

The input data is expected to be provided as single file: either single yang file (set of single source file) or a zip file (multiple
yang files). The file typeĀ to be recognized as extension. This will allow to simplify the REST API - same endpoint can be used toĀ 
serve multiple modulesets. In further perspective it makes sense to add additional to a moduleset entity for easier
management/maintenance like version and enabled/disabled flag.

To Discuss/ To Decide

  • Confirm / alternate the proposal

  • Change moduleset naming/referencing - e.g. Package, Context (as in ODL yangtools)

Discussion results (Nov 20, 2020)

  1. Moduleset is immutable collection of modules. None of modules can be added or removed independently, only
    as part of moduleset insert or delete. Also moduleset itself does not assume updates for now.

  2. The moduleset uniqueness within a dataspace to be identified by string identifier now, also by content later

  3. The modules are stored as source definition, no any processing on module entity persistence.
    On data validation it's expected the schema context (representing effective models) to be compiled
    with taking into account all the modules from the moduleset.

  4. The persistence of pre-compiled context (as moduleset content) or caching it or other performance optimizations
    to be discussed later.

DB Schema etc.



  1. All IDs in above diagram present unique DB generated IDs which typically will be numeric only. I have used alphanumeric IDs for clarity

  2. Note how the same Module M4 can be included in 2 sets for the same application S1 and S2 while still storing the module content only onceĀ 

  3. Note how the same Module M4 is also included in a set (S3) for another application while still storing the content only once

  4. The tables MODULE_SET and MODULE_SET_XREF could easily be combined into one table. This would mean that fields like NAME would be stored multiple times but this would not be a big issue. The actual implementation can choose either approach whatever suites the technologies used the best.

  5. We no longer support the idea of a single module being associated with a dataspace (application) instead set of containing one module can be used

  6. The Proposed (public) Java PAI will have a method to create a module-set by providing a name and a collection of module references (namespace an revision)Ā 

  7. The java API will NOT have a method to provide module content, instead there will be a ModuleProvider (SPI) that clients will have to implement (for now CPS will provide this role as well and internally we will add a service to store modules too)

Implementation Proposal

To implement this user story I propose the following approach

  1. Create a Rest interface method that takes in datatspace name, module set name andĀ  ZIP file containing multiple yang modules
    (alternatively the method could referred to a folder in a shared FS with the yang modules)

  2. Create an internal Service to store (and validate) yang modules using the code we had before.Ā 
    This service should also implement the ModuleProvider SPI

  3. Iterate over the yang files found and store each module while building a collection of unique Module references

  4. Use Java API to create named module set for the collection from step 3

    1. use DB SPI to create and store module_set as detailed in schema above. This includes creating entries in module table too (without content)

  5. Store content of each module, using internal service from step 2

Note. Step 2 could be simplified and step 5 removed ie. NOT processing content but handling that in a separate use story