Versions Compared

Version Old Version 101 New Version Current
Changes made by

Francesco Fiora

Francesco Fiora

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The goals of this document is to investigate about document storage, what is the impact in the performance point of view, how it could be used to adjust flexibility of Tosca Service Template Handling, and define a clear roadmap how to reach all those goals.

Table of Contents
maxLevel2

Introduction

There are different approaches that could be used to Implement a document storage: convert a the concept model to Json text or using a database document oriented. A service template stored as document, it will make simple to have more than one service template into a db.

...

  • Convert all applications to document storage
    • performance improvements
    • Keep our current APIs, all changes will be internal
    • Provide an upgrade path to the new data structure and a rollback to the current structure
    • Old ORM layer will be not removed during this step
    • namespace and imports could be defined but not implemented
    • create more the one service template (Multi-service templates) is initially disabled in ORM layer
  • Add multi-service templates support (related to POLICY-3236 - adjust flexibility of Tosca Service Template Handling)
    • Remove old ORM layer
    • Enable namespace and Multi-service templates in ORM layer
    • Adapt the current APIs to handle Multi-service templates where it needs
    • move common code in policy-model
  • Define and support any other complex features not implemented yet
    • Enable import support

Investigation about PostgreSQL and json types

PostgreSQL offers two types for storing JSON data: json and jsonb. The json data type stores an exact copy of the input text, which processing functions must reparse on each execution; while jsonb data is stored in a decomposed binary format that makes it slightly slower to input due to added conversion overhead, but significantly faster to process, since no reparsing is needed. jsonb also supports indexing, which can be a significant advantage.

...

Code Block
languagejava
titlePersistence Model
collapsetrue
@Data
@EqualsAndHashCode(callSuper = true)
public class DocToscaServiceTemplate extends DocToscaEntity<ToscaServiceTemplate> {

    @SerializedName("data_types")
    private Map<String, @Valid DocToscaDataType> dataTypes;

    -------

    public DocToscaServiceTemplate(ToscaServiceTemplate authorativeConcept) {
        this.fromAuthorativesuper(authorativeConcept);
    }    setName(DEFAULT_NAME);
 @Override     public ToscaServiceTemplate toAuthorativesetVersion(DEFAULT_VERSION);
{ 
       final var toscaServiceTemplate = new ToscaServiceTemplate();
  -------
     }
 
    public  super.setToscaEntity(toscaServiceTemplate);DocToscaServiceTemplate(ToscaServiceTemplate authorativeConcept) {
        superthis.toAuthorativefromAuthorative(authorativeConcept);
    }

    ifpublic DocToscaServiceTemplate(dataTypesfinal !=DocToscaServiceTemplate nullcopyConcept) {
            toscaServiceTemplate.setDataTypes(DocUtil.docMapMap(dataTypes, DocToscaDataType::toAuthorative))super(copyConcept);
        }this.toscaDefinitionsVersion = copyConcept.toscaDefinitionsVersion;
   -------     this.dataTypes = DocUtil.docMapToMap(copyConcept.dataTypes, DocToscaDataType::new, new  return toscaServiceTemplate;LinkedHashMap<>());
 
        -------
     }
 
     @Override
    public ToscaServiceTemplate    @OverridetoAuthorative() {
        final var toscaServiceTemplate = new ToscaServiceTemplate();
        super.setToscaEntity(toscaServiceTemplate);
        super.toAuthorative();

        toscaServiceTemplate.setDataTypes(DocUtil.docMapToMap(dataTypes, DocToscaDataType::toAuthorative));
        -------
 
        return toscaServiceTemplate;
    }

    @Override
    public void fromAuthorative(ToscaServiceTemplate toscaServiceTemplate) {
        super.fromAuthorative(toscaServiceTemplate);
 
        if (toscaServiceTemplate.getDataTypes() != null) {
            dataTypes = DocUtil.mapDocMap(toscaServiceTemplate.getDataTypes(), DocToscaDataType::new);
        }
 
        ------- 
    }

    @Override
    public int compareTo(DocToscaEntity<ToscaServiceTemplate> otherConcept) {
        int result = compareToWithoutEntities(otherConcept);
       public voidif fromAuthorative(ToscaServiceTemplate toscaServiceTemplateresult != 0) {
        super.fromAuthorative(toscaServiceTemplate);    return result;
     if (toscaServiceTemplate.getDataTypes() != null)}
{        final DocToscaServiceTemplate  other  dataTypes = DocUtil.mapDocMap(toscaServiceTemplate.getDataTypes(), DocToscaDataType::new);(DocToscaServiceTemplate) otherConcept;

        } ------- 
         ------- return ObjectUtils.compare(toscaTopologyTemplate, other.toscaTopologyTemplate);
    }
}


In the example below the implementation of JpaToscaServiceTemplate for PostgreSQL/MariaDB (full implementation could be found here: https://gerrit.nordix.org/c/onap/policy/clamp/+/13642)

...

  • Jmeter to generate requests (same used by performance tests)
  • Prometheus for monitoring
  • DMaap simulator
  • Participant simulator
  • MariaDB/PostgreSQL/MongoDB

The existing system

Hibernate/Mariadb. Tosca Service template is saved as a schema entity relation.

Using Json in MariaDB

Hibernate/Mariadb. Tosca Service Template is saved into a longtext as Json.

Using Json in Postgres

Hibernate/PostgreSQL. Tosca Service Template is saved into a text type as Json.

MongoDB

MongoDB. Tosca Service Template and all other entities (Participants and AutomationComposition) are saved as MongoDB Document.

...