In the current implementation, ACM supports multi-participant with same supported element Type but different participantId, so they need different properties file.
In order to support replica, it needs to support multi-participant using same properties file.
Note:
- In a scenario of high number of compositions, if participant is restarting it will be slow-down the restarting action: AC-runtime will send a message for each composition primed and instance deployed to the participant.
To avoid the restarting action, participant needs a database support; - In a scenario where a participant is stuck in deploying, the instance will be in TIMEOUT and the user can take action like deploy again or undeploy. In that scenario the intermediary-participant has to receive the next message, kill the thread that is stuck in deploying and create a new thread.
Solution
Add dynamic participantId support and add database support,
Can a participant share the database with other replicas? Yes, if participants do not work with same composition/instance at same time.
Shared database with same supported element Type: a composition will be connected to a specific participantId, so only one participant will do actions with this composition and his instances. But it can fetch all compositions that are sharing.
In a restarting scenario the participant will change the participantId, and it can still fetch compositions and instances. ACM-runtime decides who has to work with, and Participant will do actions based on the message.
Changes in Participant:
- UUID participantId will be generated in memory instead to fetch it in properties file.
- cosumerGroup will be empty (kafka configuration): any intermediary-participant will have unique Kafka queue, so they will receive same message that will be filtered by participantId.
- Add client support for no-sql database.
- Add no-sql database or mock for Unit Tests.
- Refactor CacheProvider to support insert/update, intermediary-participant can still use the cache in memory.
- Any new/change composition and instance will be saved in database.
- Refactor Participants that are using own cache in memory (Policy Participant saves policy and policy type in memory)
Changes in ACM-runtime:
- When participant go OFF_LINE:
- if there are compositions connected to that participant, ACM-runtime will find other ON_LINE participant with same supported element type;
- if other ON_LINE participant is present it will change the connection with all compositions and instance;
- after that, it will execute restart for all compositions and instances to the ON_LINE participant.
- When receive a participant REGISTER:
- it will check if there are compositions connected to a OFF_LINE participant with same supported element type;
- if there are, it will change the connection with all compositions and instances to that new registered participant;
- after that it will execute restart for all compositions and instances changed.
- Refactor restarting scenario to apply the restarting only for compositions and instances in transition
Changes in docker/Kubernetes environment
- Refactor CSIT to support no-sql database
- Refactor performance and stability test to support no-sql database
- Refactor OOM to support no-sql database
Database in Ericsson ADP marketplace
The no-sql database could be one that is already in ADP marketplace,
Distributed Coordinator ED (Etcd): Distributed systems use etcd as a consistent key-value store for configuration management, service discovery, and coordinating distributed work. Many organizations use etcd to implement production systems such as container schedulers, service discovery services, and distributed data storage.
- Document Database PG (PostgreSql)
- Key Value Database AG (Apache Geode): Apache Geode provides a database-like consistency model, reliable transaction processing and a shared-nothing architecture to maintain very low latency performance with high concurrency processing.(https://geode.apache.org/docs/guide/114/getting_started/intro_to_clients.html) or (https://docs.spring.io/spring-boot-data-geode-build/1.7.5/reference/html5/).
- Key Value Database RD (Redis): RDB is NOT good if you need to minimize the chance of data loss in case Redis stops working.
- Wide Column Database CD (Apache Cassandra)
- Search Engine (Open Search): It supports Rest Api, https://opensearch.org/docs/latest/clients/java/
Open Search
Create a composition index
PUT https://{{address}}:9200/compositions
{
"settings": {
"index": {
"number_of_shards": 2,
"number_of_replicas": 1
}
},
"mappings": {
"properties": {
"compositionId": {
"type": "text"
}
}
}
}
Create a composition document
POST https://{{address}}:9200/compositions/_doc/0da7f17d-68d4-47f3-9dbf-b380413c5fad
{
"compositionId": "0da7f17d-68d4-47f3-9dbf-b380413c5fad",
"automationCompositionElementDefinitionList": [
{
"acElementDefinitionId": {
"name": "onap.policy.clamp.ac.element.K8S_StarterAutomationCompositionElement",
"version": "1.2.3"
},
"automationCompositionElementToscaNodeTemplate": {
"type": "org.onap.policy.clamp.acm.K8SMicroserviceAutomationCompositionElement",
"type_version": "1.0.0",
"properties": {
"provider": "ONAP",
"startPhase": 0,
"uninitializedToPassiveTimeout": 180,
"podStatusCheckInterval": 30
},
"name": "onap.policy.clamp.ac.element.K8S_StarterAutomationCompositionElement",
"version": "1.2.3",
"metadata": {},
"description": "Automation composition element for the K8S microservice for AC Element Starter"
},
"outProperties": {}
},
{
"acElementDefinitionId": {
"name": "onap.policy.clamp.ac.element.K8S_BridgeAutomationCompositionElement",
"version": "1.2.3"
},
"automationCompositionElementToscaNodeTemplate": {
"type": "org.onap.policy.clamp.acm.K8SMicroserviceAutomationCompositionElement",
"type_version": "1.0.0",
"properties": {
"provider": "ONAP",
"startPhase": 0,
"uninitializedToPassiveTimeout": 180,
"podStatusCheckInterval": 30
},
"name": "onap.policy.clamp.ac.element.K8S_BridgeAutomationCompositionElement",
"version": "1.2.3",
"metadata": {},
"description": "Automation composition element for the K8S microservice for AC Element Bridge"
},
"outProperties": {}
},
{
"acElementDefinitionId": {
"name": "onap.policy.clamp.ac.element.K8S_SinkAutomationCompositionElement",
"version": "1.2.3"
},
"automationCompositionElementToscaNodeTemplate": {
"type": "org.onap.policy.clamp.acm.K8SMicroserviceAutomationCompositionElement",
"type_version": "1.0.0",
"properties": {
"provider": "ONAP",
"startPhase": 0,
"uninitializedToPassiveTimeout": 180,
"podStatusCheckInterval": 30
},
"name": "onap.policy.clamp.ac.element.K8S_SinkAutomationCompositionElement",
"version": "1.2.3",
"metadata": {},
"description": "Automation composition element for the K8S microservice for AC Element Sink"
},
"outProperties": {}
}
]
}
Fetch a composition document
GET https://{{address}}:9200/compositions/_doc/0da7f17d-68d4-47f3-9dbf-b380413c5fad
Create an instance index
PUT https://{{address}}:9200/instances
{
"settings": {
"index": {
"number_of_shards": 2,
"number_of_replicas": 1
}
},
"mappings": {
"properties": {
"instanceId": {
"type": "text"
}
}
}
}
Create an instance document
POST https://{{address}}:9200/instances/_doc/caf50cde-11a2-4915-a49c-609762714a6f
{
"instanceId": "caf50cde-11a2-4915-a49c-609762714a6f",
"compositionId": "0da7f17d-68d4-47f3-9dbf-b380413c5fad",
"deployState": "DEPLOYED",
"lockState": "LOCKED",
"elements": {
"709c62b3-8918-41b9-a747-d21eb79c6c23": {
"id": "709c62b3-8918-41b9-a747-d21eb79c6c23",
"definition": {
"name": "onap.policy.clamp.ac.element.Http_SinkAutomationCompositionElement",
"version": "1.2.3"
},
"deployState": "DEPLOYED",
"lockState": "LOCKED",
"properties": {
"baseUrl": "http://10.101.0.249:30802",
"httpHeaders": {
"Content-Type": "application/json",
"Authorization": "Basic YWNtVXNlcjp6YiFYenRHMzQ="
},
"configurationEntities": [
{
"configurationEntityId": {
"name": "onap.policy.clamp.ac.sink",
"version": "1.0.0"
},
"restSequence": [
{
"restRequestId": {
"name": "request3",
"version": "1.0.1"
},
"httpMethod": "POST",
"path": "/activate",
"body": "{ \"receiverId\": { \"name\": \"onap.policy.clamp.ac.sink\", \"version\": \"1.0.0\" }, \"timerMs\": 20000, \"elementType\": \"SINK\", \"topicParameterGroup\": { \"server\": \"message-router\", \"listenerTopic\": \"POLICY_UPDATE_MSG\", \"publisherTopic\": \"AC_ELEMENT_MSG\", \"fetchTimeout\": 15000, \"topicCommInfrastructure\": \"dmaap\" } }",
"expectedResponse": 201
}
]
}
]
},
"outProperties": {}
},
"709c62b3-8918-41b9-a747-d21eb79c6c24": {
"id": "709c62b3-8918-41b9-a747-d21eb79c6c24",
"definition": {
"name": "onap.policy.clamp.ac.element.Http_StarterAutomationCompositionElement",
"version": "1.2.3"
},
"deployState": "DEPLOYED",
"lockState": "LOCKED",
"properties": {
"baseUrl": "http://10.101.0.249:30800",
"httpHeaders": {
"Content-Type": "application/json",
"Authorization": "Basic YWNtVXNlcjp6YiFYenRHMzQ="
},
"configurationEntities": [
{
"configurationEntityId": {
"name": "onap.policy.clamp.ac.starter",
"version": "1.0.0"
},
"restSequence": [
{
"restRequestId": {
"name": "request1",
"version": "1.0.1"
},
"httpMethod": "POST",
"path": "/activate",
"body": "{ \"receiverId\": { \"name\": \"onap.policy.clamp.ac.startertobridge\", \"version\": \"1.0.0\" }, \"timerMs\": 20000, \"elementType\": \"STARTER\", \"topicParameterGroup\": { \"server\": \"message-router:3904\", \"listenerTopic\": \"POLICY_UPDATE_MSG\", \"publisherTopic\": \"AC_ELEMENT_MSG\", \"fetchTimeout\": 15000, \"topicCommInfrastructure\": \"dmaap\" } }",
"expectedResponse": 201
}
]
}
]
},
"outProperties": {}
},
"709c62b3-8918-41b9-a747-d21eb79c6c25": {
"id": "709c62b3-8918-41b9-a747-d21eb79c6c25",
"definition": {
"name": "onap.policy.clamp.ac.element.Http_BridgeAutomationCompositionElement",
"version": "1.2.3"
},
"deployState": "DEPLOYED",
"lockState": "LOCKED",
"properties": {
"baseUrl": "http://10.101.0.249:30801",
"httpHeaders": {
"Content-Type": "application/json",
"Authorization": "Basic YWNtVXNlcjp6YiFYenRHMzQ="
},
"configurationEntities": [
{
"configurationEntityId": {
"name": "onap.policy.clamp.ac.bridge",
"version": "1.0.0"
},
"restSequence": [
{
"restRequestId": {
"name": "request2",
"version": "1.0.1"
},
"httpMethod": "POST",
"path": "/activate",
"body": "{ \"receiverId\": { \"name\": \"onap.policy.clamp.ac.bridgetosink\", \"version\": \"1.0.0\" }, \"timerMs\": 20000, \"elementType\": \"BRIDGE\", \"topicParameterGroup\": { \"server\": \"message-router:3904\", \"listenerTopic\": \"POLICY_UPDATE_MSG\", \"publisherTopic\": \"AC_ELEMENT_MSG\", \"fetchTimeout\": 15000, \"topicCommInfrastructure\": \"dmaap\" } }",
"expectedResponse": 201
}
]
}
]
},
"outProperties": {}
}
},
"stateChangeResult": "NO_ERROR"
}
Fetch an instance document
GET https://{{address}}:9200/instances/_doc/caf50cde-11a2-4915-a49c-609762714a6f