Page Comparison

A design idea
Why not Camunda Cockpit as is: current Camunda Cockpit was designed from a BPMN process management perspective (note: need to study for TOSCA cases).
- It does not meet service-level orchestration monitoring.
- It is designed for BPMN definition/execution monitoring; require process knowledge for monitoring.

We need higher-level monitoring abstraction for both BPMN and TOSCA.

Associate Service Instance Id (or other keys) to the top-level process instance id. For the association,
Could use a process variable holding the Service Instance id (or other keys), or
Could use a database holding the association

Allow VID, UUI or external apps

Rationale

From the service/network operator perspective, the BPMN Monitoring has been realized as a pain point since the Amsterdam release. "Finding right process instance(s) and data for the service / resource request is tedious and hassle."
Camunda is not an OSS vendor, and they don't have any business value to cover the OSS opearator-friendly monitoring.
There is Camunda cockpit, but regardless community or enterprise editions, they were designed from a BPMN process management perspective.
- They require the user BPMN definition and design knowledge for monitoring.
- The user needs to know which service invokes which BPMN process definitions and how those process definitions are related to each other.
- There could be thousands process instances daily, and a top-level process instance can have multiple sub-process instances (call activity instances).

- - It is not easy to find right process instances and data
We need more than what Camunda Community/Enterprise editions offer to
- Facilitate the finding of associated process instances and data
  - Allow VID, UUI or external application users monitor process workflow process (graphically and text-based) based on
  extensible
  - their search
  keys.
We need a platform level runtime and history process activity report capabilties out of the box.
- Regardless use of Camunda Enterprise Edition or Community Edition.
- Query to Camunda/ARIA database to extract activities.
  - criteria.
- Bypassing all the tedious manual search through process definitions and instances
- Eliminating prerequisite knowledge of process definition relationships for monitoring - learn as you drill down and up
  - Needs drill down and drill up to cover sub-process instances graphically and text-based.
- Troubleshooting workflows
The following diagram depicts the high-level concept.

Image Modified

Rationale
- Search is an Camunda enterprise feature, but we need to provide searching capabilitiy for non-enterprise edition.
  - Finding right process instance(s) for a NS/VNF service request is tedious and hassle.
  - To facilitate monitoring, we need more than what Camunda Community/Enterprise edition supports.
  - provides the process monitoring (instance-search) hyperlink to the SO clients for launching process monitoring.
    - Automates tedious manual steps for finding target process instance(s), bypassing tedious search starting from the process defintions.
    - Access customized SO Service List / Camunda Cockpit widgets from VID, UUI and external APIs.
- If a service provider uses Camunda Enterprise edition, they can still utilize this SO monitoring on top of Camunda enterprise edition features.
- Many of Camunda enterprise features such as CRUDV and version control of process definitions would be part of SDC.
  - ONAP separated workflow design and runtime.
  - Several enterprise features are not part of SO monitoring features and are not applicable.
- What are current TOSCA orchestrator monitoring capabilities?
  - SO monitoring should cover both imperative and declarative orchestration.
  - Question: can we have a kind of uniform way of monitoring?

High-Level Requirements
- Dashboard views of Service lists
  - Filtering capabilities based on search criteria
  - Configurable search criteria
- Dashboard views of statistics (donuts, pie charts, etc.) for filtered service instances
- Service Instance Rendering and detail panel views
  - with sub-service instance drill-drown and drill-up capabilities
    - A service instance could be realized by multiple process instances
  - with process / task detail
  - Topology (workflow) views during/after orchestration
- Input/output data views for process/task/service task (messages, parameters)
  - Display on service / task detail panel
  - Provide message log views (could be on a pop-up widget)
- Color coding/visual indication of statistic and service type and status
- Troubleshooting capabilities by manipulating the workflow during orchestration for troubleshooting and retry from the current location (stretch goal)
- TOSCA orchestration monitoring (stretch goal)

UI Widgets
- Service List
- Service Statistic
- Service Instance Rendering and Detail

Widget Requirements and Design*** Note High-Level Requirements
- Dashboard views of Service lists
  - Filtering capabilities based on search criteria
  - Configurable search criteria
- Dashboard views of statistics (donuts, pie charts, etc.) for filtered request/service instances
- Process / Service Instance Rendering and detail panel views
  - with sub-process/service instance drill-drown and drill-up capabilities
    - A process/service instance could be realized by multiple process instances
  - with process / task detail
  - Topology (workflow) views during/after orchestration
  - Provide process activity trails to identity the execution path and failure indication
- Input/output data views for process/task/service task (messages, parameters)
  - Display on service / task detail panel
  - Provide message log views (could be on a pop-up widget)
- Color coding/visual indication of statistic and task execution trails

Design
- Collect a service list based on search criteria through the REQUEST Database table.
  - Service Instance Id,
  - Request Id
  - Service Name
  - Status
  - Date
- Display service list statistic (COMPLETED, IN_PROGRESS, FAILED, PENDING) for the selected service list.
- Provides the hyperlinks between the service list to Camunda Process instances to
  - Automate tedious manual steps for finding target process instance(s), bypassing tedious search starting from the process definitions
  - Allow VID, UUI and external API users monitor their process instances and data
- Use Camunda REST APIs to:
  - Associate the Request Id / Service Instance Id (and other search criteria) to the top-level process instance without manual intervention
  - Use a process variable holding the Request Id for the process instance search
  - Get the process definition XML for workflow rendering
  - Get the process instance details such as activities and variables
  - User process activity data for 1) execution trail, 2) drill-down and drill-up.
- Display and render collected process instance data on the pages
  - Workflow Diagram canvas (zoom in/out and scrolling)
  - Process Information panel
  - Activity Instances and Process variable panels

UI Widgets
- Service List
- Service Statistic
- Process/Service Instance Rendering and Detail

Widget Requirements and Design
- *** Note 1: actual screen layout and display can be changed based on implementation decisions. ***
- *** Note 2: the service list panel columns can be changed based on the database table columns. The important fields would be the request id / service instance id, requester id, date/time for filtering.***Service List Widget
- *** Note 3: we can add a new field, Proc_Inst-ID varchar(64) to the Infra_Active_Requests database table, to associate it (as a top-level process instance id) with the service instance id
- Service List Widget
  - Capabilities
    - Provides monitoring capabilities for processed services based on search criteria
      - Configurable Search Criteria filtering: Service ID, Operation Type, Status, User Id, Date/Time range
      - Actual filtering criteria fields could be changed based on configuration
    - The Service List panel will display the filtered Service list.
    - This widget can be triggered from other UIs (VID, UUI, external applications).
  - Design
    - The search criteria fields will be defined in a search criteria field configuration file (e.g., so-monitoring-config.yaml). So, they can be customized and support internationalization.
    - Based on the fields and buttons configuration, the Service List widget will render the corresponding fields and buttons.
    - The search criteria fields support the field validation based on the field type; e.g., length, date format, time format.
    - The Service List panel supports pagination, which will be controlled by the Navigation buttons.
    - Queries a service list from the SO Request DB through the Monitoring REST APIs.
    - Action buttons are enabled/disabled based on data population and/or context.
    - The wild characters are supported for the Search fields.
    - Note: for security (who, when and how access info), the requester info (User Id) is needed. It could be a future requirement.
      - The flexible filtering expression could be a stretch goal (it depends the BE Request DB query capabilities)

Image Removed

...

Image Added

- Search Operations:
  - Capabilities
    - Retrieve a service list thru the Service List REST API, getServiceList(...)

service-list-search-criteria-fields:

user-id:

type: String

max-length: 30

validation: alpha-numeric

service-id:

type: String

max-length: 30

validation: alpha-numeric

operation-type:

type: String

max-length: 30

validation: alpha-numeric

status:

type: combo-list

max-length: 30

default-value:

- status-item: finished

- status-item: processing

- status-item: error

date-from:

type: date

validation: date-format

time-from:

type: time

validation: time-format

date-to:

type: date

validation: date-format

time-to:

type: time

valiation: time-format

filtering-expression:

type: string

validation: regular-expression

action-buttions:

search:

statistic:

monitor:

service-list:

max-row: 25000

service-instance-rendering-details:

max-page-num: 10

- Search Operations:
  - Capabilities
    - Retrieve a service list thru the Service List REST API, getServiceList(...)
    - Provides filtering, Service ID, Operation Type, Status, Date/Time range (actual criteria could be configured).
    - Click the Search button, and the filtered service list will be displayed.
  - Design
    - After the user fills up the desired search crtieria fields, and click the Search button.
    - Use a REST API, getServiceList with search criteria parameters based on a new getInfraRequest(...)
    - Provide pagnation for the list, and enable/disable the navigation buttons based on the page location.
      - When the retreieved rows exceed the max-row configuration value, provide the user a warning to narrow the search scope.
    - When the Service List data is NOT populated, only the Search button is enabled; rest buttons are disabled.
    - When the Service List data is populated, enable the navigation, Statistic and Monitor buttons.
    - Stretch goals
      - sort the list ascending and descending order per column
      - change the column position

Image Removed

Service List Get Service Instance Monitoring
- Capabilities
  - Select a row and click the Monitor button. Then, it will open the Service instance (maps into process instance in Camunda) monitoring widget for the selected Service instance on the secondary/pop-up browser page.
- Design
  - when a user clicks on a row, highlight the selected row and enable the Monitor button.
  - Once the Monitor button is clicked, go to the Service Instance Rendering and Deail widget, passing key fields such as service instance id, which will be mapped into the process instance id to find a matched process instance id in Camunda database in the Service Instnce Rendering and Detail widget.

Image Removed

- Service Statistic Widget (stretch goal)
  - When, the static button is clicked, the Service Statistic dashboard will be shown.
  - Collect and display Service Instance statistic per filtered service list, for the filtered service range.
  - Design
    - Query for service list data for a given search criteria.
      - Use a REST API, getServiceList with search criteria parameters based on a new getInfraRequest(...)
    - Count the total number per status (Finished, Processing, Error)
    - Status Total / total number

Image Removed

- Service Instance Rendering and Detail Widget
  - Capabilities
    - Get a process definition XML through the Service Id and Process instance association.
    - use Camunda REST API, GET /process-defintion/{id}/xml
    - Get the state of a process instance from the activity-instances
    - Use Camunda REST API, get
    - Provides filtering, Service ID, Operation Type, Status, Date/Time range (actual criteria could be configured).
    - Click the Search button, and the filtered service list will be displayed.
  - Design
    - After the user fills up the desired search crtieria fields, and click the Search button.
    - Use a REST API, getServiceList with search criteria parameters based on a new getInfraRequest(...)
    - Provide pagnation for the list, and enable/disable the navigation buttons based on the page location.
      - When the retreieved rows exceed the max-row configuration value, provide the user a warning to narrow the search scope.
    - When the Service List data is NOT populated, only the Search button is enabled; rest buttons are disabled.
    - When the Service List data is populated, enable the navigation, Statistic and Monitor buttons.
    - Stretch goals
      - sort the list ascending and descending order per column
      - change the column position

Image Added

Service List Get Process / Service Instance Monitoring
- Capabilities
  - Select a row and click the Monitor button. Then, it will open the process / Service instance (maps into process instance in Camunda) monitoring widget for the selected process / Service instance on the secondary/pop-up browser page.
- Design
  - when a user clicks on a row, highlight the selected row and enable the Monitor button.
  - Once the Monitor button is clicked, go to the process / Service Instance Rendering and Deail widget, passing key fields such as process / service instance id, which will be mapped into the process instance id to find a matched process instance id in Camunda database in the Service Instance Rendering and Detail widget.

Image Added

- Service Statistic Widget
  - When, the statistics panel is clicked, the Service Statistic dashboard will be shown.
  - Collect and display Request / Service Instance statistic per filtered service list, for the filtered service range.
  - Design
    - Query for service list data for a given search criteria.
      - Use a REST API, getServiceList with search criteria parameters based on a new getServiceStatic(...) based on the getInfraRequest(...)
    - Count the total number per status (Finished, Processing, Error)
    - Status Total / total number

Image Added

- Process / Service Instance Rendering and Detail WidgetThe following is a possible alternative service instance rendering based on the Camunda Cockpit screen. If we go this path, the Camunda Cockpit customized plugin would be used.
  - Capabilities
    - Get a process definition XML through the Service Id and Process instance association.
    - use Camunda REST API, GET /process-defintion/{id}/xml
    - Get the state of a process instance from the activity-instances
    - Use Camunda REST API, get/process-instance/{id}/activity-instances
    - Render the BPMN XML with bpmn.io (or equivalent) and places markers on top of it, and provide process / Service instance detail views.
  - Design
    - This widget will be on the secondary/pop-up browser page.
      - The Service List page will be the primary page, so the user can open multiple Process / Service Instance Rendering and Detail pages.
    - The search criteria fields are for display only, and they cannot be changed.
    - The Detail panel is scrollable and display contents with the tag: value formt.
    - Query a process instance with the matched 'serviceInstanceId' process variable.
      - get a process instande id: query bases on the process variable,
        GET /history/variable-instance/{variableValue}, where {variableValue} = 'serviceIntanceId' ;
      - query BPML XML based on a process instance id
        GET /process-definition/{id}/xml
      - query the process instance detail such as process variables
        GET /process-instance/{id}/
      activity-instances
      - variables
    - Render the BPMN XML with bpmn.io and places markers on top of it, and provide Service instance detail views.
  - Design
    - This widget will be on the secondary/pop-up browser page.
      - The Service List page will be the primiary page, so the user can open multiple Service Instance Rendering and Detail pages.
      - The number of pages can be limited to the default 10, but it can be configured.
    - The search criteria fields are for display only, and they cannot be changed.
    - The Detail panel is scrollable and display contents with the tag: value formt.
    - Query a process instance with the matched 'serviceInstanceId' process variable.
      - get a process instande id: query bases on the process variable,
        GET /history/variable-instance/{variableValue}, where {variableValue} = 'serviceIntanceId' ;
      - query BPML XML based on a process instance id
        GET /process-definition/{id}/xml
      - query the process instance detail such as process variables
        GET /process-instance/{id}/variables
        Display the process variables on the Service Instance Detail panel
      - query and activity instance (tree) for the process instance id
        GET /process-instance/{id}/activity-instances
        GET /history/activity-instance/{id}
        The response provides child activity instance info. By using this we can find the parent and child process instnce id and definition id.
        If the current process instance id has its parent, enable the Drill-Up button.
        When a user clicks on a task, display the task varables
        GET /task/{id}variables
        When the task activity type is call activity, it enables the Drill-Down button.
    - Render the process instance diagram with the retrieved XML.
      - use bpmn.io javascript library for the XML rendering and event attachments for process instance tasks.

bpmnViewer.importXML (xmlDiagram, function(err) {

if (err) { ... }

var canvas = bpmnViewer.get('canvas');

canvas.zoom ('fit-viewport');

var eventBus = bpmnViewer.get('eventBus');

// you may hook into any of the following events
var events = [
'element.hover',
'element.out',
'element.click',
'element.dblclick',
'element.mousedown',
'element.mouseup'
];

events.forEach(function(event) {

eventBus.on(event, function(e) {

log(event, 'on', e.element.id);

- - - The Close button is always enabled.

Image Removed

- - - - Display the process variables on the Process / Service Instance Detail panel
      - query and activity instance (tree) for the process instance id
        GET /process-instance/{id}/activity-instances
        GET /history/activity-instance/{id}
        The response provides child activity instance info. By using this we can find the parent and child process instnce id and definition id.
        If the current process instance id has its parent, enable the Drill-Up button.
        When a user clicks on a task, display the task varables
        GET /task/{id}variables
        When the task activity type is call activity, it enables the Drill-Down button.
    - Render the process instance diagram with the retrieved XML.
      - use bpmn.io javascript library for the XML rendering and event attachments for process instance tasks.

bpmnViewer.importXML (xmlDiagram, function(err) {

if (err) { ... }

var canvas = bpmnViewer.get('canvas');

canvas.zoom ('fit-viewport');

var eventBus = bpmnViewer.get('eventBus');

// you may hook into any of the following events
var events = [
'element.hover',
'element.out',
'element.click',
'element.dblclick',
'element.mousedown',
'element.mouseup'
];

events.forEach(function(event) {

eventBus.on(event, function(e) {

log(event, 'on', e.element.id);

- - - The Close button is always enabled.

Image Added

The following is a possible alternative process / service instance rendering based on the Camunda Cockpit screen. If we go this path, the Camunda Cockpit customized plugin would be used. This path was NOT chosen.

Calling the process instance cockpit page

get the processInstanceId from the getProcessInstanceId(String serviceInstanceId).

call the process instance cockpit page, e.g., following a function like below:

$scope.getProcessInstanceUrl = function(processInstance, params) {
  var path = '#/process-instance/' + processInstance.id;
  var searches = angular.extend({}, ($location.search() || {}), (params || {}));

  var keepSearchParams = [ 'viewbox' ];
  for (var i in params) {
    keepSearchParams.push(i);
  }

  return routeUtil.redirectTo(path, searches, keepSearchParams);
};

- Task Drill-Down/Drill-Up and Detail View
  - Capabilities
    - Support Task (CAll Call Activity) drill-down/drill-up capabiltiescapabilities.
    - When a call activity task is selected, the Drill-Down button will be enabled.
    - Provide detail views for the selected Task.
    - When a task is selected including the call activity, the Detail view panel will display details for the selected task.
  - Design
    - When a selected/clicked task activity type is the Call Activity, the Drill-Down button is enabled.
      - if (activityType.equals('CallActivity') {enableDrillDown();}
    - If the current process instance has its parent process instance (from the activity tree), the Drill-Up button is enabled.
      - if (parentActivityInstanceId != null) {enableDrillUp();}

Image RemovedImage Added

- Sub-Process/Service Instance Rendering and Detail View
  - Capabilities
    - Get a sub process defintion XML through the Service Id and Process Instance association.
    - Use Camunda REST API, get/process-definition/{sub-id}/xml
    - Get the state of a sub-process instance from the activity-instances.
    - Use Camunda REST API, get/process-instance/{sub-id}/activity-instances
    - Render the BPMN XML with bpmn.io and places markers on top of it, and provides process/service instance detail views.
    - This widget uses the same code of the Process/Service Instance Rendering and Detail View with the sub-process instance id.
    - The Drill-Up button will be enabled.
  - Design
    - This widget code is the same as the Process/Service Instance Rendering and Detail.
    - The difference is the passed process id; the child process instance id will be passed.

Image RemovedImage Added

REST APIs for providing data to UIsUIs (note: actual code will be different based on the implementation decisions
- Capabilities
  - Provides REST services, by utilzing 1) Camunda REST APIs, such as BPMN XML string, process activity data, process variable, statistic, and 2) SO Request DB APIs for a service list.
  - Consolidate data responses from multple Camunda calls (both from process and history database) and feed them to UIs.
  - Use of HistoryService APIs, example, processEngine.getHistoryService().createHistoricProcessVariableQuery().xyz
  - Set a History level to ACTIVITY as a minimum; AUDIT (default) level for process variable tracing; currently, it is set to FULL
    - ProcessEngine processEngine = ProcessEngineConfiguration.createProcessEngineConfigurationFromResourceDefault().setHistory(ProcessEngineConfiguration.HISTORY_FULL).buildProcessEngine()
  - Provides workflow tracing (between parent-child workflows, interaction with other services; service task in and out); example,
    - processEngine.getRuntimeService().createExecutionQuery().processVariableValueEquals("serviceInstanceId", serviceInstanceId).singleResult();
  - Custom Query
    - Custom Query against History ACT_HI_DETAIL database table, as needed
  - Create a REST API, getServiceList with search criteria
    - Enhance MsoRequestsDbAdapter and MsoRequestsDbAdapterImpl to implement getInfraRequest(...) with additional parameters
    - invoke getInfraRequest(...) to collect service list data based on search criteria.
    - or use the RequestDB adapter directly.
  Design
  - Write REST APIs in the new mso-api-handlers/mso-api-handler-monitoring
  - getServiceList(Map searchCriteria...) invokes new getInfraRequest(Map searchCriteria...)
  - getProcessInstanceId(String serviceInstanceId);
  - getprocessDiagram(String processInstanceId);
  - getprocessInstanceDetail(String processInstanceId);
  - Custom Query
    - Custom Query against History ACT_HI_DETAIL database table, as needed
  - Create a REST API, getServiceList with search criteria
    - Enhance MsoRequestsDbAdapter and MsoRequestsDbAdapterImpl to implement getInfraRequest(...) with additional parameters
    - invoke getInfraRequest(...) to collect service list data based on search criteria.
    - or use the RequestDB adapter directly.
- Design
  - Write REST APIs in the new mso-api-handlers/mso-api-handler-monitoring
  - The following APIs are out-dated. Look at the SO-Monitor-APIs for Implementation detail, SO Monitoring API

getServiceList(Map searchCriteria...) // invokes new getInfraRequest(Map searchCriteria...)
1. getServiceStatistic(Map searchCriteria, String status); // get the statistic percentage for the status
getProcessInstanceId(String serviceInstanceId); // get the processInstanceId through GET /history/variable-instance/{id}
getprocessDiagramXML(String processInstanceId); // get a process instance, get a process definition id, and get diagram
getprocessInstanceDetail(String processInstanceId);
getHistoryActivityInstance(String processInstanceId); // GET /history/activity-instance?processInstanceId={processInstanceId}
getProcessInstanceParent (String processInstanceId);
getProcessInstanceChildren(String processInstanceId);
getProcessTaskDetail(String processInstanceId, String taskId);
getProcessTaskType(String processInstanceId, String taskid);

...

Image Removed

...

// query if this task type is "Call Activity" for drilling down
- The following diagram depicts the API interaction.

Image Added

Component Build Dependencies
- The following diagram depicts the component build dependencies.
- Components can be built in parallel with a certain level of sequential integration.
- Each estimate includes code development and unit testing plus some component integration and integration testing (since there are API dependencies).

Image RemovedImage Added

Estimates (T-Shirt Size: M)
- Note:
  - It is a rough estimate - to be refined
  - Estimate includes development and unit testing time
  - As mentioned in the above section, the components can be built in parallel.
  - 5 hours / day

Priority	Component	Development Estimate	Comments

Priority

Component

Development Estimate

Comments

1	REST APIs for providing data to UIs

80

40 hours / Java developer	REST facade for collecting and consolidating various Camunda process instance and history data to simplify GUI data collection interaction. collecting SO Request DB data
2	SO Service List widget

80

60 hours / UI developer

populate search filter criteria
Displays a service list panel
control action buttons

3

Process/Service Instance Rendering

and Detail Panel

120

60 hours / UI developer	Renders process/service instance workflow graphically provides the process/ service instance details and selected task details Provides drill-down and drill-up capabilities
4	Statistic Dashboard (stretch goal)	40 hours / UI developer	Displays statistic for the selected service scope
5	~~Custom History Producer (stretch goal)~~	~~40 hours / Java developer~~	Not need for Casablanca Produce history with additional data population This is an option, and is used only if the Camunda history data is not sufficient.

Versions Compared

Old Version 17

New Version Current

Key