Rationale
- From the service/network operator perspective, the BPMN Monitoring has been realized as a pain point since the Amsterdam release. "Finding right process instance(s) and data for the service / resource request is tedious and hassle."
- Camunda is not an OSS vendor, and they don't have any business value to cover the OSS opearator-friendly monitoring.
- There is Camunda cockpit, but regardless community or enterprise editions, they were designed from a BPMN process management perspective.
  - They require the user BPMN definition and design knowledge for monitoring.
  - The user needs to know which service invokes which BPMN process definitions and how those process definitions are related to each other.
  - There could be thousands process instances daily, and a top-level process instance can have multiple sub-process instances (call activity instances).
- - - It is not easy to find right process instances and data
- We need more than what Camunda Community/Enterprise editions offer to
  - Facilitate the finding of associated process instances and data
    - Allow VID, UUI or external application users monitor process workflow process (graphically and text-based) based on their search criteria.
  - Bypassing all the tedious manual search through process definitions and instances
  - Eliminating prerequisite knowledge of process definition relationships for monitoring - learn as you drill down and up
    - Needs drill down and drill up to cover sub-process instances graphically and text-based.
  - Troubleshooting workflows
- The following diagram depicts the high-level concept.

High-Level Requirements
- Dashboard views of Service lists
  - Filtering capabilities based on search criteria
  - Configurable search criteria
- Dashboard views of statistics (donuts, pie charts, etc.) for filtered request/service instances
- Process / Service Instance Rendering and detail panel views
  - with sub-process/service instance drill-drown and drill-up capabilities
    - A process/service instance could be realized by multiple process instances
  - with process / task detail
  - Topology (workflow) views during/after orchestration
  - Provide process activity trails to identity the execution path and failure indication
- Input/output data views for process/task/service task (messages, parameters)
  - Display on service / task detail panel
  - Provide message log views (could be on a pop-up widget)
- Color coding/visual indication of statistic and task execution trails

Design
- Collect a service list based on search criteria through the REQUEST Database table.
  - Service Instance Id,
  - Request Id
  - Service Name
  - Status
  - Date
- Display service list statistic (COMPLETED, IN_PROGRESS, FAILED, PENDING) for the selected service list.
- Provides the hyperlinks between the service list to Camunda Process instances to
  - Automate tedious manual steps for finding target process instance(s), bypassing tedious search starting from the process definitions
  - Allow VID, UUI and external API users monitor their process instances and data
- Use Camunda REST APIs to:
  - Associate the Request Id / Service Instance Id (and other search criteria) to the top-level process instance without manual intervention
  - Use a process variable holding the Request Id for the process instance search
  - Get the process definition XML for workflow rendering
  - Get the process instance details such as activities and variables
  - User process activity data for 1) execution trail, 2) drill-down and drill-up.
- Display and render collected process instance data on the pages
  - Workflow Diagram canvas (zoom in/out and scrolling)
  - Process Information panel
  - Activity Instances and Process variable panels

UI Widgets
- Service List
- Service Statistic
- Process/Service Instance Rendering and Detail

Widget Requirements and Design
- *** Note 1: actual screen layout and display can be changed based on implementation decisions. ***
- *** Note 2: the service list panel columns can be changed based on the database table columns. The important fields would be the request id / service instance id, requester id, date/time for filtering.***
- *** Note 3: we can add a new field, Proc_Inst-ID varchar(64) to the Infra_Active_Requests database table, to associate it (as a top-level process instance id) with the service instance id
- Service List Widget
  - Capabilities
    - Provides monitoring capabilities for processed services based on search criteria
      - Configurable Search Criteria filtering: Service ID, Operation Type, Status, User Id, Date/Time range
      - Actual filtering criteria fields could be changed based on configuration
    - The Service List panel will display the filtered Service list.
    - This widget can be triggered from other UIs (VID, UUI, external applications).
  - Design
    - The search criteria fields will be defined in a search criteria field configuration file (e.g., so-monitoring-config.yaml). So, they can be customized and support internationalization.
    - Based on the fields and buttons configuration, the Service List widget will render the corresponding fields and buttons.
    - The search criteria fields support the field validation based on the field type; e.g., length, date format, time format.
    - The Service List panel supports pagination, which will be controlled by the Navigation buttons.
    - Queries a service list from the SO Request DB through the Monitoring REST APIs.
    - Action buttons are enabled/disabled based on data population and/or context.
    - The wild characters are supported for the Search fields.
    - Note: for security (who, when and how access info), the requester info (User Id) is needed. It could be a future requirement.
      - The flexible filtering expression could be a stretch goal (it depends the BE Request DB query capabilities)

- Search Operations:
  - Capabilities
    - Retrieve a service list thru the Service List REST API, getServiceList(...)
    - Provides filtering, Service ID, Operation Type, Status, Date/Time range (actual criteria could be configured).
    - Click the Search button, and the filtered service list will be displayed.
  - Design
    - After the user fills up the desired search crtieria fields, and click the Search button.
    - Use a REST API, getServiceList with search criteria parameters based on a new getInfraRequest(...)
    - Provide pagnation for the list, and enable/disable the navigation buttons based on the page location.
      - When the retreieved rows exceed the max-row configuration value, provide the user a warning to narrow the search scope.
    - When the Service List data is NOT populated, only the Search button is enabled; rest buttons are disabled.
    - When the Service List data is populated, enable the navigation, Statistic and Monitor buttons.
    - Stretch goals
      - sort the list ascending and descending order per column
      - change the column position

Service List Get Process / Service Instance Monitoring
- Capabilities
  - Select a row and click the Monitor button. Then, it will open the process / Service instance (maps into process instance in Camunda) monitoring widget for the selected process / Service instance on the secondary/pop-up browser page.
- Design
  - when a user clicks on a row, highlight the selected row and enable the Monitor button.
  - Once the Monitor button is clicked, go to the process / Service Instance Rendering and Deail widget, passing key fields such as process / service instance id, which will be mapped into the process instance id to find a matched process instance id in Camunda database in the Service Instance Rendering and Detail widget.

- Service Statistic Widget
  - When, the statistics panel is clicked, the Service Statistic dashboard will be shown.
  - Collect and display Request / Service Instance statistic per filtered service list, for the filtered service range.
  - Design
    - Query for service list data for a given search criteria.
      - Use a REST API, getServiceList with search criteria parameters based on a new getServiceStatic(...) based on the getInfraRequest(...)
    - Count the total number per status (Finished, Processing, Error)
    - Status Total / total number

- Process / Service Instance Rendering and Detail Widget
  - Capabilities
    - Get a process definition XML through the Service Id and Process instance association.
    - use Camunda REST API, GET /process-defintion/{id}/xml
    - Get the state of a process instance from the activity-instances
    - Use Camunda REST API, get/process-instance/{id}/activity-instances
    - Render the BPMN XML with bpmn.io (or equivalent) and places markers on top of it, and provide process / Service instance detail views.
  - Design
    - This widget will be on the secondary/pop-up browser page.
      - The Service List page will be the primary page, so the user can open multiple Process / Service Instance Rendering and Detail pages.
    - The search criteria fields are for display only, and they cannot be changed.
    - The Detail panel is scrollable and display contents with the tag: value formt.
    - Query a process instance with the matched 'serviceInstanceId' process variable.
      - get a process instande id: query bases on the process variable,
        GET /history/variable-instance/{variableValue}, where {variableValue} = 'serviceIntanceId' ;
      - query BPML XML based on a process instance id
        GET /process-definition/{id}/xml
      - query the process instance detail such as process variables
        GET /process-instance/{id}/variables
        Display the process variables on the Process / Service Instance Detail panel
      - query and activity instance (tree) for the process instance id
        GET /process-instance/{id}/activity-instances
        GET /history/activity-instance/{id}
        The response provides child activity instance info. By using this we can find the parent and child process instnce id and definition id.
        If the current process instance id has its parent, enable the Drill-Up button.
        When a user clicks on a task, display the task varables
        GET /task/{id}variables
        When the task activity type is call activity, it enables the Drill-Down button.
    - Render the process instance diagram with the retrieved XML.
      - use bpmn.io javascript library for the XML rendering and event attachments for process instance tasks.

bpmnViewer.importXML (xmlDiagram, function(err) {

if (err) { ... }

var canvas = bpmnViewer.get('canvas');

canvas.zoom ('fit-viewport');

var eventBus = bpmnViewer.get('eventBus');

// you may hook into any of the following events
var events = [
'element.hover',
'element.out',
'element.click',
'element.dblclick',
'element.mousedown',
'element.mouseup'
];

events.forEach(function(event) {

eventBus.on(event, function(e) {

log(event, 'on', e.element.id);

- - - The Close button is always enabled.

The following is a possible alternative process / service instance rendering based on the Camunda Cockpit screen. If we go this path, the Camunda Cockpit customized plugin would be used. This path was NOT chosen.

Calling the process instance cockpit page

get the processInstanceId from the getProcessInstanceId(String serviceInstanceId).

call the process instance cockpit page, e.g., following a function like below:

$scope.getProcessInstanceUrl = function(processInstance, params) {
  var path = '#/process-instance/' + processInstance.id;
  var searches = angular.extend({}, ($location.search() || {}), (params || {}));

  var keepSearchParams = [ 'viewbox' ];
  for (var i in params) {
    keepSearchParams.push(i);
  }

  return routeUtil.redirectTo(path, searches, keepSearchParams);
};

- Task Drill-Down/Drill-Up and Detail View
  - Capabilities
    - Support Task (Call Activity) drill-down/drill-up capabilities.
    - When a call activity task is selected, the Drill-Down button will be enabled.
    - Provide detail views for the selected Task.
    - When a task is selected including the call activity, the Detail view panel will display details for the selected task.
  - Design
    - When a selected/clicked task activity type is the Call Activity, the Drill-Down button is enabled.
      - if (activityType.equals('CallActivity') {enableDrillDown();}
    - If the current process instance has its parent process instance (from the activity tree), the Drill-Up button is enabled.
      - if (parentActivityInstanceId != null) {enableDrillUp();}

- Sub-Process/Service Instance Rendering and Detail View
  - Capabilities
    - Get a sub process defintion XML through the Service Id and Process Instance association.
    - Use Camunda REST API, get/process-definition/{sub-id}/xml
    - Get the state of a sub-process instance from the activity-instances.
    - Use Camunda REST API, get/process-instance/{sub-id}/activity-instances
    - Render the BPMN XML with bpmn.io and places markers on top of it, and provides process/service instance detail views.
    - This widget uses the same code of the Process/Service Instance Rendering and Detail View with the sub-process instance id.
    - The Drill-Up button will be enabled.
  - Design
    - This widget code is the same as the Process/Service Instance Rendering and Detail.
    - The difference is the passed process id; the child process instance id will be passed.

REST APIs for providing data to UIs (note: actual code will be different based on the implementation decisions
- Capabilities
  - Provides REST services, by utilzing 1) Camunda REST APIs, such as BPMN XML string, process activity data, process variable, statistic, and 2) SO Request DB APIs for a service list.
  - Consolidate data responses from multple Camunda calls (both from process and history database) and feed them to UIs.
  - Use of HistoryService APIs, example, processEngine.getHistoryService().createHistoricProcessVariableQuery().xyz
  - Set a History level to ACTIVITY as a minimum; AUDIT (default) level for process variable tracing; currently, it is set to FULL
    - ProcessEngine processEngine = ProcessEngineConfiguration.createProcessEngineConfigurationFromResourceDefault().setHistory(ProcessEngineConfiguration.HISTORY_FULL).buildProcessEngine()
  - Provides workflow tracing (between parent-child workflows, interaction with other services; service task in and out); example,
    - processEngine.getRuntimeService().createExecutionQuery().processVariableValueEquals("serviceInstanceId", serviceInstanceId).singleResult();
  - Custom Query
    - Custom Query against History ACT_HI_DETAIL database table, as needed
  - Create a REST API, getServiceList with search criteria
    - Enhance MsoRequestsDbAdapter and MsoRequestsDbAdapterImpl to implement getInfraRequest(...) with additional parameters
    - invoke getInfraRequest(...) to collect service list data based on search criteria.
    - or use the RequestDB adapter directly.
- Design
  - Write REST APIs in the new mso-api-handlers/mso-api-handler-monitoring
  - The following APIs are out-dated. Look at the SO-Monitor-APIs for Implementation detail, SO Monitoring API

getServiceList(Map searchCriteria...) // invokes new getInfraRequest(Map searchCriteria...)
1. getServiceStatistic(Map searchCriteria, String status); // get the statistic percentage for the status
getProcessInstanceId(String serviceInstanceId); // get the processInstanceId through GET /history/variable-instance/{id}
getprocessDiagramXML(String processInstanceId); // get a process instance, get a process definition id, and get diagram
getprocessInstanceDetail(String processInstanceId);
getHistoryActivityInstance(String processInstanceId); // GET /history/activity-instance?processInstanceId={processInstanceId}
getProcessInstanceParent (String processInstanceId);
getProcessInstanceChildren(String processInstanceId);
getProcessTaskDetail(String processInstanceId, String taskId);
getProcessTaskType(String processInstanceId, String taskid); // query if this task type is "Call Activity" for drilling down
- The following diagram depicts the API interaction.

Component Build Dependencies
- The following diagram depicts the component build dependencies.
- Components can be built in parallel with a certain level of sequential integration.
- Each estimate includes code development and unit testing plus some component integration and integration testing (since there are API dependencies).

Estimates (T-Shirt Size: M)
- Note:
  - It is a rough estimate - to be refined
  - Estimate includes development and unit testing time
  - As mentioned in the above section, the components can be built in parallel.
  - 5 hours / day

Priority	Component	Development Estimate	Comments
1	REST APIs for providing data to UIs	40 hours / Java developer	REST facade for collecting and consolidating various Camunda process instance and history data to simplify GUI data collection interaction. collecting SO Request DB data
2	SO Service List widget	60 hours / UI developer	populate search filter criteria Displays a service list panel control action buttons
3	Process/Service Instance Rendering and Detail Panel	60 hours / UI developer	Renders process/service instance workflow graphically provides the process/ service instance details and selected task details Provides drill-down and drill-up capabilities
4	Statistic Dashboard (stretch goal)	40 hours / UI developer	Displays statistic for the selected service scope
5	~~Custom History Producer (stretch goal)~~	~~40 hours / Java developer~~	Not need for Casablanca Produce history with additional data population This is an option, and is used only if the Camunda history data is not sufficient.

Browser not supported