Page Comparison

...

Note: Standard metrics are already exposed for Policy DB (MariaDB) via common charts.

Metric	Prometheus Query
Memory usage	rate(jvm_memory_bytes_used[30s])*100
CPU Usage	rate(process_cpu_seconds_total[30s])*100
JVM threads	jvm_threads_current jvm_threads_daemon
Process uptime	process_start_time_seconds
Garbage Collectors	GCs per second: rate(jvm_gc_collection_seconds_sum[1m]) Avg GC time: rate(jvm_gc_collection_seconds_sum[1m]) / rate(jvm_gc_collection_seconds_count[1m])

Key metrics for Policy API

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of policy-api service

Yes

No

Exposed by policy-api healthcheck and policy-pap consolidated healthcheck.

Latency

No

To be implemented for all CRUD endpoints exposed by policy-api.

Sample s3p numbers for policy-api stress tests.

Request rate (API requests per minute)

Successful API request counter

No

Prometheus query for Number of successful API calls per minute

Failure rate (API errors per minute)

Failed API request counter	No	No	Prometheus query for Number of API calls with non 20* family of status codes per minute
SSL certificate expiry time	No	No	Can be done outside the scope of Policy Fwk

Key metrics for Policy PAP

Metric	Metric available?	Exposed via Prometheus endpoint?	Comment
*Availability of the policy-pap* service**	Yes	No	policy-pap healthcheck API
Status of PDPs as registered with policy-pap	Yes	No	policy-pap consolidated healthcheck API

Request rate (API requests per minute)

Successful API request counter

No

To be implemented for all the endpoints exposed by policy-pap.

Sample s3p numbers for policy-pap stress tests.

Failure rate (API errors per minute)

Failed API request counter

No

To be implemented for all the endpoints exposed by policy-pap.

Number of API calls with non 200 family of status codes per minute

Latency

No

To be implemented for all the endpoints exposed by policy-pap.

Policy deployment statistics

policyDeployFailureCount
policyDeploySuccessCount
totalPolicyDeployCount

Yes

No

Sample:

Code Block

language	bash
title	GET /policy/pap/v1/statistics
collapse	true

{
    "code": 200,
    "policyDeployFailureCount": 0,
    "policyDeploySuccessCount": 0,
    "policyDownloadFailureCount": 0,
    "policyDownloadSuccessCount": 0,
    "totalPdpCount": 0,
    "totalPdpGroupCount": 0,
    "totalPolicyDeployCount": 0,
    "totalPolicyDownloadCount": 0
}

SSL certificate expiry time

No

Can be done outside the scope of Policy Fwk

Key metrics for Policy Distribution

Metric	Metric available?	Exposed via Prometheus endpoint?	Comment
*Availability of the policy-distribution* service**	Yes	No	Exposed by policy-distribution healthcheck and consolidated policy-pap healthcheck

Request rate (API requests per minute)

Successful API request counter

No

To be implemented for all the endpoints exposed by policy-distribution.

Sample s3p numbers for policy-distribution stress tests.

Failure rate (API errors per minute)

Failed API request counter	No	No	To be implemented for all the endpoints exposed by policy-distribution. Number of API calls with non 200 family of status codes per minute
Latency	No	No	To be implemented for all the endpoints exposed by policy-distribution.
Policy distribution statistics distributions distribution_complete_ok distribution_complete_fail downloads downloads_ok downloads_error	Yes	No	policy-distribution Rest Endpoint Samples
SSL certificate expiry time	No	No	Can be done outside the scope of Policy Fwk

Key metrics for Policy APEX PDP

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of policy-apex-pdp

Yes

No

Exposed by policy-apex-pdp healthcheck and policy-pap consolidated healthcheck.

Policy Deployment counter (per apex-pdp instance)

policyDeployCount
policyDeploySuccessCount
policyDeployFailCount

Yes

No

Exposed by policy-pap statistics

Code Block

title	GET /policy/pap/v1/statistics/defaultGroup/apex
collapse	true

{
  "defaultGroup": {
    "apex": [
      {
        "pdpInstanceId": "devdev-policy-apex-pdp-0",
        "timeStamp": "2021-09-07T20:10:52.242Z",
        "pdpGroupName": "defaultGroup",
        "pdpSubGroupName": "apex",
        "policyDeployCount": 2,
        "policyDeploySuccessCount": 2,
        "policyDeployFailCount": 0,
        "policyExecutedCount": 0,
        "policyExecutedSuccessCount": 0,
        "policyExecutedFailCount": 0,
        "engineStats": [
          {
            "engineId": "NSOApexEngine-0:0.0.1",
            "engineWorkerState": "READY",
            "engineTimeStamp": 1630550345549,
            "eventCount": 0,
            "lastExecutionTime": 0,
            "averageExecutionTime": 0,
            "upTime": 0,
            "lastEnterTime": 0,
            "lastStart": 1630550345549
          },
          ......
        ]
      }
    ]
  }
}

Policy Execution counter (per apex-pdp instance)

# of policies executed
# of policies executed with success status
# of policies executed with a failure status

*Note: the stats currently displays APEX policy counters

No

Engine count

Can be inferred from the size of the json array object "engineStats"

Yes

No

Engine availability details (by engineID per apex-pdp instance)

engineTimestamp: timestamp at which the statistics were recorded
engineWorkerState: possible values defined in AxEngineState
upTime: time that has elapsed since the policy engine was started
lastStart: time at which the policy engine was last started

Yes

No

Count of events processed (per engine thread, per apex-pdp instance)

# of incoming trigger events processed by policy-apex-pdp
# of incoming trigger events processed successfully by policy-apex-pdp
# of incoming trigger events processed by policy-apex-pdp that resulted in a failure

*Note: the stats currently displays APEX event counters processed by the engine

No

Latency

No

Time taken for processing an incoming network trigger event by the policy

*Note: the stats currently displays execution time for processing APEX policy.

Kafka consumer lag

No

Can be implemented outside of the Policy FWK.

Monitor kafka consumer lag increase for kafka/dmaap-message-router topics related to apex-pdp

SSL certificate expiry time

No

Can be done outside the scope of Policy Fwk

Key metrics for Policy Drools PDP

*Note: The Drools PDP counters are exposed per controlloop implementation.

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of policy-drools-pdp

Yes

No

Exposed by policy-drools-pdp healthcheck and policy-pap consolidated healthcheck.

Code Block

language	bash
title	Telemetry feature-lifecycle status API
collapse	true

http://localhost:9696/policy/pdp/engine> get /policy/pdp/engine/lifecycle/state
HTTP/1.1 200 OK
Content-Length: 8
Content-Type: application/json
Date: Thu, 11 Nov 2021 16:36:13 GMT
Server: Jetty(9.4.33.v20201020)

"ACTIVE"

Policy Deployment counter (per drools-pdp instance)

policyDeployCount
policyDeploySuccessCount
policyDeployFailCount

Yes

No

Sample:

Code Block

title	GET /policy/pap/v1/statistics/defaultGroup/drools
collapse	true

{
   "defaultGroup":{
      "drools":[
         {
            "pdpInstanceId":"dev-policy-drools-pdp-0",
            "timeStamp":"2021-09-07T20:09:34.160Z",
            "pdpGroupName":"defaultGroup",
            "pdpSubGroupName":"drools",
            "policyDeployCount":54,
            "policyDeploySuccessCount":54,
            "policyDeployFailCount":0,
            "policyExecutedCount":1,
            "policyExecutedSuccessCount":1,
            "policyExecutedFailCount":0,
            "engineStats":[

            ]
         }
      ]
   }
}

Policy Execution counter (per drools-pdp instance)

policyExecutedCount
policyExecutedSuccessCount
policyExecutedFailCount

Yes

No

Latency

No

Time taken for an incoming event to be processed by drools controller.

Count of Drools facts

No

An ever increasing number of drools facts can lead to an Out of memory.

Kafka consumer lag

No

Can be implemented external to the policy FWK

Monitor kafka consumer lag increase for kafka/dmaap-message-router topics related to drools

SSL certificate expiry time

No

Can be done outside the scope of Policy Fwk

Key metrics for Policy XACML PDP

TODO: The statistics exposed can be more granular

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of policy-xacml-pdp

Yes

No

Exposed by policy-pap consolidated healthcheck. Additionally, also exposed by the XACML healthcheck API

Code Block

title	GET /policy/pdpx/v1/healthcheck
collapse	true

~ $ curl --location --request GET 'http://policy-xacml-pdp.fra-fireants-dev.svc.cluster.local:6969/policy/pdpx/v1/healthcheck' --header 'Authorization: Basic ****'
{
  "name": "Policy Xacml PDP",
  "url": "self",
  "healthy": true,
  "code": 200,
  "message": "alive"
}

Policy Deployment counter

totalPoliciesCount
totalPolicyTypesCount

Yes

No

XACML PDP statistics API

Code Block

title	GET /policy/pdpx/v1/statistics
collapse	true

~ $ curl --location --request GET 'http://policy-xacml-pdp.fra-fireants-dev.svc.cluster.local:6969/policy/pdpx/v1/statistics' --header 'Authorization: Basic ****'
{
  "code": 200,
  "totalPolicyTypesCount": 18,
  "totalPoliciesCount": 1,
  "totalErrorCount": 0,
  "permitDecisionsCount": 0,
  "denyDecisionsCount": 0,
  "indeterminantDecisionsCount": 0,
  "notApplicableDecisionsCount": 1
}

Policy execution error counter

totalErrorCount

Yes

No

Policy execution success counter by type

permitDecisionsCount
denyDecisionsCount
indeterminantDecisionsCount
notApplicableDecisionsCount

Yes

No

Latency

No

Time taken for an incoming event to be processed via the XACML policies.

Kafka consumer lag

No

Can be implemented external to the policy FWK

Monitor kafka consumer lag increase for kafka/dmaap-message-router topics related to XACML

SSL certificate expiry time

No

Can be done outside the scope of Policy Fwk

Version	Old Version 6	New Version 7
Changes made by	Rashmi Pujar	Rashmi Pujar
Saved on	Nov 11, 2021	Nov 17, 2021

Page Comparison

Versions Compared

Key

Key metrics for Policy API

Key metrics for Policy PAP

Key metrics for Policy Distribution

Key metrics for Policy APEX PDP

Key metrics for Policy Drools PDP

Key metrics for Policy XACML PDP