/
Policy Framework: Key metrics to monitor

Policy Framework: Key metrics to monitor

Owned by Rashmi Pujar
Last updated: Apr 04, 2022
5 min read

Context

Collecting application metrics is the first step towards gaining insights into Policy Fwk services and infrastructure from point of view of Availability, Performance, Reliability and Scalability.

The goal of monitoring is to achieve the below operational needs:

  • Monitoring via dashboards: Provide a visual aid to display health and key metrics for use by OPS.

  • Alerting: Something is broken, and the issue must be addressed immediately OR, something might break soon, and proactive measures are taken to avoid such a situation.

  • Conducting retrospective analysis: Rich information that is readily available to better troubleshoot issues.

  • Analyzing trends: How fast is it the usage growing? How is the incoming traffic like? Helps assess needs for scaling to meet forecasted demands.

Policy Framework Key Metrics

The principles outlined in the Four Golden Signals developed by Google Site Reliability Engineers has been adopted to define the key metrics for Policy Fwk components: API, PAP, Policy-Distribution, Policy-DB, PDPs (APEX, Drools, XACML).

  • Request Rate - Number of requests, per second as served by Policy services i.e. by API, PAP. Number of requests/events, per second as processed by the PDPs

  • Errors - Number of those requests/events processed that are failing

  • Latency/Duration (expressed as time interval) -  Amount of time those requests take, and for PDPs relevant metrics denoting the event processing times

  • Saturation - Measures the degree of fullness or % utilization of a service emphasizing the resources that are most constrained: CPU, Memory, I/O, custom metrics by domain.

System Metrics that apply to all Policy components

These metrics are available and exposed via a Prometheus endpoint since Istanbul release. 

Note: Standard metrics are already exposed for Policy DB (MariaDB) via common charts.

Metric

Prometheus Query

Metric

Prometheus Query

Memory usage

rate(jvm_memory_bytes_used[30s])*100

CPU Usage

rate(process_cpu_seconds_total[30s])*100

JVM threads

jvm_threads_current
jvm_threads_daemon

Process uptime

process_start_time_seconds

Garbage Collectors

GCs per second: rate(jvm_gc_collection_seconds_sum[1m])

Avg GC time: rate(jvm_gc_collection_seconds_sum[1m]) / rate(jvm_gc_collection_seconds_count[1m])

Note: SSL certificate expiry is a key metric to alert on, however this can be dealt with outside the scope of Policy Fwk.

Key metrics for Policy API

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of policy-api service

Yes

Yes

Exposed by policy-api healthcheck and policy-pap consolidated healthcheck.

Latency



Yes

Yes

To be implemented for all CRUD endpoints exposed by policy-api.

Sample s3p numbers for policy-api stress tests.

Successful API request counter

Yes

Yes

Prometheus query for Number of successful API calls per minute

Failed API request counter

Yes

Yes

Prometheus query for Number of API calls with non 20* family of status codes per minute

Key metrics for Policy PAP

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of the policy-pap service

Yes

Yes

policy-pap healthcheck API

Successful API request counter

Yes

Yes

To be implemented for all the endpoints exposed by policy-pap.

Sample s3p numbers for policy-pap stress tests. 

Failed API request counter

Yes

Yes

To be implemented for all the endpoints exposed by policy-pap.

Number of API calls with non 200 family of status codes per minute

Latency

Yes

Yes

To be implemented for all the endpoints exposed by policy-pap.

Policy deployment statistics

policyDeployFailureCount
policyDeploySuccessCount
totalPolicyDeployCount

Yes

Yes

Sample:

GET /policy/pap/v1/statistics
{ "code": 200, "policyDeployFailureCount": 0, "policyDeploySuccessCount": 0, "policyDownloadFailureCount": 0, "policyDownloadSuccessCount": 0, "totalPdpCount": 0, "totalPdpGroupCount": 0, "totalPolicyDeployCount": 0, "totalPolicyDownloadCount": 0 }



Key metrics for Policy Distribution

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of the policy-distribution service

Yes

Yes

Exposed by policy-distribution healthcheck and consolidated policy-pap healthcheck

Successful API request counter

Yes

Yes

To be implemented for all the endpoints exposed by policy-distribution.

Sample s3p numbers for policy-distribution stress tests. 

Failed API request counter

Yes

Yes

To be implemented for all the endpoints exposed by policy-distribution.

Number of API calls with non 200 family of status codes per minute

Latency

Yes

Yes

To be implemented for all the endpoints exposed by policy-distribution.

Policy distribution statistics

distributions
distribution_complete_ok
distribution_complete_fail
downloads
downloads_ok
downloads_error

Yes

Yes

policy-distribution Rest Endpoint Samples

Key metrics for Policy APEX PDP

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of policy-apex-pdp

Yes

Yes

Exposed by policy-apex-pdp healthcheck and policy-pap consolidated healthcheck.

TOSCA Policy Deployment counter (per apex-pdp instance)

policyDeployCount
policyDeploySuccessCount
policyDeployFailCount

Yes

Yes

Exposed by policy-pap statistics

GET /policy/pap/v1/statistics/defaultGroup/apex
{ "defaultGroup": { "apex": [ { "pdpInstanceId": "devdev-policy-apex-pdp-0", "timeStamp": "2021-09-07T20:10:52.242Z", "pdpGroupName": "defaultGroup", "pdpSubGroupName": "apex", "policyDeployCount": 2, "policyDeploySuccessCount": 2, "policyDeployFailCount": 0, "policyExecutedCount": 0, "policyExecutedSuccessCount": 0, "policyExecutedFailCount": 0, "engineStats": [ { "engineId": "NSOApexEngine-0:0.0.1", "engineWorkerState": "READY", "engineTimeStamp": 1630550345549, "eventCount": 0, "lastExecutionTime": 0, "averageExecutionTime": 0, "upTime": 0, "lastEnterTime": 0, "lastStart": 1630550345549 }, ...... ] } ] } }








TOSCA Policy Execution counter (per apex-pdp instance)

# of policies executed
# of policies executed with success status
# of policies executed with a failure status

Yes

Yes

Engine stats (by engineID per apex-pdp instance)

eventCount: number of APEX events processed
engineWorkerState: possible values defined in AxEngineState
averageExecutionTime: average time taken to process an APEX policy
lastExecutionTime: time taken to process the last APEX policy
lastStart: time at which the policy engine was last started, uptime is derived from this metric

Yes

Yes

Latency

Yes

Yes

Time taken for processing an incoming APEX event 

*Note: the stats currently displays execution time for processing APEX policy, and is a measure of system saturation and is sufficient

Kafka consumer lag

No

No

Can be implemented outside of the Policy FWK.

Monitor kafka consumer lag increase for kafka/dmaap-message-router topics related to apex-pdp

Key metrics for Policy Drools PDP

*Note: Drools PDP counters are exposed on a per controlloop implementation basis.

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of policy-drools-pdp

Yes

No

Exposed by policy-drools-pdp healthcheck and policy-pap consolidated healthcheck.

Telemetry feature-lifecycle status API
http://localhost:9696/policy/pdp/engine> get /policy/pdp/engine/lifecycle/state HTTP/1.1 200 OK Content-Length: 8 Content-Type: application/json Date: Thu, 11 Nov 2021 16:36:13 GMT Server: Jetty(9.4.33.v20201020) "ACTIVE"



Policy Deployment counter (per drools-pdp instance)

policyDeployCount
policyDeploySuccessCount
policyDeployFailCount

Yes

No

Sample:

GET /policy/pap/v1/statistics/defaultGroup/drools



Policy Execution counter (per drools-pdp instance)

policyExecutedCount
policyExecutedSuccessCount
policyExecutedFailCount

Yes

No

Latency

No

No

Time taken for an incoming event to be processed by drools controller.

Count of Drools facts

No

No

An ever increasing number of drools facts can lead to an Out of memory.

Kafka consumer lag

No

No

Can be implemented external to the policy FWK

Monitor kafka consumer lag increase for kafka/dmaap-message-router topics related to drools

Key metrics for Policy XACML PDP

TODO: The statistics exposed can be more granular

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Metric

Metric available?

Exposed via Prometheus endpoint?

Comment

Availability of policy-xacml-pdp

Yes

No

Exposed by policy-pap consolidated healthcheck. Additionally, also exposed by the XACML healthcheck API

GET /policy/pdpx/v1/healthcheck



Policy Deployment counter

totalPoliciesCount
totalPolicyTypesCount

Yes

No

XACML PDP statistics API

GET /policy/pdpx/v1/statistics



Policy execution error counter

totalErrorCount

Yes

No

Policy execution success counter by type

permitDecisionsCount
denyDecisionsCount
indeterminantDecisionsCount
notApplicableDecisionsCount

Yes

No

Latency

No

No

Time taken for an incoming event to be processed via the XACML policies.

Kafka consumer lag

No

No

Can be implemented external to the policy FWK

Monitor kafka consumer lag increase for kafka/dmaap-message-router topics related to XACML