Add gauge metric to NCMP

Owned by Lee Anjella Macabuhay
Dec 19, 2024
2 min read

Related Jira: Add gauge metric to NCMP

Recap

ADVISED

state is advised ….

  • cm handle has been registered successfully and awaits module sync

  • the module sync retries to process the cm handle again from a ‘LOCKED' state

READY

state is ready when ….

  • CM-Handle is processed by the module sync successfully

LOCKED

state is ready when ….

  • CM-Handle is processed by the module sync and it fails

DELETING

state is ready when ….

  • when CM-Handle is currently being deleted

 

Out-of-scope related notes

DELETED

state is ready when ….

  • when CM-Handle has been deleted succesfully

 

Micrometer Gauge

[see https://docs.micrometer.io/micrometer/reference/concepts/gauges.html]

Registering a gauge metric for each state under the same Gauge name

public class CmHandleStateMetrics { private final AtomicInteger advisedCmHandlesCount = new AtomicInteger(0); private final AtomicInteger readyCmHandlesCount = new AtomicInteger(0); private final AtomicInteger lockedCmHandlesCount = new AtomicInteger(0); private final AtomicInteger deletingCmHandlesCount = new AtomicInteger(0);
Gauge.builder(CMHANDLE_STATE_GAUGE, cmHandleStateMetrics, lcmMetric -> cmHandleStateMetrics.getAdvisedCmHandlesCount().get()) .tag(TAG, "ADVISED") .description("Current number of cmhandles in advised state") .register(meterRegistry);

 

/actuator/metrics

image-20241218-132813.png

 

/actuator/prometheus

image-20241218-132929.png

 

  • Every query is directed to one of the two instances

    • query result after running k6 tests

      • expected: 20,000 cm handles in state deleting

      • reality: 10,000 cm handles in state deleting in every query (relates to two instances) and uneven cm handle count in other states

        • Query 1

           

           

          Query 2

           

           

    • Alternative: use of hazelcast

      •