CPS Instrumentation Overview

Spring AOP (Aspect-Oriented Programming

• Intercept method calls within a specific package

• Measures execution time and log relevant details

• Pure Java

• Dependency

  • aspectjrt

  • spring-boot-starter-aop

• Annotation and set up

  • @Aspect
    @Component
    @Slf4j
    public class CpsLoggingAspectService {

    • sets up spring AOP

  • private static final String CPS_PACKAGE_NAME = "org.onap.cps";
    private static final String ALL_CPS_METHODS = "execution(* " + CPS_PACKAGE_NAME + "..*(..)))";
    private static final String METHOD_RETURNING_SENSITIVE_DATA = "AuthPassword";
    private static final String SENSITIVE_DATA_MASK = "***********";
    @Around(ALL_CPS_METHODS)
    @SneakyThrows
    public Object interceptMethodCall(final ProceedingJoinPoint proceedingJoinPoint) {...}

    • line 5 expression tells spring to intercept all cps methods in the given package

      • uses stopwatch to record start and end time of method execution

    • the method above also states to mask all logging data from methods that contains ‘AuthPassword’ (line 3) from the method signature name

  • in logs, sample result:

    • Measure Execution Time of CPS Service

1. Default logging level of “logging.level.org.onap.cps” is set to “INFO”.

   

   to change this, set line 194 (logging.level.org.onap.cps to DEBUG)

2. Changing logging level dynamically

   1. see Change logging level

• can control easily which packages and/or methods to run by changing the constants

• no repetitive code for debugging all method performances to see when methods are executed, their parameters, return values etc.

• “@Around"/@Before/@After annotation takes other expressions to specify to Spring which methods you want to monitor , can be based on their name, parameters, or return type…examples:

  • @Around("execution(* org.thirdparty.library.*.*(..))") - only intercepts methods of the given library

  • @Around("execution(*org.onap.cps.*.*(org.onap.TrustLevel))") - only intercepts methods with TrustLevel as first parameter

MicroMeter

• allows to collect metrics with use of annotation

• integrates with Spring boot Actuator

• TimedAspect (micrometer class) uses AOP

• Annotation and set up

  • MicroMeterConfig.java

  • wherever annotation “@Timed” is used, timing metrics is automatically applied to that method (org/onap/cps/config/MicroMeterConfig.java)

    • our methods are annotated in the following way:

      •     @Timed(value = "cps.rest.admin.controller.schemaset.create",
        description = "Time taken to create schemaset from controller")

        • specified given “value” will be the name of the metric

        • description is the details of the given metric

• enabled by default

• collected metrics are accessible via

  • endpoint of actuator/prometheus

    • see application.yml - management.endpoint.web.exposure.include: info,health,loggers,prometheus

    • http://localhost:8080/actuator/prometheus

  • Grafana - use prometheus readings to visualise

    • see docker-compose

• all metrics are stored and can be collected (from MeterRegistry) and exported to monitoring systems like Prometheus

• sample actuator/prometheus output

  • 

• sample /actuator/metrics/cps.module.service.schemaset.create output

  • add metric to actuator endpoint exposure -management.endpoint.web.exposure.include: info,health,loggers,prometheus,metric

  • 

OpenTelemetry

• uses jaeger to sample console and log for metrics

  • allows to visualise traces

• Annotation and set up

• OpenTelemetryConfig.java

• sampling happens every 30 seconds

  • int JAEGER_REMOTE_SAMPLER_POLLING_INTERVAL_IN_SECONDS = 30;

• currently excludes all CPS scheduled tasks

• run CPS via docker-compose file with ‘tracing’ profile

• access tracing through accessing http://localhost:16686 in browser

  • see docker-compose notes

• Use Jaeger's UI to spot service anomalies, high latencies, or errors in real-time.

• identify where failures or performance issues occur in the service chain

• sample operations traced after registering cm handle

• 

• can compare in UI two traces for analysis

  • 

• Can use to identify where failures occur in the service chain

  • example:

    • after registering cmhandles for dm1, i tried to get all cmhandles for that dmi and got 200 and the correct results

    • OpenTelemetry though shows where an exception occured in the background