CPS Instrumentation Overview

Spring AOP (Aspect-Oriented Programming

• Intercept method calls within a specific package

• Measures execution time and log relevant details

• Pure Java

• Dependency

  • aspectjrt

  • spring-boot-starter-aop

• Annotation and set up

  • @Aspect
    @Component
    @Slf4j
    public class CpsLoggingAspectService {

    • sets up spring AOP

  • private static final String CPS_PACKAGE_NAME = "org.onap.cps";
    private static final String ALL_CPS_METHODS = "execution(* " + CPS_PACKAGE_NAME + "..*(..)))";
    private static final String METHOD_RETURNING_SENSITIVE_DATA = "AuthPassword";
    private static final String SENSITIVE_DATA_MASK = "***********";
    @Around(ALL_CPS_METHODS)
    @SneakyThrows
    public Object interceptMethodCall(final ProceedingJoinPoint proceedingJoinPoint) {...}

    • line 5 expression tells spring to intercept all cps methods in the given package

      • uses stopwatch to record start and end time of method execution

    • the method above also states to mask all logging data from methods that contains ‘AuthPassword’ (line 3) from the method signature name

      • this method can be modified to tell what to do before and after a method runs

  • in logs, sample result:

    • Measure Execution Time of CPS Service

• all public methods in the packages configured in set up

  • no annotation needed

1. Default logging level of “logging.level.org.onap.cps” is set to “INFO”.

   

   to change this, set line 194 (logging.level.org.onap.cps to DEBUG)

2. Changing logging level dynamically

   1. see Change logging level

• can control easily which packages and/or methods to run by changing the constants

• no repetitive code for debugging all method performances to see when methods are executed, their parameters, return values etc.

• “@Around"/@Before/@After annotation takes other expressions to specify to Spring which methods you want to monitor , can be based on their name, parameters, or return type…examples:

  • @Around("execution(* org.thirdparty.library.*.*(..))") - only intercepts methods of the given library

  • @Around("execution(*org.onap.cps.*.*(org.onap.TrustLevel))") - only intercepts methods with TrustLevel as first parameter

MicroMeter

• allows to collect metrics with use of annotation

• integrates with Spring boot Actuator

• TimedAspect (micrometer class) uses AOP

• Annotation and set up

  • MicroMeterConfig.java

  • wherever annotation “@Timed” is used, timing metrics is automatically applied to that method (org/onap/cps/config/MicroMeterConfig.java)

    • our methods are annotated in the following way:

      •     @Timed(value = "cps.rest.admin.controller.schemaset.create",
        description = "Time taken to create schemaset from controller")

        • specified given “value” will be the name of the metric

        • description is the details of the given metric

• all methods with @Timed annotation

• other annotations that can be added

  • @Counted

  • @Gauge

• collected metrics are accessible via endpoint of actuator/prometheus

  • see application.yml - management.endpoint.web.exposure.include: info,health,loggers,prometheus

  • http://localhost:8080/actuator/prometheus

• Grafana - use prometheus readings to visualise

  • see docker-compose

• what can we get?

  • metrics: counters, gauges, timers

    • right now we only have timer but counters and gauges can be added to the configuration

  • HTTP request metrics response times, request counts etc

  • JVM metrics (memory usage,garbage collection, thread count, class loading stats)

  • Datasource metrics (connection pool stats)

  • System metrics (disk space, CPU usage etc)

• sample actuator/prometheus output

  • 

• sample /actuator/metrics/cps.module.service.schemaset.create output

  • add metric to actuator endpoint exposure -management.endpoint.web.exposure.include: info,health,loggers,prometheus,metric

  • 

OpenTelemetry

end-to-end tracing (tracing requests across services)

OpenTelemetry - open-source set of APIs, libraries, agents, and instrumentation to monitor application. Allows to collect metrics, logs, and traces

Jaeger - open-source distributed tracing system used with OpenTelemetry to visualise traces and understand flow of requests

** we use OpenTelemetry to send trace data to Jaeger and view it in Jaeger UI

• Annotation and set up

• OpenTelemetryConfig.java

  •     @Value("${spring.application.name:cps-application}")
        private String serviceId;
    
        @Value("${cps.tracing.exporter.endpoint:http://onap-otel-collector:4317}")
        private String tracingExporterEndpointUrl;
    
        @Value("${cps.tracing.sampler.jaeger_remote.endpoint:http://onap-otel-collector:14250}")
        private String jaegerRemoteSamplerUrl;
    
        @Value("${cps.tracing.excluded-observation-names:tasks.scheduled.execution}")
        private String excludedObservationNamesAsCsv;

    • serviceId : states the application to be monitored

    • tracingExporterEndpointUrl : (OpenTelemetry collector endpoint) this is where traced data by openTelemetry will be exported

    • jaegerRemoteSamplerUrl : (remote sampler endpoint) this controls the sampling trace behaviour

  •  @Bean
        @ConditionalOnExpression(
            "${cps.tracing.enabled} && 'grpc'.equals('${cps.tracing.exporter.protocol}')")
        public OtlpGrpcSpanExporter createOtlpExporterGrpc() {
            return OtlpGrpcSpanExporter.builder().setEndpoint(tracingExporterEndpointUrl).build();
        }
    
        @Bean
        @ConditionalOnExpression(
            "${cps.tracing.enabled} && 'http'.equals('${cps.tracing.exporter.protocol}')")
        public OtlpHttpSpanExporter createOtlpExporterHttp() {
            return OtlpHttpSpanExporter.builder().setEndpoint(tracingExporterEndpointUrl).build();
        }
    

    • **OTLP is the protocol openTelemetry uses to send traces and metrics. In this case we have two ways HTTP and grpc. NOTE: we are by default using ‘grpc’

  •     @Bean
        @ConditionalOnProperty("cps.tracing.enabled")
        public JaegerRemoteSampler createJaegerRemoteSampler() {
            return JaegerRemoteSampler.builder()
                    .setEndpoint(jaegerRemoteSamplerUrl)
          .setPollingInterval(Duration.ofSeconds(JAEGER_REMOTE_SAMPLER_POLLING_INTERVAL_IN_SECONDS))
                    .setInitialSampler(Sampler.alwaysOff())
                    .setServiceName(serviceId)
                    .build();
        }

    • creates the JaegerSampler bean specifying

      • sampling happens every 30 seconds

        • int JAEGER_REMOTE_SAMPLER_POLLING_INTERVAL_IN_SECONDS = 30;

  •     @Bean
        @ConditionalOnProperty("cps.tracing.enabled")
        public ObservationRegistryCustomizer<ObservationRegistry> skipActuatorEndpointsFromObservation() {
            final PathMatcher pathMatcher = new AntPathMatcher("/");
            return registry ->
                    registry.observationConfig().observationPredicate(observationPredicate(pathMatcher));
        }
    
        private ObservationPredicate observationPredicate(final PathMatcher pathMatcher) {
            return (observationName, context) -> {
                if (context instanceof ServerRequestObservationContext observationContext) {
                    return !pathMatcher.match("/actuator/**", observationContext.getCarrier().getRequestURI());
                } else {
                    return !excludedObservationNames.contains(observationName);
                }
            };
        }

    • the ObservationRegistry specifies how traces and metrics are handled in the application

      • in our code above, it is specified that any endpoints with “/actuator” is excluded from traces

• service interactions

• disabled by default

• run CPS via docker-compose file with ‘tracing’ profile

  • set ‘ONAP_TRACING_ENABLED’ to true for cps-and-ncmp service

    • 

• access tracing through accessing http://localhost:16686 in browser

  • see docker-compose notes

** Trace shows the journey of the request flow. It is made up of spans which shows each operations

** Each span has metadata: id ,start and end time, operation name (i.e. HTTP GET), status code, response time etc..

What gets traced?

• HTTP Requests (Inbound)

  • when requests comes in , trace is generated. The span for this could have http method, url path, status code, time taken to process the request etc…

• Internal Service Calls (Outbound)

  • when cps makes an outbound http call to another service , trace is created. span can include url of the called service,http method, status code, duration of the call

• Database Queries

• Method Calls within Your Application (Service Layer)

• Asynchronous Operations

  • @Async annotated operations

• External System Calls

  • Kafka

• Error Handling (Exceptions)

• can compare in UI two traces for analysis

  • 

• can use to identify where failures occur in the service chain

  • example:

    • after registering cmhandles for dm1, i tried to get all cmhandles for that dmi and got 200 and the correct results

    • OpenTelemetry though shows where an exception occured in the background