/
Test external Kafka access in London

Test external Kafka access in London

Owned by Andreas Geißler
Last updated: Apr 13, 2023
3 min read

Prerequisites:

  • In London release the Kafka ports are exposed through ingress.

  • Istio and Istio-Ingress is used

  • In the istio-ingress configuration the required ports (9010, 9000,9001, 9002) need to be exposed → see ONAP on ServiceMesh setup guide

  • Helm settings are configured to enable the Ingress exposure of Kafka Interfaces by:

    • global values (global.ingress.enable_all)

      global: ingress: enabled: true # enable all component's Ingress interfaces enable_all: true



    • or local setting in onap-strimzi (ingress.enabled)

      ingress: enabled: true service: - baseaddr: "kafka-bootstrap-api" name: "onap-strimzi-kafka-external-bootstrap" port: 9094 exposedPort: 9010 exposedProtocol: TLS



After the deployment the TCP interfaces are exposed through ingress and can be accessed via the following URLs and ports:

kafka-bootstrap-api.simpledemo.onap.org:9010 kafka-api.simpledemo.onap.org:9000 kafka-api.simpledemo.onap.org:9001 kafka-api.simpledemo.onap.org:9002



Test preparation

Add Kafka User for external Access

  • Login to the cluster control node

  • Create kafka-user.yaml file

tls-user.yaml
apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaUser metadata: labels: argocd.argoproj.io/instance: external-strimzi-kafka-user strimzi.io/cluster: onap-strimzi name: external-strimzi-kafka-user namespace: onap spec: authentication: type: scram-sha-512 authorization: acls: - resource: type: topic name: unauthenticated.VES_PERF3GPP_OUTPUT patternType: literal operation: Write host: "*" - resource: type: topic name: unauthenticated.VES_PERF3GPP_OUTPUT patternType: literal operation: Describe host: "*" - resource: type: topic name: unauthenticated.VES_NOTIFICATION_OUTPUT patternType: literal operation: Write host: "*" - resource: type: topic name: unauthenticated.VES_NOTIFICATION_OUTPUT patternType: literal operation: Describe host: "*" - resource: type: topic name: unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT patternType: literal operation: Write host: "*" - resource: type: topic name: unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT patternType: literal operation: Describe host: "*" - resource: type: topic name: unauthenticated.VES_MEASUREMENT_OUTPUT patternType: literal operation: Write host: "*" - resource: type: topic name: unauthenticated.VES_MEASUREMENT_OUTPUT patternType: literal operation: Describe host: "*" type: simple

  • Apply kafka-user.yaml

Create user
kubectl apply -f kafka-user.yaml

  • List kafka users

Check/List new user
root@control01-daily-master-sm:/# kubectl -n onap get kafkauser NAME CLUSTER AUTHENTICATION AUTHORIZATION READY aai-modelloader-ku onap-strimzi scram-sha-512 simple True cds-blueprints-processor-ku onap-strimzi scram-sha-512 simple True cds-sdc-listener-ku onap-strimzi scram-sha-512 simple True cps-core-ku onap-strimzi scram-sha-512 simple True cps-temporal-ku onap-strimzi scram-sha-512 simple True dcae-hv-ves-collector-ku onap-strimzi scram-sha-512 simple True dcae-ves-openapi-manager-ku onap-strimzi scram-sha-512 simple True external-strimzi-kafka-user onap-strimzi scram-sha-512 simple True multicloud-k8s-ku onap-strimzi scram-sha-512 simple True onap-cps-kafka-user onap-strimzi scram-sha-512 simple True onap-policy-kafka-user onap-strimzi scram-sha-512 simple True onap-so-sdc-list-user onap-strimzi scram-sha-512 simple True policy-clamp-ac-a1pms-ppnt-ku onap-strimzi scram-sha-512 simple True policy-clamp-ac-http-ppnt-ku onap-strimzi scram-sha-512 simple True policy-clamp-ac-k8s-ppnt-ku onap-strimzi scram-sha-512 simple True policy-clamp-ac-kserve-ppnt-ku onap-strimzi scram-sha-512 simple True policy-clamp-ac-pf-ppnt-ku onap-strimzi scram-sha-512 simple True policy-clamp-runtime-acm-ku onap-strimzi scram-sha-512 simple True policy-distribution-ku onap-strimzi scram-sha-512 simple True sdc-be-ku onap-strimzi scram-sha-512 simple True strimzi-kafka-admin onap-strimzi scram-sha-512 True

  • List strimzi secrets

List user secrets
root@control01-daily-master-sm:/# kubectl -n onap get secret|grep strimzi external-strimzi-kafka-user Opaque 2 2m7s ...

  • Get the user password

For each KafkaUser resource with scram-sha-512 auth, there will be a corresponding secret:

Get the user secret
kubectl get secret external-strimzi-kafka-user -o jsonpath='{.data.password}' -n onap | base64 --decode Ujl...lSD

Test the external client access to Kafka



  • Add hostnames to DNS (or /etc/hosts) by using the IP Address of the istio-ingressgateway LB 

sudo vi /etc/hosts ---- 10.32.242.56 kafka-bootstrap-api.simpledemo.onap.org 10.32.242.56 kafka-api.simpledemo.onap.org



  • Install KafkaCat

sudo apt install kafkacat

  • Get the Metadata (use an existing Kafka User, here "external-strimzi-kafka-user") using the sasl.password exported above:

kafkacat -L -b kafka-bootstrap-api.simpledemo.onap.org:9010 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=hCv4IZ3Q6XLR -v Metadata for all topics (from broker -1: sasl_ssl://kafka-bootstrap-api.simpledemo.onap.org:9003/bootstrap): 3 brokers: broker 0 at kafka-api.simpledemo.onap.org:9000 (controller) broker 2 at kafka-api.simpledemo.onap.org:9002 broker 1 at kafka-api.simpledemo.onap.org:9001 33 topics: topic "org.onap.dmaap.mr.PNF_REGISTRATION" with 2 partitions: partition 0, leader 2, replicas: 2, isrs: 2 partition 1, leader 1, replicas: 1, isrs: 1 ...

  • Get Metadata (use an existing Kafka User, here "external-strimzi-kafka-user"):

kafkacat -L -b kafka-bootstrap-api.simpledemo.onap.org:9010 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=e8ILJRai43LT -v Metadata for all topics (from broker -1: sasl_ssl://kafka-bootstrap-api.simpledemo.onap.org:9010/bootstrap): 3 brokers: broker 0 at kafka-api.simpledemo.onap.org:9000 broker 2 at kafka-api.simpledemo.onap.org:9002 broker 1 at kafka-api.simpledemo.onap.org:9001 (controller) 2 topics: topic "unauthenticated.VES_NOTIFICATION_OUTPUT" with 6 partitions: partition 0, leader 1, replicas: 1,0,2, isrs: 0,1,2 partition 1, leader 0, replicas: 0,2,1, isrs: 0,1,2 partition 2, leader 2, replicas: 2,1,0, isrs: 0,1,2 partition 3, leader 1, replicas: 1,2,0, isrs: 0,1,2 partition 4, leader 0, replicas: 0,1,2, isrs: 0,1,2 partition 5, leader 2, replicas: 2,0,1, isrs: 0,1,2 topic "unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT" with 6 partitions: partition 0, leader 2, replicas: 2,1,0, isrs: 0,1,2 partition 1, leader 1, replicas: 1,0,2, isrs: 0,1,2 partition 2, leader 0, replicas: 0,2,1, isrs: 0,1,2 partition 3, leader 2, replicas: 2,0,1, isrs: 0,1,2 partition 4, leader 1, replicas: 1,2,0, isrs: 0,1,2 partition 5, leader 0, replicas: 0,1,2, isrs: 0,1,2

  • Get Topic Data (use an existing Kafka User, here "external-strimzi-kafka-user"):

kafkacat -b kafka-api.simpledemo.onap.org:9000 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=e8ILJRai43LT -C -t unauthenticated.VES_NOTIFICATION_OUTPUT -v {"event":{"commonEventHeader":{"startEpochMicrosec":8745745764578,"eventId":"FileReady_1797490e-10ae-4d48-9ea7-3d7d790b25e1","timeZoneOffset":"UTC+05.30","internalHeaderFields":{"collectorTimeStamp":"Tue, 12 06 2022 01:35:59 GMT"},"priority":"Normal","version":"4.0.1","reportingEntityName":"otenb5309","sequence":0,"domain":"notification","lastEpochMicrosec":8745745764578,"eventName":"Noti_RnNode-Ericsson_FileReady","vesEventListenerVersion":"7.0.1","sourceName":"oteNB5309"},"notificationFields":{"notificationFieldsVersion":"2.0","changeType":"FileReady","changeIdentifier":"PM_MEAS_FILES","arrayOfNamedHashMap":[{"name":"test.xml.gz","hashMap":{"location":"sftp://sftp:22/test.xml.gz","fileFormatType":"org.3GPP.32.435#measCollec","fileFormatVersion":"V10","compression":"gzip"}}]}}} ...







Related content