Versions Compared

Old Version 1

changes.mady.by.user Andreas Geißler

Saved on

compared with

New Version Current

changes.mady.by.user Andreas Geißler

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • In London release the Kafka ports are exposed through ingress.

  • Istio and Istio-Ingress is used

  • In the istio-ingress configuration the required ports (9010, 9000,9001, 9002) need to be exposed → see ONAP on ServiceMesh setup guide
  • Helm settings are configured to enable the Ingress exposure of Kafka Interfaces by:

    • global values (global.ingress.enable_all)

      Code Block
      global:
  ingress:
    enabled: true
    # enable all component's Ingress interfaces
    enable_all: true


    • or local setting in onap-strimzi (ingress.enabled)

      Code Block
      ingress:
  enabled: true
  service:
  - baseaddr: "kafka-bootstrap-api"
    name: "onap-strimzi-kafka-external-bootstrap"
    port: 9094
    exposedPort: 9010
    exposedProtocol: TLS


After the deployment the TCP interfaces are exposed through ingress and can be accessed via the following URLs and ports:

Code Block
kafka-bootstrap-api.simpledemo.onap.org:9010
kafka-0-api.simpledemo.onap.org:9000
kafka-1-api.simpledemo.onap.org:9001
kafka-2-api.simpledemo.onap.org:9002


Test preparation

Add Kafka User for external Access

  • Login to the cluster control node
  • Create kafka-user.yaml file
Code Block
languagebash
themeMidnight
titletls-user.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  labels:
    argocd.argoproj.io/instance: external-strimzi-kafka-user
    strimzi.io/cluster: onap-strimzi
  name: external-strimzi-kafka-user
  namespace: onap
spec:
  authentication:
    type: scram-sha-512
  authorization:
    acls:
      - resource:
          type: topic
          name: unauthenticated.VES_PERF3GPP_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_PERF3GPP_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_NOTIFICATION_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_NOTIFICATION_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_MEASUREMENT_OUTPUT
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: topic
          name: unauthenticated.VES_MEASUREMENT_OUTPUT
          patternType: literal
        operation: Describe
        host: "*"
    type:  type: simple
  • Apply kafka-user.yaml

...

Code Block
languagebash
themeMidnight
titleCheck/List new user
root@control01-daily-master-sm:/# kubectl -n onap get kafkauser
NAME                              CLUSTER        AUTHENTICATION   AUTHORIZATION   READY
externalaai-strimzi-kafka-usermodelloader-ku               onap-strimzi   scram-sha-512    simple          True
onapcds-aaiblueprints-sdc-list-user     processor-ku       onap-strimzi   scram-sha-512    simple          True
onap-cds-sdc-listlistener-userku              onap-strimzi   scram-sha-512    simple          True
onap-cps-kafka-usercore-ku                      onap-strimzi   scram-sha-512    simple          True
onapcps-dcae-hv-ves-kafka-user       onap-strimzi   scram-sha-512temporal-ku    simple          True onap-mc-k8s-sdc-list-kafka-user   onap-strimzi   scram-sha-512    simple          True
onapdcae-hv-policyves-kafkacollector-userku            onap-strimzi   scram-sha-512    simple          True
onap-sdc-be-kafka-userdcae-ves-openapi-manager-ku      onap-strimzi   scram-sha-512    simple          True
external-strimzi-kafka-user      onap-strimzi   scram-sha-512    simple          True
strimzimulticloud-kafkak8s-adminku                onap-strimzi   scram-sha-512    simple          True
  • List strimzi secrets
Code Block
languagebash
themeMidnight
titleList user secrets
oot@control01-daily-master-sm:/# kubectl -n onap get secret|grep strimzi
external-strimzi-
onap-cps-kafka-user              onap-strimzi   scram-sha-512    simple          True
onap-policy-kafka-user        Opaque   onap-strimzi   scram-sha-512    simple          True
onap-so-sdc-list-user           2     onap-strimzi  2m7s ...
  • Get the user password

...

scram-sha-

...

Code Block
languagebash
themeMidnight
titleGet the user secret
kubectl get secret external-strimzi-kafka-user -o jsonpath='{.data.password}' -n onap | base64 --decode
Ujl...lSD

Test the external client access to Kafka

  • Add hostnames to DNS (or /etc/hosts) by using the IP Address of the istio-ingressgateway LB 
Code Block
languagebash
themeMidnight
sudo vi /etc/hosts
----
10.32.240.14 kafka-bootstrap-api.simpledemo.onap.org
10.32.240.14 kafka-api.simpledemo.onap.org
  • Install KafkaCat
Code Block
languagebash
themeMidnight
sudo apt install kafkacat
  • Get the Metadata (use an existing Kafka User, here "external-strimzi-kafka-user"):
Code Block
languagebash
themeMidnight
collapsetrue
root@control01-daily-master-sm:/# kafkacat -L -b kafka-bootstrap-api.simpledemo.onap.org:9003 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mech-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=hCv4IZ3Q6XLR -v 
Metadata for all topics (from broker -1: sasl_ssl://kafka-bootstrap-api.simpledemo.onap.org:9003/bootstrap):
 3 brokers:
  broker 0 at kafka-api.simpledemo.onap.org:9000 (controller)
  broker 2 at kafka-api.simpledemo.onap.org:9002
  broker 1 at kafka-api.simpledemo.onap.org:9001
 33 topics:
  topic "org.onap.dmaap.mr.PNF_REGISTRATION" with 2 partitions:
    partition 0, leader 2, replicas: 2, isrs: 2
    partition 1, leader 1, replicas: 1, isrs: 1 ...512    simple          True
policy-clamp-ac-a1pms-ppnt-ku    onap-strimzi   scram-sha-512    simple          True
policy-clamp-ac-http-ppnt-ku     onap-strimzi   scram-sha-512    simple          True
policy-clamp-ac-k8s-ppnt-ku      onap-strimzi   scram-sha-512    simple          True
policy-clamp-ac-kserve-ppnt-ku   onap-strimzi   scram-sha-512    simple          True
policy-clamp-ac-pf-ppnt-ku       onap-strimzi   scram-sha-512    simple          True
policy-clamp-runtime-acm-ku      onap-strimzi   scram-sha-512    simple          True
policy-distribution-ku           onap-strimzi   scram-sha-512    simple          True
sdc-be-ku                        onap-strimzi   scram-sha-512    simple          True
strimzi-kafka-admin              onap-strimzi   scram-sha-512                    True
  • List strimzi secrets
Code Block
languagebash
themeMidnight
titleList user secrets
root@control01-daily-master-sm:/# kubectl -n onap get secret|grep strimzi
external-strimzi-kafka-user                                        Opaque                                2      2m7s
...
  • Get the user password

For each KafkaUser resource with scram-sha-512 auth, there will be a corresponding secret:

Code Block
languagebash
themeMidnight
titleGet the user secret
kubectl get secret external-strimzi-kafka-user -o jsonpath='{.data.password}' -n onap | base64 --decode
Ujl...lSD

Test the external client access to Kafka


  • Add hostnames to DNS (or /etc/hosts) by using the IP Address of the istio-ingressgateway LB 
Code Block
languagebash
themeMidnight
sudo vi /etc/hosts
---- 
10.32.242.56 kafka-bootstrap-api.simpledemo.onap.org
10.32.242.56 kafka-api.simpledemo.onap.org


  • Install KafkaCat
Code Block
languagebash
themeMidnight
sudo apt install kafkacat
  • Get the Metadata (use an existing Kafka User, here "external-strimzi-kafka-user") using the sasl.password exported above:
Code Block
languagebash
themeMidnight
collapsetrue
kafkacat -L -b kafka-bootstrap-api.simpledemo.onap.org:9010 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=hCv4IZ3Q6XLR -v 
Metadata for all topics (from broker -1: sasl_ssl://kafka-bootstrap-api.simpledemo.onap.org:9003/bootstrap):
 3 brokers:
  broker 0 at kafka-api.simpledemo.onap.org:9000 (controller)
  broker 2 at kafka-api.simpledemo.onap.org:9002
  broker 1 at kafka-api.simpledemo.onap.org:9001
 33 topics:
  topic "org.onap.dmaap.mr.PNF_REGISTRATION" with 2 partitions:
    partition 0, leader 2, replicas: 2, isrs: 2
    partition 1, leader 1, replicas: 1, isrs: 1 ...
  • Get Metadata (use an existing Kafka User, here "external-strimzi-kafka-user"):
Code Block
languagebash
themeMidnight
collapsetrue
kafkacat -L -b kafka-bootstrap-api.simpledemo.onap.org:9010 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=e8ILJRai43LT -v
Metadata for all topics (from broker -1: sasl_ssl://kafka-bootstrap-api.simpledemo.onap.org:9010/bootstrap):
 3 brokers:
  broker 0 at kafka-api.simpledemo.onap.org:9000
  broker 2 at kafka-api.simpledemo.onap.org:9002
  broker 1 at kafka-api.simpledemo.onap.org:9001 (controller)
 2 topics:
  topic "unauthenticated.VES_NOTIFICATION_OUTPUT" with 6 partitions:
    partition 0, leader 1, replicas: 1,0,2, isrs: 0,1,2
    partition 1, leader 0, replicas: 0,2,1, isrs: 0,1,2
    partition 2, leader 2, replicas: 2,1,0, isrs: 0,1,2
    partition 3, leader 1, replicas: 1,2,0, isrs: 0,1,2
    partition 4, leader 0, replicas: 0,1,2, isrs: 0,1,2
    partition 5, leader 2, replicas: 2,0,1, isrs: 0,1,2
  topic "unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT" with 6 partitions:
    partition 0, leader 2, replicas: 2,1,0, isrs: 0,1,2
    partition 1, leader 1, replicas: 1,0,2, isrs: 0,1,2
    partition 2, leader 0, replicas: 0,2,1, isrs: 0,1,2
    partition 3, leader 2, replicas: 2,0,1, isrs: 0,1,2
    partition 4, leader 1, replicas: 1,2,0, isrs: 0,1,2
    partition 5, leader 0, replicas: 0,1,2, isrs: 0,1,2
  • Get Topic Data (use an existing Kafka User, here "external-strimzi-kafka-user"):
Code Block
languagebash
themeMidnight
collapsetrue
kafkacat -b kafka-bootstrap-api.simpledemo.onap.org:90039000 -X security.protocol=sasl_ssl -X enable.ssl.certificate.verification=false -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=external-strimzi-kafka-user -X sasl.password=hCv4IZ3Q6XLRe8ILJRai43LT -C -t unauthenticated.VES_NOTIFICATION_OUTPUT -v

{"event":{"commonEventHeader":{"startEpochMicrosec":8745745764578,"eventId":"FileReady_1797490e-10ae-4d48-9ea7-3d7d790b25e1","timeZoneOffset":"UTC+05.30","internalHeaderFields":{"collectorTimeStamp":"Tue, 12 06 2022 01:35:59 GMT"},"priority":"Normal","version":"4.0.1","reportingEntityName":"otenb5309","sequence":0,"domain":"notification","lastEpochMicrosec":8745745764578,"eventName":"Noti_RnNode-Ericsson_FileReady","vesEventListenerVersion":"7.0.1","sourceName":"oteNB5309"},"notificationFields":{"notificationFieldsVersion":"2.0","changeType":"FileReady","changeIdentifier":"PM_MEAS_FILES","arrayOfNamedHashMap":[{"name":"test.xml.gz","hashMap":{"location":"sftp://sftp:22/test.xml.gz","fileFormatType":"org.3GPP.32.435#measCollec","fileFormatVersion":"V10","compression":"gzip"}}]}}}
...

...