Motivation

Service Mesh simplifies infrastructure management on a an application by:

applying security policies in a centralized way
applying upgrade policies also in a centralized way
giving simple tracing view of the calls of an application

Today, ONAP uses AAF for certificates management and leaves the https implementation to the modules. Thus blacklisting a weak cipher or other security issue needs a verification on the ~200 services used by ONAP.

Having only one point of configuration would help a lot.

Troubleshooting issues is also very complex. Thus adding new tools such as tracing will help the ops people.

Service Mesh principles to apply

Generic principles

In order to make the service mesh work, here are the common principles:

Prefer use of (non headless) services. Pod to pod communication is not a first class citizen in a Service mesh (as the name stands). Thus using non headless services in the preferred way.
Listen at least on 127.0.0.1. Service Mesh sidecar will listen on the public POD IP. If the traffic is allowed, it needs to pass it to the "real" POD. For that, it sends it to 127.0.0.1. Then, the POD must listen on this interface
Name the ports in the service / pod with their protocol. In order to detect protocol, service mesh tends to use the naming of the port (on the service and pod part). Naming it "foo" doesn't help whereas "http" or "http-foo" helps the service mesh provider to know which protocol is used. allowed protocols are "tcp", "http", "https", "http2", "grpc".
Allow to disable https (or enable http). As the ssl part can be done by the service mesh, there's no need to do it on the pod side. Thus, we should be able to disable https (or enable http).

Principles specific to ONAP

As AAF is used for certificates generation and as AAF is not compliant with service mesh, we need to be able to disable AAF integration in ONAP components.

Plan of work

Focus first on the “core” part of ONAP, i.e. the one allowing to onboard and instantiate a service.

On this core, build from the ground, i.e. the services infrastructure and go up to the other services

Order of Service Mesh onboarding:

DMaaP
AAI
SDC
SO
SDN-C
Multicloud

DMaaP

Components

3 main components:

Message router
Data router
Bus controller

"Databases"

Message router relies on Kafka (which relies on Zookeeper)
Data router relies on Mariadb galera
Bus controller relies on postgreSQL

Work Done

Patch files for DMaaP (careful, it's hardcoded) and Postgres

Generic work

Jobs

Force service mesh to be disabled on every job by adding these annotations in the POD metadata (not in the Job metadata):

spec:
  template:
    metadata:
      annotations:
        linkerd.io/inject: disabled # This is not needed for Istio
        sidecar.istio.io/inject: disabled

Service Naming

Istio asks to set the type of protocol in the port name (https://istio.io/docs/ops/configuration/traffic-management/protocol-selection/). As DMaaP is exclusively http/https, I've added http- or https- in front of port names (hardcoded).

HTTP call

As istio is dealing (or not?) with mtls, http is sufficient. So every configuration with https should be move to http

Kafka

Kafka and Zookeeper from Message router seems not to be “service mesh compatible".

I've decided to to switch to banzaicloud kafka operator (which claims being istio compatible and use the same images for kafka)

Installation

I have on the cluster 3 storage classes:

default (Cinder backed by CEPH HDD)
ssd (Cinder backed by CEPH SSD)
ssd-kafka (same as SSD + the following value: volumeBindingMode: WaitForFirstConsumer) as banzaicloud propose to do (https://github.com/banzaicloud/kafka-operator/#installation-1).

command lines to install:

# Add the Helm repos
helm repo add jetstack https://charts.jetstack.io
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com/
helm repo update

# Cert Manager
kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml
kubectl create ns cert-manager
kubectl label namespace cert-manager istio-injection=enabled
helm install --name cert-manager --namespace cert-manager --version v0.11.0 jetstack/cert-manager

# Zookeeper
kubectl create ns zookeeper
helm install --name zookeeper-operator --namespace=zookeeper banzaicloud-stable/zookeeper-operator

kubectl apply -f - <<EOF
apiVersion: zookeeper.pravega.io/v1beta1
kind: ZookeeperCluster
metadata:
  name: onap
  namespace: zookeeper
spec:
  replicas: 3
  persistence:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 3G
    storageClassName: ssd
EOF

# Kafka
kubectl create ns kafka
kubectl label namespace kafka istio-injection=enabled
# the prometheus part is optional
helm install --namespace kafka --name=kafka-operator banzaicloud-stable/kafka-operator -f - << EOF
prometheus:
  serverFiles:
    alerts:
      groups:
      - name: KafkaAlerts
        rules:
        - alert: BrokerOverLoaded
          expr: avg(sum by(brokerId, kafka_cr, kubernetes_namespace) (rate(kafka_network_requestmetrics_requests_total[15m]))) > 30
          for: 5m
          labels:
            severity: alert
          annotations:
            description: 'broker {{ $labels.brokerId }} overloaded (current value is: {{ $value }})'
            summary: 'broker overloaded'
            # If brokerConfigGroup is defined it will override scaled broker config
            # brokerConfigGroup: 'default_group'
            storageClass: 'ssd-kafka'
            mountPath: '/kafkalog'
            diskSize: '2G'
            image: 'wurstmeister/kafka:2.12-2.3.0'
            command: 'upScale'
        - alert: BrokerUnderReplicated
          expr: kafka_server_replicamanager_underreplicatedpartitions > 0
          for: 5m
          labels:
            severity: alert
          annotations:
            description: 'broker {{ $labels.brokerId }} underreplicated'
            summary: 'broker underreplicated'
            storageClass: 'ssd-kafka'
            mountPath: '/kafkalog'
            diskSize: '2G'
            image: 'wurstmeister/kafka:2.12-2.3.0'
            command: 'upScale'
        - alert: PartitionCountHigh
          expr: max(kafka_server_replicamanager_partitioncount)  by (kubernetes_namespace, kafka_cr) > 100
          for: 3m
          labels:
            severity: alert
          annotations:
            description: 'broker {{ $labels.brokerId }} has high partition count'
            summary: 'high partition count'
            storageClass: 'ssd-kafka'
            mountPath: '/kafkalog'
            diskSize: '2G'
            image: 'wurstmeister/kafka:2.12-2.3.0'
            command: 'upScale'
        - alert: PartitionCountLow
          expr: min(kafka_server_replicamanager_partitioncount)  by (kubernetes_namespace, kafka_cr) < 40
          for: 3m
          labels:
            severity: alert
          annotations:
            description: 'broker {{ $labels.brokerId }} has low partition count'
            summary: 'low partition count'
            command: 'downScale'
        - alert: RemainingDiskSpaceLow
          expr: kubelet_volume_stats_available_bytes{persistentvolumeclaim=~"kafka-.*"} < 1 * 1000 * 1000 * 1000
          for: 2m
          labels:
            severity: alert
          annotations:
            description: 'broker {{ $labels.brokerId }} has low disk space'
            summary: 'low diskspace'
            storageClass: 'ssd-kafka'
            mountPath: '/kafkalog'
            diskSize: '2G'
            command: 'addPVC'
EOF

kubectl apply -f - <<EOF
apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaCluster
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: onap
  namespace: kafka
spec:
  headlessServiceEnabled: false
  zkAddresses:
    - "onap-client.zookeeper:2181"
  oneBrokerPerNode: true
  clusterImage: "wurstmeister/kafka:2.12-2.3.0"
  readOnlyConfig: |
    auto.create.topics.enable=false
  brokerConfigGroups:
    default:
      brokerConfig:
        storageConfigs:
          - mountPath: "/kafka-logs"
            pvcSpec:
              accessModes:
                - ReadWriteOnce
              storageClassName: ssd-kafka
              resources:
                requests:
                  storage: 10Gi
  brokers:
    - id: 0
      brokerConfigGroup: "default"
    - id: 1
      brokerConfigGroup: "default"
    - id: 2
      brokerConfigGroup: "default"
  rollingUpgradeConfig:
    failureThreshold: 1
  listenersConfig:
    internalListeners:
      - type: "plaintext"
        name: "plaintext"
        containerPort: 29092
        usedForInnerBrokerCommunication: true
      - type: "plaintext"
        name: "controller"
        containerPort: 29093
        usedForInnerBrokerCommunication: false
        usedForControllerCommunication: true
  cruiseControlConfig:
    topicConfig:
      partitions: 12
      replicationFactor: 3
    config: |
      # Copyright 2017 LinkedIn Corp. Licensed under the BSD 2-Clause License (the "License"). See License in the project root for license information.
      #
      # This is an example property file for Kafka Cruise Control. See KafkaCruiseControlConfig for more details.
      # Configuration for the metadata client.
      # =======================================
      # The maximum interval in milliseconds between two metadata refreshes.
      #metadata.max.age.ms=300000
      # Client id for the Cruise Control. It is used for the metadata client.
      #client.id=kafka-cruise-control
      # The size of TCP send buffer bytes for the metadata client.
      #send.buffer.bytes=131072
      # The size of TCP receive buffer size for the metadata client.
      #receive.buffer.bytes=131072
      # The time to wait before disconnect an idle TCP connection.
      #connections.max.idle.ms=540000
      # The time to wait before reconnect to a given host.
      #reconnect.backoff.ms=50
      # The time to wait for a response from a host after sending a request.
      #request.timeout.ms=30000
      # Configurations for the load monitor
      # =======================================
      # The number of metric fetcher thread to fetch metrics for the Kafka cluster
      num.metric.fetchers=1
      # The metric sampler class
      metric.sampler.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.CruiseControlMetricsReporterSampler
      # Configurations for CruiseControlMetricsReporterSampler
      metric.reporter.topic.pattern=__CruiseControlMetrics
      # The sample store class name
      sample.store.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.KafkaSampleStore
      # The config for the Kafka sample store to save the partition metric samples
      partition.metric.sample.store.topic=__KafkaCruiseControlPartitionMetricSamples
      # The config for the Kafka sample store to save the model training samples
      broker.metric.sample.store.topic=__KafkaCruiseControlModelTrainingSamples
      # The replication factor of Kafka metric sample store topic
      sample.store.topic.replication.factor=2
      # The config for the number of Kafka sample store consumer threads
      num.sample.loading.threads=8
      # The partition assignor class for the metric samplers
      metric.sampler.partition.assignor.class=com.linkedin.kafka.cruisecontrol.monitor.sampling.DefaultMetricSamplerPartitionAssignor
      # The metric sampling interval in milliseconds
      metric.sampling.interval.ms=120000
      # The partition metrics window size in milliseconds
      partition.metrics.window.ms=300000
      # The number of partition metric windows to keep in memory
      num.partition.metrics.windows=1
      # The minimum partition metric samples required for a partition in each window
      min.samples.per.partition.metrics.window=1
      # The broker metrics window size in milliseconds
      broker.metrics.window.ms=300000
      # The number of broker metric windows to keep in memory
      num.broker.metrics.windows=20
      # The minimum broker metric samples required for a partition in each window
      min.samples.per.broker.metrics.window=1
      # The configuration for the BrokerCapacityConfigFileResolver (supports JBOD and non-JBOD broker capacities)
      capacity.config.file=config/capacity.json
      #capacity.config.file=config/capacityJBOD.json
      # Configurations for the analyzer
      # =======================================
      # The list of goals to optimize the Kafka cluster for with pre-computed proposals
      default.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal
      # The list of supported goals
      goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.TopicReplicaDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.LeaderBytesInDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.kafkaassigner.KafkaAssignerDiskUsageDistributionGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.PreferredLeaderElectionGoal
      # The list of supported hard goals
      hard.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal
      # The minimum percentage of well monitored partitions out of all the partitions
      min.monitored.partition.percentage=0.95
      # The balance threshold for CPU
      cpu.balance.threshold=1.1
      # The balance threshold for disk
      disk.balance.threshold=1.1
      # The balance threshold for network inbound utilization
      network.inbound.balance.threshold=1.1
      # The balance threshold for network outbound utilization
      network.outbound.balance.threshold=1.1
      # The balance threshold for the replica count
      replica.count.balance.threshold=1.1
      # The capacity threshold for CPU in percentage
      cpu.capacity.threshold=0.8
      # The capacity threshold for disk in percentage
      disk.capacity.threshold=0.8
      # The capacity threshold for network inbound utilization in percentage
      network.inbound.capacity.threshold=0.8
      # The capacity threshold for network outbound utilization in percentage
      network.outbound.capacity.threshold=0.8
      # The threshold to define the cluster to be in a low CPU utilization state
      cpu.low.utilization.threshold=0.0
      # The threshold to define the cluster to be in a low disk utilization state
      disk.low.utilization.threshold=0.0
      # The threshold to define the cluster to be in a low network inbound utilization state
      network.inbound.low.utilization.threshold=0.0
      # The threshold to define the cluster to be in a low disk utilization state
      network.outbound.low.utilization.threshold=0.0
      # The metric anomaly percentile upper threshold
      metric.anomaly.percentile.upper.threshold=90.0
      # The metric anomaly percentile lower threshold
      metric.anomaly.percentile.lower.threshold=10.0
      # How often should the cached proposal be expired and recalculated if necessary
      proposal.expiration.ms=60000
      # The maximum number of replicas that can reside on a broker at any given time.
      max.replicas.per.broker=10000
      # The number of threads to use for proposal candidate precomputing.
      num.proposal.precompute.threads=1
      # the topics that should be excluded from the partition movement.
      #topics.excluded.from.partition.movement
      # Configurations for the executor
      # =======================================
      # The max number of partitions to move in/out on a given broker at a given time.
      num.concurrent.partition.movements.per.broker=10
      # The interval between two execution progress checks.
      execution.progress.check.interval.ms=10000
      # Configurations for anomaly detector
      # =======================================
      # The goal violation notifier class
      anomaly.notifier.class=com.linkedin.kafka.cruisecontrol.detector.notifier.SelfHealingNotifier
      # The metric anomaly finder class
      metric.anomaly.finder.class=com.linkedin.kafka.cruisecontrol.detector.KafkaMetricAnomalyFinder
      # The anomaly detection interval
      anomaly.detection.interval.ms=10000
      # The goal violation to detect.
      anomaly.detection.goals=com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal,com.linkedin.kafka.cruisecontrol.analyzer.goals.CpuCapacityGoal
      # The interested metrics for metric anomaly analyzer.
      metric.anomaly.analyzer.metrics=BROKER_PRODUCE_LOCAL_TIME_MS_MAX,BROKER_PRODUCE_LOCAL_TIME_MS_MEAN,BROKER_CONSUMER_FETCH_LOCAL_TIME_MS_MAX,BROKER_CONSUMER_FETCH_LOCAL_TIME_MS_MEAN,BROKER_FOLLOWER_FETCH_LOCAL_TIME_MS_MAX,BROKER_FOLLOWER_FETCH_LOCAL_TIME_MS_MEAN,BROKER_LOG_FLUSH_TIME_MS_MAX,BROKER_LOG_FLUSH_TIME_MS_MEAN
      ## Adjust accordingly if your metrics reporter is an older version and does not produce these metrics.
      #metric.anomaly.analyzer.metrics=BROKER_PRODUCE_LOCAL_TIME_MS_50TH,BROKER_PRODUCE_LOCAL_TIME_MS_999TH,BROKER_CONSUMER_FETCH_LOCAL_TIME_MS_50TH,BROKER_CONSUMER_FETCH_LOCAL_TIME_MS_999TH,BROKER_FOLLOWER_FETCH_LOCAL_TIME_MS_50TH,BROKER_FOLLOWER_FETCH_LOCAL_TIME_MS_999TH,BROKER_LOG_FLUSH_TIME_MS_50TH,BROKER_LOG_FLUSH_TIME_MS_999TH
      # The zk path to store failed broker information.
      failed.brokers.zk.path=/CruiseControlBrokerList
      # Topic config provider class
      topic.config.provider.class=com.linkedin.kafka.cruisecontrol.config.KafkaTopicConfigProvider
      # The cluster configurations for the KafkaTopicConfigProvider
      cluster.configs.file=config/clusterConfigs.json
      # The maximum time in milliseconds to store the response and access details of a completed user task.
      completed.user.task.retention.time.ms=21600000
      # The maximum time in milliseconds to retain the demotion history of brokers.
      demotion.history.retention.time.ms=86400000
      # The maximum number of completed user tasks for which the response and access details will be cached.
      max.cached.completed.user.tasks=100
      # The maximum number of user tasks for concurrently running in async endpoints across all users.
      max.active.user.tasks=5
      # Enable self healing for all anomaly detectors, unless the particular anomaly detector is explicitly disabled
      self.healing.enabled=true
      # Enable self healing for broker failure detector
      #self.healing.broker.failure.enabled=true
      # Enable self healing for goal violation detector
      #self.healing.goal.violation.enabled=true
      # Enable self healing for metric anomaly detector
      #self.healing.metric.anomaly.enabled=true
      # configurations for the webserver
      # ================================
      # HTTP listen port
      webserver.http.port=9090
      # HTTP listen address
      webserver.http.address=0.0.0.0
      # Whether CORS support is enabled for API or not
      webserver.http.cors.enabled=false
      # Value for Access-Control-Allow-Origin
      webserver.http.cors.origin=http://localhost:8080/
      # Value for Access-Control-Request-Method
      webserver.http.cors.allowmethods=OPTIONS,GET,POST
      # Headers that should be exposed to the Browser (Webapp)
      # This is a special header that is used by the
      # User Tasks subsystem and should be explicitly
      # Enabled when CORS mode is used as part of the
      # Admin Interface
      webserver.http.cors.exposeheaders=User-Task-ID
      # REST API default prefix
      # (dont forget the ending *)
      webserver.api.urlprefix=/kafkacruisecontrol/*
      # Location where the Cruise Control frontend is deployed
      webserver.ui.diskpath=./cruise-control-ui/dist/
      # URL path prefix for UI
      # (dont forget the ending *)
      webserver.ui.urlprefix=/*
      # Time After which request is converted to Async
      webserver.request.maxBlockTimeMs=10000
      # Default Session Expiry Period
      webserver.session.maxExpiryTimeMs=60000
      # Session cookie path
      webserver.session.path=/
      # Server Access Logs
      webserver.accesslog.enabled=true
      # Location of HTTP Request Logs
      webserver.accesslog.path=access.log
      # HTTP Request Log retention days
      webserver.accesslog.retention.days=14
    capacityConfig: |
      {
        "brokerCapacities":[
          {
            "brokerId": "-1",
            "capacity": {
              "DISK": "10000",
              "CPU": "100",
              "NW_IN": "10000",
              "NW_OUT": "10000"
            },
            "doc": "This is the default capacity. Capacity unit used for disk is in MB, cpu is in percentage, network throughput is in KB."
          }
        ]
      }
    clusterConfig: |
      {
        "min.insync.replicas": 3
      }
EOF

Using these kafka and zookeeper clusters for DMaaP Message Router

Per default, Message Router uses its own kafka and zookeeper clusters (charts cannot be disabled and configuration is set).

Here's what to do:

Change Zookeeper and Kafka Address in Message Router configuration

diff --git a/kubernetes/dmaap/components/message-router/resources/config/dmaap/MsgRtrApi.properties b/kubernetes/dmaap/components/message-router/resources/config/dmaap/MsgRtrApi.properties
index b07eaad9..714edc81 100755
--- a/kubernetes/dmaap/components/message-router/resources/config/dmaap/MsgRtrApi.properties
+++ b/kubernetes/dmaap/components/message-router/resources/config/dmaap/MsgRtrApi.properties
@@ -37,7 +37,8 @@
 ##
 #config.zk.servers=172.18.1.1
 #config.zk.servers={{.Values.zookeeper.name}}:{{.Values.zookeeper.port}}
-config.zk.servers={{.Release.Name}}-{{.Values.zookeeper.name}}-0.{{.Values.zookeeper.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.zookeeper.port}},{{.Release.Name}}-{{.Values.zookeeper.name}}-1.{{.Values.zookeeper.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.zookeeper.port}},{{.Release.Name}}-{{.Values.zookeeper.name}}-2.{{.Values.zookeeper.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.zookeeper.port}}
+#config.zk.servers={{.Release.Name}}-{{.Values.zookeeper.name}}-0.{{.Values.zookeeper.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.zookeeper.port}},{{.Release.Name}}-{{.Values.zookeeper.name}}-1.{{.Values.zookeeper.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.zookeeper.port}},{{.Release.Name}}-{{.Values.zookeeper.name}}-2.{{.Values.zookeeper.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.zookeeper.port}}
+config.zk.servers=onap-client.zookeeper:2181

 #config.zk.root=/fe3c/cambria/config

@@ -51,7 +52,9 @@ config.zk.servers={{.Release.Name}}-{{.Values.zookeeper.name}}-0.{{.Values.zooke
 ##     if you want to change request.required.acks it can take this one value
 #kafka.metadata.broker.list=localhost:9092,localhost:9093
 #kafka.metadata.broker.list={{.Values.kafka.name}}:{{.Values.kafka.port}}
-kafka.metadata.broker.list={{.Release.Name}}-{{.Values.kafka.name}}-0.{{.Values.kafka.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.kafka.port}},{{.Release.Name}}-{{.Values.kafka.name}}-1.{{.Values.kafka.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.kafka.port}},{{.Release.Name}}-{{.Values.kafka.name}}-2.{{.Values.kafka.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.kafka.port}}
+#kafka.metadata.broker.list={{.Release.Name}}-{{.Values.kafka.name}}-0.{{.Values.kafka.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.kafka.port}},{{.Release.Name}}-{{.Values.kafka.name}}-1.{{.Values.kafka.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.kafka.port}},{{.Release.Name}}-{{.Values.kafka.name}}-2.{{.Values.kafka.name}}.{{.Release.Namespace}}.svc.cluster.local:{{.Values.kafka.port}}
+kafka.metadata.broker.list=onap-all-broker.kafka:29092
+
 ##kafka.request.required.acks=-1
 #kafka.client.zookeeper=${config.zk.servers}
 consumer.timeout.ms=100

Don't wait for (already present) Zookeeper and Kafka

diff --git a/kubernetes/dmaap/components/message-router/templates/statefulset.yaml b/kubernetes/dmaap/components/message-router/templates/statefulset.yaml
index 21524ef3..d646e49b 100644
--- a/kubernetes/dmaap/components/message-router/templates/statefulset.yaml
+++ b/kubernetes/dmaap/components/message-router/templates/statefulset.yaml
@@ -30,23 +30,6 @@ spec:
         app: {{ include "common.name" . }}
         release: {{ .Release.Name }}
     spec:
-      initContainers:
-      - command:
-        - /root/ready.py
-        args:
-        - --container-name
-        - {{ .Values.kafka.name }}
-        - --container-name
-        - {{ .Values.zookeeper.name }}
-        env:
-        - name: NAMESPACE
-          valueFrom:
-            fieldRef:
-              apiVersion: v1
-              fieldPath: metadata.namespace
-        image: "{{ .Values.global.readinessRepository }}/{{ .Values.global.readinessImage }}"
-        imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
-        name: {{ include "common.name" . }}-readiness
       containers:
         - name: {{ include "common.name" . }}
           image: "{{ include "common.repository" . }}/{{ .Values.image }}"

This obviously hardcoded but we can think of a more generic approach

Mariadb Galera

Mariadb galera was working out of the box with a replica of 2. The only issue is that their internal communication is not seen and appears as 'PassthroughCluster'.

So I tried bitnami-mariadb charts (with replica of 3) but we have the same "issue" .

→ Moved back to "our" chart.

Postgresql

common postgres cluster chart doesn't work at all with service mesh:

the installation use a statefulset but with some "voodoo" in order to specialize the deployment
pgpool is not used (by DMaaP) nor can work (as passwords are hardcoded)

So I decided to recreate the chart with (specialized) deployments instead of a statefulset + make pgpool work with specific users (but only at launch for now) + use secrets with pgpool.

Here's the diff:

diff --git a/kubernetes/common/music/charts/music-cassandra-job/templates/job.yaml b/kubernetes/common/music/charts/music-cassandra-job/templates/job.yaml
index 88f0c746..0db0a95b 100644
--- a/kubernetes/common/music/charts/music-cassandra-job/templates/job.yaml
+++ b/kubernetes/common/music/charts/music-cassandra-job/templates/job.yaml
@@ -30,6 +30,9 @@ spec:
       labels:
         app: {{ include "common.name" . }}-job
         release: {{ .Release.Name }}
+      annotations:
+        linkerd.io/inject: disabled
+        sidecar.istio.io/inject: disabled
     spec:
       restartPolicy: Never
       initContainers:
diff --git a/kubernetes/common/music/charts/zookeeper/templates/job-chroots.yaml b/kubernetes/common/music/charts/zookeeper/templates/job-chroots.yaml
index b5b97233..5d7dacc7 100644
--- a/kubernetes/common/music/charts/zookeeper/templates/job-chroots.yaml
+++ b/kubernetes/common/music/charts/zookeeper/templates/job-chroots.yaml
@@ -28,6 +28,9 @@ spec:
         release: {{ .Release.Name }}
         component: jobs
         job: chroots
+      annotations:
+        linkerd.io/inject: disabled
+        sidecar.istio.io/inject: disabled
     spec:
       restartPolicy: {{ $job.restartPolicy }}
       containers:
diff --git a/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf b/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
index d8918409..580185c6 100644
--- a/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
+++ b/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
@@ -65,3 +65,4 @@
 #local   all         all                               trust
 # IPv4 local connections:
 host    all         all         0.0.0.0/0          md5
+
diff --git a/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd b/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd
deleted file mode 100644
index 3636d1de..00000000
--- a/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd
+++ /dev/null
@@ -1,2 +0,0 @@
-testuser:md599e8713364988502fa6189781bcf648f
-postgres:md53175bce1d3201d16594cebf9d7eb3f9d
diff --git a/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml b/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
index 15fa18d8..a4b0ca86 100644
--- a/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
+++ b/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
@@ -31,6 +31,33 @@ spec:
         app: {{ include "common.name" . }}
         release: {{ .Release.Name }}
     spec:
+      initContainers:
+      - name: {{ include "common.name" . }}-job
+        image: "{{.Values.repository}}/{{.Values.image}}"
+        imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
+        volumeMounts:
+        - name: pgpool-pgconf
+          mountPath: /pgconf/pgpoolconfigdir
+          readOnly: false
+        - name: pgpool-pgconf-static
+          mountPath: /configdir/
+          readOnly: false
+        command:
+        - /bin/sh
+        args:
+        - -c
+        - |
+          cp /configdir/pgpool.conf   /pgconf/pgpoolconfigdir/
+          cp /configdir/pool_hba.conf /pgconf/pgpoolconfigdir/
+          pg_md5 -f /pgconf/pgpoolconfigdir/pgpool.conf --md5auth --username=${PG_USER} ${PG_PASSWORD}
+        env:
+        - name: PG_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: {{ template "common.fullname" . }}
+              key: db-user-password
+        - name: PG_USER
+          value: {{ index .Values.credentials.pgusername }}
debian@control01-test-1:/opt/oom/kubernetes/common/postgres$ git diff -- .
diff --git a/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf b/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
index d8918409..580185c6 100644
--- a/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
+++ b/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
@@ -65,3 +65,4 @@
 #local   all         all                               trust
 # IPv4 local connections:
 host    all         all         0.0.0.0/0          md5
+
diff --git a/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd b/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd
deleted file mode 100644
index 3636d1de..00000000
--- a/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd
+++ /dev/null
@@ -1,2 +0,0 @@
-testuser:md599e8713364988502fa6189781bcf648f
-postgres:md53175bce1d3201d16594cebf9d7eb3f9d
diff --git a/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml b/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
index 15fa18d8..a4b0ca86 100644
--- a/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
+++ b/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
@@ -31,6 +31,33 @@ spec:
         app: {{ include "common.name" . }}
         release: {{ .Release.Name }}
     spec:
+      initContainers:
+      - name: {{ include "common.name" . }}-job
+        image: "{{.Values.repository}}/{{.Values.image}}"
+        imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
+        volumeMounts:
+        - name: pgpool-pgconf
+          mountPath: /pgconf/pgpoolconfigdir
+          readOnly: false
+        - name: pgpool-pgconf-static
+          mountPath: /configdir/
+          readOnly: false
+        command:
+        - /bin/sh
+        args:
+        - -c
+        - |
+          cp /configdir/pgpool.conf   /pgconf/pgpoolconfigdir/
+          cp /configdir/pool_hba.conf /pgconf/pgpoolconfigdir/
+          pg_md5 -f /pgconf/pgpoolconfigdir/pgpool.conf --md5auth --username=${PG_USER} ${PG_PASSWORD}
+        env:
+        - name: PG_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: {{ template "common.fullname" . }}
+              key: db-user-password
+        - name: PG_USER
+          value: {{ index .Values.credentials.pgusername }}
       containers:
       - image: "{{.Values.repository}}/{{.Values.image}}"
         imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
@@ -43,7 +70,10 @@ spec:
         - name: PG_USERNAME
           value: {{.Values.credentials.pgusername}}
         - name: PG_PASSWORD
-          value: {{.Values.credentials.pgpassword}}
+          valueFrom:
+            secretKeyRef:
+              name: {{ template "common.fullname" . }}
+              key: db-user-password
         ports:
         - containerPort: 5432
           name: pgpool
@@ -64,5 +94,7 @@ spec:
           readOnly: false
       volumes:
       - name: pgpool-pgconf
+        emptyDir: {}
+      - name: pgpool-pgconf-static
         configMap:
           name: {{ include "common.fullname" . }}-pgpool-configmap
diff --git a/kubernetes/common/postgres/charts/pgpool/templates/service.yaml b/kubernetes/common/postgres/charts/pgpool/templates/service.yaml
index 86442a27..4306b264 100644
--- a/kubernetes/common/postgres/charts/pgpool/templates/service.yaml
+++ b/kubernetes/common/postgres/charts/pgpool/templates/service.yaml
@@ -15,11 +15,11 @@ spec:
debian@control01-test-1:/opt/oom/kubernetes/common/postgres$ cd ..
debian@control01-test-1:/opt/oom/kubernetes/common$ git diff -- postgres
diff --git a/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf b/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
index d8918409..580185c6 100644
--- a/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
+++ b/kubernetes/common/postgres/charts/pgpool/configs/pool_hba.conf
@@ -65,3 +65,4 @@
 #local   all         all                               trust
 # IPv4 local connections:
 host    all         all         0.0.0.0/0          md5
+
diff --git a/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd b/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd
deleted file mode 100644
index 3636d1de..00000000
--- a/kubernetes/common/postgres/charts/pgpool/configs/pool_passwd
+++ /dev/null
@@ -1,2 +0,0 @@
-testuser:md599e8713364988502fa6189781bcf648f
-postgres:md53175bce1d3201d16594cebf9d7eb3f9d
diff --git a/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml b/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
index 15fa18d8..a4b0ca86 100644
--- a/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
+++ b/kubernetes/common/postgres/charts/pgpool/templates/deployment.yaml
@@ -31,6 +31,33 @@ spec:
         app: {{ include "common.name" . }}
         release: {{ .Release.Name }}
     spec:
+      initContainers:
+      - name: {{ include "common.name" . }}-job
+        image: "{{.Values.repository}}/{{.Values.image}}"
+        imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
+        volumeMounts:
+        - name: pgpool-pgconf
+          mountPath: /pgconf/pgpoolconfigdir
+          readOnly: false
+        - name: pgpool-pgconf-static
+          mountPath: /configdir/
+          readOnly: false
+        command:
+        - /bin/sh
+        args:
+        - -c
+        - |
+          cp /configdir/pgpool.conf   /pgconf/pgpoolconfigdir/
+          cp /configdir/pool_hba.conf /pgconf/pgpoolconfigdir/
+          pg_md5 -f /pgconf/pgpoolconfigdir/pgpool.conf --md5auth --username=${PG_USER} ${PG_PASSWORD}
+        env:
+        - name: PG_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: {{ template "common.fullname" . }}
+              key: db-user-password
+        - name: PG_USER
+          value: {{ index .Values.credentials.pgusername }}
       containers:
       - image: "{{.Values.repository}}/{{.Values.image}}"
         imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
@@ -43,7 +70,10 @@ spec:
         - name: PG_USERNAME
           value: {{.Values.credentials.pgusername}}
         - name: PG_PASSWORD
-          value: {{.Values.credentials.pgpassword}}
+          valueFrom:
+            secretKeyRef:
+              name: {{ template "common.fullname" . }}
+              key: db-user-password
         ports:
         - containerPort: 5432
           name: pgpool
@@ -64,5 +94,7 @@ spec:
           readOnly: false
       volumes:
       - name: pgpool-pgconf
+        emptyDir: {}
+      - name: pgpool-pgconf-static
         configMap:
           name: {{ include "common.fullname" . }}-pgpool-configmap
diff --git a/kubernetes/common/postgres/charts/pgpool/templates/service.yaml b/kubernetes/common/postgres/charts/pgpool/templates/service.yaml
index 86442a27..4306b264 100644
--- a/kubernetes/common/postgres/charts/pgpool/templates/service.yaml
+++ b/kubernetes/common/postgres/charts/pgpool/templates/service.yaml
@@ -15,11 +15,11 @@ spec:
     {{if eq .Values.service.type "NodePort" -}}
     - port: {{ .Values.service.externalPort }}
       nodePort: {{ .Values.global.nodePortPrefix | default .Values.nodePortPrefix }}{{ .Values.service.nodePort }}
-      name: {{ .Values.service.name }}
+      name: {{ .Values.service.portName }}
     {{- else -}}
     - port: {{ .Values.service.externalPort }}
       targetPort: {{ .Values.service.internalPort }}
-      name: {{ .Values.service.name }}
+      name: {{ .Values.service.portName }}
     {{- end}}
   selector:
     app: {{ include "common.name" . }}
diff --git a/kubernetes/common/postgres/charts/pgpool/values.yaml b/kubernetes/common/postgres/charts/pgpool/values.yaml
index cb732b7c..8ceff8f9 100644
--- a/kubernetes/common/postgres/charts/pgpool/values.yaml
+++ b/kubernetes/common/postgres/charts/pgpool/values.yaml
@@ -38,7 +38,8 @@ credentials:
   pgusername: testuser
   pgpassword: password
 service:
-  name: pgpool
+  name: pgpool
+  portName: tcp-postgres
   type: ClusterIP
   externalPort: 5432
   internalPort: 5432
diff --git a/kubernetes/common/postgres/templates/pv.yaml b/kubernetes/common/postgres/templates/pv.yaml
deleted file mode 100644
index 144a3f79..00000000
--- a/kubernetes/common/postgres/templates/pv.yaml
+++ /dev/null
@@ -1,58 +0,0 @@
-{{/*
-# Copyright © 2018 Amdocs, AT&T, Bell Canada
-# #
-# # Licensed under the Apache License, Version 2.0 (the "License");
-# # you may not use this file except in compliance with the License.
-# # You may obtain a copy of the License at
-# #
-# #       http://www.apache.org/licenses/LICENSE-2.0
-# #
-# # Unless required by applicable law or agreed to in writing, software
-# # distributed under the License is distributed on an "AS IS" BASIS,
-# # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# # See the License for the specific language governing permissions and
-# # limitations under the License.
-*/}}
-{{- if and .Values.persistence.enabled (not .Values.persistence.existingClaim) -}}
-kind: PersistentVolume
-apiVersion: v1
-metadata:
-  name: {{ include "common.fullname" . }}-data0
-  namespace: {{ include "common.namespace" . }}
-  labels:
-    app: {{ include "common.fullname" . }}
-    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
-    release: "{{ .Release.Name }}"
-    heritage: "{{ .Release.Service }}"
-    name: {{ include "common.fullname" . }}
-spec:
-  capacity:
-    storage: {{ .Values.persistence.size}}
-  accessModes:
-    - {{ .Values.persistence.accessMode }}
-  storageClassName: "{{ include "common.fullname" . }}-data"
-  persistentVolumeReclaimPolicy: {{ .Values.persistence.volumeReclaimPolicy }}
-  hostPath:
-    path: {{ .Values.global.persistence.mountPath | default .Values.persistence.mountPath }}/{{ .Release.Name }}/{{ .Values.persistence.mountSubPath }}0
----
-kind: PersistentVolume
-apiVersion: v1
-metadata:
-  name: {{ include "common.fullname" . }}-data1
-  namespace: {{ include "common.namespace" . }}
-  labels:
-    app: {{ include "common.fullname" . }}
-    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
-    release: "{{ .Release.Name }}"
-    heritage: "{{ .Release.Service }}"
-    name: {{ include "common.fullname" . }}
-spec:
-  capacity:
-    storage: {{ .Values.persistence.size}}
-  accessModes:
-    - {{ .Values.persistence.accessMode }}
-  storageClassName: "{{ include "common.fullname" . }}-data"
-  persistentVolumeReclaimPolicy: {{ .Values.persistence.volumeReclaimPolicy }}
-  hostPath:
-    path: {{ .Values.global.persistence.mountPath | default .Values.persistence.mountPath }}/{{ .Release.Name }}/{{ .Values.persistence.mountSubPath }}1
-{{- end -}}
diff --git a/kubernetes/common/postgres/templates/service.yaml b/kubernetes/common/postgres/templates/service.yaml
deleted file mode 100644
index 2a4e369e..00000000
--- a/kubernetes/common/postgres/templates/service.yaml
+++ /dev/null
@@ -1,95 +0,0 @@
-{{/*
-# Copyright © 2018 Amdocs, AT&T, Bell Canada
-# #
-# # Licensed under the Apache License, Version 2.0 (the "License");
-# # you may not use this file except in compliance with the License.
-# # You may obtain a copy of the License at
-# #
-# #       http://www.apache.org/licenses/LICENSE-2.0
-# #
-# # Unless required by applicable law or agreed to in writing, software
-# # distributed under the License is distributed on an "AS IS" BASIS,
-# # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# # See the License for the specific language governing permissions and
-# # limitations under the License.
-*/}}
-apiVersion: v1
-kind: Service
-metadata:
-  name: {{ .Values.service.name }}
-  namespace: {{ include "common.namespace" . }}
-  labels:
-    app: {{ include "common.name" . }}
-    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
-  annotations:
-spec:
-  type: {{ .Values.service.type }}
-  ports:
-    {{if eq .Values.service.type "NodePort" -}}
-    - port: {{ .Values.service.externalPort }}
-      nodePort: {{ .Values.global.nodePortPrefix | default .Values.nodePortPrefix }}{{ .Values.service.nodePort }}
-      name: {{ .Values.service.name }}
-    {{- else -}}
-    - port: {{ .Values.service.externalPort }}
-      targetPort: {{ .Values.service.internalPort }}
-      name: {{ .Values.service.name }}
-    {{- end}}
-  selector:
-    app: {{ include "common.name" . }}
-    release: {{ .Release.Name }}
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: {{ .Values.service.name2 }}
-  namespace: {{ include "common.namespace" . }}
-  labels:
-    app: {{ include "common.name" . }}
-    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
-  annotations:
-spec:
-  type: {{ .Values.service.type2 }}
-  ports:
-    {{if eq .Values.service.type "NodePort" -}}
-    - port: {{ .Values.service.externalPort2 }}
-      nodePort: {{ .Values.global.nodePortPrefix | default .Values.nodePortPrefix }}{{ .Values.service.nodePort2 }}
-      name: {{ .Values.service.name2 }}
-    {{- else -}}
-    - port: {{ .Values.service.externalPort2 }}
-      targetPort: {{ .Values.service.internalPort2 }}
-      name: {{ .Values.service.name2 }}
-    {{- end}}
-  selector:
-    name: "{{.Values.container.name.primary}}"
-    release: {{ .Release.Name }}
----
-apiVersion: v1
-kind: Service
-metadata:
-  name: {{ .Values.service.name3 }}
-  namespace: {{ include "common.namespace" . }}
-  labels:
-    app: {{ include "common.name" . }}
-    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
-  annotations:
-spec:
-  type: {{ .Values.service.type3 }}
-  ports:
-    {{if eq .Values.service.type "NodePort" -}}
-    - port: {{ .Values.service.externalPort3 }}
-      nodePort: {{ .Values.global.nodePortPrefix | default .Values.nodePortPrefix }}{{ .Values.service.nodePort3 }}
-      name: {{ .Values.service.name3 }}
-    {{- else -}}
-    - port: {{ .Values.service.externalPort3 }}
-      targetPort: {{ .Values.service.internalPort3 }}
-      name: {{ .Values.service.name3 }}
-    {{- end}}
-  selector:
-    name: "{{.Values.container.name.replica}}"
-    release: {{ .Release.Name }}
diff --git a/kubernetes/common/postgres/templates/statefulset.yaml b/kubernetes/common/postgres/templates/statefulset.yaml
deleted file mode 100644
index db4a256f..00000000
--- a/kubernetes/common/postgres/templates/statefulset.yaml
+++ /dev/null
@@ -1,155 +0,0 @@
-{{/*
-# Copyright © 2018 Amdocs, AT&T, Bell Canada
-# #
-# # Licensed under the Apache License, Version 2.0 (the "License");
-# # you may not use this file except in compliance with the License.
-# # You may obtain a copy of the License at
-# #
-# #       http://www.apache.org/licenses/LICENSE-2.0
-# #
-# # Unless required by applicable law or agreed to in writing, software
-# # distributed under the License is distributed on an "AS IS" BASIS,
-# # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# # See the License for the specific language governing permissions and
-# # limitations under the License.
-*/}}
-apiVersion: apps/v1beta1
-kind: StatefulSet
-metadata:
-  name: {{ include "common.fullname" . }}
-  namespace: {{ include "common.namespace" . }}
-  labels:
-    app: {{ include "common.name" . }}
-    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
-    release: {{ .Release.Name }}
-    heritage: {{ .Release.Service }}
-spec:
-  serviceName: {{ .Values.service.name }}
-  replicas: {{ .Values.replicaCount }}
-  template:
-    metadata:
-      labels:
-        app: {{ include "common.name" . }}
-        release: {{ .Release.Name }}
-    spec:
-      initContainers:
-      - command:
-        - /bin/sh
-        - -c
-        - |
-          for i in $(seq 0 $(({{ .Values.replicaCount }}-1))); do
-            if [ ! -d /podroot/data$i ]; then
-              mkdir -p /podroot/data$i;
-              chown 26:26 /podroot/data$i;
-              chmod 700 /podroot/data$i;
-            fi;
-          done
-        env:
-        - name: POD_NAME
-          valueFrom: { fieldRef: { fieldPath: metadata.name } }
-        securityContext:
-          privileged: true
-        image: {{ .Values.global.busyboxRepository | default .Values.busyboxRepository }}/{{ .Values.busyboxImage }}
-        imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
-        name: init-sysctl
-        volumeMounts:
-        - name: {{ include "common.fullname" . }}-init
-          mountPath: /podroot/
-      containers:
-      - name: {{ include "common.name" . }}
-        image: "{{ .Values.postgresRepository }}/{{ .Values.image }}"
-        imagePullPolicy: {{ .Values.global.pullPolicy | default .Values.pullPolicy }}
-        ports:
-        - containerPort: {{ .Values.service.internalPort }}
-          name: postgres
-        # disable liveness probe when breakpoints set in debugger
-        # so K8s doesn't restart unresponsive container
-        {{- if eq .Values.liveness.enabled true }}
-        livenessProbe:
-          tcpSocket:
-            port: {{ .Values.service.internalPort }}
-          initialDelaySeconds: {{ .Values.liveness.initialDelaySeconds }}
-          periodSeconds: {{ .Values.liveness.periodSeconds }}
-          timeoutSeconds: {{ .Values.liveness.timeoutSeconds }}
-        {{end -}}
-        readinessProbe:
-          tcpSocket:
-            port: {{ .Values.service.internalPort }}
-          initialDelaySeconds: {{ .Values.readiness.initialDelaySeconds }}
-          periodSeconds: {{ .Values.readiness.periodSeconds }}
-        env:
-        - name: PGHOST
-          value: /tmp
-        - name: PG_PRIMARY_USER
-          value: primaryuser
-        - name: PG_MODE
-          value: set
-        - name: PG_PRIMARY_HOST
-          value: "{{.Values.container.name.primary}}"
-        - name: PG_REPLICA_HOST
-          value: "{{.Values.container.name.replica}}"
-        - name: PG_PRIMARY_PORT
-          value: "{{.Values.service.internalPort}}"
-        - name: PG_PRIMARY_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: {{ template "common.fullname" . }}
-              key: pg-primary-password
-        - name: PG_USER
-          value: "{{.Values.config.pgUserName}}"
-        - name: PG_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: {{ template "common.fullname" . }}
-              key: pg-user-password
-        - name: PG_DATABASE
-          value: "{{.Values.config.pgDatabase}}"
-        - name: PG_ROOT_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: {{ template "common.fullname" . }}
-              key: pg-root-password
-        volumeMounts:
-        - mountPath: /pgdata
-          name: {{ include "common.fullname" . }}-data
-        - mountPath: /backup
-          name: {{ include "common.fullname" . }}-backup
-          readOnly: true
-        resources:
-{{ include "common.resources" . | indent 12 }}
-        {{- if .Values.nodeSelector }}
-        nodeSelector:
-{{ toYaml .Values.nodeSelector | indent 10 }}
-        {{- end -}}
-        {{- if .Values.affinity }}
-        affinity:
-{{ toYaml .Values.affinity | indent 10 }}
-        {{- end }}
-      volumes:
-      - name: localtime
-        hostPath:
-          path: /etc/localtime
-      - name: {{ include "common.fullname" . }}-init
-        hostPath:
-          path: {{ .Values.global.persistence.mountPath | default .Values.persistence.mountPath }}/{{ .Release.Name }}/{{ .Values.persistence.mountInitPath }}
-      - name: {{ include "common.fullname" . }}-backup
-        emptyDir: {}
-#{{ if not .Values.persistence.enabled }}
-      - name: {{ include "common.fullname" . }}-data
-        emptyDir: {}
-#{{ else }}
-  volumeClaimTemplates:
-  - metadata:
-      name: {{ include "common.fullname" . }}-data
-      labels:
-        name: {{ include "common.fullname" . }}
-    spec:
-      accessModes: [ {{ .Values.persistence.accessMode }} ]
-      storageClassName: {{ include "common.fullname" . }}-data
-      resources:
-        requests:
-          storage: {{ .Values.persistence.size }}
-      selector:
-        matchLabels:
-          name: {{ include "common.fullname" . }}
-#{{ end }}
diff --git a/kubernetes/common/postgres/values.yaml b/kubernetes/common/postgres/values.yaml
index 8dd5d0aa..3104ee79 100644
--- a/kubernetes/common/postgres/values.yaml
+++ b/kubernetes/common/postgres/values.yaml
@@ -104,14 +104,17 @@ persistence:
 service:
   type: ClusterIP
   name: pgset
+  portName: tcp-postgres
   externalPort: 5432
   internalPort: 5432
   type2: ClusterIP
-  name2: pgset-primary
+  name2: tcp-pgset-primary
+  portName2: tcp-postgres
   externalPort2: 5432
   internalPort2: 5432
   type3: ClusterIP
-  name3: pgset-replica
+  name3: tcp-pgset-replica
+  portName3: tcp-postgres
   externalPort3: 5432
   internalPort3: 5432

@@ -136,4 +139,3 @@ resources: {}
 #  requests:
 #    cpu: 2
 #    memory: 4Gi

What’s not (yet) on service mesh

Zookeeper

AAI

Components

AAI has several components:

babel
data-router
graphadmin
modelloader
resources
schema service
search data
sparky be
traversal
haproxy

"Databases"

cassandra
elasticsearch

Work Done

Cassandra

We must force Cassandra to listen on 127.0.0.1 and to use POD IP for broadcast.

diff --git a/kubernetes/common/cassandra/templates/statefulset.yaml b/kubernetes/common/cassandra/templates/statefulset.yaml
index 4be35708..57fad06a 100644
--- a/kubernetes/common/cassandra/templates/statefulset.yaml
+++ b/kubernetes/common/cassandra/templates/statefulset.yaml
@@ -115,6 +115,12 @@ spec:
             value: {{ default "GossipingPropertyFileSnitch" .Values.config.endpoint_snitch | quote }}
           - name: CASSANDRA_AUTHENTICATOR
             value: {{ default "PasswordAuthenticator" .Values.config.authenticator | quote }}
+          - name: CASSANDRA_LISTEN_ADDRESS
+            value: "127.0.0.1"
+          - name: CASSANDRA_BROADCAST_ADDRESS
+            valueFrom:
+              fieldRef:
+                fieldPath: status.podIP
           - name: POD_IP
             valueFrom:
               fieldRef:

Elasticsearch

Nothing to do

AAI components

all AAI components are using spring boot.

They're using https per default.

Adding the following lines on application.properties allow to move back to http:

security.require-ssl=false
server.ssl.enabled=false

we also need to use the different REST endpoint with http instead of https.

One example via many other: -schema.service.base.url=https://aai-schema-service.{{ include "common.namespace" . }}:8452/aai/schema-service/v1/ +schema.service.base.url=http://aai-schema-service.{{ include "common.namespace" . }}:8452/aai/schema-service/v1/

On the global part, we need to remove aaf, aaf-profile hand change client service schema to no-auth:

---
global:
   aafEnabled: false
   installSidecarSecurity: false
   config:
     profiles:
       active: production,dmaap
     schema:
       service:
         client: no-auth

HAProxy

"Main" entrance of AAI is actually an HAProxy. Instead of doing that, we could use a VirtualService from istio:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ include "common.servicename" . }}
  namespace: {{ include "common.namespace" . }}
  labels:
    app: {{ include "common.name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  hosts:
  - {{ include "common.servicename" . }}
  http:
  - name: "search-data"
    match:
    - uri:
        regex: "^/aai/v[0-9]+/search/generic-query$"
    - uri:
        regex: "^/aai/v[0-9]+/search/nodes-query$"
    - uri:
        regex: "^/aai/v[0-9]+/query$"
    - uri:
        regex: "^/aai/v[0-9]+/dsl$"
    - uri:
        prefix: "/aai/search/named-query"
    - uri:
        prefix: "/aai/search/model"
    route:
    - destination:
        host: aai-traversal
        port:
          number: 8846
  - name: "resources"
    route:
    - destination:
        host: aai-resources
        port:
          number: 8447

What’s not (yet) on service mesh

Nothing

SDC

Components

"Databases"

Work Done

What’s not (yet) on service mesh

Everything

SO

Components

"Databases"

Work Done

What’s not (yet) on service mesh

Everything

SDN-C

Components

"Databases"

Work Done

What’s not (yet) on service mesh

Everything

Multicloud

Components

"Databases"

Work Done

What’s not (yet) on service mesh

Everything