DMaaP Deployment Guide - Dublin
- Dominic Lunanuova
This page and sub-pages are intended to capture Notes for deploying DMaaP components. Due the number of integration points between DMaaP components, and also with AAF and OOM, there sometimes is a need for some minor edits to get a successful deployment.
General
Increase Helm Timeout
Depending on your environment, the deployment of all the components takes a while and can easily exceed the default helm timeout. Recommend adding --timeout 900 to your helm install command line.
Disable AAF Authentication for DMaaP MR
Authentication via AAF is the default setting in Dublin DMaaP MR. But sometimes it is useful to disable this check and the dependency on AAF (e.g. for a demo setup). This section provides some instructions for disabling AAF for the various DMaaP components.
Use WindRiver Test AAF instead of a local deployment
By default, the DMaaP OOM charts will assume a locally deployed AAF instance. Sometimes it is convenient to avoid an AAF deployment and simply use the AAF Test instance in WindRiver.
TBD
Disable persistence (Postgresql) for dmaap-bc
By default, dmaap-bc utilizes the common postgresql chart. This starts up several additional pods. But dmaap-bc does support a non-PG option (i.e. no data persistence if pod is restarted) which is handy for quick tests and full redeployments.
To disable Postresql:
edit oom/kubernetes/dmaap/components/dmaap-bc/values.yaml
change the attribute PG.enabled to false
recompile the dmaap helm charts. i.e. from kubernetes directory, make dmaap
deploy dmaap
Disable persistence (pv) for kafka and zookeeper (in message-router)
Q: Can I disable the persistent volumes in zookeeper and kafka
we may not be able to use PV's in our deployment of xyz
Sunil Unnava(10:38:46 AM): Yes. Remove the PV and PVClaimTemplate from the statefulset.yaml. You also dont need * permission-fixer init container
Change Kafka Log Retention
"Kafka Log" refers to the queue used to store messages for each topic. "Retention" is the max time an individual message should remain in the queue (regardless of whether it was read by a consumer or not) The default log retention time (i.e. how long a message will be retained) is 7 days. A topic with lots of volume might consume the amount of disk space allotted to all the kafka message queues, so in some deployment environments the retention time should be modified.
In your local deployment, if you know the log retention needs of all Kafka topics, update the default log retention by setting the following environment variable in Kafka statefulset.yaml.
Retention based on time:
KAFKA_LOG_RETENTION_HOURS ( you can also set the retention in minutes and ms using the environment variables KAFKA_LOG_RETENTION_MINUTES and KAFKA_LOG_RETENTION_MS)
Retention based on topic partition size:
KAFKA_LOG_RETENTION_BYTES
Following are the some of the useful commands to override/view/delete( the override) the topic retention at topic level at run time
To update topic level log retention based on time:
/opt/kafka/bin/kafka-configs.sh --zookeeper message-router-zookeeper:218 --entity-type topics --entity-name <<topic--name>> --alter --add-config retention.ms=<<>>
To update topic level log retention based on topic partition size:
/opt/kafka/bin/kafka-configs.sh --zookeeper message-router-zookeeper:2181 --entity-type topics --entity-name <<topic--name>> --alter --add-config retention.bytes=<<>>
View the topic level overrides:
bin/kafka-configs.sh --zookeeper message-router-zookeeper:2181 --entity-type topics --entity-name <<topic--name>> --describe
Remove the topic level overrides:
/opt/kafka/bin/kafka-configs.sh --zookeeper message-router-zookeeper:2181 --entity-type topics --entity-name <<topic--name>> --alter --<<overrided property to remove>>
Release Combinations
Earlier combinations not included.
As of March 29, 2019 (AAF 2.1.8, dbc-client 1.0.5, dmaap-bc 1.1.2, message-router 1.1.12)