Follow these steps to access Kafka from outside the Kuberenetes
1) Update the Kafka service YAML to add service type as LoadBalancer and install the DMaaP chart
Image Removed
2) Get the external IP address by running the below kubectl command. Kubernetes Dashboard also shows the external IP address.
kubectl get services/message-router-kafka -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
Image Removed
3) Update the KAFKA_ADVERTISED_LISTENERS environment variable in Kafka Deployment YAML with the external IP from step 2 and redeploy the DMaaP
Image Removed
Now Kafka can be accessed outside Kubernetes cluster at <external IP>:9092MirrorMaker is a stand-alone tool for copying data between two Apache Kafka clusters. It is a Kafka consumer and producer hooked together. Data is read from topics in the origin cluster and written to a topic with the same name in the destination cluster. You can run many such mirroring processes to increase throughput and for fault-tolerance
DMaaP Message Router offers the following REST API to create a mirror maker process, update a mirror maker process etc.
Mirror Maker
| Expand |
|---|
|
POST /mirrormakers/create This API will create a Mirror Maker. Required AAF Permissions : org.onap.dmaap.mr.mirrormaker|*|admin Request Payload:| Parameter Name | Req'd | |
|---|
| createMirrorMaker | Y | Mirror Maker Request Object | | name | Y | Unique Name for the MirrorMaker A-Z 1-0 only | | consumer | Y | Consumer hostname/IP Address | | producer | Y | Producer hostname/IP Address | | numStreams | Optional | Number of consumer threads to start. (If not provided default is 1) | | whitelist | Optional | Topic names to be whitelisted. This can be provided later using createWhitelist API | | enablelogCheck | Optional |
Enable error log monitoring on the MirrorMaker thread. Default values is False. If not provided error log is not monitored. |
{
"createMirrorMaker":
{
"name": "centraltoedge",
"consumer": "10.12.6.98:30491",
"producer": "10.12.5.108:30491",
"numStreams": "3",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": "true"
}
}
Response Payload:{
"listMirrorMaker": {
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RESTARTING",
"numStreams": 3
}
}
Error Messages: | Error Reson | Error Message |
|---|
| No AAF Mirror Maker Admin permission | No Mirror Maker Admin permission. | | Creating new MirrorMaker with existing MirrroMaker name | MirrorMaker <MM Name> already exists | | Incorrect payload | This is not a CreateMirrorMaker request. Please try again. | | Consumer field/value missing | Please provide Consumer host:port details | | Producer field/value missing | Please provide Producer host:port details | | Incorrect naming convention | Mirror Maker name can only contain alpha numeric | | Incorrect JSONIncorrect JSON object. | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. | | If Mirror Maker agent is down or it takes more than (right now it is configured 15 seconds) | listMirrorMaker is not available, please make sure MirrorMakerAgent is running |
|
| Expand |
|---|
|
POST /mirrormakers/update This API is used to update an existing MirrorMaker. Required AAF Permissions: org.onap.dmaap.mr.mirrormaker|*|admin Request Payload:| Parameter Name | Req'd | Description |
|---|
| updateMirrorMaker | Y | Mirror Maker Request Object | | name | Y | Existing MIrrorMaker name | | consumer | Optional | Consumer hostname/IP Address | | producer | Optional | Producer hostname/IP Address | | numStreams | Optional | Update Number of consumer threads. | | enablelogCheck | Optional | Enable/Disable error log monitoring on the MirrorMaker thread |
{
"updateMirrorMaker":{
"name":"centraltoedge",
"consumer": "10.12.6.98:30491",
"producer": "10.12.5.108:30491",
"numStreams":"2",
"enablelogCheck":"true"
}
} Response Payload:{
"listMirrorMaker": {
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RESTARTING",
"numStreams": 2
}
} Error Messages:| Error Reason | Error Message |
|---|
| No AAF Mirror Maker Admin permission | No Mirror Maker Admin permission. | | Incorrect MirrorMaker name that does not exist | MirrorMaker <MM Name> does not exist | | Incorrect payload | This is not a UpdateMirrorMaker request. Please try again. | | Consumer value not provided | Please provide Consumer host:port details | | Producer value not provided | Please provide Producer host:port details | | Incorrect naming convention | Mirror Maker name can only contain alpha numeric | | Could not parse JSON. | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. | | If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listMirrorMaker is not available, please make sure MirrorMakerAgent is running. |
|
| Expand |
|---|
| title | List All Mirror Maker |
|---|
|
POST /mirrormakers/listall This API returns all the MirrorMakers and their present status. Required AAF Permissions: org.onap.dmaap.mr.mirrormaker|*|admin Request:Request Payload:| Parameter Name | Req'd | Description |
|---|
| listAllMirrorMaker | Y | An empty listAllMirrorMaker JSON object |
{ "listAllMirrorMaker": { } } Response Payload:{
"listMirrorMaker": [
{
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RUNNING",
"numStreams": 3
}
],
"messageID": "99310"
} Error Messages:| Error Reason | Error Message |
|---|
| No AAF Mirror Maker Admin permission | No Mirror Maker Admin permission. | | Incorrect payload | This is not a ListAllMirrorMaker request. Please try again. | | Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. | | If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listMirrorMaker is not available, please make sure MirrorMakerAgent is running |
|
| Expand |
|---|
|
POST /mirrormakers/delete This API is used to delete an existing MirrorMaker. Required AAF Permissions: org.onap.dmaap.mr.mirrormaker|*|admin Request:Request Payload:| Parameter Name | Req'd | Description |
|---|
| deleteMirrorMaker | Y | Mirror Maker Request Object | | Name | Y | Existing MirrorMaker name |
{
"deleteMirrorMaker":
{
"name":"centraltoedge2"
}
}
Response Payload:
"listMirrorMaker": [
{
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RUNNING",
"numStreams": 3
}
],
"messageID": "99310"
}
Error Messages:| Error Reason | Error Message |
|---|
| No AAF Mirror Maker Admin permission | No Mirror Maker Admin permission. | | Incorrect MirrorMaker name that does not exist | MirrorMaker <MM Name> does not exist | | Incorrect payload | This is not a DeleteMirrorMaker request. Please try again. | | Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. | | If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listMirrorMaker is not available, please make sure MirrorMakerAgent is running. |
|
WhiteList
| Expand |
|---|
|
POST /mirrormakers/createwhitelist This API is used to add a new topic to the whitelist. AAF permissions required: org.onap.dmaap.mr.mirrormaker|*|user , org.onap.dmaap.mr.topicFactory|:org.onap.dmaap.mr.topic:org.onap.dmaap.mr|create Request Payload:| Parameter Name | Req'd | Description |
|---|
| name | Y | Mirror Maker name | | namespace | Y | a namespace | | whitelistTopicName | Y | a topic name that contains only alpha numeric value |
{
"name":"centraltoedge",
"namespace":"org.onap.dmaap.mr",
"whitelistTopicName":"org.onap.dmaap.mr.mrtesttopic2"
}
Response Payload:{
"listMirrorMaker": [
{
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic,org.onap.dmaap.mr.mrtesttopic2",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RESTARTING",
"numStreams": 3
}
],
}
Error Messages:| Error Reason | Error Message |
|---|
| No AAF Mirror Maker User permission | No Mirror Maker User permission. | | No AAF Mirror Maker User Create permission (User should have create topics permission ont he namespace listed) | No Mirror Maker User Create permission. | | Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. | | Request contain topic that already exist | The topic already exist. | | If the namespace in the topic does not match with the namespace in the request | The namespace of the topic does not match with the namespace you provided. | | If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listWhiteList is not available, please make sure MirrorMakerAgent is running | | Incorrect payload | This is not a createWhitelist request. Please try again. |
|
| Expand |
|---|
|
POST /mirrormakers/listallwhitelist This API list all the Whitelists . AAF permissions required: org.onap.dmaap.mr.mirrormaker|*|user , org.onap.dmaap.mr.topicFactory|:org.onap.dmaap.mr.topic:org.onap.dmaap.mr|create Request Payload:| Parameter Name | Req'd | Description |
|---|
| name | Y | Mirror Maker name | | namespace | Y | a namespace |
{
"name":"centraltoedge",
"namespace":"org.onap.dmaap.mr"
} Response Payload:"name": "centraltoedge",
"whitelist": "org.onap.dmaap.mr.mrtesttopic,org.onap.dmaap.mr.mrtesttopic2"
} Error Messages:| Error Reason | Error Message |
|---|
| No AAF Mirror Maker User permission | No Mirror Maker User permission. | | No AAF Mirror Maker User Create permission | No Mirror Maker User Create permission. | | Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. | | If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listallWhiteList is not available, please make sure MirrorMakerAgent is running | | Incorrect payload | This is not a ListAllWhitelist request. Please try again. |
|
| Expand |
|---|
|
POST /mirrormakers/deletewhitelist This API removes a topic from whitelist. AAF permissions required: org.onap.dmaap.mr.mirrormaker|*|user , org.onap.dmaap.mr.topicFactory|:org.onap.dmaap.mr.topic:org.onap.dmaap.mr|create Request Payload:| Parameter Name | Req'd | Description |
|---|
| name | Y | Mirror Maker name | | namespace | Y | a namespace | | whitelistTopicName | Y | a topic name that contains only alpha numeric value |
{
"name":"centraltoedge",
"namespace":"org.onap.dmaap.mr",
"whitelistTopicName":"org.onap.dmaap.mr.mrtesttopic2"
} Response Payload:{
"listMirrorMaker": [
{
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RESTARTING",
"numStreams": 3
}
],
}
Error Messages:| Error Reason | Error Message |
|---|
| No AAF Mirror Maker User permission | No Mirror Maker User permission. | | No AAF Mirror Maker User Create permission | No Mirror Maker User Create permission. | | Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. | | Request contain topic does not exist | The topic does not exist. | | If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listWhiteList is not available, please make sure MirrorMakerAgent is running | | Incorrect payload | This is not a DeleteAllWhitelist request. Please try again. |
|
Recommendations:
1) Topic partitions must be same in both producer and consumer Kafka clusters
2) For better performance numStreams should match to the topic partitions