- Created by sunil unnava, last modified on Apr 18, 2019
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 4 Current »
MirrorMaker is a stand-alone tool for copying data between two Apache Kafka clusters. It is a Kafka consumer and producer hooked together. Data is read from topics in the origin cluster and written to a topic with the same name in the destination cluster. You can run many such mirroring processes to increase throughput and for fault-tolerance
DMaaP Message Router offers the following REST API to create a mirror maker process, update a mirror maker process etc.
Mirror Maker
POST /mirrormakers/create
This API will create a Mirror Maker.
Required AAF Permissions : org.onap.dmaap.mr.mirrormaker|*|admin
Request Payload:
| Parameter Name | Req'd | Description |
|---|---|---|
| createMirrorMaker | Y | Mirror Maker Request Object |
| name | Y | Unique Name for the MirrorMaker A-Z 1-0 only |
| consumer | Y | Consumer hostname/IP Address |
| producer | Y | Producer hostname/IP Address |
| numStreams | Optional | Number of consumer threads to start. (If not provided default is 1) |
| whitelist | Optional | Topic names to be whitelisted. This can be provided later using createWhitelist API |
| enablelogCheck | Optional | Enable error log monitoring on the MirrorMaker thread. Default values is False. If not provided error log is not monitored. |
{
"createMirrorMaker":
{
"name": "centraltoedge",
"consumer": "10.12.6.98:30491",
"producer": "10.12.5.108:30491",
"numStreams": "3",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": "true"
}
}
Response Payload:
{
"listMirrorMaker": {
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RESTARTING",
"numStreams": 3
}
}
Error Messages:
| Error Reson | Error Message |
|---|---|
| No AAF Mirror Maker Admin permission | No Mirror Maker Admin permission. |
| Creating new MirrorMaker with existing MirrroMaker name | MirrorMaker <MM Name> already exists |
| Incorrect payload | This is not a CreateMirrorMaker request. Please try again. |
| Consumer field/value missing | Please provide Consumer host:port details |
| Producer field/value missing | Please provide Producer host:port details |
| Incorrect naming convention | Mirror Maker name can only contain alpha numeric |
| Incorrect JSONIncorrect JSON object. | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. |
| If Mirror Maker agent is down or it takes more than (right now it is configured 15 seconds) | listMirrorMaker is not available, please make sure MirrorMakerAgent is running |
POST /mirrormakers/update
This API is used to update an existing MirrorMaker.
Required AAF Permissions: org.onap.dmaap.mr.mirrormaker|*|admin
Request Payload:
| Parameter Name | Req'd | Description |
|---|---|---|
| updateMirrorMaker | Y | Mirror Maker Request Object |
| name | Y | Existing MIrrorMaker name |
| consumer | Optional | Consumer hostname/IP Address |
| producer | Optional | Producer hostname/IP Address |
| numStreams | Optional | Update Number of consumer threads. |
| enablelogCheck | Optional | Enable/Disable error log monitoring on the MirrorMaker thread |
{
"updateMirrorMaker":{
"name":"centraltoedge",
"consumer": "10.12.6.98:30491",
"producer": "10.12.5.108:30491",
"numStreams":"2",
"enablelogCheck":"true"
}
}
Response Payload:
{
"listMirrorMaker": {
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RESTARTING",
"numStreams": 2
}
}
Error Messages:
| Error Reason | Error Message |
|---|---|
| No AAF Mirror Maker Admin permission | No Mirror Maker Admin permission. |
| Incorrect MirrorMaker name that does not exist | MirrorMaker <MM Name> does not exist |
| Incorrect payload | This is not a UpdateMirrorMaker request. Please try again. |
| Consumer value not provided | Please provide Consumer host:port details |
| Producer value not provided | Please provide Producer host:port details |
| Incorrect naming convention | Mirror Maker name can only contain alpha numeric |
| Could not parse JSON. | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. |
| If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listMirrorMaker is not available, please make sure MirrorMakerAgent is running. |
POST /mirrormakers/listall
This API returns all the MirrorMakers and their present status.
Required AAF Permissions: org.onap.dmaap.mr.mirrormaker|*|admin
Request:
Request Payload:
| Parameter Name | Req'd | Description |
|---|---|---|
| listAllMirrorMaker | Y | An empty listAllMirrorMaker JSON object |
{
"listAllMirrorMaker":
{
}
}
Response Payload:
{
"listMirrorMaker": [
{
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RUNNING",
"numStreams": 3
}
],
"messageID": "99310"
}
Error Messages:
| Error Reason | Error Message |
|---|---|
| No AAF Mirror Maker Admin permission | No Mirror Maker Admin permission. |
| Incorrect payload | This is not a ListAllMirrorMaker request. Please try again. |
| Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. |
| If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listMirrorMaker is not available, please make sure MirrorMakerAgent is running |
POST /mirrormakers/delete
This API is used to delete an existing MirrorMaker.
Required AAF Permissions: org.onap.dmaap.mr.mirrormaker|*|admin
Request:
Request Payload:
| Parameter Name | Req'd | Description |
|---|---|---|
| deleteMirrorMaker | Y | Mirror Maker Request Object |
| Name | Y | Existing MirrorMaker name |
{
"deleteMirrorMaker":
{
"name":"centraltoedge2"
}
}
Response Payload:
"listMirrorMaker": [
{
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RUNNING",
"numStreams": 3
}
],
"messageID": "99310"
}
Error Messages:
| Error Reason | Error Message |
|---|---|
| No AAF Mirror Maker Admin permission | No Mirror Maker Admin permission. |
| Incorrect MirrorMaker name that does not exist | MirrorMaker <MM Name> does not exist |
| Incorrect payload | This is not a DeleteMirrorMaker request. Please try again. |
| Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. |
| If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listMirrorMaker is not available, please make sure MirrorMakerAgent is running. |
WhiteList
POST /mirrormakers/createwhitelist
This API is used to add a new topic to the whitelist.
AAF permissions required: org.onap.dmaap.mr.mirrormaker|*|user , org.onap.dmaap.mr.topicFactory|:org.onap.dmaap.mr.topic:org.onap.dmaap.mr|create
Request Payload:
| Parameter Name | Req'd | Description |
|---|---|---|
| name | Y | Mirror Maker name |
| namespace | Y | a namespace |
| whitelistTopicName | Y | a topic name that contains only alpha numeric value |
{
"name":"centraltoedge",
"namespace":"org.onap.dmaap.mr",
"whitelistTopicName":"org.onap.dmaap.mr.mrtesttopic2"
}
Response Payload:
{
"listMirrorMaker": [
{
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic,org.onap.dmaap.mr.mrtesttopic2",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RESTARTING",
"numStreams": 3
}
],
}
Error Messages:
| Error Reason | Error Message |
|---|---|
| No AAF Mirror Maker User permission | No Mirror Maker User permission. |
| No AAF Mirror Maker User Create permission (User should have create topics permission ont he namespace listed) | No Mirror Maker User Create permission. |
| Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. |
| Request contain topic that already exist | The topic already exist. |
| If the namespace in the topic does not match with the namespace in the request | The namespace of the topic does not match with the namespace you provided. |
| If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listWhiteList is not available, please make sure MirrorMakerAgent is running |
| Incorrect payload | This is not a createWhitelist request. Please try again. |
POST /mirrormakers/listallwhitelist
This API list all the Whitelists .
AAF permissions required: org.onap.dmaap.mr.mirrormaker|*|user , org.onap.dmaap.mr.topicFactory|:org.onap.dmaap.mr.topic:org.onap.dmaap.mr|create
Request Payload:
| Parameter Name | Req'd | Description |
|---|---|---|
| name | Y | Mirror Maker name |
| namespace | Y | a namespace |
{
"name":"centraltoedge",
"namespace":"org.onap.dmaap.mr"
}
Response Payload:
"name": "centraltoedge",
"whitelist": "org.onap.dmaap.mr.mrtesttopic,org.onap.dmaap.mr.mrtesttopic2"
}
Error Messages:
| Error Reason | Error Message |
|---|---|
| No AAF Mirror Maker User permission | No Mirror Maker User permission. |
| No AAF Mirror Maker User Create permission | No Mirror Maker User Create permission. |
| Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. |
| If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listallWhiteList is not available, please make sure MirrorMakerAgent is running |
| Incorrect payload | This is not a ListAllWhitelist request. Please try again. |
POST /mirrormakers/deletewhitelist
This API removes a topic from whitelist.
AAF permissions required: org.onap.dmaap.mr.mirrormaker|*|user , org.onap.dmaap.mr.topicFactory|:org.onap.dmaap.mr.topic:org.onap.dmaap.mr|create
Request Payload:
| Parameter Name | Req'd | Description |
|---|---|---|
| name | Y | Mirror Maker name |
| namespace | Y | a namespace |
| whitelistTopicName | Y | a topic name that contains only alpha numeric value |
{
"name":"centraltoedge",
"namespace":"org.onap.dmaap.mr",
"whitelistTopicName":"org.onap.dmaap.mr.mrtesttopic2"
}
Response Payload:
{
"listMirrorMaker": [
{
"name": "centraltoedge",
"producer": "10.12.5.108:30491",
"whitelist": "org.onap.dmaap.mr.mrtesttopic",
"enablelogCheck": true,
"consumer": "10.12.6.98:30491",
"status": "RESTARTING",
"numStreams": 3
}
],
}
Error Messages:
| Error Reason | Error Message |
|---|---|
| No AAF Mirror Maker User permission | No Mirror Maker User permission. |
| No AAF Mirror Maker User Create permission | No Mirror Maker User Create permission. |
| Incorrect JSON | Incorrect JSON object.Could not parse JSON. Please correct the JSON format and try again. |
| Request contain topic does not exist | The topic does not exist. |
| If Mirror Maker agent is down or it takes more than (right now it is configured 15 second) | listWhiteList is not available, please make sure MirrorMakerAgent is running |
| Incorrect payload | This is not a DeleteAllWhitelist request. Please try again. |
Recommendations:
1) Topic partitions must be same in both producer and consumer Kafka clusters
2) For better performance numStreams should match to the topic partitions
- No labels