Page Comparison

This howto is still work in progress.

It is sometimes convenient to have a local installation of DMaaP on your machine for testing purposes. For example, for testing policies, you can connect the Policy Framework to a local installation of DMaaP and use curl to issue the policy REST calls to the local DMaaP. For this howto, in order to avoid a dependency on A&AI, checks towards A&AI are commented out. This howto was written on the Amsterdam versions of DMaaP. It was executed on a Macbook Pro running macOS 10.13.3.

...

1. Create a new directory, let's call it "local-dmaap", and go into the directory

   Code Block
   language bash
   mkdir <path-to>/local-dmaap cd <path-to>/local-dmaap

2. Clone the DMaaP msgrtr and messageservice repositories

   Code Block
   language bash
   git clone https://gerrit.onap.org/r/dmaap/messagerouter/msgrtr git clone https://gerrit.onap.org/r/dmaap/messagerouter/messageservice

3. Check out the Amsterdam branches of the DMaaP repos

   Code Block
   cd msgrtr git co amsterdam cd ../messageservice git co amsterdam cd ..

4. In order to remove the need to install A&AI, comment out as follows in msgrtr src/main/java/com/att/nsa/cambria/service/impl/TopicServiceImpl.java:

   Code Block
   language java
   diff --git a/src/main/java/com/att/nsa/cambria/service/impl/TopicServiceImpl.java b/src/main/java/com/att/nsa/cambria/service/impl/TopicServiceImpl.java index c12be2f..e5ca1fcd11dda0 100644 --- a/src/main/java/com/att/nsa/cambria/service/impl/TopicServiceImpl.java +++ b/src/main/java/com/att/nsa/cambria/service/impl/TopicServiceImpl.java @@ -196,7 +196,7 @@ public class TopicServiceImpl implements TopicService { String appName=dmaapContext.getRequest().getHeader("AppName"); String enfTopicName= com.att.ajsc.beans.PropertiesMapBean.getProperty(CambriaConstants.msgRtr_prop,"enforced.topic.name.AAF"); - if(user != null) + /*if(user != null) { key = user.getKey(); @@ -214,7 +214,7 @@ public class TopicServiceImpl implements TopicService { } //else if (user==null && (null==dmaapContext.getRequest().getHeader("Authorization") && null == dmaapContext.getRequest().getHeader("cookie")) ) { - else if (user == null && null==dmaapContext.getRequest().getHeader("Authorization") && + else */if (user == null && null==dmaapContext.getRequest().getHeader("Authorization") && (null == appName && null == dmaapContext.getRequest().getHeader("cookie"))) { LOGGER.error("Failed to create topic"+topicBean.getTopicName()+", Authentication failed."); @@ -243223,7 +243223,7 @@ public class TopicServiceImpl implements TopicService { permission = mrFactoryVal+nameSpace+"|create"; errorMessages.getCreateTopicFail()+" "+errorMessages.getNotPermitted1()+" create "+errorMessages.getNotPermitted2()); DMaaPAAFAuthenticator aaf = new DMaaPAAFAuthenticatorImpl(); LOGGER.info(errRes.toString()); - throw new DMaaPAccessDeniedException(errRes); - if(!aaf.aafAuthentication(dmaapContext.getRequest(), permission)) + } + /*if(!aaf.aafAuthentication(dmaapContext.getRequest(), permission)) }*/ { if (user == null && (null!=dmaapContext.getRequest().getHeader("Authorization") || null != LOGGERdmaapContext.error("Failed to create topic"+topicBean.getTopicName()+", Authentication failed.");getRequest().getHeader("cookie"))) { @@ -254243,7 +254243,7 @@ public class TopicServiceImpl implements TopicService { permission LOGGER.info(errRes.toString())= mrFactoryVal+nameSpace+"|create"; DMaaPAAFAuthenticator aaf = throw new DMaaPAccessDeniedExceptionDMaaPAAFAuthenticatorImpl(errRes); - - }else{ if(!aaf.aafAuthentication(dmaapContext.getRequest(), permission)) + /*if(!aaf.aafAuthentication(dmaapContext.getRequest(), permission)) }else{*/ { // if user is null and aaf authentication is ok then key should be "" //key = ""; LOGGER.error("Failed to create topic"+topicBean.getTopicName()+", /** Authentication failed."); @@ -264254,7 +264254,7 @@ public class TopicServiceImpl implements TopicService { key = dmaapContext.getRequest().getUserPrincipal().getName().toString();LOGGER.info(errRes.toString()); LOGGER.info("key ==================== "+key); throw new DMaaPAccessDeniedException(errRes); - }else{ + /*}else{*/ } // if user tryis {null @@ -311,7 +311,7 @@ public class TopicServiceImpl implements TopicService {and aaf authentication is ok then key should be "" LOGGER.info("Deleting topic " + topicName); final NsaApiKey user = DMaaPAuthenticatorImpl.getAuthenticatedUser(dmaapContext)//key = ""; - if (user == null && null!=dmaapContext.getRequest().getHeader("Authorization")) { + /** @@ -264,7 +264,7 @@ public class /*if (user == null && null!=dmaapContext.getRequest().getHeader("Authorization")) TopicServiceImpl implements TopicService { LOGGER.info("Authenticating the user, as ACL authentication is not provided");key //= dmaapContext.getRequest().getUserPrincipal().getName().toString(); String permission = "com.att.dmaap.mr.topic"+"|"+topicName+"|"+"manage"; LOGGER.info("key ==================== "+key); String permission = ""; @@ -331,7 +331,7 @@ public class TopicServiceImpl implements TopicService { - } } + /*}*/ - } } + try { @@ -311,7 +311,7 @@ public class TopicServiceImpl }*/implements TopicService { final Broker metabroker = getMetaBroker(dmaapContextLOGGER.info("Deleting topic " + topicName); final TopicNsaApiKey topicuser = metabrokerDMaaPAuthenticatorImpl.getTopicgetAuthenticatedUser(topicNamedmaapContext);

5. Build the msgrtr, skipping tests if you wish. Once the build completes, we're finished in msgrtr.

   Code Block
   language bash
   cd msgrtr mvn clean install

6. In messageservice, the start-kafka.sh script breaks with the latest version of Scala and Kafka unless a new line is added to the end of the server.properties file. Edit the messageservice src/main/resources/docker-compose/start-kafka.sh as follows:

   Code Block
   language bash
   cd ../messageservice diff --git a/src/main/resources/docker-compose/start-kafka.sh b/src/main/resources/docker-compose/start-kafka.sh index 87047ad..e1bf99c 100644 --- a/src/main/resources/docker-compose/start-kafka.sh +++ b/src/main/resources/docker-compose/start-kafka.sh @@ -47,6 +47,9 @@ if [[ -z "$KAFKA_ADVERTISED_HOST_NAME" && -n "$HOSTNAME_COMMAND" ]]; then export KAFKA_ADVERTISED_HOST_NAME=$(eval $HOSTNAME_COMMAND) fi +# The replacement below fails if there is not a new line at the end of the $KAFKA_HOME/config/server.properties file +echo "" >> $KAFKA_HOME/config/server.properties + for VAR in `env` do if [[ $VAR =~ ^KAFKA_ && ! $VAR =~ ^KAFKA_HOME ]]; then

7. Ensure that the version of msgrtr dependency used in messageservice pom.xml is the local one created above, check your pom.xml in messageservice, specifically you must specify if msgrtr is a SNAPSHOT version. for example, see below:

   Code Block
   language bash
   <dependency>- if (user == null && null!=dmaapContext.getRequest().getHeader("Authorization")) { + /*if (user == null && null!=dmaapContext.getRequest().getHeader("Authorization")) { LOGGER.info("Authenticating the user, as ACL authentication is not provided"); // String permission = "com.att.dmaap.mr.topic"+"|"+topicName+"|"+"manage"; String permission = ""; @@ -331,7 +331,7 @@ public class TopicServiceImpl implements TopicService { <groupId>org.onap.dmaap.messagerouter.msgrtr</groupId> } <artifactId>msgrtr</artifactId> <version>1.0.1-SNAPSHOT</version>

   Build the message service, skipping tests if you wish.

   Code Block
   language bash
   mvn clean install

8. Ensure Docker is running on your machine. Clear down all relevant containers and images in your docker that might interfere with DMaaP. Use the commands below WITH CARE!

   Code Block
   docker ps -a | cut -f1 -d' ' | awk '{printf("docker rm %s\n", $1)}' docker images | cut -c45-60 | grep -v ubuntu | awk '{printf("docker rmi --force %s\n", $1)}'

9. In messageservice, copy the "appl" directory to target/classes/docker. This copies the configuration for DMaaP into the location in which we will build the docker images and the docker-compose setup.

   Code Block
   pwd local-dmaap/messageservice cp -pr target/swm/package/nix/dist_files/appl target/classes/docker

10. If /var/tmp/MsgRtr.properties file or directory exists on /var/tmp,delete it. This is the file where we will specify the local properties for Kafka/Zookeeper for DMaaP.

    Code Block
    language bash
    rm -fr /var/tmp/MsgRtrApi.properties

11. Copy the template MsgRtr properties to /var/tmp

    - } + }*/ final Broker metabroker = getMetaBroker(dmaapContext); final Topic topic = metabroker.getTopic(topicName);

12. Build the msgrtr, skipping tests if you wish. Once the build completes, we're finished in msgrtr.

    Code Block
    language bash
    cp bundleconfig-local/etc/appprops/MsgRtrApi.properties /var/tmp

    Edit /var/tmp/MsgRtrApi.properties as follows to put in the host names and ports for Kafka and Zookeeper in our docker-compose setup

    cd msgrtr mvn clean install

13. In messageservice, the start-kafka.sh script breaks with the latest version of Scala and Kafka unless a new line is added to the end of the server.properties file. Edit the messageservice src/main/resources/docker-compose/start-kafka.sh as follows:

    Code Block
    language bash
    0c40cd < config.zk.servers=<zookeeper_host>./messageservice diff --- > config.zk.servers=zookeeper 52c52 < kafka.metadata.broker.list=<kafka_host>:<kafka_port> --- > kafka.metadata.broker.list=kafka:9092

14. Now we prepare to build the Docker image for the DMaaP message service. Edit the messageservice startup.sh script as shown below, replacing all occurrences of "${project.artifactId}" with "dmaapMR1" and all occurrences of "${ajscRuntimeVersion}" with "1.0.0"

    Code Block
    language bash
    cd target/classes/docker diff startup.sh.orig startup.sh 22,24c22,24 < root_directory="/appl/${project.artifactId}" < config_directory="/appl/${project.artifactId}/bundleconfig" < runner_file="appl/${project.artifactId}/lib/ajsc-runner-${ajscRuntimeVersion}.jar" --- > root_directory="/appl/dmaapMR1" > config_directory="/appl/dmaapMR1/bundleconfig" > runner_file="appl/dmaapMR1/lib/ajsc-runner-1.0.0.jar"

15. Build the adapted DMaaP docker image, it will be tagged as dmaap:localadapt

    Code Block
    language bash
    docker build -t dmaap:localadapt .

16. Now we prepare the docker-compose setup for DMaaP. Go to messageservice/target/classes/docker-compose and edit Dockerfile, changing the KAFKA_VERSION to 1.0.0 and the SCALA_VERSION to 2.12

    Code Block
    cd ../docker-compose diff Dockerfile.orig Dockerfile 7c7 < ENV KAFKA_VERSION="0.8.1.1" SCALA_VERSION="2.9.2" --- > ENV KAFKA_VERSION="1.0.0" SCALA_VERSION="2.12"

17. Edit docker-compose.yml to change the name of th DMaaP docker image to use our locally adapted DMaaP image:

    Code Block
    language bash
    diff docker-compose.yml.orig docker-compose.yml 21c21 < image: attos/dmaap --- > image: dmaap:localadapt

18. On macOS only, edit docker-compose.yml to change var/tmp to /private/var/tmp (ensure you have /private/var/tmp shared with Docker on macOS).

19. Start DMaaP, crossing your fingers really tightly!

    Code Block
    language bash
    docker-compose up

20. Check that DMaaP is running with a curl request to list topics

    Code Block
    language bash
    curl --request GET localhost:3904/topics {"topics": []}

21. Create a topic called MyTpoic

    Code Block
    language bash
    curl \ --header "Content-type: application/json" \ --request POST \ --data '{ "topicName":"MyTopic", "partitionCount":"1", "replicationCount":"1", "transactionEnabled":"false" }' \ http://localhost:3904/topics/create git a/src/main/resources/docker-compose/start-kafka.sh b/src/main/resources/docker-compose/start-kafka.sh index 87047ad..e1bf99c 100644 --- a/src/main/resources/docker-compose/start-kafka.sh +++ b/src/main/resources/docker-compose/start-kafka.sh @@ -47,6 +47,9 @@ if [[ -z "$KAFKA_ADVERTISED_HOST_NAME" && -n "$HOSTNAME_COMMAND" ]]; then export KAFKA_ADVERTISED_HOST_NAME=$(eval $HOSTNAME_COMMAND) fi +# The replacement below fails if there is not a new line at the end of the $KAFKA_HOME/config/server.properties file +echo "" >> $KAFKA_HOME/config/server.properties + for VAR in `env` do if [[ $VAR =~ ^KAFKA_ && ! $VAR =~ ^KAFKA_HOME ]]; then

22. Ensure that the version of msgrtr dependency used in messageservice pom.xml is the local one created above, check your pom.xml in messageservice, specifically you must specify if msgrtr is a SNAPSHOT version. for example, see below:

    Code Block
    language bash
    <dependency> <groupId>org.onap.dmaap.messagerouter.msgrtr</groupId> <artifactId>msgrtr</artifactId> <version>1.0.1-SNAPSHOT</version>

23. Build the message service, skipping tests if you wish.

    Code Block
    language bash
    mvn clean install

24. Ensure Docker is running on your machine. Clear down all relevant containers and images in your docker that might interfere with DMaaP. Use the commands below WITH CARE!

    Code Block
    docker ps -a | cut -f1 -d' ' | awk '{printf("docker rm %s\n", $1)}' docker images | cut -c45-60 | grep -v ubuntu | awk '{printf("docker rmi --force %s\n", $1)}'

25. In messageservice, copy the "appl" directory to target/classes/docker. This copies the configuration for DMaaP into the location in which we will build the docker images and the docker-compose setup.

    Code Block
    pwd local-dmaap/messageservice cp -pr target/swm/package/nix/dist_files/appl target/classes/docker

26. In messageservice, copy the "startup.sh" script target/classes/docker. This is the startup shell script for the DMaaP message service in Docker.

    Code Block
    cp target/swm/package/nix/dist_files/startup.sh target/classes/docker

27. If /var/tmp/MsgRtr.properties file or directory exists on /var/tmp,delete it. This is the file where we will specify the local properties for Kafka/Zookeeper for DMaaP.

    Code Block
    language bash
    rm -fr /var/tmp/MsgRtrApi.properties

28. Copy the template MsgRtr properties to /var/tmp

    Code Block
    language bash
    cp bundleconfig-local/etc/appprops/MsgRtrApi.properties /var/tmp

29. Edit /var/tmp/MsgRtrApi.properties as follows to put in the host names and ports for Kafka and Zookeeper in our docker-compose setup:

    Code Block
    language bash
    0c40 < config.zk.servers=<zookeeper_host> --- > config.zk.servers=zookeeper 52c52 < kafka.metadata.broker.list=<kafka_host>:<kafka_port> --- > kafka.metadata.broker.list=kafka:9092

30. Build the adapted DMaaP docker image, it will be tagged as dmaap:localadapt

    Code Block
    language bash
    cd target/classes/docker docker build -t dmaap:localadapt .

31. Now we prepare the docker-compose setup for DMaaP. Go to messageservice/target/classes/docker-compose and edit Dockerfile, changing the KAFKA_VERSION to 1.0.0 and the SCALA_VERSION to 2.12

    Code Block
    cd ../docker-compose diff Dockerfile.orig Dockerfile 7c7 < ENV KAFKA_VERSION="0.8.1.1" SCALA_VERSION="2.9.2" --- > ENV KAFKA_VERSION="1.0.0" SCALA_VERSION="2.12"

32. Edit docker-compose.yml to change the name of th DMaaP docker image to use our locally adapted DMaaP image:

    Code Block
    language bash
    diff docker-compose.yml.orig docker-compose.yml 21c21 < image: attos/dmaap --- > image: dmaap:localadapt

33. On macOS only, edit docker-compose.yml to change var/tmp to /private/var/tmp (ensure you have /private/var/tmp shared with Docker on macOS).

34. Start DMaaP, crossing your fingers really tightly!

    Code Block
    language bash
    docker-compose up

35. Check that DMaaP is running with a curl request to list topics

    Code Block
    language bash
    curl --request GET localhost:3904/topics ANSWER {"topics": []}

36. Create a topic called MyTopic

    Code Block
    language bash
    curl \ > --header "Content-type: application/json" \ > --request POST \ > --data '{ > "topicName":"MyTopic", > "partitionCount":"1", > "replicationCount":"1", > "transactionEnabled":"false" > }' \ > http://localhost:3904/topics/create ANSWER: { "owner": "", "readerAcl": { "enabled": true, "users": [] }, "name": "MyTopic", "description": "", "writerAcl": { "enabled": true, "users": [] } }

37. List the topics again, our topic should now be there:

    Code Block
    language bash
    curl --request GET localhost:3904/topics ANSWER: {"topics": [ "msgrtr.apinode.metrics.dmaap", "MyTopic" ]}

    
    

    
    

    
    

    
    

Related articles

...