CPS-1613 Explore How to Integrate Kafka Tests in the CSITs
References
https://jira.onap.org/browse/CPS-1601
Assumptions
# | Issue | Notes | Decisions |
|---|---|---|---|
1 | kafka container port issue | The exposed port of kafka container changed to 9092 in docker-compose.yml | As per the technical discussion, there is no hard-coded port given in any java classes. |
2 | KAFKA_ADVERTISED_LISTENERS hostname resolution issue | The port of KAFKA_ADVERTISED_LISTENERS has been changed to 29092. | As per the technical discussion, there is no hard-coded port given in any java classes. |
Issues & Decisions
# | Issue | Notes | Decisions |
|---|---|---|---|
1 | kafka container port issue | Open Questions: 1- Can we change the port to 9092 | Since docker-compose.yml is being used for local env. setup and CSIT, it is possible to change to 9092. |
2 | KAFKA_ADVERTISED_LISTENERS hostname resolution issue | Open Questions: 1- Can we change the port to 29092 | As per local tests run by the Team 2, and ONAP build, the required change has no negative effect. |
Overview
Can Robot Framework test Kafka clusters and events?
In the latest version of Robot Framework (6.0.2), there is no support for Kafka Event Streaming.
Is there any third-party library that would aid in this?
ConfluentKafkaLibrary is a Robot Framework library to verify Kafka Event Streaming.
Issues
1-In the current implementation of ConfluentKafkaLibrary, the consumer module connects to Kafka cluster from the port "9092".
Consumer init
def __init__(
self,
server='127.0.0.1',
port='9092',
topics='',
group_id=None,
only_value=True,
**kwargs
):Whereas, CPS's kafka container expose its port from "19092"
The previous Kafka container config
kafka:
image: confluentinc/cp-kafka:6.2.1
container_name: kafka
ports:
- "19092:19092"
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,CONNECTIONS_FROM_HOST://localhost:19092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
In order to solve this conflict, the current port has been changed to 9092 in docker-compose.yml
The new Kafka container config
kafka:
image: confluentinc/cp-kafka:6.2.1
container_name: kafka
ports:
- '9092:9092'
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,CONNECTIONS_FROM_HOST://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
2-The ConfluentKafkaLibrary also try to access the KAFKA_ADVERTISED_LISTENERS (PLAINTEXT://kafka:9092) natively.
However, it gets fail because of hostname to IP address resolution issue. There is a possible solution for this problem.
A change in the KAFKA_ADVERTISED_LISTENERS port to 29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,CONNECTIONS_FROM_HOST://localhost:909