Functional Tests
Basic Functionality Test:
- Verify that all microservices work correctly with a single replica.
- Increase the number of replicas and ensure all functionality still works as expected.
Message Processing Test:
- Ensure that messages sent via Kafka are consumed and processed correctly by multiple replicas.
- Validate that there are no duplicate message processing or message loss.
- Ensure runtime policy operation topic sends to only one replica.
- Ensure sync topic sends to all replicas
Database Access Test: - Maybe
- Verify that the microservice accessing the database handles concurrent access correctly with multiple replicas.
- Ensure that read and write operations are consistent and correct.
Data Consistency Test:
- Test that the data remains consistent across multiple replicas when performing CRUD operations for ACM through REST API.
Performance Tests – add testing with replicas...
Load Testing:
- Simulate high load on the microservices and monitor their performance.
- Verify that the system can handle the load with multiple replicas without degradation.
- Comparison of this at the end of testing will be useful
Throughput Testing:
- Measure the throughput of message processing with different numbers of replicas.
- Ensure that increasing replicas improves or maintains the throughput.
Reliability and Scalability Tests
Replica Scaling Test:
- Dynamically scale the number of replicas up and down and ensure the system continues to function correctly.
- Verify that new replicas start correctly and join the existing system seamlessly.
Failover Test:
- Simulate failure of one or more replicas and verify that the system continues to function correctly.
- Ensure that there is no data loss and in-flight messages are processed by other replicas.
- Test that, after crashing a participant, and a new replica comes up –operations still work as expected.
- Test that, after crashing a participant replica, in the middle of operations like DEPLOY, and a new replica comes up – confirm the user can successfully re-run the operation.
Integration Tests
End-to-End Testing:
- Test the entire workflow involving all microservices with multiple replicas.
- Verify that the communication between microservices over Kafka is correct and the system meets the expected behavior.
Consistency and Synchronization Tests
Session Management Test:
- Verify that user sessions are managed correctly across multiple replicas.
- Ensure that session data is consistent and available to all replicas.
Distributed Locking Test:
- Test any distributed locking mechanisms to ensure they work correctly with multiple replicas.
- Verify that locks are acquired and released properly to prevent data corruption or race conditions.
Monitoring and Logging Tests
Logging and Monitoring Test:
- Ensure that logging works correctly across all replicas.
- Verify that monitoring tools correctly report the status and metrics of each replica.
Health Check and Auto-recovery Test:
- Implement and test health checks to ensure that unhealthy replicas are detected and handled correctly.
- Verify that auto-recovery mechanisms (like restarting failed replicas) work as expected.
Regression Tests
Backward Compatibility Test:
- Ensure that the new multi-replica setup does not break any existing functionality.
- Run all existing test cases to verify backward compatibility.
Stress Tests
Stress Testing:
- Push the system to its limits with maximum load and number of replicas.
- Observe the system's behavior and ensure it degrades gracefully without crashing.