Async task execution failed by TimeoutException

Description

Hi,

We have encountered an issue during the Topology Discovery phase of Subsystem registration.

During that process we are sending down requests to the NCMP through the /v1/data endpoint which forwards the request to the ENM Adapter. As we have seen a few of the requests hasn't arrived to the ENM Adapter and we have seen the issues in the NCMP logs:

We have tried to increase this parameter in the NCMP but even with the increased value we were able to reproduce this issue:

NCMP_DMI_HTTPCLIENT_IDLECONNECTIONEVICTIONTHRESHOLDINSECONDS

The default value for this parameter was 5 seconds and we increased it 30 seconds.

There is one timeout variable defined in the DmiDataOperations class

  • private static final long DEFAULT_ASYNC_TASK_EXECUTOR_TIMEOUT_IN_MILLISECONDS = 30000L;

We would like to increase this value to see if that would fix this issue.

EDIT: We have built an NCMP image with the aforementioned CPS changes but the issue still persist

Adding the NCMP logs to the ticket.

Thank you!

Environment

None

Attachments

1

relates to

Activity

Show:

Daniel Hollos March 20, 2024 at 1:53 PM

Hi  

Thank you for letting me know and thank you for all your help with this issue!

Best regards,

Daniel

Priyank Maheshwari March 20, 2024 at 1:24 PM

Hi  

I will close this bug now.

Thanks for all your help.

Regards

Priyank

Priyank Maheshwari March 20, 2024 at 10:27 AM

Fault Slip Through Analysis

  • Bug Category: Performance

  • Reason for Slippage: Timing Issue

  • Action needed to prevent slippage: Appears during the load testing with 20k nodes discovery

  • Phase it should have been found in: Microservice CI (CIST)

  • Category of test it should have been found in: Performance

  • Description of test it should have been found in: Doing ENM discovery with high load in the system can cause the threads to be exhausted and not freeing up the memory.

Priyank Maheshwari March 15, 2024 at 11:11 AM
Edited

Hi  

We merged the changes.

137491: [Bug] Removing inner TaskExecutor to call DMI | https://gerrit.onap.org/r/c/cps/+/137491

If you want to test it again using the master branch, feel free to do so.

Priyank Maheshwari March 14, 2024 at 9:08 AM

Hi  

No worries at all.

I will raise an official fix now with the tested values and get my patch merged.

Once my patch is merged I will close the Bug.

Thanks for your help.

Done

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

Created March 6, 2024 at 1:32 PM
Updated June 14, 2024 at 10:40 AM
Resolved March 20, 2024 at 1:45 PM