Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

DL-Feeder log error and configuration issue

Description

Following (3) issues noticed on recent deployment

  1. Run.sh used from docker writes into consul (need to verify which other services reads off consul still)

  2. OOM charts for DL-feeder includes below properties which are deprecated

    • CONSUL_HOST: consul

    • CONFIG_BINDING_SERVICE: 30408

  3. Error reported on Feeder on startup; DMAAP/zookeeper connection configuration to be fixed

022-06-03 17:13:00,116 |ERROR| [                main] org.onap.datalake.feeder.service.TopicConfigPollingService:86 - error connection to HDFS. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dmaapService': Invocation of init method failed; nested exception is java.net.UnknownHostException: message-router-zookeeper: System error         at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:139)         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419)         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1737)         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:576)         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)         at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:338)         at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:224)         at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1114)         at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveBean(DefaultListableBeanFactory.java:407)         at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:341)         at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1107)         at org.onap.datalake.feeder.service.TopicConfigPollingService.poll(TopicConfigPollingService.java:186)         at org.onap.datalake.feeder.service.TopicConfigPollingService.poll(TopicConfigPollingService.java:170)         at org.onap.datalake.feeder.service.TopicConfigPollingService.init(TopicConfigPollingService.java:84)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)         at java.base/java.lang.reflect.Method.invoke(Unknown Source)         at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)         at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)         at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419)         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1737)         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:576)         at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:498)         at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)         at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)         at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)         at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)         at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:846)         at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:863)         at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:546)         at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)         at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)         at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)         at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)         at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260)         at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248)         at org.onap.datalake.feeder.Application.main(Application.java:46)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)         at java.base/java.lang.reflect.Method.invoke(Unknown Source)         at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)         at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)         at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)         at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) Caused by: java.net.UnknownHostException: message-router-zookeeper: System error         at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)         at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown Source)         at java.base/java.net.InetAddress.getAddressesFromNameService(Unknown Source)         at java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown Source)         at java.base/java.net.InetAddress.getAllByName0(Unknown Source)         at java.base/java.net.InetAddress.getAllByName(Unknown Source)         at java.base/java.net.InetAddress.getAllByName(Unknown Source)         at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)         at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)         at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)         at org.onap.datalake.feeder.service.DmaapService.connect(DmaapService.java:116)         at org.onap.datalake.feeder.service.DmaapService.init(DmaapService.java:90)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)         at java.base/java.lang.reflect.Method.invoke(Unknown Source)         at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)         at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)         at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)         ... 46 common frames omitted 2022-06-03 17:13:02,972 |INFO | [                main] org.onap.datalake.feeder.Application:59  - Started Application in 10.904 seconds (JVM running for 11.897) 2022-06-03 17:13:02,985 |INFO | [                main] org.onap.datalake.feeder.service.PullService:84 - PullService starting ...

 

HV-VES Kafka updates reference

https://gerrit.onap.org/r/c/dcaegen2/collectors/hv-ves/+/127005 here are the changes made for hv-ves
especially: https://gerrit.onap.org/r/c/dcaegen2/collectors/hv-ves/+/127005/12/sources/hv-collector-core/src/main/kotlin/org/onap/dcae/collectors/veshv/impl/adapters/kafka/KafkaSenderOptionsFactory.kt

Environment

None

Confluence content

mentioned on

Activity

Show:

Former user April 20, 2023 at 4:30 PM

As we are past M3 for London,  moving this to M release.

Former user November 17, 2022 at 3:43 PM

I am facing errors with the changes I have done. Could you please move it to the L-release.

Former user November 8, 2022 at 5:28 PM
Edited

Hi , the plan overall for kafka is to move to kraft. This will remove the need for zookeeper as kafka will handle it's own meta data.

The strimzi kafka cluster for now will have both but ultimately it will not allow direct comms to zookeeper.

Former user November 8, 2022 at 5:08 PM

Thanks   for assessment on other impacts.  Reg. zookeeper, what is replacement in the target architecture? Is this something planned for London or beyond?

  - For the current container to work the sasl mechanism should be updated (similar to adaptaion done in HV-VES). Have you had chance to start on those updates?

Former user November 8, 2022 at 4:06 PM

Hi,

From briefly looking at the code, there will be some changes needed.

Strimzi does not permit direct access to zookeeper so this will have to go. Also, zookeeper is being phased out in the future.

In ONAP, we decided to use SCRAM-SHA-512 sasl.mechanism with SASL_PLAINTEXT so this will have to align.

Also, any dynamic/runtime topic management is not permitted. All required topics must be created via helm CRDs in the OOM project.

Details

Assignee

Reporter

Labels

Components

Fix versions

Affects versions

Priority

Parent

Created June 3, 2022 at 8:14 PM
Updated April 20, 2023 at 4:31 PM
Resolved June 3, 2022 at 8:14 PM

Flag notifications