We will create a POC for Dublin release to prove the concept and usefulness of this project. As DataLake infrastructure itself will be considered outside of ONAP scope, the POC will focus on delivering one of key component of DL-Feeder as service under DCAE project.
Architecture
POC Scope
- DataLake Feeder delivery as service under DCAE (dcaegen2/services/dl-handler/dl-feeder)
- DLFeeder will support pluggable sources and datastore support
- DMAAP/Kafka as primary source (option to poll kafka broker for new topic?)
- Druid/Couchbase/Elasticsearch/Hadoop datastore support
- ConfigBingingService integration to fetch deployment/run-time configuration
- Alternate option will be to leverage k8s Configmap
- DLAdmin delivery as service under DCAE (dcaegen2/services/dl-handler/dl-admin). Will integrate the DL Admin UI into DCAE dashboard, which is built on portal SDK and serves as UAM for DCAE users/operation.
- Integrate DLAdmin into DCAE-Dashboard (See note below)
Other extension cases (stretch goal)
- RDMS datastore support
- Support other input data stream beside Dmaap/kafka
- Integrate with PNDA/datastore
- DCAE Platform integration for dynamic topic configuration
- Logging project integration
Deliverable
- DL Feeder container
- DL Admin GUI container
- Documentation (wiki and readme files) to setup/configure above
- Instruction for Test storage and UI setup and DL configuration management to demo data flow and visualization
...
- Karl Chiang
- Kate Hsuan
- Ekko Chang
IBM
Driptaroop Das
...