Immediately, we’re excited to announce that DataFlow Capabilities (DFF), a function inside Cloudera DataFlow for the Public Cloud, is now usually out there for AWS, Microsoft Azure, and Google Cloud Platform. DFF supplies an environment friendly, price optimized, scalable approach to run NiFi flows in a totally serverless style. That is the primary full no-code, no-ops improvement expertise for capabilities, permitting customers to save lots of time and sources.
Fig1: First no-code UI within the business to rapidly develop and deploy capabilities to cloud suppliers’ serverless compute companies.
First no-code UI for serverless capabilities
Beforehand, builders needed to write code and depend on code samples to get began with capabilities. Now, they will use DataFlow’s no-code UI to be extra productive – they will rapidly design new NiFi flows after which run them as capabilities in AWS Lambda, Azure Capabilities, and Google Cloud Capabilities.
Fig2: DataFlow Capabilities runtime environments can be found in
AWS Lambda, Azure Capabilities, and Google Cloud Capabilities.
Optimize price and eradicate infrastructure administration
For the reason that knowledge flows are working in serverless environments within the public clouds, infrastructure administration is a factor of the previous. The circulation is barely executed when an occasion triggers the perform, providing a really environment friendly method of deploying event-driven use instances with out requiring builders to expend invaluable sources on operational tasks. For example, a file touchdown in an object retailer (S3, ADLS, or GCS) triggers the execution of a knowledge circulation, which then processes the file and sends the end result elsewhere.
Fig3: A pattern use case the place a file that lands in an object retailer triggers a perform that processes that file and sends outcomes to a vacation spot.
DataFlow Capabilities supplies an environment friendly, price optimized, scalable approach to run NiFi flows in a totally serverless style for event-driven use instances.
The fitting runtime on your use instances
There at the moment are two methods to run your Apache NiFi knowledge flows within the Cloudera DataFlow service: DataFlow deployments and DataFlow Capabilities:
- Deployments runtime is optimized for high-throughput, low-latency streaming use instances
- Capabilities runtime is greatest suited to event-driven, short-lived use instances
Fig4: Runtime choices within the public cloud: DataFlow Deployments and DataFlow Capabilities
Beneath is a extra detailed breakdown of the 2 NiFi runtime choices within the public cloud:
| Runtime choices within the Public Cloud | ||
| Characteristic | DataFlow Deployments | DataFlow Capabilities |
| Cloud Runtime | NiFi clusters utilizing
Kubernetes/containers |
NiFi flows working on cloud suppliers’ serverless compute companies (AWS Lambda, Azure Capabilities, and Google Cloud Capabilities) |
| Use Case | Use instances that want low latency for prime throughput workloads requiring always-running NiFi flows | Occasion pushed, micro-bursty use instances with no sub-second latency requirement the place NiFi flows don’t must run repeatedly |
| Advantages | Auto-scaling Kubernetes clusters for lengthy working workflows with centralized monitoring | Environment friendly, price optimized, scalable approach to run NiFi flows serverless, permitting builders to give attention to enterprise logic |
Abstract
DataFlow Capabilities supplies a brand new, environment friendly approach to run your event-driven Apache NiFi knowledge flows.
With DataFlow Capabilities you may deploy your circulation purposes in minutes by leveraging the serverless structure of all main public cloud suppliers (AWS, Azure, and Google Cloud Platform), and also you shouldn’t have to fret concerning the operational overhead of managing and sustaining NiFi circulation runtime environments.
To study extra on the way to arrange and run DataFlow Capabilities in AWS Lambda, Azure Capabilities, and Google Cloud Capabilities, checkout our technical weblog, or take a product tour for a light-weight step-by-step expertise.
