Introducing Cloudera DataFlow Designer: Self-service, No-Code Dataflow Design

Posted in Technical |
December 09, 2022 7 min learn

Cloudera has been offering enterprise help for Apache NiFi since 2015, serving to tons of of organizations take management of their information motion pipelines on premises and within the public cloud. Working with these organizations has taught us loads in regards to the wants of builders and directors with regards to growing new dataflows and supporting them in mission-critical manufacturing environments. 

In 2021 we launched Cloudera DataFlow for the Public Cloud (CDF-PC), addressing operational challenges that directors face when operating NiFi flows in manufacturing environments. Present NiFi customers can now deliver their NiFi flows and run them in our cloud service by creating DataFlow Deployments that profit from auto-scaling, one-button NiFi model upgrades, centralized monitoring by way of KPIs, multi-cloud help, and automation by way of a robust command-line interface (CLI). Lately, we introduced the overall availability of DataFlow Capabilities, permitting NiFi flows to be executed in serverless compute environments, corresponding to AWS Lambda, Azure Capabilities, or Google Cloud Capabilities. 

With DataFlow Deployments and DataFlow Capabilities being obtainable, movement directors can now decide the best choice for operating their dataflows in manufacturing within the public cloud. Now, we shift deal with the wants of builders and addressing the challenges they face when constructing dataflows within the cloud.

Enabling self-service for builders

Builders have to onboard new information sources, chain a number of information transformation steps collectively, and discover information because it travels by way of the movement. They worth NiFi’s visible, no-code, drag-and-drop UI, the 450+ out-of-the-box processors and connectors, in addition to the flexibility to interactively discover information by beginning particular person processors within the movement and instantly seeing the affect as information streams by way of the movement. 

We’ve noticed organizations utilizing an increasing number of information sources and locations, in addition to anticipating a extra various vary of builders to construct information motion flows. This remark additional emphasizes the necessity for common developer accessibility, which makes certain that developer tooling is straightforward to make use of for newcomers whereas giving energy customers the superior choices they want. A essential facet of common developer accessibility is to offer dataflow growth as a self-service providing to builders. This can be a problem as a result of builders are both required to handle their very own native Apache NiFi set up, or a platform workforce is required to handle a centralized growth setting that every one builders can use. 

What if there was a solution to not require builders to handle their very own Apache NiFi set up with out placing that burden on platform directors? What if we might present an easy-to-manage, self-service growth setting for builders that anybody can begin utilizing instantly? 

These are the questions we requested ourselves, and I’m excited to announce the technical preview of DataFlow Designer, making self-service dataflow growth a actuality for Cloudera clients.

A reimagined visible editor to spice up developer productiveness and allow self service

On the core of our new self-service developer expertise is the brand new DataFlow Designer, which reinforces NiFi’s hottest options whereas making key enhancements to the person expertise—all introduced in a recent feel and appear. 

Determine 1: The Designer canvas with a model new feel and appear

A key enchancment over the normal Apache NiFi canvas is the brand new expandable configuration facet panel, permitting builders to rapidly edit processor configurations with out dropping focus of what’s taking place on the canvas. The facet panel is context-sensitive and immediately shows related configuration info as you navigate by way of your movement elements.

Determine 2: Don’t lose sight of the canvas whereas making use of configuration adjustments within the facet panel

One other instance of how the brand new movement designer makes a developer’s life simpler is the flexibility to immediately add information by way of the designer UI. In conventional NiFi growth environments, builders would both require SSH entry to the NiFi cases to add information or ask their directors to do it for them. Being able to add information like JDBC Drivers, Python scripts, and so on. immediately within the designer makes constructing new flows much more self-service.

Determine 3: Simply add information immediately by way of the designer with out requiring SSH entry to servers

Talking of parameters—they’re an vital idea to make your dataflows moveable. In any case, it’s very doubtless that you’re growing your movement towards take a look at techniques however in manufacturing it must run towards manufacturing techniques, that means that your supply and vacation spot connection configuration must be adjusted. The easiest way to do that is by parameterizing these connection configuration values permitting you to plug in several values when making a movement deployment in manufacturing. You may set default values for parameters in addition to mark them as delicate, which ensures that nobody can see the worth that was set.

Determine 4: Central administration of movement parameters

The Designer helps on-the-fly parameter creation when configuring elements in addition to auto-complete by urgent CTRL+SPACE when offering a configuration worth. Because of this, parameter administration is at all times at your fingertips proper the place you want it with out requiring you to modify between views to look them up.

Determine 5: Parameter references within the configuration panel and auto-complete

Interactivity when wanted whereas saving prices

Certainly one of NiFi’s distinctive options is the flexibility to work together with every element in a dataflow individually with out having to cease your entire movement. This enables builders to make adjustments to their processing logic on the fly whereas operating some take a look at information by way of their movement and validating that their adjustments work as supposed. For instance, in case your dataflow is studying occasions from a Kafka subject, which you need to filter and course of however you’re undecided in regards to the actual schema the occasions are in, you would possibly need to peek on the occasions earlier than writing your filter situation. With NiFi you possibly can configure your supply processor and run it independently of another processors to retrieve information. After getting retrieved the information, NiFi shops it in a queue, which lets you discover the content material and metadata attributes of the occasions. As soon as you understand how your occasions look, you possibly can transfer to the subsequent step in your movement and outline the filter situation and additional processing logic. This makes it straightforward for builders to iterate and validate every processing step in addition to onboard new information sources that they’re not conversant in.

We needed to protect the speedy, interactive growth course of whereas conserving the associated fee for required infrastructure low—particularly throughout instances when builders are usually not engaged on their flows. To fulfill this want we’ve launched a brand new idea known as take a look at classes with the DataFlow Designer. 

When a developer creates a brand new dataflow, they’re instantly directed to the Designer and may begin constructing their movement with out having to attend for any assets to be created. They’ll drag and drop processors to the canvas instantly, create parameters and providers, and apply configuration adjustments. 

Determine 6: Builders can begin constructing dataflows instantly with out requiring any NiFi assets to be allotted—notice the grayed out processors indicating that no take a look at session is lively

As quickly as they need to run a processor and take a look at their movement logic, they will provoke a take a look at session, which provisions NiFi assets on the fly inside minutes. 

Determine 7: Check classes present an interactive expertise that NiFi builders love

As soon as a take a look at session is lively, builders can begin or cease particular person processors and providers and discover information within the movement to validate their movement design. When the take a look at session is not wanted, builders can terminate it, liberating up the assets and saving prices. Check classes act like on-demand NiFi sandboxes for builders.

Determine 8: As soon as a take a look at session has been began, builders can work together with processors and monitor information as it’s processed by their dataflow

A streamlined deployment course of from growth to manufacturing

Creating and testing dataflows is step one within the dataflow life cycle, and must combine properly with deploying and monitoring dataflows in manufacturing environments. With the designer turning into obtainable in CDF-PC, we will now help movement builders and movement directors alike by way of a streamlined course of. 

Determine 9: Builders can create new draft flows as wanted

Builders create draft flows, construct them out, and take a look at them with the designer earlier than they’re printed to the central DataFlow catalog. As soon as they’re within the DataFlow catalog, movement directors can deploy them of their cloud supplier of selection (AWS or Azure) and profit from the aforementioned options like auto-scaling, one-button NiFi model upgrades, centralized monitoring by way of KPIs, and automation by way of a robust CLI. 

Determine 10a: As soon as a draft movement has been validated utilizing a take a look at session, builders can publish them to the DataFlow catalog for manufacturing deployments

Determine 10b: As a part of the publication step, builders can go away feedback and are redirected to the catalog from the place they will provoke a deployment

Wanting forward and subsequent steps

The DataFlow Designer technical preview represents an vital step to ship on our imaginative and prescient of a cloud-native service that organizations can use for all their information distribution wants, and is accessible to any developer no matter their technical background. Cloudera DataFlow for the Public Cloud (CDF-PC) now covers your entire dataflow lifecycle from growing new flows with the Designer by way of testing and operating them in manufacturing utilizing DataFlow Deployments or DataFlow Capabilities relying on the use case.

Determine 11: Cloudera DataFlow for the Public Cloud (CDF-PC) permits Common Knowledge Distribution

The DataFlow Designer is now obtainable to CDP Public Cloud clients as a technical preview. Please attain out to your Cloudera account workforce or to Cloudera Help to request entry.

Keep tuned for extra info as we work in the direction of making the DataFlow Designer typically obtainable to CDP Public Cloud clients and join our upcoming DataFlow webinar or try the DataFlow Designer technical preview documentation.