Databricks Notebooks provides builders a managed authoring expertise the place information and AI groups can effectively collaborate on initiatives collectively. The group right here is working onerous as we put together to share thrilling new improvements for Notebooks on the Information + AI Summit later this month, and we hope you’ll be part of us at our session Develop Like a Professional in Databricks Notebooks, led by Weston Hutchins and Neha Sharma. As a heat up, we needed to take a fast look again at some latest additions in Notebooks.
Run Databricks Notebooks on SQL Warehouses
SQL is the 2nd hottest language in Notebooks behind Python, and to higher help our customers who love SQL, we’re bringing SQL warehouses to Notebooks. SQL warehouses are the identical sources that energy Databricks SQL, they usually ship higher price-performance for SQL execution in comparison with all-purpose clusters. This function is rolling out now, so maintain a watch out!
Whereas connected to a SQL warehouse, solely your pocket book’s SQL cells will execute. Cells utilizing different languages (like Python or Scala) shall be skipped. Markdown cells will proceed to be rendered. To be taught extra, go to our documentation.
View information from Notebooks, SQL editor, and Information Explorer in the identical expertise
The brand new unified schema browser helps you to view all the information within the Unity Catalog metastore with out leaving a pocket book or the SQL editor. You may choose “For you” to filter the listing to the energetic tables in your pocket book.
As you sort your search request into the filter field, the show actively updates to point out solely these objects that include that textual content. This may search for objects which are at present open or have been opened earlier within the present session. Be taught extra right here.
Share Notebooks utilizing Delta Sharing
Now you can use Delta Sharing to share pocket book recordsdata with the Databricks-to-Databricks sharing move. You get the convenience and safety of Delta Sharing. Sharing notebooks empowers you to collaborate throughout metastores and accounts. This permits individuals who share information to unpack the worth of that information with notebooks.
To be taught extra, right here is how one can add notebooks to a share (for suppliers) and how you can learn shared notebooks (for recipients).
Debug your Notebooks with the Variable Explorer
The Variable Explorer shows the state of all of the Python variables in your pocket book growth session. The identify, sort, and worth are surfaced for all easy variable sorts. The Variable Explorer additionally surfaces further metadata for Spark and Pandas DataFrames. The form and column names can be found at-a-glance, and a full view of the schema is obtainable on hover.
The Variable Explorer additionally means that you can step via and debug Python code by leveraging the help for pdb in Databricks Notebooks. You may set breakpoints with breakpoint() or pdb.set_trace(). If you run the cell, the execution will pause on the breakpoint and the Variable Explorer will mechanically replace with the state of the pocket book at that breakpoint. See our documentation for extra info.
See you at Summit
At Information + AI Summit 2023, we’ll have deep dive periods into utilizing Notebooks. We may even discuss concerning the latest methods to be extra environment friendly whereas utilizing Databricks. We hope to see you there.
