Iceberg is an rising open-table format designed for big analytic workloads. The Apache Iceberg undertaking continues creating an implementation of Iceberg specification within the type of Java Library. A number of compute engines comparable to Impala, Hive, Spark, and Trino have supported querying knowledge in Iceberg desk format by adopting this Java Library offered by the Apache Iceberg undertaking.
Totally different question engines comparable to Impala, Hive, and Spark can instantly profit from utilizing Apache Iceberg Java Library. A variety of Iceberg desk evaluation comparable to itemizing desk’s knowledge file, deciding on desk snapshot, partition filtering, and predicate filtering could be delegated via Iceberg Java API as an alternative, obviating the necessity for every question engine to implement it themself. Nonetheless, Iceberg Java API calls should not at all times low cost.
On this weblog, we’ll focus on efficiency enchancment that Cloudera has contributed to the Apache Iceberg undertaking with reference to Iceberg metadata reads, and we’ll showcase the efficiency profit utilizing Apache Impala because the question engine. Nonetheless, different question engines comparable to Hive and Spark may profit from this Iceberg enchancment as nicely.
Repeated metadata reads downside in Impala + Iceberg
Apache Impala is an open supply, distributed, massively parallel SQL question engine. There are two parts of Apache Impala: again finish executor and entrance finish planner. The Impala again finish executor is written in C++ to supply quick question execution. Then again, Impala entrance finish planner is written in Java and is accountable for analyzing SQL queries from customers and planning the question execution. Throughout question planning, Impala entrance finish will analyze desk metadata comparable to partition data, knowledge information, and statistics to provide you with an optimized execution plan. For the reason that Impala entrance finish is written in Java, Impala can straight analyze many elements of the Iceberg desk metadata via the Java Library offered by Apache Iceberg undertaking.
Determine 1. store_sales desk scanned 15 occasions with various filter predicates in TPC-DS Q9
In Hive desk format, the desk metadata comparable to partition data and statistics are saved in Hive Metastore (HMS). Impala can entry Hive desk metadata quick as a result of HMS is backed by RDBMS, comparable to mysql or postgresql. Impala additionally caches desk metadata in CatalogD and Coordinator’s native catalog, making desk metadata evaluation even sooner if the focused desk metadata and information have been beforehand accessed. This caching is essential for Impala since it might analyze the identical desk a number of occasions throughout concurrent question planning and likewise inside single question planning. Determine 1 reveals a TPC-DS Q9 plan the place one frequent desk, store_sales, is analyzed 15 occasions to plan 15 separate desk scan fragments.
Determine 2. Iceberg desk metadata reads by Iceberg Java API
In distinction, Iceberg desk format shops its metadata as a set of information within the file system, subsequent to the information information. Determine 2 illustrates the three sorts of metadata information: snapshot information, manifest listing, and manifest information. The information information and metadata information in Iceberg format are immutable. New DDL/DML operation over the Iceberg desk will create a brand new set of information as an alternative of rewriting or changing prior information. Each desk metadata question via Iceberg Java API requires studying a subset of those metadata information. Subsequently, every of them additionally incurs a further storage latency and community latency overhead, even when a few of them are analyzing the identical desk. This downside is described in IMPALA-11171.
Determine 3. A number of socket learn actions throughout TPC-DS Q9 planning over Iceberg desk on S3.
Iceberg manifest file cache design
Impala entrance finish have to be cautious in utilizing Iceberg Java API whereas nonetheless sustaining quick question planning efficiency. Lowering the a number of distant reads of Iceberg metadata information requires implementing related caching methods as Impala does for Hive desk format, both in Impala entrance finish facet or embedded in Iceberg Java Library. Finally, we decide to do the latter so we will contribute one thing that’s helpful for the entire neighborhood and all the compute engines can profit from it. It’s also essential to implement such a caching mechanism as much less intrusive as attainable.
Determine 4. Iceberg manifest caching design.
We filed pull request 4518 within the Apache Iceberg undertaking to implement this caching mechanism. The thought is to cache the binary content material of manifest information (the bottom of Iceberg metadata information hierarchy) in reminiscence, and let Iceberg Java Library learn from the reminiscence if it exists as an alternative of studying once more from the file system. That is made attainable by the truth that Iceberg metadata information, together with the manifest information, are immutable. If an Iceberg desk evolves, new manifest information can be written as an alternative of modifying the previous manifest information. Thus, Iceberg Java Library will nonetheless learn any new further manifest information that it wants from the file system, populating manifest cache with new content material and expiring the previous ones within the course of.
Determine 4 illustrates the two-tiered design of the Iceberg manifest cache. The primary tier is the FileIO degree cache, mapping a FileIO into its personal ContentCache. FileIO itself is the first interface between the core Iceberg library and underlying storage. Any file learn and write operation by the Iceberg library will undergo the FileIO interface. By default, this primary tier cache has a most of eight FileIO that may concurrently have ContentCache entries in reminiscence. This quantity could be elevated via Java system properties iceberg.io.manifest.cache.fileio-max.
The second tier cache is the ContentCache object, which is a mapping of file location path to the binary file content material of that file path. The binary file content material is saved in reminiscence as a ByteBuffer of 4MB chunks. Each of the tiers are carried out utilizing the Caffeine library, a high-performance caching library that’s already in use by Iceberg Java Library, with a mixture of weak keys and tender values. This mix permits automated cache entries elimination on the occasion of JVM reminiscence strain and rubbish assortment of the FileIO. This cache is carried out within the core Iceberg Java Library (ManifestFiles.java), making it obtainable for instant use by totally different FileIO implementations with out altering their code.
Impala Coordinator and CatalogD can do a quick file itemizing and knowledge file pruning over an Iceberg desk utilizing this Iceberg manifest cache. Word that Iceberg manifest caching doesn’t eradicate the position of CatalogD and Coordinator’s native catalog. Some desk metadata comparable to desk schema, file descriptors, and block places (for HDFS backed tables) are nonetheless cached inside CatalogD. Collectively, CatalogD cache and Iceberg manifest cache assist obtain quick question planning in Impala.
Desk 1. CatalogD and Iceberg Manifest Cache position throughout desk codecs.
Particular person per-FileIO ContentCache could be tuned via their respective Iceberg Catalog properties. The outline and default values for every properties are following:
Efficiency enchancment in Impala
Determine 5. Inverse CDF of Impala question compilation time over TPC-DS queries 3TB scale
Impala and different question engines can leverage the manifest caching function ranging from Apache Iceberg model 1.1.0. Determine 5 reveals the development in compilation time by Impala entrance finish. The x-axis represents the proportion of queries in TPC-DS workload, and the y-axis represents the question compilation time in milliseconds. In comparison with Iceberg with out manifest caching (Vanilla Iceberg), enabling Iceberg manifest caching can enhance question compilation 12 occasions sooner (Iceberg + caffeine), virtually matching the efficiency over the Hive exterior desk. One caveat is that, in Impala, the io.manifest.cache.expiration-interval-ms config is elevated greater to at least one hour in Coordinator. That is helpful for Impala as a result of for the default Hive catalog, Impala entrance finish maintains a singleton of Iceberg’s HiveCatalog that’s lengthy lived. Setting an extended expiration is ok since Iceberg information are immutable, as defined within the design part above. A protracted expiration time can even permit cache entries to stay longer and be utilized for a number of question planning concentrating on the identical tables.
Impala presently reads its default Iceberg catalog properties from core-site.xml. To allow Iceberg manifest caching with a one hour expiration interval, set the next configuration in Coordinator and CatalogD service core-site.xml:
Abstract
Apache Iceberg is an rising open-table format designed for big analytic workloads. The Apache Iceberg undertaking continues creating the Iceberg Java Library, which has been adopted by many question engines comparable to Impala, Hive, and Spark. Cloudera has contributed the Iceberg manifest caching function to Apache Iceberg to cut back repeated manifest file reads issues. We showcase the good thing about Iceberg manifest caching by utilizing Apache Impala because the question engine, and present that Impala is ready to achieve as much as 12 occasions speedup in question compilation time on Iceberg tables with Iceberg manifest caching enabled.
To be taught extra:
- For extra on Iceberg manifest caching configuration in In Cloudera Knowledge Warehouse (CDW), please consult with https://docs.cloudera.com/cdw-runtime/cloud/iceberg-how-to/subjects/iceberg-manifest-caching.html.
- Watch our webinar Supercharge Your Analytics with Open Knowledge Lakehouse Powered by Apache Iceberg. It features a stay demo recording of Iceberg capabilities.
Strive Cloudera Knowledge Warehouse (CDW), Cloudera Knowledge Engineering (CDE), and Cloudera Machine Studying (CML) by signing up for a 60 day trial, or check drive CDP. If you have an interest in chatting about Apache Iceberg in CDP, let your account group know.
