Introduction
Apache XTable, previously known as OneTable, is a groundbreaking solution that is currently incubating under the Apache Software Foundation. It’s not a new or separate format, but a tool that provides abstractions for the translation of lakehouse table format metadata. This means it reads the existing metadata of your table and writes out metadata for one or more other table formats, allowing your existing data to be read as though it was written using other popular formats like Delta, Hudi, or Iceberg.
How Does Apache XTable Work?
Apache XTable works by reading the existing metadata of your table and writing out metadata for one or more other table formats. This is achieved by leveraging the existing APIs provided by each table format project. The metadata is then persisted under a directory in the base path of your table (_delta_log for Delta, metadata for Iceberg, and .hoodie for Hudi). This allows your existing data to be read as though it was written using Delta, Hudi, or Iceberg. For example, a Spark reader can use spark.read.format (“delta | hudi | iceberg”).load (“path/to/data”).
Apache XTable vs Delta Lake Uniform
While both Apache XTable and Delta Lake Uniform aim to provide interoperability between different table formats, there are key differences between the two. Apache XTable provides abstraction interfaces that allow omni-directional interoperability across Delta, Hudi, Iceberg, and any other future lakehouse table formats such as Apache Paimon. It is a standalone project that provides a neutral space for all the lakehouse table formats to constructively collaborate together. On the other hand, Delta Lake Uniform is a one-directional conversion from Delta Lake to Apache Hudi or Apache Iceberg and is governed inside the Delta Lake repo.
When to Consider Apache XTable
Apache XTable can be used to easily switch between any of the table formats or even benefit from more than one simultaneously. Some organizations use Apache XTable today because they have a diverse ecosystem of tools with polarized vendor support of table formats. Some users want lightning fast ingestion or indexing from Hudi and photon query accelerations of Delta Lake inside of Databricks. Some users want managed table services from Hudi, but also want write operations from Trino to Iceberg. Regardless of which combination of formats you need, Apache XTable ensures you can benefit from all 3 projects.
Current Limitations
While Apache XTable is a powerful tool, it does have some limitations. Currently, Hudi and Iceberg MoR tables are not supported. However, as the project continues to develop, these limitations may be addressed in future updates.
Conclusion
Apache XTable is a promising solution for those looking to improve their data management practices in a lakehouse environment. By providing a means for seamless data exchange between different table formats, it enhances the flexibility and efficiency of data management. As the project continues to develop, it will be interesting to see the new capabilities and improvements that will be introduced. Stay tuned for more updates on this exciting project!