Fixed Issues in Apache Iceberg

Review the list of Iceberg issues that are resolved in Cloudera Runtime 7.3.2, its service packs and cumulative hotfixes.

Cloudera Runtime 7.3.2.10000 SP1

CDPD-97613, CDPD-100780: Materialized view rebuild fails for Iceberg tables

7.3.2.10000

Previously, materialized view rebuild failed when the base table was an Iceberg table and the materialized view was implicitly created as an ACID table due to metastore.create.as.acid=true resulting in an error. During rebuild, no transaction started but an entry was created in the MATERIALIZATION_REBUILD_LOCKS table and not cleared, causing subsequent attempts to fail due to an existing lock entry.

This issues is now fixed by setting the metastore.create.as.acid property value to false.

This issue also occurs when the materialized view is an Iceberg table and is tracked in upstream documentation.

Apache Jira: HIVE-29436

CDPD-104751, CDPD-105219: OPTIMIZE TABLE fails on Iceberg V3 tables migrated from V2

7.3.2.10000

Previously, running OPTIMIZE TABLE on Iceberg V3 tables that were migrated from V2 failed with an IllegalStateException in ColumnLineageGraph during query planning. The failure occurred during column lineage processing for hidden columns on migrated V3 tables, which blocked the optimization cycle needed to clear legacy V2 position delete files on newly promoted V3 tables.

This issue is now fixed. Impala excludes OPTIMIZE TABLE from the column lineage graph. OPTIMIZE TABLE is a maintenance operation that rewrites an Iceberg table to a compact form without changing its content.

Apache Jira: IMPALA-15072

CDPD-95072, CDPD-99123: Iceberg metadata scan scheduled to a dedicated executor causes a crash

7.3.2.10000

Previously, on Impala clusters with dedicated coordinators and executors, certain queries on Iceberg metadata tables could schedule an Iceberg metadata scan fragment to a dedicated executor instead of the coordinator. This caused the executor to crash. The issue could occur when a query plan included a UNION over Iceberg metadata scans, even though an earlier fix required metadata scan fragments to run on the coordinator.

This issue is now fixed. Impala now schedules Iceberg metadata scan fragments, including those created by UNION plans, on the coordinator.

Apache Jira: IMPALA-14638

CDPD-102486, CDPD-103287: INSERT fails on Iceberg V3 tables with IndexOutOfBoundsException

7.3.2.10000

Previously, running INSERT on Iceberg V3 tables from Impala failed during query planning with an IndexOutOfBoundsException such as toIndex = 7. CREATE TABLE and SELECT on the same tables worked as expected. The failure occurred when lineage tracking for queries is enabled.

This issue is now fixed. Impala now plans and executes INSERT on Iceberg V3 tables correctly.

Apache Jira: IMPALA-14952

CDPD-102654, CDPD-105436: Iceberg DELETE hangs when the tuple cache is enabled

7.3.2.10000

Previously, when the Impala tuple cache was enabled and MT_DOP was greater than 2, DELETE statements on Iceberg tables could hang indefinitely and remain in a running state. The issue could occur when you ran multiple DELETE statements on the same table.

This issue is now fixed. Impala now completes Iceberg DELETE statements correctly when the tuple cache is enabled.

Apache Jira: IMPALA-14951

CDPD-105713, CDPD-105716: Iceberg V2 count(*) optimization is incorrectly applied to queries that have constant expressions only and no count(*), causing row loss

7.3.2.10000

Previously, on Iceberg V2 tables that contain delete files, queries that have constant expressions only and no count(*) in the select list (for example, SELECT 1 FROM tbl) returned fewer rows than they should.

This issue is now fixed. Impala no longer applies the Iceberg V2 count(*) optimization to queries that do not include count(*).

Apache Jira: IMPALA-14993

CDPD-102876, CDPD-103045: Time-travel queries can result in UnsupportedOperationException in IcebergScanPlanner

7.3.2.10000

Previously, time-travel queries on Iceberg tables from Impala could result in an UnsupportedOperationException during query planning in IcebergScanPlanner. The failure occurred when a time-travel query had to read files on data nodes that are not used by the data files in the current snapshot.

This issue is now fixed. Impala now successfully plans and executes scans for valid Iceberg snapshots in time-travel queries.

Apache Jira: IMPALA-14970

CDPD-99467: Puffin statistics reading is enabled by default

7.3.2.10000

Previously, the initial support for Puffin statistics disabled reading them by default.

This issue is now fixed. Reading Puffin statistics is enabled by default. Impala queries now take available Puffin statistics into consideration.

Apache Jira: IMPALA-14848

CDPD-72897: Inconsistent handling of NOT NULL fields in Iceberg tables

7.3.2.10000

Previously, since IMPALA-12729, inserting a NULL value into a non-nullable column in an Iceberg table could succeed and Impala wrote the NULL value in the Parquet data file. Query results could be inconsistent.

This issue is now fixed. Impala now does not allow writing NULL values into non-nullable columns.

Apache Jira: IMPALA-13273

CDPD-50404: Optimize MIN(part_col), MAX(part_col), and COUNT(DISTINCT part_col) queries for Iceberg tables

7.3.2.10000

Previously, the OPTIMIZE_PARTITION_KEY_SCANS optimization could optimize MIN(key_column), MAX(key_column), and COUNT(DISTINCT key_column) queries using HMS metadata. For Iceberg tables, partitioning statistics are not stored in HMS and are obtained through the Iceberg API instead, so these queries were not optimized in the same way.

This issue is now fixed. For Iceberg V1 and V2 tables without deletes, Impala now optimizes MIN(key_column), MAX(key_column), and COUNT(DISTINCT key_column) queries using partitioning statistics obtained through the Iceberg API when the partition transform is identity. The optimization partially applies to Iceberg V2 tables with deletes.

Apache Jira: IMPALA-11986

CDPD-101398, CDPD-85719: Query compilation failure in Hive Iceberg with ClassCastException

7.3.2.10000

Previously, queries on Hive Iceberg tables with MV rewrite feature failed during the compilation phase with a ClassCastException error.

This issue occurred due to a bug in the code and was triggered when the main query contained a condition that always resulted to false and MV rewrite was applied.

This issue is now fixed.

Cloudera Lakehouse Optimizer

CDPD-93120, CDPD-91628: In-progress task metadata is cleaned up periodically: 7.3.2.10000; Previously, running the task cleanup executor for Cloudera Lakehouse Optimizer cleaned up the tasks that were in progress.
This issue is now fixed. The executor cleans up only the completed and failed tasks.
CDPD-93759: Improper output encoding in API responses: 7.3.2.10000; Previously, certain Cloudera Lakehouse Optimizer API endpoints returned content without properly encoding special characters such as <, >, &, and ".
This issue is now fixed.
CDPD-93760: Missing security response headers in API responses: 7.3.2.10000; Previously, standard security response headers were not included in Cloudera Lakehouse Optimizer API endpoint responses. The missing headers included Content-Security-Policy, X-Content-Type-Options, and Strict-Transport-Security.
This issue is now fixed.
CDPD-99255: Cloudera Lakehouse Optimizer API returns misleading Missing Bearer Token error (401): 7.3.2.10000; Previously, when you accessed the Cloudera Lakehouse Optimizer REST API through the Knox gateway with a valid bearer token but was not a member of the required authorization groups, the API returned a 401 Unauthorized response with a misleading error message indicating that the bearer token was missing.
This issue is now fixed. The API now returns an appropriate authorization error message.
CDPD-98341: Cloudera Lakehouse Optimizer does not handle deleted Iceberg tables correctly: 7.3.2.10000; Previously, when an Iceberg table was dropped, Cloudera Lakehouse Optimizer continued to hold stale policy-table associations in the registry, expose deleted table names through read-only APIs, and dispatch policy evaluation tasks against non-existent tables. This caused misleading metrics, phantom associations in the UI and errors from in-flight tasks targeting dropped tables.
This issue is now fixed.

Cloudera Runtime 7.3.2.100 CHF1

CDPD-84534: New configuration to disable block location retrieval

7.3.2.100

Impala is designed to optimize reads by retrieving file block locations from storage systems like HDFS and Ozone, and scheduling read operations on the same nodes where the data resides. In on-premise environments, Impala nodes are typically colocated with storage nodes. However, in scenarios where they are not colocated, retrieving the block location information is a resource-intensive operation.

This issue is now fixed by introducing new Hadoop configuration properties that allow you to disable the retrieval of block location information during table loading. Disabling the retrieval of block location information can be configured in the following ways:

Globally, using the Hadoop Safety Valve for core-site.xml advanced configuration snippet.
```
'impala.preload-block-locations-for-scheduling': 'false'
```
By using a Filesystem scheme to disable it for a specific type of storage.
```
'impala.preload-block-locations-for-scheduling.scheme.hdfs': 'false'
```
By authority, to disable block location retrieval for a specific endpoint when multiple storage systems are configured with the same scheme.
```
'impala.preload-block-locations-for-scheduling.authority.mycluster': 'false'
```
This disables block location retrieval only for URIs as hdfs://mycluster/warehouse/tablespace/....

Apache Jira: IMPALA-14138

CDPD-99853, CDPD-99080: Incorrect results when querying multiple Iceberg snapshots

7.3.2.100

Previously, queries on Iceberg tables that referenced multiple snapshots (branch, tag, or current) returned incorrect results because the query planner read data from the same snapshot across all the table references.

This issue is now fixed. The query planner now correctly resolves snapshots for each table reference, ensuring accurate query results.

Apache Jira: HIVE-29509

Cloudera Lakehouse Optimizer

CDPD-102237: Registry is not marked dirty after uploading a policy definition: 7.3.2.10000; Previously, after you uploaded a policy definition, the registry was not marked dirty. As a result, the scheduler did not pick up new policy definitions until you reschedule explicitly.
This issue is now fixed.
CDPD-103101: Task event log files accumulate in HDFS after log ingestion: 7.3.2.100; Previously, the task event log files in HDFS required manual monitoring and cleanup after the log files were ingested.
This issue is now fixed. The task event log files from HDFS are cleaned up automatically after the log files are successfully ingested into the Iceberg table.
CDPD-103026: Cloudera Lakehouse Optimizer task ingestion action fails due to strict timestamp parsing in Spark3: 7.3.2.100; Previously, the task ingestion action failed during Spark validation step because of the strict timestamp parsing in the Spark 3 service.
This issue is now fixed.
CDPD-101842, CDPD-101706: Cloudera Lakehouse Optimizer event policies are triggered even after the policies are deleted: 7.3.2.100; Previously, when you tried to associate the event-based policy with an individual table, the association failed abruptly if the policy was already associated with the entire namespace. After you deleted the policy, the event-based scheduler retained the triggers and continued to schedule the table maintenance, which was incorrect.
This issue is now fixed.
CDPD-101534, CDPD-101213: List policy API returns unauthorized status for wildcard namespace: 7.3.2.100; Previously, the GET /namespaces/{namespace}/policies API returned the 401 Unauthorized error HTTP status code when you entered the asterisk (*) wildcard for the {namespace} parameter.
This issue is now fixed.
CDPD-101284, CDPD-100562: The Edit Association page displays all namespaces as associated to the Cloudera Lakehouse Optimizer policy: 7.3.2.100; Previously, when you edited an association for a Cloudera Lakehouse Optimizer policy, the UI incorrectly displayed all the namespaces as associated with the policy or incorrectly displayed all the namespaces as selected.
This issue is now fixed.
CDPD-101183, CDPD-100274, CDPD-101104: API-related performance issues impact Cloudera Lakehouse Optimizer UI response time: Previously, API-related performance issues impacted the View policy page load and the "Create Policy" process response time in the Lakehouse Optimizer UI.
This issue is now fixed.
CDPD-84718: Cloudera Lakehouse Optimizer uses the latest com.google.guava:guava 28.1-jre library dependency.: 7.3.2.100; Cloudera Lakehouse Optimizer uses the latest com.google.guava:guava 28.1-jre library dependency.
This issue is now fixed.

Cloudera Runtime 7.3.2

Cloudera Runtime 7.3.2 resolves Iceberg issues and incorporates fixes from the service packs and cumulative hotfixes from 7.3.1.100 through 7.3.1.706. For a comprehensive record of all fixes in Cloudera Runtime 7.3.1.x, see Fixed Issues.

CDPD-78686: Iceberg tables created in 7.2.17 are not captured in Atlas when using 7.2.18 or 7.3.1 Atlas server

7.3.2

This issue occurred due to incompatibility between Data Services, Hive, and Impala hooks in 7.2.17 and the Atlas server in 7.2.18 and 7.3.1.

This fix resolves the compatibility issue and Iceberg tables created in 7.2.17 are now correctly captured and displayed in the Atlas UI in later versions.

CDPD-97171: Concurrency issues between compaction and concurrent write operations

7.3.2

This fix resolves an issue where compaction conflicted with concurrent write operations causing data corruption, by improving concurrency handling to ensure stable operations and data consistency.

Apache Jira: HIVE-29437

CDPD-74040: Dropping Iceberg table with complex type containing timestamp fails

7.3.2

This fix resolves an issue where dropping an Iceberg table with complex types such as array, map, or struct containing timestamp fields failed due to unsupported Hive timestamp type handling, by ensuring proper type conversion and successful table deletion.

CDPD-89402: Event processing invalidates Iceberg tables due to reload failures

7.3.2

This fix resolves an issue where CatalogServiceCatalog.reloadTableIfExists() resulted in a ClassCastException during event processing, invalidating Iceberg tables and triggering full table reloads instead of incremental loading.

Apache Jira: IMPALA-14358

CDPD-72383: COUNT(*) optimization returns incorrect results in UNION queries on Iceberg V2 tables

7.3.2

A faulty COUNT(*) query optimization caused incorrect results in UNION queries on Iceberg V2 tables that have delete files, leading to inconsistent query outputs.

The fix corrects the optimization logic to ensure accurate results across UNION queries, including scenarios involving post data changes.

Apache Jira: IMPALA-13756 IMPALA-13249

CDPD-81076: LEFT ANTI JOIN fails on Iceberg V2 tables with Delete files

7.3.2

Queries using a LEFT ANTI JOIN fail with an AnalysisException if the right-side table is an Iceberg V2 table containing delete files. For example, consider the following query:

SELECT * FROM table_a a
LEFT ANTI JOIN iceberg_v2_table b
ON a.id = b.id;

The error Illegal column/field reference'b.input_file_name' of semi-/anti-joined table 'b' is displayed because semi-joined tuples need to be explicitly made visible for paths pointing inside them to be resolvable.

The fix updates the IcebergScanPlanner to ensure that the tuple containing the virtual fields is made visible when it is semi-joined.

Apache Jira: IMPALA-13888

CDPD-78427: Enable MERGE statement for Iceberg tables with equality deletes

7.3.2

This patch fixes an issue that caused MERGE statements to fail on Iceberg tables that use equality deletes.

The failure occurred because the delete expression calculation was missing the data sequence number, even though the underlying data description included it. This mismatch caused row evaluation to fail.

The fix ensures the data sequence number is correctly included in the result expressions, allowing MERGE operations to complete successfully on these tables.

Apache Jira: IMPALA-13674

CDPD-77773: Tolerate missing data files during Iceberg table loading

7.3.2

This fix addresses an issue where an Iceberg table would fail to load completely if any of its data files were missing from the file system. This TableLoadingException left the table in an incomplete state, blocking all operations on it.

Impala now tolerates missing data files during the table loading process. An exception will only be executed if a query subsequently attempts to read one of the specific files that is missing.

This change allows other operations that do not depend on the missing data—such as ROLLBACK, DROP PARTITION, or SELECT statements on valid partitions—to execute successfully.

Apache Jira: IMPALA-13654

CDPD-78508: Skip reloading Iceberg tables when metadata JSON file is the same

7.3.2

This patch optimizes metadata handling for Iceberg tables, particularly those that are updated frequently.

Previously, if an event processor was lagging, Impala might receive numerous update events for the same table (for example, 100 events). Impala would attempt to reload the table 100 times, even if the table's state was already up-to-date after processing the first event.

With this fix, Impala now compares the path of the incoming metadata JSON file with the one that is currently loaded. If the metadata file location is the same, Impala skips the reload, correctly assuming the table is already unchanged. This significantly reduces unnecessary metadata processing.

Apache Jira: IMPALA-13718

CDPD-82415: TABLESAMPLE clause of the COMPUTE STATS statement has no effect on Iceberg tables

7.3.2

This fix resolves a regression introduced by IMPALA-13737. For example, the following query scans the entire Iceberg table to calculate statistics, whereas it should ideally use only about 10% of the data.

COMPUTE STATS t TABLESAMPLE SYSTEM system(10);

This fix introduces proper table sampling logic for Iceberg tables, which can be utilized for COMPUTE STATS. The sampling algorithm previously located in IcebergScanNode.getFilesSample() is now relocated to FeIcebergTable.Utils.getFilesSample().

Apache Jira: IMPALA-14014

CDPD-85228: IllegalStateException with Iceberg table with DELETE

7.3.2

Running a query on an Iceberg table fails with an IllegalStateException error in the following scenario:

The Iceberg table has delete files for every data file (no data files without delete files) AND
An anti-join operation is performed on the result of the Iceberg delete operation (IcebergDeleteNode or HashJoinNode)

This fix resolves the issue by setting the TableRefIds of the node corresponding to the Iceberg delete operation (IcebergDeleteNode or HashJoinNode) to only the table reference associated with the data files, excluding the delete files.

Apache Jira: IMPALA-14154

CDPD-87405: Error unnesting arrays in Iceberg tables with DELETE files

7.3.2

The following error occurred when unnesting a nested array (a 2D array) from an Iceberg table. This issue was triggered specifically when the table contained delete files for some, but not all, of its data files.

Filtering an unnested collection that comes from a UNION [ALL] is not supported yet.

Reading an Iceberg table with this mixed data and delete file configuration creates a UNION ALL node in the query execution plan. The system had a check that explicitly blocked any filtering on an unnested array.

This fix relaxes the validation check, allowing the operation to proceed if all UNION operands share the same tuple IDs. This ensures the query can successfully unnest the array.

Apache Jira: IMPALA-14185

Cloudera Lakehouse Optimizer

CDPD-91628, CDPD-98014: In-progress task metadata is cleaned up periodically: Previously, when the task cleanup executor daemon for Cloudera Lakehouse Optimizer ran, it cleaned up the tasks that were in progress .
This issue is now fixed. The daemon cleans up only the completed and failed tasks that are not in progress.
CDPD-88828: Duplicate tasks run on table: Previously, Cloudera Lakehouse Optimizer allowed duplicate tasks to run on a table.
This issue is now fixed.
CDPD-94583: Filter expression does not transform as expected and fails during the validation step: Previously, when an Iceberg table was created using a DDL command with a bucket transformation specification, and you created a Cloudera Lakehouse Optimizer policy on the table to perform the compaction or rewrite data files tasks, the tasks failed during the validation step.
This issue is now fixed.
CDPD-97815: Associations are lost after Cloudera Lakehouse Optimizer restarts: 7.3.2.10000; Previously, the following issues occurred when Cloudera Lakehouse Optimizer restarted:

The associations defined using the Cloudera Lakehouse Optimizer UI were lost because they were not persisted.

The namespace-level associations for the tables were not initialized in the registry.

These issues are now fixed.