What's new in Cloudera Runtime 7.1.9 SP2 CHF 2

Ozone

Configurable replica counts for diverged quasi-closed containers

Ozone now features configurable replica counts for diverged quasi-closed containers, enhancing data resilience.

Previously, Ozone maintained a fixed two copies for each origin of a diverged quasi-closed container. With this update, you can customize replica counts to allocate more copies to the origin with the highest Block Commit Sequence ID (BCSID). Increasing the replication factor for the healthiest, most up-to-date origin significantly reduces the risk of data loss during node failures.

The following new dynamic configuration keys are now available to manage these replica counts:

Table 1. Replica count configuration keys

Configuration key	Description	Default value
hdds.scm.replication.quasi.closed.stuck.best.origin.copies	Defines the number of copies to maintain for the origin with the highest BCSID among healthy replicas.	3
hdds.scm.replication.quasi.closed.stuck.other.origin.copies	Defines the number of copies to maintain for all other origins of the same container.	2

Ozone CLI supports acknowledging and suppressing missing container reports

Ozone now allows administrators to acknowledge or suppress missing containers using new command-line interface (CLI) options in the Storage Container Manager (SCM). Previously, missing containers persisted in reports and metrics indefinitely, even if the underlying data was deleted or the containers were never created (such as when a pipeline closed before the first write).

This update includes the following capabilities in the Storage Container Manager:

• Container report suppression — The ozone admin container report command now includes mutually exclusive --suppress and --unsuppress flags. These flags accept multiple container IDs passed directly through the command line, stdin, or input files. SCM applies these updates during its next Replication Manager cycle, either removing suppressed containers from the report and the missing container count metric or restoring unsuppressed containers to them.
• Filtering suppressed containers — The ozone admin container list command now supports the --suppressed filtering option, allowing administrators to display and audit suppressed or unsuppressed containers.

Ozone Container Balancer expands support for non-standard and quasi-closed containers

The Ozone Container Balancer can now move containers that contain non-closed replicas (such as QUASI_CLOSED), if the container meets minimum closed replica requirements. Previously, the balancer only processed containers with all replicas strictly in a CLOSED state. This improvement optimizes data distribution and cluster balance when certain replicas have not fully transitioned to a closed state despite the container being finalized in Storage Container Manager (SCM).

This feature is disabled by default. To enable it, set hdds.container.balancer.include.non.standard.containers configuration to true.

When enabled, the Container Balancer can process the following container states:

• CLOSED and OVER_REPLICATED containers with CLOSED replicas — Includes containers that meet the minimum required number of closed replicas based on their replication configuration (such as Ratis or Erasure Coding schemes).
• CLOSED and OVER_REPLICATED containers with QUASI_CLOSED replicas — Includes containers with non-empty QUASI_CLOSED replicas, if the minimum required number of closed replicas are present.
• QUASI_CLOSED and OVER_REPLICATED containers — Includes containers with excess QUASI_CLOSED replicas and a non-empty source replica.
• QUASI_CLOSED and HEALTHY containers — Includes containers with the exact required count of QUASI_CLOSED replicas and a non-empty source replica.

Ozone Storage Container Manager (SCM) Replication Manager supports configurable container sample limits

Ozone SCM Replication Manager allows administrators to dynamically configure the number of containers sampled in each iteration. Previously, SCM hardcoded this limit to 100 containers per state per iteration. This improvement provides greater flexibility for monitoring container health and managing cluster state in large-scale deployments without requiring a service restart.

This enhancement introduces the following configuration property:

• Property name: hdds.scm.replication.container.sample.limit
• Default value: 100
• Description: Defines the maximum number of containers in each state to sample per Replication Manager iteration.
• Dynamic update: Supported. You can apply changes dynamically at runtime using the ozone admin reconfig command without restarting SCM.

Ozone volume health check toggle

Ozone introduces a configuration switch that enables or disables the RocksDB open test during volume health checks. This provides administrators with the ability to customize health check validations based on their specific environment requirements.

DataNode supports configurable disk health check timeout tolerance

You can now configure the number of tolerated consecutive timeout failures for DataNode disk health checks. This prevents volumes from being incorrectly marked as failed due to transient stalls or temporary system latency. This update includes the following configuration options and benefits:

• New configuration property — Introduces the hdds.datanode.disk.check.timeout.tolerated property to set the allowed number of consecutive timeout failures.
• Enhanced resilience — Reduces false-positive disk failure alerts by accommodating temporary latency before flagging a volume as failed.
• Reduced operational overhead — Minimizes unnecessary disk isolations and manual interventions caused by transient system delays.

Ozone client supports configurable retry limits for exponential backoff

Ozone clients now support a configurable maximum number of retry attempts for ExponentialBackoffPolicy. Previously, the client retry count defaulted to a maximum integer value, which caused clients to wait for extended periods during DataNode failures.

This enhancement offers the following key capabilities and benefits:

• Configurable retry limit — Introduces the hdds.ratis.client.exponential.backoff.max.retries configuration property to tune retry behavior based on cluster requirements.
• Accelerated failover — Enables clients to time out faster and attempt requests on alternative DataNodes sooner when a node fails.
• Improved responsiveness — Enhances overall system responsiveness and reduces application wait times during DataNode outages.

Ozone supports configurable RocksDB bottommost-level compaction strategy

Ozone now supports a configurable BottommostLevelCompaction strategy for RocksDB. Previously hardcoded to kForce, this setting is now adjustable using configuration properties. This improvement allows for better optimization of background compaction processes by selecting from kSkip, kForce, or kIfHaveCompactionFilter based on your cluster performance needs.

Ozone SCM safemode status command supports checking all SCM nodes

Ozone enhances Storage Container Manager (SCM) safemode monitoring by adding the --all flag to the ozone admin safemode status command. This update allows you to view the safemode status of all SCM nodes in a High Availability (HA) setup simultaneously, making it easier to verify SCM state during startup and maintenance windows.