- CDPD-67848: Rocks tools native libraries are
incorrectly packaged inside the Java JAR file
- 7.3.2.10000
- Previously, the Rocks tools native libraries
were packaged inside the Java JAR file. Because native libraries were
platform-specific while JAR files were platform-agnostic, this caused
potential incompatibilities when the same JAR was shipped across different
operating systems. This issue is now resolved and moving the shared library
files outside of the JAR ensures that native libraries load from the correct
platform-specific paths.
- CDPD-74200: Ozone Recon fails to synchronize data with
Ozone Manager (OM) database
- 7.3.2.10000
- Previously, Ozone Recon failed to synchronize
its database with the Ozone Manager (OM) database. This resulted in the
Recon UI displaying incorrect or missing information for volumes, buckets,
and keys. This issue is now resolved and Recon UI displays accurate
namespace data.
- CDPD-77957: Ozone Manager follower bootstrap failures
with snapshots
- 7.3.2.10000
- Previously, a race condition during the Ozone
Manager (OM) bootstrap process could corrupt the database on follower nodes
when snapshots were in use. This could cause the OM bootstrap to fail and
impact cluster stability. This issue is now fixed and introduces a lock to
prevent this race condition, ensuring that the OM bootstrapping is reliable
and that the database remains consistent.
- Apache JIRA:
HDDS-12090
- CDPD-78023: OzoneKey isFile returns true for
directories
- 7.3.2.10000
- Previously, the
OzoneKey.isFile() method incorrectly returned true when
called on a directory. This issue is now resolved.
- Apache JIRA:
HDDS-12094
- CDPD-82738: Ozone clients incorrectly attempt to use
gRPC ports for streaming when streaming is disabled
- 7.3.2.10000
- Previously, when Ozone streaming was disabled,
pipelines did not contain DataNode streaming port information. If a client
attempted to use streaming for these pipelines, it incorrectly defaulted to
using the gRPC port, resulting in a connection failure. This issue is now
resolved.
- Apache JIRA:
HDDS-12992
- CDPD-82805: Container replication fails for
over-allocated containers due to inaccurate space reservation
- 7.3.2.10000
- Previously, when exporting or importing
containers during DataNode replication, Ozone used a conservative estimate
(2x the maximum container size) to reserve disk space. In environments with
over-allocated containers, the actual container size exceeded this estimate,
leading to inaccurate space accounting and failed replications. This issue
is now resolved and ensures 2 * actual container size space is reserved to
prevent replication failures for over-allocated containers.
- Apache JIRA:
HDDS-12998
- CDPD-87218: Storage space availability checks for
container allocation were performed at a suboptimal stage in the pipeline
process
- 7.3.2.10000
- Previously, validation checks for sufficient
storage space in a pipeline occurred too early in the allocation workflow,
which led to inconsistencies during container creation. This issue is now
resolved by moving this logic to the
allocateContainer
function to ensure that space availability is verified at the most
appropriate stage of the allocation process, improving the reliability of
disk space handling in Ozone DataNodes.
- Apache JIRA:
HDDS-13338
- CDPD-88421: Ozone Recon UI displays blank data during
full OM database snapshot bootstrap
- 7.3.2.10000
- Previously, when Ozone Recon performed a full
snapshot fetch of the OM database, it would truncate or clean existing task
tables before reprocessing. This caused the Recon UI to appear blank until
the synchronization was complete, resulting in a temporary loss of data
visibility for users. This issue is now resolved by introducing a staging
database mechanism that stores processed data in the background and only
updates the final database once reprocessing is finished, ensuring
continuous data availability in the UI.
- Apache JIRA:
HDDS-13515
- CDPD-88636: DataNode decommissioning fails when other
DataNodes are offline during Erasure Coded (EC) replication
- 7.3.2.10000
- Previously, when attempting to decommission a
DataNode while other DataNodes holding Erasure Coded (EC) blocks were
offline, the process failed. This occurred because the Storage Container
Manager (SCM) attempted to use an affinity node that had been removed from
the network topology, resulting in a placement error that prevented
decommissioning from completing. This issue is now resolved.
- Apache JIRA:
HDDS-13544
- CDPD-89023: Ozone Tiering profile parameters
incorrectly accept negative values
- 7.3.2.10000
- Previously, the Ozone Tiering API accepted
negative values (such as -1) for the lessThanEqual and
keySizeAboveBytes parameters without returning an
error. This caused confusion because these values were treated as "not set"
rather than being validated. This issue is now resolved, and the API
correctly validates these fields, ensuring that only values greater than or
equal to 0 are accepted.
- CDPD-89072: The ozone admin container
info command did not validate container IDs before
execution
- 7.3.2.10000
- Previously, the ozone admin container
info command did not validate the syntax of container IDs
passed through the command line before attempting to process them. This led
to unnecessary processing or delayed error reporting for invalid IDs. This
issue is now resolved, and the command now validates all container IDs
upfront and fails immediately if any ID with invalid syntax is provided,
ensuring faster feedback and more robust execution.
- Apache JIRA:
HDDS-13592
- CDPD-90077: Ozone Tiering workflows fail immediately
when the Ozone Manager (OM) Leader node is unavailable
- 7.3.2.10000
- Previously, Ozone Tiering workflows would fail
immediately without any retry mechanism if they encountered transient
external dependency issues, such as the Ozone Manager (OM) Leader node being
unavailable during an election or maintenance. This required manual
intervention to resume the workflow and could cause up to a 24-hour delay in
data processing. This issue is now resolved, and a unified retry and timeout
framework has now been implemented, allowing tasks to automatically retry
with exponential backoff when transient errors occur, improving the overall
resilience of the tiering service.
- CDPD-91077: Ozone Recon fails to synchronize with
Ozone Manager due to duplicate metrics source error
- 7.3.2.10000
- Previously, Ozone Recon encountered a
MetricsException stating that the metrics source
userTableCache already existed during the
synchronization process with the Ozone Manager (OM). This occurred when
Recon attempted to initialize a new RocksDB store for a full snapshot,
leading to a failure in closing the database and subsequent Rocks
Database is closed errors during delta updates. This issue is
now resolved by ensuring that metrics sources are correctly managed during
the re-initialization of metadata tables.
- Apache JIRA:
HDDS-13710
- CDPD-91507: Mis-replication check fails when a
maintenance node becomes unreachable
- 7.3.2.10000
- Previously, when a node in maintenance mode
became unreachable and was removed from the network topology, Ozone
container placement validation failed. This occurred because the system
attempted to check the node's containers for mis-replication using the
topology from which the node had already been removed. This issue is now
resolved by ensuring the node's network location is correctly referenced
even if it is no longer present in the active topology.
- Apache JIRA:
HDDS-13762
- CDPD-91511: Duplicate purge requests in Ozone could
lead to incorrect bucket quota usage calculations
- 7.3.2.10000
- Previously, the
KeyDeletingService and
DirectoryDeletingService services in Ozone did not
track inflight purge requests in memory. This resulted in duplicate purge
requests being submitted, which led to inaccurate bucket quota usage
reporting. This issue is now resolved by tracking the last purge transaction
information in memory to prevent redundant requests and ensure accurate
storage accounting.
- Apache JIRA:
HDDS-13760
- CDPD-91529: Ozone services fail to export traces to
Jaeger due to invalid endpoint format
- 7.3.2.10000
- Previously, Ozone services (such as OM, SCM,
DN, and S3G) failed to export traces to Jaeger because the
OTEL_EXPORTER_OTLP_ENDPOINT variable was not configured
as a full URL. This resulted in an IllegalArgumentException
stating that the endpoint must start with http:// or https://. This issue is
now resolved.
- Apache JIRA:
HDDS-13741
- CDPD-91531: Excessive logging of full message payloads
during WriteChunk or PutBlock failures
- 7.3.2.10000
- Previously, when a client failed to send a
WriteChunk or PutBlock request, the
system logged the entire message payload, including verbose checksum data.
This resulted in excessively large and chatty log files. This issue is now
resolved.
- Apache JIRA:
HDDS-13739
- CDPD-91594: Snapshot bucket quota usage is not
correctly reduced during background purge operations
- 7.3.2.10000
- Previously, when the background
KeyDeletingService and
DirectoryDeletingService services purged keys or
directories, the SnapshotUsedBytes and
snapshotUsedNamespace values were not decreased. This
resulted in inaccurate bucket quota tracking and reporting when snapshots
were in use. This issue is now resolved, and quota usage is correctly
updated during purge requests.
- Apache JIRA:
HDDS-13764
- CDPD-92382: Ozone Manager DB Insights Open Keys table
displays incorrect replication information for Erasure Coded (EC) keys
- 7.3.2.10000
- Previously, the Open Keys table in Ozone
Manager DB Insights displayed confusing or partial information for Erasure
Coded (EC) keys because it used separate columns for replication type and
factor. This issue is now resolved and a combined replication column
correctly formats and displays EC replication details.
- Apache JIRA:
HDDS-13835
- CDPD-92409: Namespace summary API returns incorrect
directory and key counts
- 7.3.2.10000
- Previously, the Namespace summary API
(
/api/v1/namespace/summary) returned incorrect file
sizes and counts for directories. This occurred because metadata propagation
logic skipped the immediate parent directory when a new directory containing
files was added to the tree, updating only higher-level ancestors. This
issue is now resolved.
- Apache JIRA:
HDDS-13841
- CDPD-92656: Ozone DataNode startup failure on
co-located hosts due to storage directory locking
- 7.3.2.10000
- Previously, when Ozone SCM or OM HA and
DataNodes were co-located on the same host, the DataNode failed to start
with an
OverlappingFileLockException. This occurred because
the DataNode fell back to using the ozone.metadata.dirs
path for Ratis storage when
dfs.container.ratis.datanode.storage.dir was not explicitly
configured, leading to resource contention and directory locking conflicts
with other processes. This issue is now resolved by ensuring proper storage
directory handling to prevent locking conflicts on colocated hosts.
- Apache JIRA:
HDDS-13866
- CDPD-92660: Recon UI fails to load Disk Usage view
with large number of nested directories and keys
- 7.3.2.10000
- Previously, when there were a large number of
entries, the Recon UI Disk Usage (DU) page took time to load or populate the
tree. This issue is resolved now and the Disk Usage page no longer loads the
root path by default. The user also gets a warning in case they try to
submit root path DU. The root path DU can be fetched only after explicit
submission by the user.
- CDPD-92869: Deadlock in Ozone background services
causes intermittent timeouts during snapshot and key deletion
- 7.3.2.10000
- Previously, a deadlock occurred in Ozone
background services because the bootstrap lock was acquired after a snapshot
was already opened. The bootstrap flow held the bootstrap lock while waiting
for a snapshot cache lock, while the background service held the snapshot
open and waited for the bootstrap lock, resulting in intermittent
TimeoutException errors during background tasks such as
snapshot and key deletion. This issue is now resolved.
- Apache JIRA:
HDDS-13889
- CDPD-93006: Snapshot read cache lock is not released
during runtime exceptions
- 7.3.2.10000
- Previously, if a runtime exception occurred
during Ozone Manager (OM) snapshot processing, the snapshot read cache lock
could remain held instead of being released. This unreleased lock could
stall downstream operations, leading to deadlocks or failures during the OM
bootstrap process. This issue has been resolved by ensuring that exception
handling routines properly and safely release the read cache lock.
- Apache JIRA:
HDDS-13904
- CDPD-93585: Ozone client retry is inconsistent when a
DataNode has some full disks
- 7.3.2.10000
- Previously, when an Ozone DataNode had some
full disks while others remained available, the client hung or repeatedly
retried for several minutes. This occurred because the client attempted to
access a full disk instead of failing immediately and reallocating a new
block, leading to performance delays. This issue is now resolved.
- Apache JIRA:
HDDS-14040
- CDPD-93952: Recon fails to initialize properly after
upgrade due to a race condition
- 7.3.2.10000
- Previously, a race condition occurred during
the Recon startup sequence when background services and metrics were
initialized before the database schema upgrade was finalized. This caused
Recon to fail with a
SQLSyntaxErrorException because it
attempted to query the RECON_TASK_STATUS table before the
required last_task_run_status column was added. This issue
is now resolved.
- Apache JIRA:
HDDS-14046
- CDPD-94032: Missing default value for
hdds.datanode.kerberos.principal configuration
causes authentication failures
- 7.3.2.10000
- Previously, the
hdds.datanode.kerberos.principal configuration
property in ozone-default.xml had an empty default
value. This inconsistency with other Ozone components required manual
configuration and caused Kerberos authentication errors, such as
Server has invalid Kerberos principal, particularly
when executing DiskBalancer commands in secure clusters. This issue is now
resolved by providing a standard default value for the DataNode Kerberos
principal.
- Apache JIRA:
HDDS-14064
- CDPD-94035: Ozone Manager logs do not identify the
command type for large responses
- 7.3.2.10000
- Previously, when the Ozone Manager generated a
response exceeding the size threshold, logs indicated only the response size
and protocol method (
submitRequest). This made it difficult
to identify the specific command causing the large response without
cross-referencing audit logs. This issue is now resolved.
- Apache JIRA:
HDDS-14061
- CDPD-94262: Recon fails to start during cluster
upgrades due to database schema race conditions
- 7.3.2.10000
- Previously, Recon failed to start during
cluster upgrades due to race conditions between database schema upgrades and
service initialization. Multiple components attempted to access database
tables or columns before the upgrade framework successfully added them,
resulting in
SQLSyntaxErrorException errors and startup
crashes. This issue is now resolved.
- Apache JIRA:
HDDS-14079
- CDPD-94836: Ozone Tiering workflow tasks fail due to
Hive query timeouts
- 7.3.2.10000
- Previously, Ozone Tiering workflow runs
(specifically the
AggregateAuditRecords task) failed with a
timeout error when the external Hive service was slow or unavailable,
because the default connection and socket read timeouts were set to 60
seconds. This resulted in inconsistent workflow success rates. This issue is
now resolved, and the default Hive client connection and socket read timeout
configuration values are increased to improve the reliability of workflow
executions that depend on Hive.
- CDPD-94970: Redundant object creation in
ContainerInfo.fromProtobuf method
- 7.3.2.10000
- Previously, the
ContainerInfo.fromProtobuf method in Ozone created a
throwaway instance of ContainerInfo before returning the
final object. This resulted in unnecessary memory allocation and overhead.
This issue is resolved now.
- Apache JIRA:
HDDS-14196
- CDPD-95031: Ozone administrator privilege checks are
inconsistent across the system
- 7.3.2.10000
- Previously, Ozone administrator checks were
inconsistent because some operations directly checked for administrator
privileges while others first verified if Access Control Lists (ACLs) were
enabled (ozone.acl.enabled). This led to inconsistent
behavior compared to HDFS, where permission checks were verified before
superuser privileges. This issue has been resolved by ensuring that Ozone
now consistently checks if permissions are enabled
(ozone.acl.enabled) before validating administrator
privileges.
- Apache JIRA:
HDDS-14207
- CDPD-95195: Ozone Manager fails to start due to
missing Java annotation dependencies
- 7.3.2.10000
- Previously, the Ozone Manager (OM) failed to
start with a
java.lang.NoClassDefFoundError:
javax/annotation/Priority error when running on Java 11 or
higher. This occurred because the RangerOzoneAuthorizer
required specific Java annotation and HK2 libraries that were not included
in the Ozone classpath. This issue has been resolved by adding the missing
dependencies (javax.annotation-api,
hk2-api, hk2-utils, and
hk2-locator) to the dependency list.
- CDPD-95245: Ozone Manager fails to start in non-HA
mode with minimal configuration
- 7.3.2.10000
- Previously, the Ozone Manager (OM) failed to
start in non-high availability (non-HA) mode when using a minimal
configuration. This occurred because the system could not automatically
retrieve the SCM address from default port logic, requiring manual
configuration of ozone.scm.client.address and
ozone.scm.datanode.address. This issue is now resolved.
- Apache JIRA:
HDDS-14249
- CDPD-95648: Ozone Tiering Workflow API incorrectly
reports a running status after a task failure
- 7.3.2.10000
- Previously, the Ozone Tiering Workflow API
continued to report the workflow state as
running: true
even if one or more workflow tasks failed. This resulted in a misleading
status where the workflow appeared active despite stopping due to an error.
This issue is resolved now.
- CDPD-95682: Recon displays incorrect pipeline
associations for containers
- 7.3.2.10000
- Previously, the Recon UI incorrectly displayed
all available pipelines for every container. This issue is resolved
now.
- Apache JIRA:
HDDS-14368
- CDPD-95716: File descriptor leak in the Ozone Manager
during checkpoint transfers
- 7.3.2.10000
- Previously, a directory stream was not properly
closed in the
OMDBCheckpointServletInodeBasedXfer.writeDBToArchive()
method. This caused file descriptors to accumulate on the Ozone Manager
host, potentially leading to performance degradation or service instability
over time. This issue has been resolved by ensuring the directory stream is
explicitly closed after execution.
- Apache JIRA:
HDDS-14376
- CDPD-95936: Ozone SCM unnecessarily processes block
deletion commands for empty files
- 7.3.2.10000
- Previously, Storage Container Manager (SCM)
processed block deletion commands for empty files that contained no data
blocks, resulting in unnecessary audit log entries and system overhead. This
issue is resolved now.
- Apache JIRA:
HDDS-14418
- CDPD-95970: Recon UI displays "N/A" instead of "0 B"
for zero-byte values in Summary Cards
- 7.3.2.10000
- Previously, in the Open Keys
Summary and Delete Pending Keys
Summary cards, fields with a value of 0 (zero bytes or zero
count) were displayed as N/A. This made it difficult
to differentiate between valid zero values and missing data or error
responses. This issue is now resolved, and zero values are explicitly
displayed as 0 B or 0.
- Apache JIRA:
HDDS-14422
- CDPD-96003: Redundant transaction deletion and
replica-level size updates in Ozone
- 7.3.2.10000
- Previously, transaction deletion logic in Ozone
executed twice, causing redundant delete calls to the database.
Additionally, transaction size updates were triggered for each replica
instead of at the transaction level, resulting in unnecessary processing and
potential aggregation errors. This issue is now resolved.
- Apache JIRA:
HDDS-14429
- CDPD-96234: Ozone bucket table cache is not updated
with usage statistics upon key commit
- 7.3.2.10000
- Previously, when a key was committed in Ozone,
bucket usage statistics (used bytes and namespace) were updated in the
backend but were not reflected in the bucket table cache, resulting in stale
or inconsistent usage data being reported. This issue is now resolved.
- Apache JIRA:
HDDS-14467
- CDPD-96416: The Containers page incorrectly displays
the "No of Keys" label instead of "No of Blocks"
- 7.3.2.10000
- Previously, the Containers
page in the Ozone UI incorrectly used the label No of
Keys to represent block counts. This was inconsistent with
the underlying terminology and the legacy UI. This issue is now
resolved.
- Apache JIRA:
HDDS-14492
- CDPD-96710: Non-recursive directory deletion fails
with the Directory is not empty error
- 7.3.2.10000
- Previously, the S3 clients could unexpectedly
encounter a Directory is not empty error when attempting
to recursively delete a directory through the S3 Gateway on Ozone. This was
driven by a transient timing race condition that occurred when the Ozone
Manager's internal double buffer had not yet flushed child deletion
transactions to the database. This issue is fixed now by properly checking
key tombstones in cache in
checkSubFileExists or
checkSubDirectoryExists.
- Apache JIRA:
HDDS-14600
- CDPD-96831: Ozone-tiering build overwrites core Ozone
JAR files in the CDH parcel
- 7.3.2.0 and 7.3.2.10000
- Previously, the Ozone-tiering build process
bundled all dependencies, including core Ozone and Hadoop JARs, into its
output. During CDH parcel assembly, the Ozone-tiering RPM was extracted
after the base Ozone RPM, overwriting patch-level Ozone JARs with older or
mismatched versions and potentially reverting patch changes. This issue is
now resolved.
- CDPD-97209: Container Balancer attempts to move
unhealthy containers due to inconsistent validation
- 7.3.2.10000
- Previously,
ContainerBalancerSelectionCriteria applied a less
stringent health check than MoveManager. This allowed unhealthy containers
to be selected for balancing, only to be rejected later by MoveManager's
stricter criteria, resulting in wasted cycles and inefficient balancing
operations. This issue has been resolved by unifying the validation
logic, ensuring that the same rigorous health checks are applied
upfront. Additionally, if a balancing attempt fails, source DataNodes
are now correctly returned to the priority queue so they remain eligible
for subsequent balancing cycles.
- Apache JIRA:
HDDS-14614
- CDPD-98087: Performance degradation in SCM
listContainer API at scale
- 7.3.2.10000
- Previously, the
StorageContainerLocationProtocol.listContainer() API
experienced significant performance degradation when handling a large number
of containers (200K+). Excessive RPC payloads and per-container lock
acquisition caused high CPU usage on the Storage Container Manager (SCM) and
slowed Recon sync operations. This issue is now resolved.
- Apache JIRA:
HDDS-14730
- CDPD-98452: Ozone Manager followers could serve stale
checkpoints during bootstrap requests
- 7.3.2.10000
- Previously, metadata checkpoints could be
downloaded from either a leader or follower Ozone Manager (OM) instance.
Fetching a checkpoint from a follower during a manual bootstrap or download
resulted in stale metadata being served. This issue is now resolved.
- Apache JIRA:
HDDS-14775
- CDPD-98892: Incorrect file size distribution range
displayed in Recon UI
- 7.3.2.10000
- Previously, the Recon UI displayed an incorrect
file size distribution range (for example, showing 512KB to 1MB for files
actually in the GB range) when a large number of buckets existed within a
volume. This issue has been resolved, and the Recon UI now correctly
calculates and displays the file size distribution range.
- Apache JIRA:
HDDS-14827
- CDPD-99243: In
OzoneManagerSyncMetrics class, the
incrNumDeltaRequestsFailed method increments the wrong
value
- 7.3.2.10000
- Previously, the metrics for Ozone Manager
transaction delta request failures in Recon were not accounted for properly
because the
incrNumDeltaRequestsFailed method incremented
the incorrect variable. This issue is resolved now.
- Apache JIRA:
HDDS-14848
- CDPD-99292: Ozone
ThrottledAsyncChecker schedules volume checks in a
non-thread-safe manner
- 7.3.2.10000
- Previously, a race condition occurred in
ThrottledAsyncChecker where multiple threads could
simultaneously schedule checks for the same target or volume. This occurred
because the internal map tracking checks in progress was updated only after
a check was initiated, resulting in redundant checks or undefined behavior
when a volume check was overdue or had not yet run. This issue is resolved
now.
- Apache JIRA:
HDDS-14860
- CDPD-99365: DataNode health-check latch timeouts cause
false-positive volume failures
- 7.3.2.10000
- Previously, the DataNode
StorageVolumeChecker used a single latch for all volume
health checks. If this latch timed out due to a transient stall, all pending
volumes were immediately marked as failed, even if they were healthy, which
led to false-positive volume failures. This issue is now resolved and
DataNodes now tolerate per-volume health-check latch timeouts before marking
volumes as failed.
- Apache JIRA:
HDDS-14871
- CDPD-99398: DataNodes may crash during volume failure
handling due to active RocksDB iterators
- 7.3.2.10000
- Previously, when a storage volume failure
occurred, the
StorageVolumeChecker closed the RocksDB
instance for that volume without notifying background scanner processes,
such as BackgroundContainerDataScanner or
OnDemandContainerScanner. If these scanners attempted
to use an active iterator on the closed database, a JVM crash occurred. This
issue is resolved now.
- Apache JIRA:
HDDS-14800
- CDPD-99431: The
ContainerBalancerStatus command displays incorrect
"Max Size to Move per Iteration" value
- 7.3.2.10000
- Previously, the
ContainerBalancerStatus command often displayed
0GB for the Max Size to Move per
Iteration field. This occurred because the value was
incorrectly mapped to the Max Datanodes to Involve per Iteration
(percentage) field instead of the Max Size to
Move field. This issue is resolved now, and the command
correctly displays the maximum size to move per iteration.
- Apache JIRA:
HDDS-14883
- CDPD-99831: Ozone DataNode decommission command hangs
when monitoring multiple nodes
- 7.3.2.10000
- Previously, the Ozone DataNode decommission
command in Cloudera Manager became stuck indefinitely when
decommissioning multiple nodes simultaneously. This occurred because the
monitoring script used imprecise string-matching logic (grep) that
incorrectly handled IP addresses sharing a common prefix (for example,
10.129.39.23 and 10.129.39.233), causing the script to wait for nodes that
had already completed decommissioning. This issue is resolved now.
- CDPD-100872: Ozone snapshot rename operation is
disabled in the CLI
- 7.3.2.10000
- Previously, the Ozone snapshot rename operation
was displayed in the Command Line Interface (CLI) but was not fully
supported or functional in Cloudera Base on premises, resulting in
unexpected behavior when you attempted to rename snapshots. This issue is
resolved now, and the snapshot rename functionality has now been disabled
from the CLI to prevent unsupported usage.
- CDPD-101434: Risk of missing blocks in Ratis pipelines
due to initial under-replication
- 7.3.2.10000
- Previously, when data was written through a
Ratis pipeline, blocks were occasionally written only to the leader
DataNode, despite the system returning a successful write acknowledgment to
the client. Consequently, if the leader DataNode failed or went offline
before the data was replicated to other nodes, the associated blocks or
chunk data went missing, resulting in temporary data unavailability. This
vulnerable state persisted until the Storage Container Manager (SCM)
background processes detected the under-replicated container and completed
the proper replication. This issue is resolved now.
- Apache JIRA:
HDDS-15052
- CDPD-101613: Unhealthy containers with sufficient
replicas are not correctly tracked in container reports
- 7.3.2.10000
- Previously, the
RatisUnhealthyReplicationCheckHandler failed to
increment the UNHEALTHY container count in reports for
containers that were sufficiently replicated but contained unhealthy
replicas. While under-replicated and over-replicated unhealthy containers
were correctly identified, those with the correct replica count were missed,
leading to inaccurate health reporting in Ozone. This issue is resolved
now.
- Apache JIRA:
HDDS-15261
- CDPD-101838: Ozone Manager terminates during startup
due to race condition
- 7.3.2.10000
- Previously, a race condition between the Ozone
Manager (OM) startup flow and the OM snapshot installation flow caused the
OM to terminate unexpectedly with an
IllegalStateException.
This occurred when both processes attempted to start the delegation token
secret manager simultaneously. This issue is resolved now.
- Apache JIRA:
HDDS-15103
- CDPD-101970: Ozone client hangs for a long duration
during
writeStateMachineData failures
- 7.3.2.10000
- Previously, when a failure occurred during the
writeStateMachineData operation on an Ozone Datanode
(such as a disk space issue or other exceptions), the client would hang for
a long duration while the leader node repeatedly retried the operation until
the pipeline closed. This resulted in significant delays before the
operation was finally marked as a failure. This issue is now resolved by
optimizing the failure handling, reducing the client waiting time to a few
seconds in most failure scenarios.
- Apache Issue:
HDDS-15122
- CDPD-102272: Ozone DataNode scanner flags healthy
containers as UNHEALTHY due to system resource limits
- 7.3.2.10000
- Previously, background container scanners in
Ozone incorrectly marked containers as
UNHEALTHY when they
encountered system resource issues, such as file-descriptor exhaustion ("Too
many open files"), instead of actual data corruption. This led to false
positives where healthy containers were flagged due to temporary system
limits. This issue is now resolved, and the logic is updated to catch these
resource-related exceptions (FileNotFoundException or
FileSystemException), ensuring that containers remain
in their current state if the scanner could not perform its check due to
system constraints.
- Apache JIRA:
HDDS-15150
- CDPD-102811: Ozone files marked for deletion were not
being reclaimed from snapshots
- 7.3.2.10000
- Previously, files marked for deletion in Ozone
were often not reclaimed because their contents were perpetually moved to
the next active snapshot during snapshot deletion. This caused deleted files
to remain in the cluster indefinitely, consuming storage space. This issue
is now resolved by enabling snapshot deep cleaning by default, ensuring that
deleted keys are correctly moved through the snapshot chain until they are
fully purged.
- Apache JIRA:
HDDS-15187
- CDPD-102907: Ozone Tiering service fails to start due
to missing configuration classes
- 7.3.2.10000
- Previously, the Ozone Tiering service failed to
start with a
java.lang.NoClassDefFoundError for
org/apache/commons/configuration2/Configuration. This
occurred because of a version mismatch in the classpath dependencies. This
issue is now resolved.
- CDPD-103013: Delay in safemode exit on follower
causing intermittent behaviour leading to leader switch test failure
- 7.3.2.10000
- Previously, Ozone Storage Container Manager (SCM)
follower nodes could remain stuck in safe mode after a restart. This
occurred because the
ContainerSafeModeRule would refresh
its container list and potentially add newly created containers that
DataNodes (DNs) had already registered, leading to a situation where the 99%
reporting threshold could not be met. Consequently, safe mode exit was
delayed until a new Ratis transaction was applied, causing intermittent
failures in leader switch tests and impacting small clusters. This issue has
been resolved by ensuring the safe mode rule maintains the container list
initialized at startup and only removes deleted containers, correctly
updating the threshold and cutoff values.
- Apache Issue:
HDDS-15238
- CDPD-103659: The ozone admin container list
--all command returns duplicate containers
- 7.3.2.10000
- Previously, the ozone admin container
list --all could produce duplicate container entries because
SCM returned each batch sorted by container lastUsed timestamp, while the
CLI paginated using the last container ID in the batch. When the last item
in a batch did not have the highest container ID, the next page re-fetched
containers that had already been listed. This is now fixed by sorting list
results by container ID on the server.
- Apache JIRA:
HDDS-15305
- CDPD-103728: Inconsistent task state transitions in
Tiering Workflow API
- 7.3.2.10000
- Previously, the Tiering Workflow API endpoint
returned inconsistent task state transitions, causing tasks that were
reported as
PASSED to later transition to
SKIPPED. This issue is resolved now.
- CDPD-103890: Ozone Tiering UI remains blank or stuck
in a loading state during the key conversion process
- 7.3.2.10000
- Previously, when a data preparation or key
conversion task was triggered in the Ozone Tiering UI, the page remained
blank or displayed a loading skeleton until the process finished. This
occurred because the UI hid existing data during background refreshes and
used a long polling interval, leading to page timeouts and a lack of
progress feedback. This issue is resolved now.
- CDPD-104319: Tiering UI fails to display SKIPPED
status and messages for conversion tasks
- 7.3.2.10000
- Previously, when a tiering workflow skipped a
task (for example, if all selected keys were already Erasure Coded), the
Overview UI did not handle the
SKIPPED state. This caused
the workflow to appear as Scheduled with empty metrics,
obscuring the skip messages provided by the API. This issue is resolved
now.
- CDPD-104329: Tiering API workflow returns inconsistent
task states for Hive Metastore failures
- 7.3.2.10000
- Previously, when the Tiering workflow
encountered a Hive Metastore connectivity failure, task statuses were
reported inconsistently across different runs and workflow stages. In some
instances, tasks were marked as
FAILED, while in others
they were marked as SKIPPED, despite having the same
underlying root cause. This issue is resolved now.
- CDPD-104553: Tiering Workflow API incorrectly reports
running status after task failure
- 7.3.2.10000
- Previously, the Tiering Workflow API continued
to report a status of
running=true even after a task within
the workflow reached a terminal FAILED state. This resulted
in an inconsistent reporting state where the workflow appeared active
despite the failure. This issue is resolved now.
- CDPD-104558: Incorrect task status transition for
AggregateAuditRecords
- 7.3.2.10000
- Previously, the
AggregateAuditRecords task was incorrectly marked as
COMPLETED in the initial execution snapshot even when
the Hive Metastore was down, before it eventually transitioned to
FAILED. This race condition resulted in inconsistent
status reporting during task execution. This issue is resolved now.
- CDPD-106341: SCM logs are flooded with WARN messages
when Datanode command count information is missing
- 7.3.2.10000
- Previously, Storage Container Manager (SCM)
logs were flooded with WARN messages stating "
No command count
information for datanode... Assuming zero" during block
deletion services, even when the condition was expected or non-critical.
This resulted in excessive log noise. This issue is now resolved.
- Apache JIRA:
HDDS-15588
- CDPD-107080: Race condition in Ozone block deletion
transaction handling
- 7.3.2.10000
- Previously, a race condition occurred between
the
readOnlyIterator and onFlush
operations during Ozone block deletion. The iterator skipped elements
already in the deletion queue, but if another thread cleared the queue
before the iterator finished, completed transactions were incorrectly
reloaded for deletion. This issue is now resolved.
- Apache JIRA:
HDDS-15665
- CDPD-109489: Snapshot defragmentation fails when
directly ingesting a single live RocksDB SST file
- 7.3.2.10000
- Previously, when the incremental snapshot
defragmentation service identified exactly one candidate SST file for a
table, it bypassed the standard rewrite path and attempted to ingest the
live RocksDB SST file directly. This caused ingestion failures (such as
External file version not found) or installed incorrect
data because the direct path bypassed critical bucket-prefix filtering,
snapshot comparisons, and tombstone generation. This issue is now
resolved.
- Apache JIRA:
HDDS-15860
- CDPD-110020: Ozone EC offline reconstruction could
result in data loss due to premature scrubbing of idle RECOVERING
containers.
- 7.3.2.10000
- Previously, data loss occurred during EC
offline reconstruction when idle RECOVERING containers were scrubbed and
auto-recreated as partial CLOSED replicas. This issue is now resolved.
- Apache JIRA:
HDDS-15791
- CDPD-119182: Ozone Recon disk space exhaustion due to
leaked checkpoints
- 7.3.2.10000
- Previously, Ozone Recon failed to clean up
temporary checkpoints created during task reinitialization. This caused
checkpoint files to accumulate in the
/var/lib/hadoop-ozone/recon directory, eventually leading to
disk space exhaustion and causing Recon to become unhealthy. This issue is
now resolved.
- CDPD-111367: Recon Overview displays zero counts for
volumes, buckets, and keys after upgrade
- 7.3.2.10000
- Previously, when upgrading from 7.1.9 SP2 to
7.3.2, the Recon Overview page displayed zero counts for volumes, buckets,
and keys. This occurred because both versions assigned different features to
the same layout version number, causing Recon to skip a database schema
update that added required columns to the
RECON_TASK_STATUS
table and silently failing Ozone Manager (OM) data synchronization. This
issue is now resolved.
- CDPD-111714: Recommissioning a new Ozone Manager node
fails during bootstrap
- 7.3.2.10000
- Previously, when adding or recommissioning an
Ozone Manager (OM) node, the OM bootstrap process failed if catching up
required an OM checkpoint installation from the leader (for example, when
leader logs were purged). The checkpoint installation request was rejected
because the OM was in the
BOOTSTRAPPING state, causing a
deadlock during Ratis configuration staging and preventing the new node from
joining the cluster. This issue is now resolved.
- Apache JIRA:
HDDS-16057
