SU522: [Impact: Critical] StorageGRID 11.3 through 11.6 releases using S3 multipart objects may incorrectly purge multipart parts leading to data loss

Views:: 1,974

Last Updated:: 2/7/2023, 1:25:01 AM

免责声明：
联想凌拓科技有限公司（“Lenovo NetApp”）不对本页面中提供的任何信息或建议的准确性、可靠性或可维护性，或通过使用这些信息或遵守本文中提供的建议可能获得的任何结果，提供任何陈述或保证。本页面中的信息是按原样分发的，使用这些信息或实施本文中的任何建议或技术是客户的责任，取决于客户评估这些信息并将其整合到客户的运营环境中的能力。本页面及其包含的信息只能与本页面中讨论的 NetApp 产品结合使用。在任何情况下，Lenovo NetApp 均不承担因与使用或执行本页面上提供的信息有关的或导致的任何特殊的、间接的或随之而来的任何损失，或者因使用、数据或利润损失（无论是否在合同履行中）、疏忽或其它侵权行为导致的任何损失。

更多最新信息请参考 NetApp 官网支持公告

Summary

StorageGRID 11.3 through 11.6 releases using S3 multipart objects may incorrectly purge multipart parts leading to data loss.

[Impact: Critical – Data loss/corruption]

Issue Description

StorageGRID versions 11.3 through 11.6 using S3 multipart objects that required the retry of the multipart complete to an unversioned or versioning suspended bucket may incorrectly purge multipart parts leading to data loss. This specific issue only occurs when:

The multipart complete fails with a HTTP 5xx error due to a specific internal metadata failure that may be caused by network/communications impairment between Storage Nodes. The metadata failure required is an extremely rare condition.
The same multipart complete is retried and successful.

The result is the deletion of the multipart parts for the multipart object.

If an object is affected by this issue, HEADs will succeed, but GETs will fail to return data as the multipart parts will be missing.

The log message signature for this issue appears in the Storage Node bycast-err.log as

ERROR 0354 7e5a420d6f7e8220 SCAN: Couldn't retrieve segment object (container uuid = 35FE2B76-4767-4D84-ADA8-390FA07C3C5A, segment uuid = CDC08CC1-0550-4193-B7C9-9C99BE7AD7FB)

Where all of the following are true:

The “container uuid” is the same across multiple log messages *and*
Same log messages occur across multiple ILM scan periods *and*
Approximately 15 days have passed since the “container uuid” Creation Time.

These ERROR-level log messages may appear on any Storage Node. The ILM scan period is shown on the Grid Manager UI Dashboard as “Scan Period – Estimated”.

The issue may be additionally confirmed using Object Lookup in the Grid Manager UI:

Lookup of the container UUID returns metadata as normal
Lookup of segment UUIDs returns “not found” confirming the multipart parts are not found

Note: FabricPool does not use multipart objects. FabricPool’s single part objects are not impacted by this issue.

Workaround

There is no workaround for this issue. As a result, the solution must be followed to fix the software issue.

Solution

For all StorageGRID environments running 11.5.0.10 or earlier, it is recommended to upgrade to StorageGRID 11.5.0.11 or 11.6.0.7 and later available on the NetApp Support site.

For all StorageGRID environments running 11.6.0.5 or earlier 11.6 versions, it is recommended to upgrade to StorageGRID 11.6.0.7, or later available on the NetApp Support site.

NetApp provides no representations or warranties regarding the accuracy or reliability or serviceability of any information or recommendations provided in this publication or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS and the use of this information or the implementation of any recommendations or techniques herein is a customer's responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document.