SU367: [Impact: High] Excess Data Writing to mSATA on E-Series Systems and StorageGRID Appliances

Summary

NetApp has become aware of an issue related to the excess writing of statistical data to the onboard mSATA SSD on certain NetApp^® E-Series and EF-Series controllers. E2800 and EF280 controllers that run NetApp SANtricity^® OS 11.30.x and E2800, EF280, E5700, and EF570 controllers that run SANtricity OS 11.40.x, 11.50.0, and 11.50.1 are affected.

If they run an affected SANtricity OS release, E2800 controllers in NetApp StorageGRID^® SG5712, SG5760, and SG6060 appliances and EF570 controllers in StorageGRID SGF6024 appliances might also experience this issue.

The affected mSATA SSD is used only for E-Series software and system data and is not visible for any host data.

Note: For StorageGRID appliances, review the specific StorageGRID procedures that are described in this bulletin

Issue Description

This issue can occur on E2800, EF280, E5700, and EF570 controllers and on SG5712, SG5760, SG6060, and SGF6024 StorageGRID appliances. When the defect is encountered, excess statistical data might be written to the onboard mSATA SSD. This excess writing of data might exceed the endurance of the mSATA SSD.

If the mSATA SSD endurance is exceeded on one controller, the mSATA device will start returning errors to some read/write operations from the controller. Subsequently, the device can end up failing causing write cache to be disabled, but the controller will continue to serve host I/O operations.

Symptom

When the endurance of the mSATA device on one or both controllers is exceeded in the storage array and a health check for the mSATA device fails, write caching is disabled. Users will receive relevant failures in recovery guru in SANtricity System Manager.

In SANtricity OS 11.50.3, users will receive a failure entry for mSATA endurance exceeded (end of life) before the mSATA device fails a health check. This notification helps users prepare for replacing the controller before the mSATA device fails.

Workaround

When the issue results in controller(s) exceeding the mSATA device endurance level, the controller must be replaced. If both controllers are impacted, it is recommended to attempt replacing the controllers one at a time.

NetApp Technical Support can help evaluate which controller to replace first based on the maximum erase count (P/E) value described in How to determine the remaining wear life of E-Series mSATA devices article (under additional information).

In a StorageGRID appliance with only one E2800 storage controller (SG5712 and SG5760), if the E2800 controller has failed, only the E2800 controller must be replaced. The E5700SG controller runs StorageGRID firmware and is not vulnerable to this issue.

Users should also update to the latest recommended SANtricity OS controller software versions, as listed in the "Solution" section.

Solution

NetApp requires all users of any E-Series or EF-Series system with E2800, EF280, E5700, or EF570 controllers running SANtricity OS 11.30.x, 11.40.x, 11.50.0, or 11.50.1 to perform a SANtricity OS controller software update as soon as possible. This update prevents the excess writing from occurring. The update cannot be applied directly to a controller that has already failed. If the controller has failed, it must be replaced.

If the failure has already occurred, users should follow the procedure in the preceding "Workaround" section.

NetApp also requires users to perform a SANtricity OS controller software update as soon as possible for:

• SG5712, SG5760, or SG6060 StorageGRID appliances with E2800 storage controllers running SANtricity OS 11.30.x, 11.40.x, 11.50.00, or 11.50.1
• SGF6024 StorageGRID appliances with EF570 storage controllers running SANtricity OS 11.50.0 or 11.50.1

This update will prevent the excess writing from occurring. The update cannot be applied directly to a controller that has already failed. If the controller has failed, it must be replaced.

If the failure has already occurred, follow the procedure in the preceding "Workaround" section.

How to Obtain SANtricity OS Controller Software

Users can find E-Series SANtricity OS controller software at the NetApp Support site Downloads page.

To download the latest revision, on the Downloads page, users select from the drop-down or enter the appropriate OS version into the drop-down menu. Following is a list of the latest revisions that resolve this issue.

E2800 and EF280:

• 11.30.5R3 contains the Partial Excessive Write fix and wear level data
• 11.40.3R2 contains the Partial Excessive Write fix and wear level data
• 11.50.2 contains the Partial Excessive Write fix and wear level data
• 11.50.3 or later - contains the Excessive Write fix, wear level data and warning messages (Recommend upgrade to this or later releases)
• 11.60.2R1 or later - contains the Excessive Write fix, wear level data and warning messages (Recommend upgrade to this or later releases)

E5700 and EF570:

• 11.50.2 contains the Partial Excessive Write fix and wear level data
• 11.50.3 or later - contains the Excessive Write fix, wear level data and warning messages (Recommend upgrade to this or later releases)
• 11.60.2R1 or later - contains the Excessive Write fix, wear level data and warning messages (Recommend upgrade to this or later releases)

Note: The procedure to update NetApp SANtricity OS on a StorageGRID appliance differs from the procedure to update SANtricity OS on a standard E-Series array. To avoid potential data loss, users must follow the correct procedure. For details, users should refer to the KB article How to Upgrade E-Series Firmware on a StorageGRID appliance:

Alert Notifications

In some instances, SANtricity OS 11.50.2 or later versions that run on an E2800, EF280, E5700, or EF570 controller issue alert notifications.

If the endurance of the mSATA device on one or both controllers is exceeded in the storage array, the array firmware generates an alert notification to replace one or both controllers that have the failed mSATA device.

If the endurance of the mSATA device on one or both controllers is exceeded in the storage array and the health check for the mSATA device fails, write caching is disabled. The array firmware also generates an alert notification to replace one or both controllers that have the failed mSATA device.

Note: If the download of SANtricity OS controller software fails and the user observes that SANtricity OS can detect only one of the controllers in a dual-controller E-Series array, the following "Additional Notes" provide more information.

If the download of SANtricity OS controller software fails, and the user observes that SANtricity OS can detect both controllers in a dual-controller E-Series array, the user should contact NetApp Support for assistance.

Additional Notes for Inability to Download SANtricity OS Software to Controller

An issue has been found that might affect a user’s ability to download the SANtricity OS controller software. If a user cannot download the software, they should follow these workaround steps:

For E5700 and EF570 controllers in E-Series arrays:

1. While I/O is running:
   1. Remove controller A from the system.
   2. Remove the battery from controller A.
   3. Wait 5 minutes, then reinsert the same battery into controller A.
      Note: All volumes are automatically moved to controller B, and all I/O continues to run.
   4. Reinsert controller A into the system and allow controller A to complete startup for at least 5 minutes.
2. While I/O is running:
   1. Remove controller B from the system.
   2. Remove the battery from controller B.
   3. Wait 5 minutes, then reinsert the same battery into controller B
      Note: All volumes are automatically moved to controller A, and all I/O continues to run.
   4. Reinsert controller B into the system and allow controller B to complete startup for at least 5 minutes.
3. Attempt to download the SANtricity OS controller software again.

For E2800 and EF280 controllers in E-Series arrays:

1. While I/O is running:
   1. Reboot controller A.
   2. Allow controller A to complete startup for at least 5 minutes.
   3. Reboot controller B.
   4. Allow controller B to complete startup for at least 5 minutes.
2. Attempt to download the SANtricity OS controller software again.

For StorageGRID appliances:

1. Make sure that the StorageGRID appliance is in maintenance mode.
2. Follow the procedure for the storage controller model (E2800 or EF570) in your appliance.

Contact Information: Contact NetApp Support with questions concerning this bulletin.

Important: The purpose of this communication is for NetApp to notify its installed-base end users about urgent and important product information that may affect product performance or reliability. The information contained herein and the distribution lists are NetApp confidential materials that are subject to restrictions on redistribution and that cannot be shared outside of this email distribution list.

***************************************************
*** NETAPP CONFIDENTIAL – FOR LIMITED USE ONLY ***
***************************************************