Resolved -
This incident has been resolved.
Mar 11, 12:43 CDT
Monitoring -
The recovery process has gotten far enough to re-enable the filesystem safely. Please validate your data. If you find any data corruption please report that to itrss-support@umsystem.edu.
Feb 13, 09:28 CST
Update -
The Pixstor disk recovery process is approximately 31% completed. We expect this process to take longer than the 4-day estimate provided. At the current rate we estimate to have data availability fully restored by next Friday. We appreciate your patience and understanding while we work to resolve the issue.
Feb 6, 14:51 CST
Update -
The Pixstor disk recovery process is approximately 31% completed. We expect this process to take longer than the 4-day estimate provided. At the current rate we estimate to have data availability fully restored by next Friday. We appreciate your patience and understanding while we work to resolve the issue.
Feb 6, 14:51 CST
Update -
The previous time estimate of 'a couple days' has been revised based on the current progress to 4 days. We are confident that no data will be permanently lost if the recovery process completes successfully. We are proceeding with caution to reduce the risk of data loss. We are monitoring the progress of the recovery, and working with vendors regarding the timing of returning the storage to full production. While some data written before Monday afternoon may be inaccessible, you can safely write new data to the filesystem and read it back out without concern. Thank you for your patience while we work through this issue.
Feb 4, 13:43 CST
Update -
At current rate, the recovery process is estimated to take a couple more days to reach a fault tolerant point in recovery where we can restore the filesystem while it continues to recover the remaining disks. We have replacement parts inbound to try to prevent the instigating event from occurring again. Thank you for your patience while we work through this issue.
Feb 4, 07:58 CST
Update -
We are continuing to work on the issue. A critical component of the filesystem has experienced a failure. Recovery has begun and parts are inbound, however the recovery process will take some time. We will be checking the status of recovery tomorrow and provide an update at that time or earlier if the status changes.
Feb 3, 16:33 CST
Identified -
A storage array is showing failures. We're currently working with the vendor to bring services back to fully operational status.
This is resulting in some files having IO errors throughout the filesystem including but not limited to Hellbender home directories.
Feb 3, 14:27 CST