Root cause traced to network glitch between active and passive controllers. Unable to identify the specific nature of the glitch.
Posted Mar 14, 2024 - 13:26 MDT
Monitoring
A ticket with the vendor open to determine RCA, but as far as we can tell, the service is restored. Will update as we learn more.
Posted Mar 04, 2024 - 14:26 MST
Update
We are continuing to investigate this issue.
Posted Mar 04, 2024 - 10:28 MST
Update
While investigating this issue. The TrueNAS controller started a failover at 9:33, and the failover event completed at 9:36.
We are validating the status of the storage, but early tests indicate normal function, and the issue noted earlier has been resolved by the failover.
Posted Mar 04, 2024 - 09:57 MST
Investigating
We have identified some cases where groups in CSUDOM are not getting the correct ID mapping on TrueNAS. The result is that some users are being inappropriately denied access to directories where they were working as recently as March-01.
Posted Mar 04, 2024 - 09:34 MST
This incident affected: Research (RSTOR (Research-Specific Computing and Applications) - Fort Collins) and Infrastructure and Network (Storage (Server and Storage Management) - Fort Collins).