Users may be unable to access or use some Microsoft 365 services and features
Incident
Report for UM IT Systems
Resolved
Nov 26, 2024, 6:53 AM CST
We’re continuing our period of monitoring service telemetry, which shows the service availability has remained healthy.
Nov 26, 2024, 4:59 AM CST
From monitoring service telemetry, most users should now experience relief. We’ve completed our optimizations and we're continuing our period of extended monitoring to ensure the availability remains stable.
Nov 26, 2024, 4:02 AM CST
While we continue our period of extended monitoring, the availability of the Outlook on the Web service has reached expected availability levels. We’re continuing to optimize the environment to address the remaining impact.
Nov 26, 2024, 3:03 AM CST
We're still addressing the remaining impact to the Outlook on the web service that is affecting some users. We’ve applied mitigation actions to reduce the mail queues and we’re continuing the extended period of monitoring to ensure stability continues.
Nov 26, 2024, 2:04 AM CST
We’ve isolated the cause of mail queue delays and have restarted the affected infrastructure to drain stalled queues. We’ll remain in an extended monitoring phase until this draining is completed and we can consider the incident fully recovered.
Nov 25, 2024, 11:04 PM CST
We're continuing to address lingering impact to the Outlook on the web service that is still affecting some users. In parallel, we're investigating some mail queuing delays that is resulting in mail taking longer than expected to be delivered. Due to the impact of this incident, we will enter a period of extended monitoring prior to declaring this issue resolved.
Nov 25, 2024, 10:32 PM CST
Impact to core services have been restored with the exception of Outlook on the web, which we’ll continue to monitor and actively troubleshoot until full recovery. This quick update is designed to give the latest information on this issue.
Nov 25, 2024, 9:31 PM CST
We’ve successfully restored functionality for all previously impacted services and users with the exception of Outlook on the web, which is showing prolonged impact for a small number of users. We’ll continue carefully monitoring the service health and focus on troubleshooting this persisting impact to fully recover for the remaining affected users. We'll provide a new timeline within the next update.
Nov 25, 2024, 8:10 PM CST
Our monitoring indicates that a large portion of affected users and services are seeing recovery following our mitigation efforts. We're working on addressing the lingering regions that are still seeing small impact to fully restore service availability, which we still expect to complete by Monday, November 25, 2024 at 9:00 PM CST
Nov 25, 2024, 6:01 PM CST
The service is continuing to incrementally recover. We're forecasting that impact will be fully remediated by Monday, November 25, 2024 at 9:00 PM CST , though we're closely monitoring our progress and will provide an updated timeline should the estimate change.
We’re continuing our period of monitoring service telemetry, which shows the service availability has remained healthy.
Nov 26, 2024, 4:59 AM CST
From monitoring service telemetry, most users should now experience relief. We’ve completed our optimizations and we're continuing our period of extended monitoring to ensure the availability remains stable.
Nov 26, 2024, 4:02 AM CST
While we continue our period of extended monitoring, the availability of the Outlook on the Web service has reached expected availability levels. We’re continuing to optimize the environment to address the remaining impact.
Nov 26, 2024, 3:03 AM CST
We're still addressing the remaining impact to the Outlook on the web service that is affecting some users. We’ve applied mitigation actions to reduce the mail queues and we’re continuing the extended period of monitoring to ensure stability continues.
Nov 26, 2024, 2:04 AM CST
We’ve isolated the cause of mail queue delays and have restarted the affected infrastructure to drain stalled queues. We’ll remain in an extended monitoring phase until this draining is completed and we can consider the incident fully recovered.
Nov 25, 2024, 11:04 PM CST
We're continuing to address lingering impact to the Outlook on the web service that is still affecting some users. In parallel, we're investigating some mail queuing delays that is resulting in mail taking longer than expected to be delivered. Due to the impact of this incident, we will enter a period of extended monitoring prior to declaring this issue resolved.
Nov 25, 2024, 10:32 PM CST
Impact to core services have been restored with the exception of Outlook on the web, which we’ll continue to monitor and actively troubleshoot until full recovery. This quick update is designed to give the latest information on this issue.
Nov 25, 2024, 9:31 PM CST
We’ve successfully restored functionality for all previously impacted services and users with the exception of Outlook on the web, which is showing prolonged impact for a small number of users. We’ll continue carefully monitoring the service health and focus on troubleshooting this persisting impact to fully recover for the remaining affected users. We'll provide a new timeline within the next update.
Nov 25, 2024, 8:10 PM CST
Our monitoring indicates that a large portion of affected users and services are seeing recovery following our mitigation efforts. We're working on addressing the lingering regions that are still seeing small impact to fully restore service availability, which we still expect to complete by Monday, November 25, 2024 at 9:00 PM CST
Nov 25, 2024, 6:01 PM CST
The service is continuing to incrementally recover. We're forecasting that impact will be fully remediated by Monday, November 25, 2024 at 9:00 PM CST , though we're closely monitoring our progress and will provide an updated timeline should the estimate change.
Update
Nov 25, 2024, 2:25 PM CST
Services are showing significant signs of restoration following our infrastructure optimizations to redistribute traffic and restart affected servers, upwards of 90 percent recovery as we continue to increase processing capability. We're continuing to perform appropriate optimizations on the subset of remaining affected machines to resolve the issue for all users.
Nov 25, 2024, 1:52 PM CST
We're continuing to redirect traffic and apply targeted remediation to impacted servers. We've observed initially positive results in some areas, and we're monitoring service health closely as our mitigation workstreams continue in parallel.
Nov 25, 2024, 1:15 PM CST
Our mitigative actions haven't provided relief as expected, and a portion of infrastructure remains in an unhealthy state. We determined that some of the targeted server restarts did not succeed due to processing issues, which are under investigation. We’re currently focused on spreading traffic to healthy infrastructure, and we're seeing some recovery.
The issue is still ongoing, and we will continue to provide updates as the cloud provider continues to work and resolve the issue.
Thank you!
Ed McKinzie
Services are showing significant signs of restoration following our infrastructure optimizations to redistribute traffic and restart affected servers, upwards of 90 percent recovery as we continue to increase processing capability. We're continuing to perform appropriate optimizations on the subset of remaining affected machines to resolve the issue for all users.
Nov 25, 2024, 1:52 PM CST
We're continuing to redirect traffic and apply targeted remediation to impacted servers. We've observed initially positive results in some areas, and we're monitoring service health closely as our mitigation workstreams continue in parallel.
Nov 25, 2024, 1:15 PM CST
Our mitigative actions haven't provided relief as expected, and a portion of infrastructure remains in an unhealthy state. We determined that some of the targeted server restarts did not succeed due to processing issues, which are under investigation. We’re currently focused on spreading traffic to healthy infrastructure, and we're seeing some recovery.
The issue is still ongoing, and we will continue to provide updates as the cloud provider continues to work and resolve the issue.
Thank you!
Ed McKinzie
Investigating
The affected scenarios are as follows:
1. Users may be unable to access Exchange Online using the following impacted connection methods:
• Outlook on the web
• Outlook desktop client
• Representational State Transfer (REST)
• Exchange ActiveSync (EAS)
2. Users may be unable to use the following features within Microsoft Teams:
• Users are unable to create or update Virtual Events, including webinars and Town Halls.
• Users may be unable to access or modify their calendar in Microsoft Teams. This would include loading calendar, viewing meetings, creating/updating meetings and joining meetings.
• Users are unable to create chat, add users and create or edited meetings.
• Users are unable to create or modify new teams and channels.
• Users may be unable to update presence.
• Users may be unable to use the search function.
• Users may not see updated list of files and links failing to load within the Chat shared tab.
3. Users may experience the following issues with Microsoft Purview:
• Users may be unable to access the Purview Portal, or Purview Solutions.
• Users may experience delays in policy stamping and with Adaptive Scope Evaluations.
4. Users may be unable to export content or set and view labels within Microsoft Fabric.
• Some Microsoft Fabric users with Purview Information Protection Policies with sensitivity labels enabled, may be unable to use interactive operations on Power BI Desktop format files and reports, including export operations on Fabric artifacts with Sensitivity labels applied.
5. Users may be unable to use the search feature within SharePoint Online.
6. Users may be unable to perform the following actions within Microsoft Defender for Office365.
• Users may be unable to create simulations, simulation payloads or end user notifications.
• Users may experience issues with delivery for end user notifications and simulation messages
• Some users may experience failures in manual or AIR approved Remediation Actions submitted through ThreatExplorer, Advanced Hunting or the Action Center.
• Users may experiences issues with viewing simulation reports, and content.
7. Users may also be unable to print via Universal Print.
8. Users may experience errors running flows that utilize cloud connectors in Power Automate for Desktop.
9. Users may be unable to access their bookings within Microsoft Bookings.
10. Users may be unable to use some Microsoft Copilot features including:
• Users are unable to use the personal Copilot panel in meetings and post meetings.
• Users are unable to see historic Copilot conversation history in meetings and post meetings.
Scope of impact
Impact is specific to users who are served through the affected infrastructure.
Preliminary root cause
A recent change has resulted in a portion of infrastructure not operating as expected.
Most Recent Microsoft Status Update:
Nov 25, 2024, 8:27 AM CST
We’re monitoring the progress of the fix, which has been deployed to approximately 60% of the affected environments. We’re continuing our manual restarts on the remaining impacted machines. This quick update is designed to give the latest information on this issue.
1. Users may be unable to access Exchange Online using the following impacted connection methods:
• Outlook on the web
• Outlook desktop client
• Representational State Transfer (REST)
• Exchange ActiveSync (EAS)
2. Users may be unable to use the following features within Microsoft Teams:
• Users are unable to create or update Virtual Events, including webinars and Town Halls.
• Users may be unable to access or modify their calendar in Microsoft Teams. This would include loading calendar, viewing meetings, creating/updating meetings and joining meetings.
• Users are unable to create chat, add users and create or edited meetings.
• Users are unable to create or modify new teams and channels.
• Users may be unable to update presence.
• Users may be unable to use the search function.
• Users may not see updated list of files and links failing to load within the Chat shared tab.
3. Users may experience the following issues with Microsoft Purview:
• Users may be unable to access the Purview Portal, or Purview Solutions.
• Users may experience delays in policy stamping and with Adaptive Scope Evaluations.
4. Users may be unable to export content or set and view labels within Microsoft Fabric.
• Some Microsoft Fabric users with Purview Information Protection Policies with sensitivity labels enabled, may be unable to use interactive operations on Power BI Desktop format files and reports, including export operations on Fabric artifacts with Sensitivity labels applied.
5. Users may be unable to use the search feature within SharePoint Online.
6. Users may be unable to perform the following actions within Microsoft Defender for Office365.
• Users may be unable to create simulations, simulation payloads or end user notifications.
• Users may experience issues with delivery for end user notifications and simulation messages
• Some users may experience failures in manual or AIR approved Remediation Actions submitted through ThreatExplorer, Advanced Hunting or the Action Center.
• Users may experiences issues with viewing simulation reports, and content.
7. Users may also be unable to print via Universal Print.
8. Users may experience errors running flows that utilize cloud connectors in Power Automate for Desktop.
9. Users may be unable to access their bookings within Microsoft Bookings.
10. Users may be unable to use some Microsoft Copilot features including:
• Users are unable to use the personal Copilot panel in meetings and post meetings.
• Users are unable to see historic Copilot conversation history in meetings and post meetings.
Scope of impact
Impact is specific to users who are served through the affected infrastructure.
Preliminary root cause
A recent change has resulted in a portion of infrastructure not operating as expected.
Most Recent Microsoft Status Update:
Nov 25, 2024, 8:27 AM CST
We’re monitoring the progress of the fix, which has been deployed to approximately 60% of the affected environments. We’re continuing our manual restarts on the remaining impacted machines. This quick update is designed to give the latest information on this issue.
This incident affected: Faculty/Staff Email, Student Email, Other MU IT Services, and Other UM IT Services.