CNYRIC Incident Report

Citrix Outage 9/10/2026

All dates/times relative to EST

Summary

On September 10, 2026, CNYRIC experienced a service interruption starting at 7:20 AM, where end users encountered black screens when attempting to launch Citrix applications. Symptoms were consistent from an August 19th incident. Collaborating with Citrix Support, CNYRIC determined that the graphics driver for Citrix was crashing, and the service was fully restored by 11:58 AM after systematically reinstalling the Microsoft Visual C++ v14.51.36247 package across all Citrix servers. CNYRIC is currently working with Citrix to perform a deep-dive analysis of the collected trace logs to identify the underlying root cause of the graphics driver failure and prevent future occurrences.  Specific details of the technical troubleshooting are located below in the detailed section.

Start Time

7:20am

Resolved

11:58am

Duration

4 Hours 38 Minutes

Components affected

Citrix and citrix hosted applications

Detailed Description of Incident

At 7:20 AM, end users reported that launching Citrix applications resulted in a black screen due to application video rendering failures. CNYRIC immediately engaged Citrix Support and began reviewing the environment for recent changes. Log analysis revealed that all Citrix VDA servers had received Microsoft's September updates. These updates were applied in accordance with CNYRIC patching policies, as they contained critical fixes for two actively exploited Windows vulnerabilities. Because the symptoms matched a previous Citrix outage on August 19th, details and root cause analysis from that incident are provided below for context.

August 19th Root Cause:

On August 18, 2026, CNYRIC conducted scheduled maintenance to upgrade all Citrix servers and clients to version 25.07 CU2 to resolve security vulnerabilities and introduce new functionality. The upgrade was completed successfully at 12:30 AM on August 19th. However, post-upgrade testing revealed that launching Citrix applications displayed a black screen. A CNYRIC technician troubleshooted the issue until 4:30 AM before escalating to management and opening a critical case with Citrix Support. At 8:30 AM, a Citrix engineer established a troubleshooting bridge with the CNYRIC team and executed the following steps:

1. Modified registry keys related to hardware graphics acceleration, which did not resolve the issue.
2. Reviewed system and application logs on both Citrix control and application servers, finding no errors.
3. Captured session logs using CDF Control, which yielded no recorded errors.
4. Analyzed Process Monitor logs via CDF Control, discovering that the Citrix graphics driver was crashing due to an improper call to the Microsoft Visual C++ v14 standard library DLL (MSVCP140.dll).

This driver failure was confirmed as the root cause of the black screen issue. Unsuccessful initialization calls and silent memory corruption within the rendering thread prevented proper display rendering, matching user reports.

Upon identifying the affected DLL, Citrix and CNYRIC verified that Microsoft Visual C++ v14 Redistributable (version 14.51.36247) was installed and compatible with Citrix applications. Further file inspection revealed a version mismatch between the deployed MSVCP140.dll file and the package version included with the C++ v14.51.36247 installer. To resolve this, the CNYRIC team uninstalled and reinstalled Microsoft Visual C++ v14.51.36247, which restored normal functionality for users running Citrix Receiver client 25.07 LTSC. Remaining affected endpoints were resolved once updated to the 25.07 LTSC client release.

Leveraging lessons learned from the August 19th incident, the CNYRIC team verified the MSVCP140.dll version during the September 10th outage. The file version correctly matched the C++ v14.51.36247 package. At 8:08 AM, Citrix engineers joined a troubleshooting bridge to analyze logs and perform standard diagnostics. Given the recent Windows update deployment, Citrix requested a patch rollback; however, the rollback failed, and the black screen issue persisted. Following escalation to tier-3 support, Citrix engineers recommended reinstalling Microsoft Visual C++ v14.51.36247 based on the symptom profile. Reinstalling the package successfully resolved the black screen issue. Trace logs were captured from active servers prior to remediation to determine if the same MSVCP140.dll was resulting in the graphic crash. DLL crashed. CNYRIC teams systematically reinstalled C++ v14.51.36247 across all Citrix servers, fully restoring services by 11:58 AM on September 10th.

CNYRIC continues to collaborate with Citrix Support to analyze the collected trace logs and identify the root cause of the graphics driver crash. As logs showed the original issue with the version mismatch was not present during the September 10th outage.  Further details will be added as investigation results become available.

Update 9/11/2025

Further review of logs identified that the Citrix servers upon windows updates being applied has a cached C++ Runtime version 14.40.33810 in C:\ProgramData\Package Cache\. Though uninstall was run the cached version was not removed by the native windows uninstall process. These cached versions of Runtime were removed and server were rebooted and testing resulted in no Graphic driver crashes.

Remediation and Prevention

Review Root Cause Procedures: Evaluate the incident response workflow to ensure faster identification and escalation of recurring technical issues, minimizing downtime for end users.

Review Citrix Environment: Assess the Citrix environment for potential underlying issues that could impact performance or stability, and explore automated monitoring and reporting solutions.

Deep-Dive Log Analysis: Continue collaborating with Citrix Support to analyze the September 10th trace logs and determine the root cause of the graphics driver failure.