Memory Leak in mstscax.dll (RDP Connection Workflow)

Question

Hi Team,

We’ve identified a memory leak in XenCenter related to repeated RDP connections/disconnections. Here’s the breakdown:

1. Issue: Memory usage grows indefinitely during repeated rdpClient.connect()/rdpClient.disconnect() cycles.

Root Cause (WPA): The leak originates from mstscax.dll (Windows RDP control).

Version Dependency:

Leak observed: mstscax.dll v10.0.26100.3037

  **No leak observed:** `mstscax.dll` v10.0.22621.4830
```**Questions:**

Why does the leak occur in v10.0.26100.3037 but not in v10.0.22621.4830?

What mitigation steps (code/workaround) would resolve this?

**Context:**

XenCenter references `MSTSCLib.dll`, which wraps the OS-provided `mstscax.dll`.

Suspect improper cleanup during RDP session teardown (e.g., unmanaged resource retention).

Feel free to reach out for more details. I appreciate it!

Answer 1

This is the WPA result screenshot.

Answer 2

Hello,

Below is some background and potential mitigation advice addressing your concerns:

──────────────────────────────

1. Why does the leak occur in v10.0.26100.3037 but not in v10.0.22621.4830?

• In v10.0.26100.3037 the leak appears to be caused by a change in how mstscax.dll manages and cleans up its internal resources when a session is torn down.

• Although both versions expose the same COM API via MSTSCLib.dll, the newer version (v10.0.26100.3037) seems to be retaining some unmanaged resources after a disconnect—likely due to incomplete cleanup of COM or related objects (e.g., internal window handles or RDP session state).

• In contrast, v10.0.22621.4830 correctly deallocates all resources after the rdpClient.disconnect() call. This suggests that either a bug was introduced in v10.0.26100.3037 or that internal changes (perhaps intended for performance or state reuse) inadvertently left a leak.

──────────────────────────────

2. What mitigation steps (code/workaround) would resolve this?

A. Explicit Resource Cleanup

 ○ In environments like .NET where you host MSTSCLib.dll, make sure you’re explicitly releasing COM objects. For example, after calling disconnect(), call Marshal.ReleaseComObject on your rdpClient (and any other related COM interfaces) to force cleanup.

 ○ Ensure that any event handlers or callbacks hooked to the RDP control are unsubscribed; lingering subscriptions can prevent garbage collection.

B. Forcing Disposal/GC (when appropriate)

 ○ If you’re using disposable wrappers for the RDP control, wrap its lifetime in using statements or explicitly call Dispose() after disconnect.

 ○ In cases where you’re certain that no more references exist, you might trigger GC.Collect/Garbage Collection post-disconnect. (Be cautious, though, as this is not a silver bullet but sometimes helps manage unmanaged leaks.)

C. Reuse RDP Sessions

 ○ Instead of creating and tearing down connections repeatedly, consider reusing the same RDP session instance wherever possible. If the leak only manifests during repeated instantiation, this can reduce frequency of leak-trigger conditions.

D. Roll Back or Side-Load an Older Version

 ○ If possible and acceptable from a support standpoint, use the known-good version (v10.0.22621.4830) of mstscax.dll until the issue is resolved in a future release.

 ○ Coordinate with your platform/OS vendor if side-loading (or the equivalent version control mechanism) is viable.

──────────────────────────────

In summary, the most direct code-level mitigations are to enforce explicit release of COM objects and possibly change the lifecycle of your RDP connections to avoid repeated create/dispose cycles.

---

If the Answer is helpful, please click "Accept Answer" and upvote it.