Internet Speed Throttled to 13.5Mb

Question

This is the first ever time I've used Microsoft Ask a Question, because this is the first ever time I've seen an issue like this and i'm prepared to log the hell of this to help others in the future as I cannot express my frustration.

The Setup

My laptop is wired in, it goes via one unmanaged gigabit switch down to the Unifi Security Gateway, which authenticates to my ISP via PPPoE.
I have a 1Gb FTTP (Fibre to the Premises) Internet connection with a 110Mb upload and on a typical day I can comfortably get 950Mb Down and 107 Up.
There is no QoS, bandwidth limitations or other impacting configurations set on any of my home infrastructure.

I connect through a Dock, Dell WD19 USB-C, which might sound like the cause, but I have also used a USB-A to Ethernet adaptor as well as a USB-C to Ethernet adaptor. There is no built in Ethernet with this laptop. Note that this laptop has worked perfectly fine up until the problem.

The Problem

As of Sunday 5th December 2021, my laptop was found to be off, despite me leaving it on over the weekend. I turned it back on without any issues but then discovered by internet speed was 13.5Mb/s (lower case b) on all speed tests, except Ookla.

Interestingly Ookla's Speedtest.net revealed speeds of up to 400Mb down and 98Up, which is still slower than normal, but gives me a good indication something is broken somewhere. All other speed tests tell me exactly 13.5Mb, consistently and actual download speed is indeed 13.5Mb, despite Speedtest.net's result of >400Mb, my actual download speed is indeed 13.5Mb.

The speed tests carried out are via Edge, Chrome and Firefox browsers.

It affects both Wired and Wireless. Yes, this is not just wired. 13.5Mb on both!

Iperf to external servers reveals 13.5Mbs, I am indeed throttled.

My upload is unaffected running >90Mb/s.
All other devices on my network are running happily at >900Mb/s.

I know the difference between Mb/s and MB/s, I'm measuring network performance not disk performance.
Yes, I've done sfc /scannow, if someone says that I might flip a lid.

Troubleshooting

Let's begin ruling stuff out.

• Windows 10 Operating System - Fully up to date, even optional updates included
• BIOS - Fully up to date
• Drivers (Dock's NIC, Chipset, Wireless Card) - Fully up to date
• Dock Drivers - Fully up to date
• Only this laptop effected. Other devices, including those that connect via the same switch, running at >900Mb/s
• Network card is capable, this laptop, in the same configuration, worked at >900Mb/s prior to the date it failed.
• Network cable replaced just in case, running now off a Cat 6a, massively overkill, still slow.
• IPerf tests reveal speeds to other computers in my network at 1Gb/s, telling me the network card, cabling and infrastructure is working fine.
• Linux Portable Distro, running on same laptop, in the same configuration, gets 1Gb/s internet fine.
• Reset Network card with all netsh commands already
• Changed policies to turn off Windows Update 80% bandwidth reservation
• Plugged in a new USB-C to Ethernet adapter, still get 13.5Mb/s
• Bypassed my Unifi Security Gateway and plugged straight into ISP provided router, same 13.5Mb/s
• Bypassed and routed directly PPPoE from my network adapter, thus exposing my laptop to the raw internet, turning my laptop into a router, still 13.5Mb/s
• At a friends home, still 13.5Mb/s.
• No Proxies or VPNs.
• Wireshark revealed 13.5Mb/s speed when reviewing bandwidth speeds, EXCEPT, when running speedtest.net of which I see higher speeds. ****.
• There are no running background processes utilising any networking.
• Firewall has been turned off and on
• Disconnected all USB Devices, still slow

Variables

My laptop is managed by my company's Intune. There are no bandwidth limitation policies in place, but this is a variable I cannot verify.

Conclusions

The fact my laptop can connect to local devices at 1Gb, rules out hardware (cable, switches, docks)
The fact my laptop is still limited at another location, rules out network configuration external to the laptop, including ISP
The fact that Linux Distro works at a Gigabit on the same laptop, tells me it's the Windows OS.

It seems that potentially Windows 10 is indeed throttling my internet connection.

Usually I would just rebuild at this stage, but for a change I want to find the root cause.
I've read people have rebuilt and it still remains, but that would surprise me.
I might have to do that, but naturally as a intune managed device, its going to be a pain.

Hardware

Dell Latitude 7420
i7 1185G7 (4 core, 8 threads)
16GB RAM
NVME 256GB

Answer 1

I have discovered the true cause of this problem.

My laptop has various Dell apps that have had no discovered negative effects up until recently, whereby an application known as Dell Optimizer (ironic), took control of my network card and limited it's performance to 13.5Mb/s.

Here is the video of evidence of this:

Dell Optimizer App - Throttling my Internet Connection

It led me to believe that the Windows Operating System was the cause as I was under the illusion that the only thing that had changed recently was Windows updates.
In fact, this application self updates and there is no evidence of any change to it or it’s version. Strange I know but I think the application has self repaired, restoring it to defaults, thus not changing any files but instead deleting its own config. Unfortunately, by default, is that it takes over the network adapter. What a stupid default.

It was discovered off the back of the fact that the only setting in the network adapter that made any impact was the Priority and VLAN setting.
Looking up what this setting does, it is indeed related to 802.1p (QoS). It seems that this Dell Optimizer tool is essentially a QoS management tool and when enabled, it bottlenecked my connection to 13.5Mb.

Priority and VLAN setting impacts the "Priority Level Definition", which in short is the configuration you can make within task manager where you can prioritise CPU Resources to specific services. I noticed when carrying out my speed tests to any internet location, not a LAN location, that the Dell Optimiser App began demonstrating CPU/Memory spikes along with networking traffic.

Once I found the app, I realised that it was enabled to "optimise" my network cards. (Wifi and Ethernet).
Once turned off my speed test immediately went back up to the actual speed provided to me by my Internet Service Provider.

May I pass on my thanks tremendously to @Anonymous for his incredible determination to help discover the cause by analysing deep level knowledge of network data.
It was always going to be something stupid, but I can let Microsoft off the hook with this one and point all my blame to the OEM Dell. It felt like software, it was software, case resolved.

Dell - Fix your Optimiser, what on earth is the point of it.

Answer 2

Hello @Anonymous ,

If you are prepared to share trace data, then I would be happy to look at it. There are a few threads where posters have agreed to share data (two that I could quickly find are https://learn.microsoft.com/en-us/answers/questions/330172/extreamly-slow-upload-speed-in-windows-all-other-o.html and https://learn.microsoft.com/en-us/answers/questions/390860/remote-connections-speed-limited-to-about-15-mb-s.html).

The trace data needed is more extensive than just a Wireshark trace. One command that captures the data is pktmon start --capture --comp nics --flags 0x17 --trace -p Microsoft-Windows-TCPIP -k 0x200500000000 -l 16 -p Microsoft-Windows-WFP -k 0x7FFFFFFFFFFFFFFF -l 255 --file-name why.etl and the capture can be stopped with the command pktmon stop.

Gary

Answer 3

Hello Dan,

Thanks for that. It will take a while to work through all of that data. The tool that I developed to investigate Windows 10/11 congestion control (affects upload speeds) shows:

The top-left graph (starting 26 seconds into the trace) shows an upload speed of about 110 megabytes in 10 seconds (11 megabytes per second, 90 megabits per second) - probably the speed test with Google.

The other graphs show four parallel connections (starting 76 seconds into the trace), each with an upload speed of about 25 megabytes in 10 seconds - probably the speed test with Ookla.

The test with WhatsApp appears to be using QUIC (HTTP/3, UDP).

The big differences are in the download speeds and I have not had a chance to look at those yet. It is getting late here in Switzerland - I will report on what I find out about the download speed tomorrow.

Gary

Answer 4

Hello Dan,

Here is an update on what I have found.

You performed 3 tests. The final test was a download from WhatsApp - this seems to have used QUIC. As you may know, most of the QUIC packet is encrypted - including the flow control information. It is difficult to perform performance analysis of this traffic without using tools specific to the QUIC implementation in use.

The first test was a speed test with Google. The trace data here appears to be incomplete, but the trace says no events were lost and there seems to be a pattern to the missing data. Event Tracing for Windows (ETW) is usually very reliable when reporting whether any events were lost, so I am still confused about what could have happened.

The second test was a speed test with Ookla. The trace command that I proposed for the Microsoft-Windows-TCPIP provider (which gives insights into what the TCP/IP stack is "thinking" (i.e. helps to explain its behaviour)) only specified the keywords ut:SendPath,ut:ConnectPath,ut:Summary (those are the symbolic names for the bits in 0x200500000000). I now regret not including ut:ReceivePath - this would provide insight into the TCP acknowledgement behaviour. The reason is that for the Ookla downloads, the Windows client only sends one acknowledgement for approximately every 4 full-sized incoming segments, whereas an acknowledgement for every second such segment would be usual. This is not a TCP protocol violation, but it does potentially cause difficulties for the congestion control mechanism on the sender (the Ookla server) which uses acknowledgement heuristics to judge the level of congestion. Apart from this, nothing in the behaviour of the client seems to reduce the speed of the download.

Adding ut:ReceivePath would also perhaps help understand what is happening in the Google test.

If you want to make a new trace (of the Google and Ookla tests), with ut:ReceivePath data included, 0x200500000000 needs to be changed to 0x200700000000 ( pktmon start --capture --comp nics --flags 0x17 --trace -p Microsoft-Windows-TCPIP -k 0x200700000000 -l 16 -p Microsoft-Windows-WFP -k 0x7FFFFFFFFFFFFFFF -l 255 --file-name why.etl ).

Gary

Answer 5

Hello Dan,

Thanks for that new trace data and the very useful accompanying description. You can extract Wireshark compatible data from the trace file by using the pktmon etl2pcap command.

There is a lot to be reviewed in the trace data, but here is one trace extract that caught my eye (in the Google download speed test), concerning the 2 segment ssequence from 2310381109 to 2310384013:

10:56:22.571831 212.113.31.37.443 > 192.168.1.152.62194: . 2310381109:2310382561(1452) ack 4142586958 win 501
10:56:22.571832 212.113.31.37.443 > 192.168.1.152.62194: P 2310382561:2310384013(1452) ack 4142586958 win 501
10:56:22.573428 212.113.31.37.443 > 192.168.1.152.62194: . 2310384013:2310385465(1452) ack 4142586958 win 501
10:56:22.573429 212.113.31.37.443 > 192.168.1.152.62194: P 2310385465:2310386917(1452) ack 4142586958 win 501
10:56:22.574990 212.113.31.37.443 > 192.168.1.152.62194: . 2310386917:2310388369(1452) ack 4142586958 win 501
10:56:22.574990 212.113.31.37.443 > 192.168.1.152.62194: P 2310388369:2310389821(1452) ack 4142586958 win 501
10:56:22.576647 212.113.31.37.443 > 192.168.1.152.62194: . 2310389821:2310391273(1452) ack 4142586958 win 501
10:56:22.576648 212.113.31.37.443 > 192.168.1.152.62194: P 2310391273:2310392725(1452) ack 4142586958 win 501
10:56:22.578081 212.113.31.37.443 > 192.168.1.152.62194: . 2310392725:2310394177(1452) ack 4142586958 win 501
10:56:22.578082 212.113.31.37.443 > 192.168.1.152.62194: P 2310394177:2310395629(1452) ack 4142586958 win 501
10:56:22.579819 212.113.31.37.443 > 192.168.1.152.62194: . 2310395629:2310397081(1452) ack 4142586958 win 501
10:56:22.579820 212.113.31.37.443 > 192.168.1.152.62194: P 2310397081:2310398533(1452) ack 4142586958 win 501
10:56:22.581200 212.113.31.37.443 > 192.168.1.152.62194: . 2310398533:2310399985(1452) ack 4142586958 win 501
10:56:22.581200 212.113.31.37.443 > 192.168.1.152.62194: P 2310399985:2310401437(1452) ack 4142586958 win 501
10:56:22.582762 212.113.31.37.443 > 192.168.1.152.62194: . 2310401437:2310402889(1452) ack 4142586958 win 501
10:56:22.582763 212.113.31.37.443 > 192.168.1.152.62194: P 2310402889:2310404341(1452) ack 4142586958 win 501
10:56:22.584259 212.113.31.37.443 > 192.168.1.152.62194: . 2310404341:2310405793(1452) ack 4142586958 win 501
10:56:22.584260 212.113.31.37.443 > 192.168.1.152.62194: P 2310405793:2310407245(1452) ack 4142586958 win 501
10:56:22.585787 212.113.31.37.443 > 192.168.1.152.62194: . 2310407245:2310408697(1452) ack 4142586958 win 501
10:56:22.585787 212.113.31.37.443 > 192.168.1.152.62194: P 2310408697:2310410149(1452) ack 4142586958 win 501
10:56:22.587361 212.113.31.37.443 > 192.168.1.152.62194: . 2310410149:2310411601(1452) ack 4142586958 win 501
10:56:22.587362 212.113.31.37.443 > 192.168.1.152.62194: P 2310411601:2310413053(1452) ack 4142586958 win 501
10:56:22.588959 212.113.31.37.443 > 192.168.1.152.62194: . 2310413053:2310414505(1452) ack 4142586958 win 501
10:56:22.588959 212.113.31.37.443 > 192.168.1.152.62194: P 2310414505:2310415957(1452) ack 4142586958 win 501
10:56:22.590487 212.113.31.37.443 > 192.168.1.152.62194: . 2310415957:2310417409(1452) ack 4142586958 win 501
10:56:22.590487 212.113.31.37.443 > 192.168.1.152.62194: P 2310417409:2310418861(1452) ack 4142586958 win 501
10:56:22.592025 212.113.31.37.443 > 192.168.1.152.62194: . 2310418861:2310420313(1452) ack 4142586958 win 501
10:56:22.592026 212.113.31.37.443 > 192.168.1.152.62194: P 2310420313:2310421765(1452) ack 4142586958 win 501
10:56:22.593605 212.113.31.37.443 > 192.168.1.152.62194: . 2310421765:2310423217(1452) ack 4142586958 win 501
10:56:22.593605 212.113.31.37.443 > 192.168.1.152.62194: P 2310423217:2310424669(1452) ack 4142586958 win 501
10:56:22.595152 212.113.31.37.443 > 192.168.1.152.62194: . 2310424669:2310426121(1452) ack 4142586958 win 501
10:56:22.595152 212.113.31.37.443 > 192.168.1.152.62194: P 2310426121:2310427573(1452) ack 4142586958 win 501
10:56:22.595235 192.168.1.152.62194 > 212.113.31.37.443: . ack 2310331741 win 2053 (DF)
10:56:22.595277 192.168.1.152.62194 > 212.113.31.37.443: . ack 2310337549 win 2053 (DF)
10:56:22.595295 192.168.1.152.62194 > 212.113.31.37.443: . ack 2310340453 win 2053 (DF)
10:56:22.595329 192.168.1.152.62194 > 212.113.31.37.443: . ack 2310343357 win 2053 (DF)
10:56:22.595343 192.168.1.152.62194 > 212.113.31.37.443: . ack 2310346261 win 2053 (DF)
10:56:22.595355 192.168.1.152.62194 > 212.113.31.37.443: . ack 2310349165 win 2053 (DF)
10:56:22.595375 192.168.1.152.62194 > 212.113.31.37.443: . ack 2310352069 win 2053 (DF)
10:56:22.596615 212.113.31.37.443 > 192.168.1.152.62194: . 2310427573:2310429025(1452) ack 4142586958 win 501
10:56:22.596615 212.113.31.37.443 > 192.168.1.152.62194: P 2310429025:2310430477(1452) ack 4142586958 win 501
10:56:22.598379 212.113.31.37.443 > 192.168.1.152.62194: . 2310430477:2310431929(1452) ack 4142586958 win 501
10:56:22.598380 212.113.31.37.443 > 192.168.1.152.62194: P 2310431929:2310433381(1452) ack 4142586958 win 501
10:56:22.599753 212.113.31.37.443 > 192.168.1.152.62194: . 2310433381:2310434833(1452) ack 4142586958 win 501
10:56:22.599754 212.113.31.37.443 > 192.168.1.152.62194: P 2310434833:2310436285(1452) ack 4142586958 win 501
10:56:22.634427 192.168.1.152.62194 > 212.113.31.37.443: . ack 2310384013 win 2053 (DF)

The 34 millisecond delay at the end of the trace extract and total of 62 millisecond delay between receiving the packet and acknowledging it are both very long.

The ETW trace shows this:

The reason for the long delay seems to be that the TCP/IP stack was first informed about the arrival of the data after 62 milliseconds in an RSC (Receive Segment Coalescing) event.

It is much too early in the analysis to draw any firm conclusions, but if you have the time and inclination, you could try experimenting with RSC, perhaps using the PowerShell cmdlets Get-NetAdapterRsc and Set-NetAdapterRsc.

Gary