Windows container version compatibility
Applies to: Windows Server 2022, Windows Server 2019, Windows Server 2016
Windows Server 2016 and Windows 10 Anniversary Update (both version 14393) were the first Windows releases that could build and run Windows Server containers. Containers built using these versions can run on newer releases, but there are a few things you need to know before you start.
The architecture of Windows differs vastly from that of Linux. Linux has a monolithic kernel, while in Windows User and Kernel mode are more tightly bound. Until the introduction of containers, Windows User and Kernel mode were shipped in synchrony, thus resulting in container compatibility requirements on Windows that differ from the norm in Linux.
Decoupling the User/Kernel boundary in Windows is a monumental task and highly non-trivial, however, we have been working hard to stabilize this boundary across all of Windows to provide our customers the flexibility to run down-level containers. Starting with Windows 11 and Windows Server 2022 we are enabling the ability to run process-isolated WS2022 containers on Windows 11 hosts. We've done our best to capture the areas which break the boundary, but now want to open the feature to developers on Windows 11 for feedback. We are committed to enabling this experience for you, so please let us know when you experience issues.
For any other scenario where there is a mismatch in host/guest Windows versioning compatibility between User/Kernel mode is possible, but not guaranteed, and thus the container image will be prevented from running on the host. For any mismatched version, running with Hyper-V isolation provides the container with a set of matching Kernel binaries and does not depend on the version of the host. See the tables below for a detailed compatibility matrix.
Windows Server host OS compatibility
Container base image OS version | Supports Hyper-V isolation | Supports process isolation |
---|---|---|
Windows Server 2022 | ✔ | ✔ |
Windows Server 2019 | ✔ | ❌ |
Windows Server 2016 | ✔ | ❌ |
Windows Client host OS compatibility
Container base image OS version | Supports Hyper-V isolation | Supports process isolation |
---|---|---|
Windows Server 2022 | ✔ | ✔(preview) |
Windows Server 2019 | ✔ | ❌ |
Windows Server 2016 | ✔ | ❌ |
Note
Windows 10 version 1809 and Windows Server 2019 had the same build number at GA moment. Since then, they received independent updates resulting in build number mismatch. Process isolation on Windows client is available in preview for Windows 11 with Windows Server 2022 images - with build number mismatch. If you have a requirement to run process isolated containers on a Windows 10, please let us know in our GitHub issues.
Matching container host version with container image versions
Windows Server containers
Build number (new release of Windows)
The Windows operating system has four levels of versioning: major, minor, build and revision. For example, version 10.0.14393.103 would have a major version of 10, a minor version of 0, a build number of 14393, and a revision number of 103. The build number only changes when new versions of the OS are published, and the revision number is updated as Windows updates are applied.
With the exception of WS2022 + Windows 11, Windows Server containers are blocked from starting when the build number between the container host and the container image are different. For example, when the container host is version 10.0.14393.* (Windows Server 2016) and you attempt to run a container with an image version 10.0.16299.* (Windows Server, version 1709) the OS compute service will return a version incompatibility error.
Windows Server 2016 Restrictions
Windows Server 2016-based containers will not run in a system where the revision numbers of the container host and the container image are different. For example, if the container host is version 10.0.14393.1914 (Windows Server 2016 with KB4051033 applied) and the container image is version 10.0.14393.1944 (Windows Server 2016 with KB4053579 applied), then the image might not start.
For hosts or images using Windows Server, version 1809 and later, this rule doesn't apply - the host and container image do not need matching revisions.
Note
We strongly recommend you update both your host and containers with the latest patches and updates to stay secure and compatible. For important guidance for how to update Windows containers, see Update Windows Server containers.
Practical application
Example 1: The container host is running Windows Server 2016 with KB4041691 applied. Any Windows Server container deployed to this host must be based on the version 10.0.14393.1770 container base images. If you apply KB4053579 to the host container, you must also update the images to make sure the host container supports them.
Example 2: The container host is running Windows Server version 1809 with KB4534273 applied. Any Windows Server container deployed to this host must be based on a Windows Server version 1809 (10.0.17763) container base image, but doesn't need to match the host KB. If KB4534273 is applied to the host, the container images will still be supported, but we recommend you update them to address any potential security issues.
Querying version
Method 1: Introduced in version 1709, the cmd prompt and ver command now return the revision details.
Microsoft Windows [Version 10.0.16299.125]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\>ver
Microsoft Windows [Version 10.0.16299.125]
Method 2: Query the following registry key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion
For example:
C:\>reg query "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion" /v BuildLabEx
Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
PS C:\Users\Administrator> (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\').BuildLabEx
14393.321.amd64fre.rs1_release_inmarket.161004-2338
To check what version your base image uses, review the tags on the Docker hub or the image hash table provided in the image description. The Windows 10 update history page lists when each build and revision was released.
Hyper-V isolation for containers
You can run Windows containers with or without Hyper-V isolation. Hyper-V isolation creates a secure boundary around the container with an optimized VM. Unlike standard Windows containers that share the kernel between containers and the host, each Hyper-V isolated container has its own instance of the Windows kernel. This means you can have different OS versions in the container host and image (for more information, see the following compatibility matrix).
To run a container with Hyper-V isolation, simply add the tag --isolation=hyperv
to your docker run command.
Errors from mismatched versions
If you try to run an unsupported combination, you'll get the following error:
docker: Error response from daemon: container b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839 encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839","Owner":"docker","IsDummy":false,"VolumePath":"\\\\?\\Volume{2443d38a-1379-4bcf-a4b7-fc6ad4cd7b65}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839","Layers":[{"ID":"1532b584-8431-5b5a-8735-5e1b4fe9c2a9","Path":"C:\\ProgramData\\docker\\windowsfilter\\b2b88bc2a47abcc682e422507abbba9c9b6d826d34e67b9e4e3144cc125a1f80"},{"ID":"a64b8da5-cd6e-5540-bc73-d81acae6da54","Path":"C:\\ProgramData\\docker\\windowsfilter\\5caaedbced1f546bccd01c9d31ea6eea4d30701ebba7b95ee8faa8c098a6845a"}],"HostName":"b81ed896222e","MappedDirectories":[],"HvPartition":false,"EndpointList":["002a0d9e-13b7-42c0-89b2-c1e80d9af243"],"Servicing":false,"AllowUnqualifiedDNSQuery":true}.
There are three ways you can resolve this error:
- Rebuild the container based on the correct version of
mcr.microsoft.com/microsoft-windows-nanoserver
ormcr.microsoft.com/windows/servercore
- If the host is newer, run docker run --isolation=hyperv ...
- Try running the container on a different host with the same Windows version
Choose which container OS version to use
Note
As of April 16, 2019, the "latest" tag is no longer published or maintained for the Windows Server, Windows Server Core, and Nano Server base OS container images. You must declare a specific tag when pulling or referencing images from these repos.
You must know which version you need to use for your container. For example, if you want Windows Server version 1809 as your container OS and want to have the latest patches for it, you should use the tag 1809
when specifying which version of the base OS container images you want, like so:
FROM mcr.microsoft.com/windows/nanoserver:1809
...
However, if you want a specific patch of Windows Server version 1809, you can specify the KB number in the tag. For example, to get a Nano Server base OS container image from Windows Server version 1809 with the KB4493509 applied to it, you would specify it like so:
FROM mcr.microsoft.com/windows/nanoserver:1809-KB4493509
...
You can also specify the exact patches you need with the schema we have used previously, by specifying the OS version in the tag:
FROM mcr.microsoft.com/windows/nanoserver:10.0.17763.437
...
The Server Core base images based on Windows Server 2022 and Windows Server 2019 are Long-Term Servicing Channel (LTSC) releases. If you for instance want Windows Server 2019 as your Server Core image's container OS and want to have the latest patches for it, you can specify LTSC releases like so:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
...
Matching versions using Docker Swarm
Docker Swarm doesn't currently have a built-in way to match the version of Windows that a container uses to a host with the same version. If you update the service to use a newer container, it will run successfully.
If you need to run multiple versions of Windows for a long period of time, there are two approaches you can take: either configure the Windows hosts to always use Hyper-V isolation or use label constraints.
Finding a service that won't start
If a service won't start, you'll see that the MODE
is replicated
but REPLICAS
will get stuck at 0. To see if the OS version is the problem, run the following commands:
Run docker service ls to find the service name:
ID NAME MODE REPLICAS IMAGE PORTS
xh6mwbdq2uil angry_liskov replicated 0/1 windows/servercore/iis
Run docker service ps (service name) to get the status and latest attempts:
C:\Program Files\Docker>docker service ps angry_liskov
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
klkbhn742lv0 angry_liskov.1 windows/servercore/iis WIN-BSTMQDRQC2E Ready Ready 3 seconds ago
y5blbdum70zo \_ angry_liskov.1 windows/servercore/iis WIN-BSTMQDRQC2E Shutdown Failed 24 seconds ago "starting container failed: co…"
yjq6zwzqj8kt \_ angry_liskov.1 windows/servercore/iis WIN-BSTMQDRQC2E Shutdown Failed 31 seconds ago "starting container failed: co…"
ytnnv80p03xx \_ angry_liskov.1 windows/servercore/iis WIN-BSTMQDRQC2E Shutdown Failed about a minute ago "starting container failed: co…"
xeqkxbsao57w \_ angry_liskov.1 windows/servercore/iis WIN-BSTMQDRQC2E Shutdown Failed about a minute ago "starting container failed: co…"
If you see starting container failed: ...
, you can see the full error with docker service ps --no-trunc (container name):
C:\Program Files\Docker>docker service ps --no-trunc angry_liskov
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
dwsd6sjlwsgic5vrglhtxu178 angry_liskov.1 windows/servercore/iis@sha256:868bca7e89e1743792e15f78edb5a73070ef44eae6807dc3f05f9b94c23943d5 WIN-BSTMQDRQC2E Running Starting less than a second ago
y5blbdum70zoh1f6uhx5nxsfv \_ angry_liskov.1 windows/servercore/iis@sha256:868bca7e89e1743792e15f78edb5a73070ef44eae6807dc3f05f9b94c23943d5 WIN-BSTMQDRQC2E Shutdown Failed 39 seconds ago "starting container failed: container e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0 encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0","Owner":"docker","VolumePath":"\\\\?\\Volume{2443d38a-1379-4bcf-a4b7-fc6ad4cd7b65}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0","Layers":[{"ID":"bcf2630f-ea95-529b-b33c-e5cdab0afdb4","Path":"C:\\ProgramData\\docker\\windowsfilter\\200235127f92416724ae1d53ed3fdc86d78767132d019bdda1e1192ee4cf3ae4"},{"ID":"e3ea10a8-4c2f-5b93-b2aa-720982f116f6","Path":"C:\\ProgramData\\docker\\windowsfilter\\0ccc9fa71a9f4c5f6f3bc8134fe3533e454e09f453de662cf99ab5d2106abbdc"},{"ID":"cff5391f-e481-593c-aff7-12e080c653ab","Path":"C:\\ProgramData\\docker\\windowsfilter\\a49576b24cd6ec4a26202871c36c0a2083d507394a3072186133131a72601a31"},{"ID":"499cb51e-b891-549a-b1f4-8a25a4665fbd","Path":"C:\\ProgramData\\docker\\windowsfilter\\fdf2f52c4323c62f7ff9b031c0bc3af42cf5fba91098d51089d039fb3e834c08"},{"ID":"1532b584-8431-5b5a-8735-5e1b4fe9c2a9","Path":"C:\\ProgramData\\docker\\windowsfilter\\b2b88bc2a47abcc682e422507abbba9c9b6d826d34e67b9e4e3144cc125a1f80"},{"ID":"a64b8da5-cd6e-5540-bc73-d81acae6da54","Path":"C:\\ProgramData\\docker\\windowsfilter\\5caaedbced1f546bccd01c9d31ea6eea4d30701ebba7b95ee8faa8c098a6845a"}],"HostName":"e7b5d3adba7e","HvPartition":false,"EndpointList":["298bb656-8800-4948-a41c-1b0500f3d94c"],"AllowUnqualifiedDNSQuery":true}"
This is the same error as CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101)
.
Fix - Update the service to use a matching version
There are two considerations for Docker Swarm. In the case where you have a compose file that has a service that uses an image you didn't create, you'll want to update the reference accordingly. For example:
version: '3'
services:
YourServiceName:
image: windows/servercore:1709
...
The other consideration is if the image you are pointing to is one that you've created yourself (for example, contoso/myimage):
version: '3'
services:
YourServiceName:
image: contoso/myimage
...
In this case, you should use the method described in Errors from mismatched versions to modify that dockerfile instead of the docker-compose line.
Mitigation - Use Hyper-V isolation with Docker Swarm
Windows containers supports using Hyper-V isolation on a per-container basis, which requires changing the Docker service configuration, and then restarting the Docker engine.
Edit
C:\ProgramData\docker\config\daemon.json
Add a line with
"exec-opts":["isolation=hyperv"]
Note
The daemon.json file does not exist by default. If you find that this is the case when you peek into the directory, you must create the file. Then you'll want to copy in the following:
{ "exec-opts":["isolation=hyperv"] }
Close and save the file, then restart the docker engine by running the following cmdlets in PowerShell:
Stop-Service docker Start-Service docker
After you've restarted the service, launch your containers. Once they're running, you can verify the isolation level of a container by inspecting the container with the following cmdlet:
docker inspect --format='{{json .HostConfig.Isolation}}' $instanceNameOrId
It will return either "process" or "hyperv". If you have modified and set your daemon.json as described above, it should show the latter.
Mitigation - Use labels and constraints
Here's how to use labels and constraints to match versions:
Add labels to each node.
On each node, add two labels:
OS
andOsVersion
. This assumes you're running locally but could be modified to set them on a remote host instead.docker node update --label-add OS="windows" $ENV:COMPUTERNAME docker node update --label-add OsVersion="$((Get-ComputerInfo).OsVersion)" $ENV:COMPUTERNAME
Afterwards, you can check those by running the docker node inspect command, which should show the newly added labels:
"Spec": { "Labels": { "OS": "windows", "OsVersion": "10.0.16296" }, "Role": "manager", "Availability": "active" }
Add a service constraint.
Now that you've labeled each node, you can update constraints that determine placement of services. In the following example, replace "contoso_service" with the name of your actual service:
docker service update \ --constraint-add "node.labels.OS == windows" \ --constraint-add "node.labels.OsVersion == $((Get-ComputerInfo).OsVersion)" \ contoso_service
This enforces and limits where a node may run.
To learn more about how to use service constraints, check out the service create reference.
Matching versions using Kubernetes
The same issue described in Matching versions using Docker Swarm can happen when pods are scheduled in Kubernetes. This issue can be avoided with similar strategies:
- Rebuild the container based on the same OS version in development and production. To learn how, see Choose which container OS version to use.
- Use node labels and nodeSelectors to make sure pods are scheduled on compatible nodes if both Windows Server 2016 and Windows Server version 1709 nodes are in the same cluster
- Use separate clusters based on OS version
Finding pods failed on OS mismatch
In this case, a deployment included a pod that was scheduled on a node with a mismatched OS version, and without Hyper-V isolation enabled.
The same error is shown in the events listed with kubectl describe pod <podname>
. After several attempts, the pod status will probably be CrashLoopBackOff
.
$ kubectl -n plang describe pod fabrikamfiber.web-789699744-rqv6p
Name: fabrikamfiber.web-789699744-rqv6p
Namespace: plang
Node: 38519acs9011/10.240.0.6
Start Time: Mon, 09 Oct 2017 19:40:30 +0000
Labels: io.kompose.service=fabrikamfiber.web
pod-template-hash=789699744
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"plang","name":"fabrikamfiber.web-789699744","uid":"b5062a08-ad29-11e7-b16e-000d3a...
Status: Running
IP: 10.244.3.169
Created By: ReplicaSet/fabrikamfiber.web-789699744
Controlled By: ReplicaSet/fabrikamfiber.web-789699744
Containers:
fabrikamfiberweb:
Container ID: docker://eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a
Image: patricklang/fabrikamfiber.web:latest
Image ID: docker-pullable://patricklang/fabrikamfiber.web@sha256:562741016ce7d9a232a389449a4fd0a0a55aab178cf324144404812887250ead
Port: 80/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: container eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a","Owner":"docker","IsDummy":false,"VolumePath":"\\\\?\\Volume{037b6606-bc9c-461f-ae02-829c28410798}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a","Layers":[{"ID":"f8bc427f-7aa3-59c6-b271-7331713e9451","Path":"C:\\ProgramData\\docker\\windowsfilter\\e206d2514a6265a76645b9d6b3dc6a78777c34dbf5da9fa2d564651645685881"},{"ID":"a6f35e41-a86c-5e4d-a19a-a6c2464bfb47","Path":"C:\\ProgramData\\docker\\windowsfilter\\0f21f1e28ef13030bbf0d87cbc97d1bc75f82ea53c842e9a3250a2156ced12d5"},{"ID":"4f624ca7-2c6d-5c42-b73f-be6e6baf2530","Path":"C:\\ProgramData\\docker\\windowsfilter\\4d9e2ad969fbd74fd58c98b5ab61e55a523087910da5200920e2b6f641d00c67"},{"ID":"88e360ff-32af-521d-9a3f-3760c12b35e2","Path":"C:\\ProgramData\\docker\\windowsfilter\\9e16a3d53d3e9b33344a6f0d4ed34c8a46448ee7636b672b61718225b8165e6e"},{"ID":"20f1a4e0-a6f3-5db3-9bf2-01fd3e9e855a","Path":"C:\\ProgramData\\docker\\windowsfilter\\7eec7f59f9adce38cc0a6755da58a3589287d920d37414b5b21b5b543d910461"},{"ID":"c2b3d728-4879-5343-a92a-b735752a4724","Path":"C:\\ProgramData\\docker\\windowsfilter\\8ed04b60acc0f65f541292a9e598d5f73019c8db425f8d49ea5819eab16a42f3"},{"ID":"2973e760-dc59-5800-a3de-ab9d93be81e5","Path":"C:\\ProgramData\\docker\\windowsfilter\\cc71305d45f09ce377ef497f28c3a74ee027c27f20657d2c4a5f157d2457cc75"},{"ID":"454a7d36-038c-5364-8a25-fa84091869d6","Path":"C:\\ProgramData\\docker\\windowsfilter\\9e8cde1ce8f5de861a5f22841f1ab9bc53d5f606d06efeacf5177f340e8d54d0"},{"ID":"9b748c8c-69eb-55fb-a1c1-5688cac4efd8","Path":"C:\\ProgramData\\docker\\windowsfilter\\8e02bf5404057055a71d542780a2bb2731be4b3707c01918ba969fb4d83b98ec"},{"ID":"bfde5c26-b33f-5424-9405-9d69c2fea4d0","Path":"C:\\ProgramData\\docker\\windowsfilter\\77483cedfb0964008c33d92d306734e1fab3b5e1ebb27e898f58ccfd108108f2"},{"ID":"bdabfbf5-80d1-57f1-86f3-448ce97e2d05","Path":"C:\\ProgramData\\docker\\windowsfilter\\aed2ebbb31ad24b38ee8521dd17744319f83d267bf7f360bc177e27ae9a006cf"},{"ID":"ad9b34f2-dcee-59ea-8962-b30704ae6331","Path":"C:\\ProgramData\\docker\\windowsfilter\\d44d3a675fec1070b61d6ea9bacef4ac12513caf16bd6453f043eed2792f75d8"}],"HostName":"fabrikamfiber.web-789699744-rqv6p","MappedDirectories":[{"HostPath":"c:\\var\\lib\\kubelet\\pods\\b50f0027-ad29-11e7-b16e-000d3afd2878\\volumes\\kubernetes.io~secret\\default-token-rw9dn","ContainerPath":"c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0}],"HvPartition":false,"EndpointList":null,"NetworkSharedContainerName":"586870f5833279678773cb700db3c175afc81d557a75867bf39b43f985133d13","Servicing":false,"AllowUnqualifiedDNSQuery":false}
Exit Code: 128
Started: Mon, 09 Oct 2017 20:27:08 +0000
Finished: Mon, 09 Oct 2017 20:27:08 +0000
Ready: False
Restart Count: 10
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rw9dn (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-rw9dn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rw9dn
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=windows
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
49m 49m 1 default-scheduler Normal Scheduled Successfully assigned fabrikamfiber.web-789699744-rqv6p to 38519acs9011
49m 49m 1 kubelet, 38519acs9011 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-rw9dn"
29m 29m 1 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning Failed Failed to pull image "patricklang/fabrikamfiber.web:latest": rpc error: code = 2 desc = Error response from daemon: {"message":"Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: no such host"}
49m 3m 12 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Pulling pulling image "patricklang/fabrikamfiber.web:latest"
33m 3m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Pulled Successfully pulled image "patricklang/fabrikamfiber.web:latest"
33m 3m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Created Created container
33m 2m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning Failed Error: failed to start container "fabrikamfiberweb": Error response from daemon: {"message":"container fabrikamfiberweb encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {\"SystemType\":\"Container\",\"Name\":\"fabrikamfiberweb\",\"Owner\":\"docker\",\"IsDummy\":false,\"VolumePath\":\"\\\\\\\\?\\\\Volume{037b6606-bc9c-461f-ae02-829c28410798}\",\"IgnoreFlushesDuringBoot\":true,\"LayerFolderPath\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\fabrikamfiberweb\",\"Layers\":[{\"ID\":\"f8bc427f-7aa3-59c6-b271-7331713e9451\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\e206d2514a6265a76645b9d6b3dc6a78777c34dbf5da9fa2d564651645685881\"},{\"ID\":\"a6f35e41-a86c-5e4d-a19a-a6c2464bfb47\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\0f21f1e28ef13030bbf0d87cbc97d1bc75f82ea53c842e9a3250a2156ced12d5\"},{\"ID\":\"4f624ca7-2c6d-5c42-b73f-be6e6baf2530\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\4d9e2ad969fbd74fd58c98b5ab61e55a523087910da5200920e2b6f641d00c67\"},{\"ID\":\"88e360ff-32af-521d-9a3f-3760c12b35e2\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\9e16a3d53d3e9b33344a6f0d4ed34c8a46448ee7636b672b61718225b8165e6e\"},{\"ID\":\"20f1a4e0-a6f3-5db3-9bf2-01fd3e9e855a\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\7eec7f59f9adce38cc0a6755da58a3589287d920d37414b5b21b5b543d910461\"},{\"ID\":\"c2b3d728-4879-5343-a92a-b735752a4724\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\8ed04b60acc0f65f541292a9e598d5f73019c8db425f8d49ea5819eab16a42f3\"},{\"ID\":\"2973e760-dc59-5800-a3de-ab9d93be81e5\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\cc71305d45f09ce377ef497f28c3a74ee027c27f20657d2c4a5f157d2457cc75\"},{\"ID\":\"454a7d36-038c-5364-8a25-fa84091869d6\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\9e8cde1ce8f5de861a5f22841f1ab9bc53d5f606d06efeacf5177f340e8d54d0\"},{\"ID\":\"9b748c8c-69eb-55fb-a1c1-5688cac4efd8\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\8e02bf5404057055a71d542780a2bb2731be4b3707c01918ba969fb4d83b98ec\"},{\"ID\":\"bfde5c26-b33f-5424-9405-9d69c2fea4d0\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\77483cedfb0964008c33d92d306734e1fab3b5e1ebb27e898f58ccfd108108f2\"},{\"ID\":\"bdabfbf5-80d1-57f1-86f3-448ce97e2d05\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\aed2ebbb31ad24b38ee8521dd17744319f83d267bf7f360bc177e27ae9a006cf\"},{\"ID\":\"ad9b34f2-dcee-59ea-8962-b30704ae6331\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\d44d3a675fec1070b61d6ea9bacef4ac12513caf16bd6453f043eed2792f75d8\"}],\"HostName\":\"fabrikamfiber.web-789699744-rqv6p\",\"MappedDirectories\":[{\"HostPath\":\"c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\b50f0027-ad29-11e7-b16e-000d3afd2878\\\\volumes\\\\kubernetes.io~secret\\\\default-token-rw9dn\",\"ContainerPath\":\"c:\\\\var\\\\run\\\\secrets\\\\kubernetes.io\\\\serviceaccount\",\"ReadOnly\":true,\"BandwidthMaximum\":0,\"IOPSMaximum\":0}],\"HvPartition\":false,\"EndpointList\":null,\"NetworkSharedContainerName\":\"586870f5833279678773cb700db3c175afc81d557a75867bf39b43f985133d13\",\"Servicing\":false,\"AllowUnqualifiedDNSQuery\":false}"}
33m 11s 151 kubelet, 38519acs9011 Warning FailedSync Error syncing pod
32m 11s 139 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning BackOff Back-off restarting failed container
Mitigation - using node labels and nodeSelector
Run kubectl get node to get a list of all nodes. After that, you can run kubectl describe node (node name) to get more details.
In the following example, two Windows nodes are running different versions:
$ kubectl get node
NAME STATUS AGE VERSION
38519acs9010 Ready 21h v1.7.7-7+e79c96c8ff2d8e
38519acs9011 Ready 4h v1.7.7-25+bc3094f1d650a2
k8s-linuxpool1-38519084-0 Ready 21h v1.7.7
k8s-master-38519084-0 Ready 21h v1.7.7
$ kubectl describe node 38519acs9010
Name: 38519acs9010
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_D2_v2
beta.kubernetes.io/os=windows
failure-domain.beta.kubernetes.io/region=westus2
failure-domain.beta.kubernetes.io/zone=0
kubernetes.io/hostname=38519acs9010
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Fri, 06 Oct 2017 01:41:02 +0000
...
System Info:
Machine ID: 38519acs9010
System UUID:
Boot ID:
Kernel Version: 10.0 14393 (14393.1715.amd64fre.rs1_release_inmarket.170906-1810)
OS Image:
Operating System: windows
Architecture: amd64
...
$ kubectl describe node 38519acs9011
Name: 38519acs9011
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_DS1_v2
beta.kubernetes.io/os=windows
failure-domain.beta.kubernetes.io/region=westus2
failure-domain.beta.kubernetes.io/zone=0
kubernetes.io/hostname=38519acs9011
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Fri, 06 Oct 2017 18:13:25 +0000
Conditions:
...
System Info:
Machine ID: 38519acs9011
System UUID:
Boot ID:
Kernel Version: 10.0 16299 (16299.0.amd64fre.rs3_release.170922-1354)
OS Image:
Operating System: windows
Architecture: amd64
...
Let's use this example to show how to match the versions:
Take note of each node name and
Kernel Version
from the system info.In our example, the info will look like this:
Name Version 38519acs9010 14393.1715.amd64fre.rs1_release_inmarket.170906-1810 38519acs9011 16299.0.amd64fre.rs3_release.170922-1354 Add a label to each node called
beta.kubernetes.io/osbuild
. Windows Server 2016 needs both major and minor versions (14393.1715 in this example) to be supported without Hyper-V isolation. Windows Server version 1709 only needs the major version (16299 in this example) to match.In this example, the command to add the labels looks like this:
$ kubectl label node 38519acs9010 beta.kubernetes.io/osbuild=14393.1715 node "38519acs9010" labeled $ kubectl label node 38519acs9011 beta.kubernetes.io/osbuild=16299 node "38519acs9011" labeled
Check the labels are there by running kubectl get nodes --show-labels.
In this example, the output will look like this:
$ kubectl get nodes --show-labels NAME STATUS AGE VERSION LABELS 38519acs9010 Ready,SchedulingDisabled 3d v1.7.7-7+e79c96c8ff2d8e beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=windows,beta.kubernetes.io/osbuild=14393.1715,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=38519acs9010 38519acs9011 Ready 3d v1.7.7-25+bc3094f1d650a2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_DS1_v2,beta.kubernetes.io/os=windows,beta.kubernetes.io/osbuild=16299,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=38519acs9011 k8s-linuxpool1-38519084-0 Ready 3d v1.7.7 agentpool=linuxpool1,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=k8s-linuxpool1-38519084-0,kubernetes.io/role=agent k8s-master-38519084-0 Ready 3d v1.7.7 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=k8s-master-38519084-0,kubernetes.io/role=master
Add node selectors to deployments. In this example case, we'll add a
nodeSelector
to the container spec withbeta.kubernetes.io/os
= windows andbeta.kubernetes.io/osbuild
= 14393.* or 16299 to match the base OS used by the container.Here's a full example for running a container built for Windows Server 2016:
apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: kompose.cmd: kompose convert -f docker-compose-combined.yml kompose.version: 1.2.0 (99f88ef) creationTimestamp: null labels: io.kompose.service: fabrikamfiber.web name: fabrikamfiber.web spec: replicas: 1 strategy: {} template: metadata: creationTimestamp: null labels: io.kompose.service: fabrikamfiber.web spec: containers: - image: patricklang/fabrikamfiber.web:latest name: fabrikamfiberweb ports: - containerPort: 80 resources: {} restartPolicy: Always nodeSelector: "beta.kubernetes.io/os": windows "beta.kubernetes.io/osbuild": "14393.1715" status: {}
The pod can now start with the updated deployment. The node selectors are also shown in
kubectl describe pod <podname>
, so you can run that command to verify they were added.The output for our example is as follows:
$ kubectl -n plang describe po fa Name: fabrikamfiber.web-1780117715-5c8vw Namespace: plang Node: 38519acs9010/10.240.0.4 Start Time: Tue, 10 Oct 2017 01:43:28 +0000 Labels: io.kompose.service=fabrikamfiber.web pod-template-hash=1780117715 Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"plang","name":"fabrikamfiber.web-1780117715","uid":"6a07aaf3-ad5c-11e7-b16e-000d3... Status: Running IP: 10.244.1.84 Created By: ReplicaSet/fabrikamfiber.web-1780117715 Controlled By: ReplicaSet/fabrikamfiber.web-1780117715 Containers: fabrikamfiberweb: Container ID: docker://c94594fb53161f3821cf050d9af7546991aaafbeab41d333d9f64291327fae13 Image: patricklang/fabrikamfiber.web:latest Image ID: docker-pullable://patricklang/fabrikamfiber.web@sha256:562741016ce7d9a232a389449a4fd0a0a55aab178cf324144404812887250ead Port: 80/TCP State: Running Started: Tue, 10 Oct 2017 01:43:42 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-rw9dn (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: default-token-rw9dn: Type: Secret (a volume populated by a Secret) SecretName: default-token-rw9dn Optional: false QoS Class: BestEffort Node-Selectors: beta.kubernetes.io/os=windows beta.kubernetes.io/osbuild=14393.1715 Tolerations: <none> Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 5m 5m 1 default-scheduler Normal Scheduled Successfully assigned fabrikamfiber.web-1780117715-5c8vw to 38519acs9010 5m 5m 1 kubelet, 38519acs9010 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-rw9dn" 5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Pulling pulling image "patricklang/fabrikamfiber.web:latest" 5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Pulled Successfully pulled image "patricklang/fabrikamfiber.web:latest" 5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Created Created container 5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Started Started container