The Basics of Single Instance Storage (SIS) in WSS 2003 R2 and WUDSS 2003
What is SIS
Single Instance Storage (SIS) is a feature of Windows Storage Server (WSS) 2003 R2 and Windows Unified Storage Server (WUDSS) 2003 that helps save disk space by de-duplicating files.
If you have two or more identical files in a volume, SIS will store a single copy of those bits, replacing the files with links (or “reparse points”) to that single store.
The files don’t need to be on the same folder, have the same name or have the same date, but they do need to be in the same volume, have exactly the same size and the contents of both need to be exactly the same.
As you enable SIS on each volume, a “SIS Common Store” folder will be created on that volume. This is where the single copy is stored (as a file with the SIS extension).
A service (called Groveler) will monitor the volume enabled for SIS to find and de-duplicate identical files.
The History of SIS
SIS was originally introduced as part of the RIS (Remote Installation Services) in Windows Server 2000, but at that time it was restricted to volumes used by RIS to hold multiple installations of Windows.
RIS has since been replaced by Windows Deployment Services (WDS), which leverages the WIM file format for the de-duplication of Windows installation files.
Today, SIS is only included as part of WSS 2003 R2 and WUDSS 2003.
SIS is not available in Windows XP, Windows Server 2003, Windows Server 2003 R2, Windows Vista or Windows Server 2008.
Checking if SIS is enabled
You can check if SIS is enabled on a volume by looking at the “Advanced” tab for the properties of that volume.
To get there, Click “Start” then “My Computer” and on the list of volumes right-click the disk and select “Properties”. Then go to the “Advanced” tab.
You should see something similar to one the windows below:
In the left you have a system without SIS enabled. On the right SIS is enabled on the system and the checked box indicates it’s also enabled for that specific volume.
You can also check if SIS is enabled using a command-line. You will use the SISADMIN.EXE tool for that. If you just run SISADMIN with no parameters, you will get:
C:\> sisadmin Valid commands: /? /h Displays usage information /i Install SIS (volumes must be specified) /f Set the groveler to foreground mode /b Set the groveler to background mode /r Initiate a full scan /e Enable and start the groveler service /d Stop and disable the groveler service /v Analyze volumes for overall SIS usage information /l List SIS controlled files /s Display information about SIS files Where: Is a list of volumes of the form "e: f: g: q:". If no volumes are specified, the operation will be performed on all volumes where SIS is enabled. Is a list of space delimited files. SIS is not enabled on any volumes. C:\>
Note the message on the SISADMIN output indicating that SIS is not enabled.
You can also check the status of the Groveler service. It is initially stopped and disabled.
C:\> sc qc Groveler [SC] QueryServiceConfig SUCCESS SERVICE_NAME: Groveler TYPE : 10 WIN32_OWN_PROCESS START_TYPE : 4 DISABLED ERROR_CONTROL : 1 NORMAL BINARY_PATH_NAME : C:\WINDOWS\system32\grovel.exe LOAD_ORDER_GROUP : TAG : 0 DISPLAY_NAME : Single Instance Storage Groveler DEPENDENCIES : SIS SERVICE_START_NAME : LocalSystem C:\>sc query Groveler SERVICE_NAME: Groveler TYPE : 10 WIN32_OWN_PROCESS STATE : 1 STOPPED (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)) WIN32_EXIT_CODE : 1077 (0x435) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0 C:\>
Enabling SIS
Before you enable SIS, you need to make sure the components are added to your system. This is done by default on WSS 2003 R2 and WUDSS 2003, but it does not hurt to check.
SIS should be in the component list you can access from “Control Panel”, “Add/Remove Programs”, “Add/Remove Windows Components”.
From there, select “Other Network File and Print Services”, click on “Details” and confirm that “Single Instance Storage” is checked like in the window below:
The next step is to enable SIS on the system. There is no GUI for that, so you will need to use the /i option of the SISADMIN command line tool, making sure to specify the volume (or volumes) where you want to install SIS.
There are a few restrictions. The volume must be formatted with the NTFS file system, it has to be a fixed disk and you cannot do it on the volume where Windows itself is installed. There is also a limit of 6 volumes per server.
Here is what you should see while enabling SIS on the E: disk:
C:\>sisadmin /i E: SIS has been initialized on volume 'E:'. Restarting the groveler service to begin groveling new SIS volumes. C:\>sc qc Groveler [SC] QueryServiceConfig SUCCESS SERVICE_NAME: Groveler TYPE : 10 WIN32_OWN_PROCESS START_TYPE : 2 AUTO_START ERROR_CONTROL : 1 NORMAL BINARY_PATH_NAME : C:\WINDOWS\system32\grovel.exe LOAD_ORDER_GROUP : TAG : 0 DISPLAY_NAME : Single Instance Storage Groveler DEPENDENCIES : SIS SERVICE_START_NAME : LocalSystem C:\>sc query Groveler SERVICE_NAME: Groveler TYPE : 10 WIN32_OWN_PROCESS STATE : 4 RUNNING (STOPPABLE, PAUSABLE, ACCEPTS_SHUTDOWN) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0 C:\>sisadmin /v Analyzing volume 'E:\'. Analyzing the SIS Common Store directory... Analyzing the SIS Common Store directory... 0 processed, complete. Sorting SIS common store file list... Sorting SIS common store file list... complete. Analyzing reparse points... Analyzing reparse points... 0 processed, 0 not analyzed, complete. Analyzing data... Analyzing data... complete. === Analysis of volume 'E:\' on Server1 === Common store files: 0 Link files: 0 Inaccessible link files: 0 Space saved: 0 KB C:\>
You can see that the “Groveler” service is now running and set to auto-start.
Testing SIS
Now we move on to actually prove that SIS is working, by creating a few duplicate files and verifying if the Groveler does its work.
At this point, all you need to do is add a few files that are copies of each other, wait for the Groveler to act and verify the space savings from SIS.
Here’s an example:
E:\>copy C:\WINDOWS\system32\shell32.dll E:\File1.DLL 1 file(s) copied. E:\>copy C:\WINDOWS\system32\shell32.dll E:\File2.DLL 1 file(s) copied. E:\>copy C:\WINDOWS\system32\shell32.dll E:\File3.DLL 1 file(s) copied. E:\>dir Volume in drive E is New Volume Volume Serial Number is E870-DC26 Directory of E:\ 11/07/2007 05:49 AM 8,360,448 File1.DLL 11/07/2007 05:49 AM 8,360,448 File2.DLL 11/07/2007 05:49 AM 8,360,448 File3.DLL 3 File(s) 25,081,344 bytes 0 Dir(s) 17,077,125,120 bytes free E:\>sisadmin /v Analyzing volume 'E:\'. Analyzing the SIS Common Store directory... Analyzing the SIS Common Store directory... 1 processed, complete. Sorting SIS common store file list... Sorting SIS common store file list... complete. Analyzing reparse points... Analyzing reparse points... 3 processed, 0 not analyzed, complete. Analyzing data... Analyzing data... complete. === Analysis of volume 'E:\' on Server1 === Common store files: 1 Link files: 3 Inaccessible link files: 0 Space saved: 16336 KB E:\>
You can also tell that a file has been de-duplicated by SIS by looking at its properties in Windows Explorer:
You can see that the size of the file does not change, but the “Size on disk” property shows only 4KB.
SISADMIN also has a command line to tell exactly which files have been moved to the “SIS Common Store” in more detail.
Here’s that output:
E:\>sisadmin /l E: Listing SIS controlled files on volume 'E:'. EE3C0955-AF46-11DC-BBF5-0003FF6F7322.sis <- E:\File1.DLL EE3C0955-AF46-11DC-BBF5-0003FF6F7322.sis <- E:\File2.DLL EE3C0955-AF46-11DC-BBF5-0003FF6F7322.sis <- E:\File3.DLL 3 SIS controlled files found on volume 'E:'. E:\>
The SIS Common Store
In case you’re wondering, the “SIS Common Store” folder has special permissions to prevent users from seeing it or changing it in any away.
What’s inside is just a set of configuration files, the database used by SIS for tracking changes to the volume and the SIS files themselves.
Here’s what it looks like:
You should not change the permissions of the “SIS Common Store” folder or attempt to modify those files directly.
Always use the SISADMIN tool to manage SIS.
Disabling SIS
You can use SISADMIN also to disable and stop the Groveler service from running. Here’s how you would do it:
E:\>sisadmin /d Stopping and disabling the groveler service. E:\>sc query Groveler SERVICE_NAME: Groveler TYPE : 10 WIN32_OWN_PROCESS STATE : 1 STOPPED (NOT_STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE : 0 (0x0) CHECKPOINT : 0x0 WAIT_HINT : 0x0 E:\>sc qc Groveler [SC] QueryServiceConfig SUCCESS SERVICE_NAME: Groveler TYPE : 10 WIN32_OWN_PROCESS START_TYPE : 4 DISABLED ERROR_CONTROL : 1 NORMAL BINARY_PATH_NAME : C:\WINDOWS\system32\grovel.exe LOAD_ORDER_GROUP : TAG : 0 DISPLAY_NAME : Single Instance Storage Groveler DEPENDENCIES : SIS SERVICE_START_NAME : LocalSystem E:\>
However, keep in mind that files already moved into the “SIS Common Store” won’t be moved out and the links (or “reparse points”) will continue to be there even after you have disabled the Groveler.
If you delete the “SIS Common Store” system folder or disable the SIS service (even after you disabled the Groveler service using SISADMIN), you will risk losing data.
There is no SISADMIN option to remove the SIS reparse points and the simplest way to accomplish that is to disable the Groveler and then perform a backup followed by a restore of the entire volume. You will need additional space in the volume as you perform that restore operation, since the previously de-duplicated files will be duplicated again.
Conclusion
SIS is a important feature of Windows Storage Server and Windows Unified Data Storage Server.
With the amount of file duplication created by users these days, you can expect to save a significant amount of disk space.
In fact, a Microsoft IT White Paper on their implementation of SIS pointed to savings of 25 to 40 percent on file servers. That’s hard to ignore…
Links and references
Here are a few additional links about the Single Instance Storage feature of WSS 2003 R2 and WUDSS 2003:
- https://download.microsoft.com/download/0/c/a/0cad7d83-2ef5-498a-af51-7911a10175b0/SIS_TWP.doc
White paper on how SIS was used by Microsoft IT - https://blogs.technet.com/filecab/archive/tags/Single+Instance+Store+_2800_SIS_2900_/default.aspx
Blog post on SIS from the “The Filing Cabinet” - https://research.microsoft.com/sn/Farsite/WSS2000.pdf
White Paper from Microsoft Research on the inner workings of SIS - https://blogs.technet.com/josebda/archive/2007/12/07/installing-the-evaluation-version-of-wudss-2003-with-the-microsoft-iscsi-software-target-on-a-vm.aspx
Step-by-step instructions on how to install an evaluation version of WUDSS on a VM
Comments
- Anonymous
January 01, 2003
Overview I am glad to share that Windows Storage Server 2008 (WSS 2008) with the Microsoft iSCSI Software - Anonymous
January 01, 2003
The comment has been removed - Anonymous
June 12, 2016
There is now an option to move files out of the “SIS Common Store” by using the new /u option. While SIS is good for space saving, it is known to cause series issue for backup. Not all backup solution support SIS and may end up backing up link files instead of actual files. Even for those that are SIS aware, there are complains over their poor performance on SIS storage.