Script recipe of the week: how to copy an opened file

發行項
01/05/2005

I just read recently a post on the newsgroups - someone wanted to backup her PST file. The problem is that Outlook keeps this file open all the time, therefore we are unable to just copy this file once in a while. Argh...

A cute little script

Fortunately, we have Shadow copies in XP and Server 2003! So I just wrote a quick script that allows this PST to be quickly backed up. The script (called CopyWithVss.cmd) has a syntax which is very similar with the COPY command in shell.

CopyWithVss source_file destination file

Here is the script contents:

setlocal

@REM test if we are called by VSHADOW
if NOT "%CALLBACK_SCRIPT%"=="" goto :IS_CALLBACK

@REM
@REM Get the source and destination path
@REM

set SOURCE_DRIVE_LETTER=%~d1
set SOURCE_RELATIVE_PATH=%~pnx1
set DESTINATION_PATH=%2

@REM
@REM Create the shadow copy - and generate env variables into a temporary script.
@REM
@REM Then, while the shadow is still live
@REM recursively execute the same script.
@REM

@echo ...Determine the scripts to be executed/generated...

set CALLBACK_SCRIPT=%~dpnx0
set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd

@echo ...Creating the shadow copy...

%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%

del /f %TEMP_GENERATED_SCRIPT%

@goto :EOF

:IS_CALLBACK
setlocal

@REM
@REM This generated script should set the SHADOW_DEVICE_1 env variable
@REM

@echo ...Obtaining the shadow copy device name...

call %TEMP_GENERATED_SCRIPT%

@REM
@REM This should copy the file to the right location
@REM

@echo ...Copying from the shadow copy to the destination path...

copy "%SHADOW_DEVICE_1%\%SOURCE_RELATIVE_PATH%" %DESTINATION_PATH%

How does it work?

Let's go through all the instructions, step by step.

1) The initialization phase:

setlocal

@REM test if we are called by VSHADOW
if NOT "%CALLBACK_SCRIPT%"=="" goto :IS_CALLBACK

@REM
@REM Get the source and destination path
@REM

set SOURCE_DRIVE_LETTER=%~d1
set SOURCE_RELATIVE_PATH=%~pnx1
set DESTINATION_PATH=%2

Here, "setlocal" is used to make sure that any variables that we set in this script will not be defined after its execution is done (we'll see later why). Let's skip the "if NOT" test for a second, and go to the next SET statements. Here, we are just interpreting the script arguments. So, if the script was invoked with the arguments CopyWithVss c:\path1\foo.txt d:\path2\bar.txt then, %1 will be c:\path1\foo.txt, and %2 will be d:\path2\bar.txt.

And the weird-looking expression "%~d1" is nothing more than the drive letter of the path contained in %1. ("d" stands for the "drive letter"). Therefore %~d1 will become c: . How about %~pnx1 ? Similarly, "p" stands for "path", "n" for "name", and "x" for extension. And more, %~pnx1 is just a shortcut for %~p1%~n1%~x1 which represents the concatenated path, name and extension of the file contained in %1. In other words, the string path1\foo.txt.

For more documentation on these expressions, you can do a "CALL /?" in your CMD shell.

2) Let's continue...

@echo ...Determine the scripts to be executed/generated...

set CALLBACK_SCRIPT=%~dpnx0
set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd

Now, similarly with the logic above, the variable CALLBACK_SCRIPT will be set to the full path (including drive letter, path, name and extension) of the actual script (since %0 is the name of the currently executing script). So, if you execute the script from z:\foo\CopyWithVss.cmd, then %~dpnx0 will represent this fully qualified path. Why we are doing all this manipulation will become apparent in a second.

3) The following line is the fun part:

%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%

Now, what is this vshadow.exe tool? It is a simple command-line tool that allows anyone to create shadow copies. You can find the tool already compiled in the latest downloadable VSS SDK. Just copy VSHADOW.EXE in the same directory as the CMD above. What this tool does is that it creates a shadow copy of the volume specified as its last parameter. Aha! so this is why we had to isolate the drive letter of the source path. To go with the example above (%1 being c:\path1\foo.txt), then the last parameter is the C: drive letter, meaning that we will create a shadow copy of C:. Remember that shadow copies are per volume, not per file or directory! But don't worry, the shadow copy creation is a relatively painless operation in terms of performance. Most likely it takes less than the time needed to copy your PST file :-)

But what is a shadow copy, anyway? It is a regular volume that appears in the system for a brief time (at least in XP, because in Windows Server 2003 you have the ability to create a persistent shadow copy - just use the "-p" option in VSHADOW for example). This volume is identified by a temporary device name, looking like this: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253. The fact that the volume is somewhat temporary makes our script a little bit harder. How do we retrieve the shadow copy device in our CMD script? VSHADOW solves this in a simple way, offering you the ability to generate a temporary script, containing a bunch of SET commands that will save the new devices as environment variables. Then, you can simply refer these devices in some shell commands, like COPY. So if you execute this script, it will set a variable called SHADOW_DEVICE_1 (and subsequently SHADOW_DEVICE_2,... N) for each volume to be shadowed. So this is the explanation for the "-script" option above: it will generate a script called GeneratedVarsTempScript.cmd that we have to execute to get the actual shadow device.

Now, here is another challenging problem: the shadow copy device dissapears immediately after the VSHADOW program exits! This is by design with all non-persistent shadow copies created by VSS. Internally, VSHADOW acts as a COM client, and the VSS service as a COM server. When the last referece to the COM interface referred by VSHADOW goes away, all the created shadow copies are going away too. In fact, this is a convenient feature for a backup application; for example if that application crashes, then you won't have leaking shadow copies: COM will automatically release the COM reference after a 6 minutes idle time, therefore automatically deleting the shadow copy.

But, this seems to be an impossible problem now - if the shadow copy is gone by the time VSHADOW exits, how we can still access its contents? Fortunately, VSHADOW has the ability to execute a "callback" program while the shadow copy is still alive. This callback script is passed using the second "-exec" option. Whew!

So what is being passed in the "-exec" option? Nothing more than the fully-qualified path of the original script! In other words, the script is recursively calling itself. We avoid the infinite cycle in a very easy way. Remember the "if NOT" instruction at the beginning of the script?

@REM test if we are called by VSHADOW
if NOT "%CALLBACK_SCRIPT%"=="" goto :IS_CALLBACK

That does the trick - since the env variable CALLBACK_SCRIPT is defined inside our script, it is (almost) guaranteed to not be set when the script is invoked for the first time and being set when invoked recursively for the second time. This is a neat trick that allowed us to avoid defining a separate CMD script. At the same time, the test allows us to detect what to do when we are inside VSHADOW... remember, all we wanted to do is to copy a file. This also explains why do we need a "setlocal" at the beginning of the script - otherwise, succesive executions in the same CMD instance will simply not work since the first execution will leave CALLBACK_SCRIPT defined, and the script will think that it is already in VSHADOW!

All this seems pretty complicated, but if you think a little bit about it, you can see that there is no other way around it. This is actually the simplest way to execute some custom commands while the shadow copy is still alive...

4) Let's jump now to the :IS_CALLBACK part - this is what is being executed when we are called by VSHADOW:

:IS_CALLBACK
setlocal

@REM
@REM This generated script should set the SHADOW_DEVICE_1 env variable
@REM

@echo ...Obtaining the shadow copy device name...

call %TEMP_GENERATED_SCRIPT%

@REM
@REM This should copy the file to the right location
@REM

@echo ...Copying from the shadow copy to the destination path...

copy "%SHADOW_DEVICE_1%\%SOURCE_RELATIVE_PATH%" %DESTINATION_PATH%

We can see above the call to the generated script (that should define the SHADOW_DEVICE_1 environment variable). Then, we just invoke the COPY command, passing the source and the destination path. Now it is clear why we needed the SOURCE_RELATIVE_PATH variable in the first place - since the file path on the shadow copy is actually relative to the root of the shadow copy volume.

5) Now, let's jump back to the original code branch:

%~dp0\vshadow.exe -script=%TEMP_GENERATED_SCRIPT% -exec=%CALLBACK_SCRIPT% %SOURCE_DRIVE_LETTER%

del /f %TEMP_GENERATED_SCRIPT%

@goto :EOF

Obviously, this deletes the temporarily-generated CMD script and exits the main script. Done!

Let's give it a try

Nothing easier... here is the generated output.

   Z:\outlook_backup>CopyWithVss.cmd x:\Store\PST\2004-01-27.pst x:\Store\PST\backup.pst Z:\outlook_backup>setlocal Z:\outlook_backup>if NOT "" == "" goto :IS_CALLBACK Z:\outlook_backup>set SOURCE_DRIVE_LETTER=x: Z:\outlook_backup>set SOURCE_RELATIVE_PATH=\Store\PST\2004-01-27.pst Z:\outlook_backup>set DESTINATION_PATH=x:\Store\PST\backup.pst
...Determine the scripts to be executed/generated... Z:\outlook_backup>set CALLBACK_SCRIPT=Z:\outlook_backup\CopyWithVss.cmd Z:\outlook_backup>set TEMP_GENERATED_SCRIPT=GeneratedVarsTempScript.cmd
...Creating the shadow copy... Z:\outlook_backup>Z:\outlook_backup\\vshadow.exe -script=GeneratedVarsTempScript.cmd -exec=Z:\outlook_backup\CopyWithVss.cmd x: VSHADOW.EXE 2.0 - Volume Shadow Copy sample client
Copyright (C) 2004 Microsoft Corporation. All rights reserved. 
(Option: Generate SETVAR script 'GeneratedVarsTempScript.cmd')
(Option: Execute binary/script after shadow creation 'Z:\outlook_backup\CopyWithVss.cmd')
(Option: Create shadow copy set)
(Gathering writer metadata...)
(Waiting for the asynchronous operation to finish...)
Initialize writer metadata ...
Discover directly excluded components ...
- Excluding writer 'MSDEWriter' since it has no selected components for restore.
Discover components that reside outside the shadow set ...
- Component '\System Files' from writer 'System Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\WMI' from writer 'WMI Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\Event Logs' from writer 'Event Log Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\Registry' from writer 'Registry Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\COM+ REGDB' from writer 'COM+ REGDB Writer' is excluded from backup (it requires C:\ in the shadow set)
- Component '\IISMETABASE' from writer 'IIS Metabase Writer' is excluded from backup (it requires C:\ in the shadow set)
Discover all excluded components ...
Discover excluded writers ...
- The writer 'System Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'WMI Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'Event Log Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'Registry Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'COM+ REGDB Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
- The writer 'IIS Metabase Writer' is now entirely excluded from the backup:
  (it does not contain any components that can be potentially included in the backup)
Discover explicitly included components ...
Verifying explicitly specified writers/components ...
Select explicitly included components ...
Creating shadow set {e8d4d81c-337a-4df1-bda0-b7be96c5e0da} ...
- Adding volume \\?\Volume{785cc4a6-3d68-11d7-9cc5-505054503030}\ [X:\] to the shadow set...
Preparing for backup ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...)
Creating the shadow (DoSnapshotSet) ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...)
Shadow copy set succesfully created. List of created shadow copies: 
Querying all shadow copies with the SnapshotSetID {e8d4d81c-337a-4df1-bda0-b7be96c5e0da} ... * SNAPSHOT ID = {0b2fc776-3ebc-459c-9437-af02b4fbd46f} ...
   - Shadow copy Set: {e8d4d81c-337a-4df1-bda0-b7be96c5e0da}
   - Original count of shadow copies = 1
   - Original Volume name: \\?\Volume{785cc4a6-3d68-11d7-9cc5-505054503030}\ [X:\]
   - Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253
   - Originating machine: aoltean-dev.ntdev.corp.microsoft.com
   - Service machine: aoltean-dev.ntdev.corp.microsoft.com
   - Not Exposed
   - Provider id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   - Attributes:  Auto_Release Differential Generating the SETVAR script (GeneratedVarsTempScript.cmd) ...
Executing command 'Z:\outlook_backup\CopyWithVss.cmd' ...
----------------------------------------------------- Z:\outlook_backup>setlocal Z:\outlook_backup>if NOT "Z:\outlook_backup\CopyWithVss.cmd" == "" goto :IS_CALLBACK Z:\outlook_backup>setlocal
...Obtaining the shadow copy device name... Z:\outlook_backup>call GeneratedVarsTempScript.cmd [This script is generated by VSHADOW.EXE for the shadow set {e8d4d81c-337a-4df1-bda0-b7be96c5e0da}] 
Z:\outlook_backup>SET SHADOW_SET_ID={e8d4d81c-337a-4df1-bda0-b7be96c5e0da} Z:\outlook_backup>SET SHADOW_ID_1={0b2fc776-3ebc-459c-9437-af02b4fbd46f} Z:\outlook_backup>SET SHADOW_DEVICE_1=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253
...Copying from the shadow copy to the destination path... Z:\outlook_backup>copy "\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy253\\Store\PST\2004-01-27.pst"  x:\Store\PST\backup.pst<br>        1 file(s) copied. 
-----------------------------------------------------
- Mark all writers as succesfully backed up...
Completing the backup (BackupComplete) ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...) Snapshot creation done. Z:\outlook_backup>del /f GeneratedVarsTempScript.cmd

In the end, we copied the original PST file from x:\Store\PST\2004-01-27.pst into x:\Store\PST\backup.pst (even the original PST was opened by Outlook).

That's it!

WARNING: Note that there are almost identical two versions of this tool: one for XP, and another one for Server 2003. Do not attempt to run the XP version on server or viceversa - it won't work!

Comments

Anonymous
January 04, 2005
Very useful script !.
I sucess to do it.

but, If SQL Server service are running,
CopyWithVss is failed.

ERROR Messages is:
----
ERROR: Selected writer 'MSDEWriter' is in failed state!
- Status: 8 (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)
- Writer Failure code: 0x800423f4 (<Unknown error code>)
- Writer ID: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
- Instance ID: {ef567f78-4f42-49be-bb1c-2386a7f489cf}
----

I want to copy Database/Log Files by this script, because I'm MVP for SQL Server.

Regards,
Anonymous
January 05, 2005
>> ERROR Messages is:
----
ERROR: Selected writer 'MSDEWriter' is in failed state!
- Status: 8 (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)
- Writer Failure code: 0x800423f4 (<Unknown error code>)
- Writer ID: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
- Instance ID: {ef567f78-4f42-49be-bb1c-2386a7f489cf}

Intersting - this should not happen. Are there any VSS errors in the Application event logs? If yes, please cut & paste them in your reply.

Note that with SQL, if the database and the logs are on different volumes, then you might want to change the script to snapshot simultaneously both the database and the log volumes. If everything is on the same volume, it should still work.

By the way, the VSS SDK contains a set of test VSS scripts that can be used to backup and restore a SQL/MSDE database, and then verify that the restore was succesful.

Thanks, Adi
Anonymous
January 05, 2005
Hello, Adi

Application Event log:
Type: ERROR
SOURCE: VSS
Category: non
Event: : 6004
User: N/A
Description:
"Sqllib ERROR: Database testdb is not simple".
** I use Japanese Environment (OS, SQL Server).
Therefore, real errormessage is in japanese,

If i detach database 'testdb' by SQL Server Enterprise Manger, Or If I change 'testdb' recovery mode from 'full' to 'simple',
CopyWithVSS command do complete !.

Therefore,
if any database's recovery mode is not simple, CopyWithVSS fail.

Regards,
Anonymous
January 05, 2005
I have two more questions: Do you have XP or Windows Server on your system? Also, is this a SQL Server 2000 database?

Thanks, Adi
Anonymous
January 05, 2005
Hello, Adi

Yes, I have
Windows XP Pro + SP2 (Japanese Edition)
Windows Server 2003 Std (Japanese Edition)
SQL Server 2000 Std SP3a (Japanese Edition)

and, this problem seems to relate below kb.

*Error 800423f4 appears in the backup log file when you back up a volume by using the Volume Shadow Copy service in Windows Server 2003
http://support.microsoft.com/default.aspx?scid=kb;en-us;828481

and Windows Server 2003 SP1 will fix this problem.
-- how about XP ?

Regards,
Anonymous
January 06, 2005
The comment has been removed
Anonymous
January 21, 2005
Will work for lotus(domino server) databases?
Anonymous
January 22, 2005
It depends. The recommended procedure is to have a VSS writer that listens for backup events (shadow copy creation events) and flushes the database in order to minimize the recovery time.

If there is no writer involved, then the contents of the shadow copy will look as if the machine was suddenly powered off. If Lotus can recover after a sudden reboot (as anyone would expect) then, yes, you can use this technique for Lotus databases.
Anonymous
March 22, 2005
As I was enjoying a fine dinner of Lipton Italian Sides one night, I came across Adi's post on how to copy an open file with shadow copies. Not bad, I thought, and stored that little nugget of knowledge away for when I might need to do just that.

A...
Anonymous
July 03, 2006
PingBack from http://nerhood.homeip.net/wordpress/archives/2006/07/03/i-want-to-backup-both-my-mac-and-pc/
Anonymous
December 06, 2006
And you thought it couldn't be done: copying locked files, such as your .pst file from Outlook, while
Anonymous
December 06, 2006
And you thought it couldn't be done: copying locked files, such as your .pst file from Outlook, while
Anonymous
August 05, 2007
PingBack from http://netweb.wordpress.com/2007/08/06/links-for-2007-08-06/
Anonymous
October 07, 2007
Last week, I completely virtualized my life. Well, at least my server life is all virtualized. I'd been
Anonymous
March 02, 2008
PingBack from http://caballe.cat/wp/2008/03/03/copiar-fitxers-oberts-en-windows/
Anonymous
April 01, 2008
PingBack from http://www.atomice.com/blog/?p=26
Anonymous
June 07, 2008
PingBack from http://xaviernewssite.k2free.com/howtocopyfiles.html
Anonymous
June 22, 2008
PingBack from http://isabell.ranchcredit.com/scriptdp0.html
Anonymous
July 08, 2008
PingBack from http://jaydendailynews.iifree.net/vshadowexemsdn.html
Anonymous
August 23, 2008
PingBack from http://www.shoretelforums.com/forums/administrators/2048-backup-data-remote-server-dr.html#post6144
Anonymous
November 29, 2008
PingBack from http://xaegr.wordpress.com/2008/11/30/shadow-copy-%d0%b8%d0%b7-%d0%ba%d0%be%d0%bc%d0%b0%d0%bd%d0%b4%d0%bd%d0%be%d0%b9-%d1%81%d1%82%d1%80%d0%be%d0%ba%d0%b8/
Anonymous
January 17, 2009
PingBack from http://www.hilpers.it/2742093-backup-partizione-nella-partizione-su
Anonymous
January 20, 2009
PingBack from http://www.hilpers-esp.com/247319-logs-de-seguridad
Anonymous
January 21, 2009
PingBack from http://www.keyongtech.com/1672650-outlook-pst-files-locked-while
Anonymous
January 22, 2009
PingBack from http://www.hilpers.pl/412024-kopiowanie-otwartych-zablokowanych-plikow

共用方式為

Script recipe of the week: how to copy an opened file

Comments

其他資源