Duplicate Files

Need a way to check if two files are the same?  Calculate a hash of the files.  Here is one way to do it:

 

## Calculates the hash of a file and returns it as a string.

function Get-MD5([System.IO.FileInfo] $file = $(throw 'Usage: Get-MD5 [System.IO.FileInfo]'))

{

  $stream = $null;

  $cryptoServiceProvider = [System.Security.Cryptography.MD5CryptoServiceProvider];

 

  $hashAlgorithm = new-object $cryptoServiceProvider

  $stream = $file.OpenRead();

  $hashByteArray = $hashAlgorithm.ComputeHash($stream);

  $stream.Close();

 

  ## We have to be sure that we close the file stream if any exceptions are thrown.

  trap

  {

    if ($stream -ne $null)

    {

      $stream.Close();

    }

   

    break;

  }

 

  return [string]$hashByteArray;

}

 

I think about the only new thing here is the trap statement.  It’ll get called if any exception is thrown, otherwise its just ignored.  Hopefully nothing will go wrong with the function but if anything does I want to be sure to close any open streams.  Anyway, keep this function around, we’ll use it along with AddNotes and group-object to write a simple script that can search directories and tell us all the files that are duplicates.  Now… an example of this function in use:

 

MSH>"foo" > foo.txt

MSH>"bar" > bar.txt

MSH>"foo" > AlternateFoo.txt

MSH>dir *.txt | foreach { get-md5 $_ }

33 69 151 28 248 32 88 177 8 34 154 58 46 59 255 53

54 122 136 147 125 209 249 229 12 105 236 19 140 5 107 169

33 69 151 28 248 32 88 177 8 34 154 58 46 59 255 53

MSH>

 

 

Note that two of the files have the same hash, as expected since they have the same content.  Of course, it is possible for two files to have the same hash and not the same content so if you are really paranoid you might want to check something else in addition to the MD5 hash.

 

- Marcel

Comments

• Anonymous
  September 13, 2005
  The comment has been removed
• Anonymous
  September 13, 2005
  Oops, sorry, I just saw you recognized that at the end of your post.
• Anonymous
  September 13, 2005
  The comment has been removed
• Anonymous
  September 18, 2005
  That's cool...so now how would I use that in a script to list all the files in a given folder (recursively) that are (very likely) the same?
• Anonymous
  September 20, 2005
  I think byte arrays should be displayed in hexadecimal, when turned into strings.
• Anonymous
  October 10, 2005
  NoClone uncovers duplicate files by comparing byte by byte, not by CRC or MD5, so duplicates found are totally identical. Try it.
  
  Alan Wo
  The author
• Anonymous
  January 26, 2006
  Directory Report finds duplicate files based on name and/or size and/or CRC. It checks the size before calculating the file's CRC.
  
  Download the program at
  http://www.file-utilities.com/downloads/wdir.zip
  
  Allan Braun
  The author
• Anonymous
  January 21, 2009
  PingBack from http://www.keyongtech.com/2832238-howto-create-an-md5-of