Partilhar via

Duplicate Files

  • Artigo

Need a way to check if two files are the same?  Calculate a hash of the files.  Here is one way to do it:

 

## Calculates the hash of a file and returns it as a string.

function Get-MD5([System.IO.FileInfo] $file = $(throw 'Usage: Get-MD5 [System.IO.FileInfo]'))

{

  $stream = $null;

  $cryptoServiceProvider = [System.Security.Cryptography.MD5CryptoServiceProvider];

 

  $hashAlgorithm = new-object $cryptoServiceProvider

  $stream = $file.OpenRead();

  $hashByteArray = $hashAlgorithm.ComputeHash($stream);

  $stream.Close();

 

  ## We have to be sure that we close the file stream if any exceptions are thrown.

  trap

  {

    if ($stream -ne $null)

    {

      $stream.Close();

    }

   

    break;

  }

 

  return [string]$hashByteArray;

}

 

I think about the only new thing here is the trap statement.  It’ll get called if any exception is thrown, otherwise its just ignored.  Hopefully nothing will go wrong with the function but if anything does I want to be sure to close any open streams.  Anyway, keep this function around, we’ll use it along with AddNotes and group-object to write a simple script that can search directories and tell us all the files that are duplicates.  Now… an example of this function in use:

 

MSH>"foo" > foo.txt

MSH>"bar" > bar.txt

MSH>"foo" > AlternateFoo.txt

MSH>dir *.txt | foreach { get-md5 $_ }

33 69 151 28 248 32 88 177 8 34 154 58 46 59 255 53

54 122 136 147 125 209 249 229 12 105 236 19 140 5 107 169

33 69 151 28 248 32 88 177 8 34 154 58 46 59 255 53

MSH>

 

 

Note that two of the files have the same hash, as expected since they have the same content.  Of course, it is possible for two files to have the same hash and not the same content so if you are really paranoid you might want to check something else in addition to the MD5 hash.

 

- Marcel

Comments

  • Anonymous
    September 13, 2005
    The comment has been removed
  • Anonymous
    September 13, 2005
    Oops, sorry, I just saw you recognized that at the end of your post.
  • Anonymous
    September 13, 2005
    The comment has been removed
  • Anonymous
    September 18, 2005
    That's cool...so now how would I use that in a script to list all the files in a given folder (recursively) that are (very likely) the same?
  • Anonymous
    September 20, 2005
    I think byte arrays should be displayed in hexadecimal, when turned into strings.
  • Anonymous
    October 10, 2005
    NoClone uncovers duplicate files by comparing byte by byte, not by CRC or MD5, so duplicates found are totally identical. Try it.

    Alan Wo
    The author
  • Anonymous
    January 26, 2006
    Directory Report finds duplicate files based on name and/or size and/or CRC. It checks the size before calculating the file's CRC.

    Download the program at
    http://www.file-utilities.com/downloads/wdir.zip

    Allan Braun
    The author
  • Anonymous
    January 21, 2009
    PingBack from http://www.keyongtech.com/2832238-howto-create-an-md5-of