Share via

Active Directory PowerShell SIDHistory Module Update 1.5

  • Article

To see all of the articles in this series visit https://aka.ms/SIDHistory .

I would like to thank everyone who has been using the Active Directory SIDHistory PowerShell module and sending me feedback.  Your input helps guide future releases like the one I am publishing today.

I’ve been sitting on some updates for a while, because I prefer to release code that has been field-tested.  I also wanted to time this release with the upcoming PowerShell Deep Dives book where I have a chapter discussing the origins of this module.  The last update was version 1.4 in June of 2012.  This is update 1.5 in July of 2013.

Summary of Changes

I am excited to announce the following key improvements in this release:

  • SID history updates in ACLs can be added instead of replaced.
  • Create SID map files for security translation without needing SID history.
  • Track down old domains after their trusts have been removed.
  • Get error logging for file server paths that fail the ACL scan.
  • Automate SID history data collection across many servers and shares.

Note:   This module version is compatible with PowerShell v2 and any newer versions.  The next release will require PowerShell v3 as a minimum level.

Change Details

This release includes some significant changes and additions, which I have highlighted below. Here is a list of the functions in this release:

 PS C:\> Get-Command -Module SIDHistory

CommandType Name                     ModuleName
----------- ----                     ----------
Function    Convert-SIDHistoryNTFS   SIDHistory
Function    Export-DomainSIDs        SIDHistory
Function    Export-SIDHistoryShare   SIDHistory
Function    Export-SIDMapping        SIDHistory
Function    Export-SIDMappingCustom  SIDHistory
Function    Get-ADObjectADSI         SIDHistory
Function    Get-DomainSIDWordCount   SIDHistory
Function    Get-SIDHistory           SIDHistory
Function    Get-SIDHistoryDuplicates SIDHistory
Function    Merge-CSV                SIDHistory
Function    Remove-SIDHistory        SIDHistory
Function    Search-SIDHistoryACL     SIDHistory
Function    Update-SIDMapping        SIDHistory
Function    Watch-Job                SIDHistory

Due to the large number of changes I am not going to include code samples for each function in this article.  Please use Get-Help -Full to see complete details and examples for each module member.

 PS C:\> Get-Help Convert-SIDHistoryNTFS -Full

PS C:\> Get-Help Export-SIDMappingCustom -Full

PS C:\> Get-Help Get-DomainSIDWordCount -Full

Etc...

 

 

Functions Added

Export-SIDMappingCustom

  • I had a customer who was not able to use the Active Directory Migration Tool for a domain migration.  The newly acquired subsidiary was not allowed to create a trust for compliance reasons.  Obviously we need a trust to do a migration.  Or do we?
  • To work around the situation I wrote a function that will map SIDs between accounts in two different domains where there is no SID history.  The trick is having a common, unique attribute like samAccountName, employeeID, employeeNumber, mail, etc.
  • The customer manually exported the accounts from the old domain and imported them into the new domain.  Next they made sure that their HR system populated the same unique Employee ID in the new domain.  Now they had a common, unique key between the two domains.
  • In the old 2003 domain we used an ADSI script to create a file containing the objectSID and employeeID attributes.  (See Get-ADObjectADSI described below.)  This is all we needed to create our mapping file.  We copied this file across international borders to the local new domain.
  • Now we can use the new Export-SIDMappingCustom function to create a mapping file between the exported SID data from the old domain and the live accounts in the new domain.  This is only one possible scenario with the function.  It can use any combination of live connection or export file from the old and new domains.
  • In this particular case the customer only wanted to migrate the file server from the old domain.  With this new SID mapping file they were able to run Convert-SIDHistoryNTFS against the old server to re-ACL the resources.  This is POWERFUL.  Essentially, you can do a NAS ACL migration without SID history in place.
  • This works for both users and groups.  At another customer they wanted to migrate several domain local groups.  The groups had been recreated with the same name in the new domain without SID history.  The Export-SIDMappingCustom function was able to create a mapping file by matching the group names between both old and new live domains and then putting both SIDs into a mapping file.

Get-SIDHistoryDuplicates

  • There are processes to resolve duplicate SIDs on accounts but not for duplicate SID history.
  • This should never never never happen.  But it happened for one customer, the same customer where I discovered 100 unique old domains.  We’re not sure how the duplicate SID history entries got there, because there were years of migration history with a variety of migration tools.
  • Duplicates in SID history are a violation of everything we know to be good and true.  There can only be one.  If duplicates are found, then it would invalidate the SID mapping process.
  • To find these I wrote this function to generate a report.  Once identified, the customer chose to simply delete the accounts with the duplicates, because they were old empty groups.

Search-SIDHistoryACL

  • In this module two main functions do all of the work to discover SID history on shares and files:  Convert-SIDHistoryNTFS and Export-SIDHistoryShare.  In order to fully document SID history on your file servers you need to call each of these functions for each share on each server.  That’s a lot of calls.
  • This is a “meta-function” that calls these two functions for a batch of servers or shares listed in either an OU or a text file.  Using PowerShell background jobs it scans them all in parallel to reduce the discovery time required.
  • My only caution is that there is no throttling built in.  Run these in small batches of servers until you see how the performance works out with your hardware.
  • Use the following Watch-Job function to manage these background jobs.

Watch-Job

  • This function conveniently manages receiving the output from the jobs spun up by Search-SIDHistoryACL.  It displays a PowerShell progress bar indicating how many jobs are still running.  Once they are all complete it will receive the output into a consolidated job log and report on each job’s start and end times.  All of the ACL output files are rendered separately by their respective jobs.
  • Optionally you can have the function clear the job queue with the -Remove switch.
  • This is a utility function, and it could be modified easily for many other background job scenarios.

Get-DomainSIDWordCount

  • The Export-DomainSIDs function crawls all trusts in the forest to identify domain names and domain SIDs.  When old trusts to migrated domains are gone you can no longer identify where SID history originated… until now.
  • The function creates a list of word counts gleaned from popular string attributes found on accounts from each old domain SID identified in the forest.  This gives you clues about where the accounts in the unidentified SID history domain may have originated.  For example, employees migrated from the old ContosoPartner domain may have that company name string in their description, department, display name, or OU path.  By finding and counting the common strings across all migrated accounts you can usually identify the old domain.
  • This process gives you the information needed to manually update the DomainSIDs.csv file.  After updating the file re-run the Update-SIDMapping function to add these old domain names to your master SID history report.
  • Pipe the output to Export-CSV or Out-GridView for easy viewing.

Get-ADObjectADSI

  • I wrote this utility function for a customer with legacy DCs where they were not going to install the AD web service.  They installed PowerShell, and we used this function to create the account listing for Export-SIDMappingCustom.  It uses ADSI to mimic Get-ADObject, but it was a point solution not intended for full parity with the AD module cmdlet.  This function is intentionally undocumented, because most environments will not need it.  Feel free to modify it to meet your needs.

 

Functions Modified

Export-SIDMapping

  • I added description and whenCreated attributes to the SIDReport.CSV.  These properties help identify accounts in the list, especially old accounts that are likely stale.
  • By mistake I was running the AD query twice in this function, so I corrected it to only run once.  Obviously this will greatly improve performance.

Update-SIDMapping

  • Added description and whenCreated to SIDReportUpdated.CSV for same reasons listed above.

Convert-SIDHistoryNTFS

  • Added -Add switch.  The previous functionality did an ACE replace only.  Now you have the option of doing a replace or an add. This way you can co-exist with both old and new SIDs temporarily until you are ready to completely remove all old SIDs from the ACLs.  Running without the -Add switch gives you a replace, leaving the new SID and removing the old SID.
  • As a result of the new -Add switch I had to modify the CSV output and log files to report the status of both Old SID and New SID in the ACLs.  The “Both” column will be true if both the old and new SIDs are present in the ACL.  The “Operation” column will now say “Add” or “Replace”.  Added new column for NewDomainSID.
  • After more field testing I discovered that there were some issues scanning folders.  In prior releases you would see red text in the PowerShell console, but you didn’t know where the errors occurred. I added error logging so that you get a list of the reasons and the paths where the ACL scans fail down inside the folder tree.  Primarily I’ve found three issues:
  • 1. Path too long.   I spent a lot of time investigating this issue.  The Get-ACL -Path parameter is limited to paths of 260 characters or less.  I found this TechNet Wiki article and this blog series, but none of the proposed work-arounds were applicable.  As a temporary work-around if you run into this problem you may be able to map a drive down farther in the folder structure of longer paths and run the scan again for that newly mapped path.  This is due to an underlying .NET limitation.  Unfortunately this error only gets logged when running in PowerShell v3.  PowerShell v2 will not report this error in a way that we can trap.
  • 2. Invalid character in path.   I discovered an issue with Get-ACL that does not return an object when the path contains a ‘[‘ character.  After researching this on the Microsoft PowerShell Connect site I found that it is resolved in PowerShell v3 with the new -LiteralPath parameter.  For now I am logging the error until I update the module to use PowerShell v3 features.
  • 3. Access denied.   Using this function you often find file shares where Administrator rights have been removed.  To resolve this either rescan the path in question with appropriate credentials or modify the permissions on the path.

Export-SIDHistoryShare

  • Modified CSV columns to match the output from Convert-SIDHistoryNTFS.  This allows all results to be consolidated into a single CSV report.

 

 

Time-Saving Tip

When using Convert-SIDHistoryNTFS -WhatIf to scan servers and shares it can take hours or days depending on the size of the environment.  Sometimes you may only find a few SIDHistory entries that need to be remediated.  Rather than rescan the entire server or share to fix these ACLs you can target the SID history cleanup this way to save time:

 PS C:\> Import-CSV ACL_SID_History.csv |
 Select-Object -Unique -ExpandProperty Folder |
 ForEach-Object {Convert-SIDHistoryNTFS $_}

In this example ACL_SID_History.csv is the NTFS permission scan output from Convert-SIDHistoryNTFS -WhatIf.  We simply get a unique list of folders and feed those into the conversion function.  Obviously this is much faster than rescanning every ACL of every folder in the share subtree.

What’s next?

I have a long list of pending enhancements for the module, but my top priority is to update it with PowerShell v3 features:

  • Use -LiteralPath in place of -Path to avoid the ‘[‘ issue in Convert-SIDHistoryNTFS.
  • Rewrite the Export-SIDHistoryShare function using the new SmbShare module.
  • Create a function that will update share ACLs using the new SmbShare module.
  • Create a -Server parameter for Get-SIDHistory and Export-SIDMapping to target a specific DC.
  • Capture all output streams correctly for Watch-Job logging.

Please use the comment area below to tell me what features you would like to see improved or added in the module. I would also appreciate any feedback for how this has helped your projects.

Download

You can download the updated Active Directory SIDHistory PowerShell module from the TechNet Script Gallery.

Comments

  • Anonymous
    September 22, 2014
    Ashley, what an excellent article!
  • Anonymous
    November 18, 2014
    Two questions:
    1) Can I run the SIDHistory commands against the root of a remote server? In other words something like:
    Convert-SIDHistoryNTFS -StartPath \Server1c$ ... bla bla bla

    2) Will these scripts run on a Windows 2003 server (assuming that PowerShell is installed)?

    Thanks!
    Ted
  • Anonymous
    November 26, 2014
    Hi Ted,
    Great questions.
    1. Yes.
    2. Yes and No. You can target a 2003 file server with the scripts, but the scripts must run from 2008 R2 or newer.
    Ashley
    @GoateePFE
  • Anonymous
    January 12, 2016
    I am trying to use this procedure to migrate between two domains, however, we don't have any SIDHistory and employeeID is not populated. The "user logon name" will remain the same between the domains and we would love to use this value instead in the mapping file. Can you point me in the right direction on how we can make this happen?
  • Anonymous
    January 12, 2016
    Nevermind, found much more information inside of the actual psm1 documentation, so figuring it out now! Thanks
  • Anonymous
    July 28, 2016
    For many, it also may be worth checking out "repacls" which is like a multi-threaded version of subinacl that works on NetApps (which subinacl doesn't). It has a various of options to detect and remediate a variety of ACL related woes. Open source and feature suggestions welcome. More information is here: https://nomorefood.github.io/Repacls
  • Anonymous
    June 04, 2017
    Not active