Us a script cache to speed up repetitive directory lookups

In an Active Directory environment there will be many instances where you’ll be doing report scripting that will involve repetitive resolution of identity references. To demonstrate, I’ll bring back the unused vacation notification script from a previous post on splatting parameters:

Param (
$ImportFile     =  './user_vacation_list.csv',
$EmailFrom      =  'HR@mycompany.com',
$EmailSubject   =  'Unused vacation reminder',
$SMTPServer     =  'mail.mycompany.com'
)

$AllEmailParams =
        @{
          From       = $EmailFrom
          Subject    = $EmailSubject
          SMTPServer = $SMTPServer
          }

Foreach ($User in inport-csv $ImportFile)
 {
   $ADUser = Get-ADUser $User.Name -Propertes Mail,Manager
   $Manager = Get-ADUser $User.Manager -Properties Mail

   $ThisEmailParams = 
        @{
          To   = $ADUser.Mail
          Cc   = $Manager.Mail
          Body = "You have $($User.VacationDays) unused vacation days."
         }

 Send-MailMessage @AllEmailParams @ThisEmailParams
 }

Now, in most any organization this script can easily result in having to look up the email address for the same manager many times, since there will be relatively few managers compared to the number of employees, and multiple employees having the same manager. We can speed up our script and reduce the load on our DC's by creating an in-script cache of manager email addresses we've already looked up, so we don't have to keep doing it over and over again.


Param (
$ImportFile     =  './user_vacation_list.csv',
$EmailFrom      =  'HR@mycompany.com',
$EmailSubject   =  'Unused vacation reminder',
$SMTPServer     =  'mail.mycompany.com'
)
 
$AllEmailParams =
        @{
          From       = $EmailFrom
          Subject    = $EmailSubject
          SMTPServer = $SMTPServer
          }

#Create an empty hash table for the cache
$Manager = @{} 
 
Foreach ($User in inport-csv $ImportFile)
 {
   $ADUser = Get-ADUser $User.Name -Propertes Mail,Manager

   #Check to see if the entry we want is already in cache, if not add it.
   if ($Manager[$ADUser.Manager] -eq $null)
     {
       $Manager[$ADUser.Manager] = 
        (Get-ADUser $User.Manager -Properties Mail).Mail
     }
   
   #Use the identity reference to get the entry from cache.
   $ThisEmailParams = 
        @{
          To   = $ADUser.Mail
          Cc   = $Manager[$ADUser.Manager]
          Body = "You have $($User.VacationDays) unused vacation days."
         }
 
 Send-MailMessage @AllEmailParams @ThisEmailParams
 }

By using the hash table created ahead of entering the processing loop:


#Create an empty hash table for the cache
$Manager = @{} 

We can test that to see if we already have an entry cached for this manager. If we don’t have it yet, we’ll go ahead and do that lookup and add it to our cache:

if ($Manager[$ADUser.Manager] -eq $null)
     {
       $Manager[$ADUser.Manager] = 
        (Get-ADUser $User.Manager -Properties Mail).Mail
     }

Then use the cache reference to get our manager’s email address

$ThisEmailParams = 
        @{
          To   = $ADUser.Mail
          Cc   = $Manager[$ADUser.Manager]
          Body = "You have $($User.VacationDays) unused vacation days."
         }

Now we only go to our DC once for each manager the first time we encounter a user with that manager in our input data, and for every subsequent user with that same manager we just retrieve it from our hash table.

Reporting scripts will frequently be focused on a particular group or type of objects in the directory and associations between them are linked by an identity reference in AD (DN, SID, guid) you’ll find many instances where you will end up having to resolve the same identity references and retrieve the same information for that object over and over again, and using a cache of already resolved indentities and data can both speed up your script and reduce the impact on your critical infrastructure servers.

Random Password generator

SANS:

$length = 15
{
    If ($length -lt 4) { $length = 4 }   #Password must be at least 4 characters long in order to satisfy complexity requirements.

    Do {
        $password =     $null 
        $hasupper =     $false   #Has uppercase letter character flag.
        $haslower =     $false   #Has lowercase letter character flag.
        $hasnumber =    $false   #Has number character flag.
        $hasnonalpha =  $false   #Has non-alphanumeric character flag.
        $isstrong =     $false   #Assume password is not strong until tested otherwise.
        
        For ($i = $length; $i -gt 0; $i--)
        {
            $x = get-random -min 33 -max 126              #Random ASCII number for valid range of password characters.
                                                          #The range eliminates the space character, which causes problems in other scripts.        
            If ($x -eq 34) { $x-- }                       #Eliminates double-quote.  This is also how it is possible to get "!" in a password character.
            If ($x -eq 39) { $x-- }                       #Eliminates single-quote, causes problems in scripts.
            If ($x -eq 47) { $x-- }                       #Eliminates the forward slash, causes problems for net.exe.
            If ($x -eq 96) { $x-- }                       #Eliminates the backtick, causes problems for PowerShell.
            If ($x -eq 48 -or $x -eq 79) { $x++ }         #Eliminates zero and capital O, which causes problems for humans. 
            
            $password = $password + [System.Char] $x      #Convert number to an ASCII character, append to password string.

            If ($x -ge 65 -And $x -le 90)  { $hasupper = $true }
            If ($x -ge 97 -And $x -le 122) { $haslower = $true } 
            If ($x -ge 48 -And $x -le 57)  { $hasnumber = $true } 
            If (($x -ge 33 -And $x -le 47) -Or ($x -ge 58 -And $x -le 64) -Or ($x -ge 91 -And $x -le 96) -Or ($x -ge 123 -And $x -le 126)) { $hasnonalpha = $true } 
            If ($hasupper -And $haslower -And $hasnumber -And $hasnonalpha) { $isstrong = $true } 
        } 
    } While ($isstrong -eq $false)
    
    $password

Replaced with:


Length = 15

$PasswordCharCodes = {33..126}.invoke()

#Exclude ",',/,`,O,0
34,39,47,96,48,79 | foreach {[void]$PasswordCharCodes.Remove($_)}

$PasswordChars = [char[]]$PasswordCharCodes 

do { 
    $NewPassWord =  $(foreach ($i in 1..$length) 
     { Get-Random -InputObject $PassWordChars }) -join '' 
   }

 until (
         ( $NewPassword -cmatch '[A-Z]' ) -and
         ( $NewPassWord -cmatch '[a-z]' ) -and
         ( $NewPassWord -imatch '[0-9]' ) -and 
         ( $NewPassWord -imatch '[^A-Z0-9]' )
       ) 
        
 $NewPassword 

Testing V4’s Where method and the pipeline

Continuing with the topic of the where() method and return types, I decided to to some bench testing.

I can get the same object type return from the where() collection method that I do from using the Where-Object comdlet by piping the collection to Set-Variable. But did I just give up the performance advantage using the method by sticking another pipeline into it?

Here’s my test setup:

$counter = 1..100

"`n"
'Where-Object cmdlet'

  Measure-Command {

   foreach ($count in $counter)
    {$result = $counter | where {$_ -match '33'}}

  } | select TotalMilliseconds | fl 

"`n"
'Where method, pipe to Set-Variable'
Measure-Command {

foreach ($count in $counter)
 {$counter.where({$_ -match '33'}) | Set-Variable result}

} | select TotalMilliseconds | fl 


"`n"
'Where method, return collection'
Measure-Command {

foreach ($count in $counter)
{$result = $counter.Where({$_ -match '33'})}

} | select TotalMilliseconds | fl 

And the results:

Where-Object cmdlet
TotalMilliseconds : 704.2918

Where method, pipe to Set-Variable
TotalMilliseconds : 203.6046

Where method, return collection
TotalMilliseconds : 160.8534

It gave up some performance piping to Set-Variable, but not much.

The Where-Object pipeline had to handle every item in the array.
The where() pipeline only had to handle one.
Filter left.

Generic Collections and Arrays revisited

I noticed a pingback from Simon Wahlid’s blog on Boe’s post on the V4 Foreach and Where array methods.

Simon uses this example

$list = New-Object -TypeName System.Collections.ArrayList
1,2 | Foreach {
    $object = [PSCustomObject]@{
        ID = "$_"
        Array = @("first","second")
    }
    [void]$list.Add($object)   
}

and notes that this doesn’t work:

$entry2 = $list.Where({$_.ID -eq 2})
$entry2.Array += "third"
$list

because of the return type being a collection, but does work with the cmdlet which returns a single object from his array:

$entry1 = $list | Where-Object {$_.ID -eq 1}
$entry1.Array += "third"
$list

and offers a couple of workarounds to fix his problem:

$entry2[0].Array += "fourth"
$list

or

$entry2.Item(0).Array += "fifth"

The difference in return types is more result of how these methods are implemented than the methods themselves, and the difference is the pipeline. Remember that pipelines “unroll” objects and collections into a stream of elements, and that Powershell will package up any expression that returns multiple objects into an array when you assign it to a variable, or a scalar (single object) if just one object is returned. The where-object example uses the pipeline, and we see that in the returned result. Using indirection (left-hand assignment) of the where() method return doesn’t use the pipeline, and we get the collection.

So if we use the pipeline to set our variable rather than indirection we can get the same return type we would get using the where-object comdlet

$entry2 = $list.Where({$_.ID -eq 2})
$entry2.gettype()


IsPublic IsSerial Name                                     BaseType                                  
-------- -------- ----                                     --------                                  
True     True     Collection`1                             System.Object  

$list.Where({$_.ID -eq 2}) | Set-Variable entry2
$entry2.GetType()


IsPublic IsSerial Name                                     BaseType                                  
-------- -------- ----                                     --------                                  
True     False    PSCustomObject                           System.Object 

So we can make that work by simply changing the way we do the variable assigment:

$list = New-Object -TypeName System.Collections.ArrayList
1,2 | Foreach {
    $object = [PSCustomObject]@{
        ID = "$_"
        Array = @("first","second")
    }
    [void]$list.Add($object)   
}

$list.Where({$_.ID -eq 2}) | Set-Variable entry2
$entry2.Array += "third"
$list



ID                                                 Array                                             
--                                                 -----                                             
1                                                  {first, second}                                   
2                                                  {first, second, third} 

More on generic collections in Powershell V4, and a gotcha.

In a previous post about generic collections I talked about using the invoke() method of a script block to get a collection.

Boe Prox covered the new Foreach and Where methods of arrays in V4 in this this post .

Turns out they have something in common – they both return the same type of generic collection.

Starting with a generic collection using {}.invoke(), and an array:


$col = {1..10}.invoke()
$col.gettype() | ft -AutoSize


IsPublic IsSerial Name         BaseType     
-------- -------- ----         --------     
True     True     Collection`1 System.Object

$arr = 1..10
$arr.gettype() | Format-Table -AutoSize


IsPublic IsSerial Name     BaseType    
-------- -------- ----     --------    
True     True     Object[] System.Array

Running that array through it’s where() method and checking the type returned:

$x = $arr.where({$_})
$x.gettype()| Format-Table -AutoSize

IsPublic IsSerial Name         BaseType     
-------- -------- ----         --------     
True     True     Collection`1 System.Object

Where the Where-Object cmdlet would return an array:


$y = $arr | where {$_}
$y.gettype() | Format-Table -AutoSize


IsPublic IsSerial Name     BaseType    
-------- -------- ----     --------    
True     True     Object[] System.Array

Another notable difference is that the where() method will always return that collection even if nothing passed and it’s empty:


$x = $arr.where({})
$x.gettype()| Format-Table -AutoSize

IsPublic IsSerial Name         BaseType     
-------- -------- ----         --------     
True     True     Collection`1 System.Object

Where the cmdlet will return $null:

$y = $arr | where {}
$y.gettype()| Format-Table -AutoSize

You cannot call a method on a null-valued expression.
At line:3 char:1
+ $y.gettype()| Format-Table -AutoSize
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

Which can sometimes lead to unexpected results:

Starting with an array of values, and a threshold value to test for:

$arr = -10..10
$Warn = 15

Then test the array for values over the threshold:

$warnings = $arr | where { $_ -gt $Warn } 
  if ($warnings -eq $null)
    { 'All OK' }
   else { 'Over limit' }
All OK

But if we change that to using the where() method:

$warnings = $arr.where({ $_ -gt $Warn }) 
  if ($warnings -eq $null)
    { 'All OK' }
   else { 'Over limit' }
Over limit

Splatting Parameters Pt 2 – Remote Possibilities

In my last post I talked about using splatted parameters to shorten long command lines and help organize code.

Splatting parameters is usually discussed as a way to pass parameters to cmdlets, but it can also be used to pass parameters to scripts, functions, filters and even an anonymous script block.

Here’s a simple script block that just spits out whatever arguments it’s given:

{$args}

Let’s invoke that, and splat one of the hash tables from the vacation notice script:

Param (
$ImportFile     =  './user_vacation_list.csv',
$EmailFrom      =  'HR@mycompany.com',
$EmailSubject   =  "'Unused vacation reminder'",
$SMTPServer     =  'mail.mycompany.com'
)
 
$AllEmailParams =
        @{
          From       = $EmailFrom
          Subject    = $EmailSubject
          SMTPServer = $SMTPServer
          }

&{$args} @AllEmailParams

-Subject:
'Unused vacation reminder'
-From:
HR@mycompany.com
-SMTPServer:
mail.mycompany.com

The splat operation took the hash table keys and added a hyphen to the front and a colon to the end to make them conform with Powershell parameter syntax, and then spit out each key and it’s value.

That will get implicitly cast as [string] when it’s gets used as a parameter set. Let’s do that explicitly to see what it will look like:

"$(&{$args} @AllEmailParams)"

-Subject: 'Unused vacation reminder' -From: HR@mycompany.com -SMTPServer: mail.mycompany.com

And that’s how splatting works.

Now, back on the vacation notification front, it turns out my script has stopped working because the SMTP server I was using is no longer available. I do have an Exchange server, but by default Exchange receive connectors won’t do anonymous mail relay without special configuration of the access controls. But here a little secret about those receive connectors – the access controls only apply to network connections. An Exchange transport server will relay mail for a script running locally using ‘localhost’ as it’s relay host, even if it’s receive connectors aren’t configured for relay.

So I can use that Exchange server for my mail relay if I change my -SMTPServer to ‘localhost’, and run my script there.
I have remoting enabled on that server, so I should be able to do that using Invoke-Command to run the Send-MailMessage command there. But now all those parameters that are using local variables as arguments are a problem. I can’t put the hash tables in he script block because the variables in them are on my computer, and Invoke-Command won’t let me splat them to it’s Scriptblock. It will only take an argument list, and you can’t used named parameters there. I’ll have hope all those parameters are positional, figure out what the position is for each one, and construct and argumentlist in that order. If I can’t do that, I’ll have to rewrite it to use explicit $args elements, or re-write it by explicitly scoping all the variables with the $Using: prefix. But there is another way.

Sometimes it’s easier to get a script block using local variables by creating it from an expandable string. Using what we know about splatting, we can take our existing hash tables of arguments, and modify the existing script to run that Send-MailMessage command on the Exchange server like this:

Param (
$ImportFile     =  './user_vacation_list.csv',
$EmailFrom      =  'HR@mycompany.com',
$EmailSubject   =  'Unused vacation reminder',
$SMTPServer     =  'localhost' #Changed to 'localhost'
)
 
$AllEmailParams =
        @{
          From       = $EmailFrom
          Subject    = $EmailSubject
          SMTPServer = $SMTPServer
          }
 
Foreach ($User in import-csv $ImportFile)
 {
   $ADUser = Get-ADUser $User.Name -Propertes Mail,Manager
   $Manager = Get-ADUser $User.Manager -Properties Mail
 
   $ThisEmailParams = 
        @{
          To   = $ADUser.Mail
          Cc   = $Manager.Mail
          Body = "You have $($User.VacationDays) unused vacation days."
         }
 
#These lines added in place of the Send-MailMessage command

  $Scriptblock = [Scriptblock]::Create(
   "Send-MailMessage $(&{$args} @AllEmailParams) @ThisEmailParams)")
 Invoke-Command -ScriptBlock $Scriptblock -ComputerName ExchangeServer
 }

Create the script block using the Send-MaillMessage command and the splat strings, and it’s ready to go. All the local variables are already expanded to literal values in the script block, and it’s ready to be invoked on the remote system.
Since I’ve only made one parameter change and added a couple of extra lines to make this work with the existing code, it would be relatively easy to add a another parameter set to include the Exchange server and have it work either way.

Splatting Parameters

This is a quick post on using hash tables to splat parameters to help organize and tidy up your powershell scripts.

I’m going to assume that everyone is familiar with the concept and simply present a quick sample script. It’s intentionally very basic (no parameter validation or error checking). The premise is that you have a csv file from HR with a list of user names and the number of vacation days they have remaining. You need to send them an email notification and CC their manager.

First, this script without any splatting:


Param (
$ImportFile     =  './user_vacation_list.csv',
$EmailFrom      =  'HR@mycompany.com',
$EmailSubject   =  'Unused vacation reminder',
$SMTPServer     =  'mail.mycompany.com'
)

Foreach ($User in inport-csv $ImportFile)
 {
   $ADUser = Get-ADUser $User.Name -Propertes Mail,Manager
   $Manager = Get-ADUser $User.Manager -Properties Mail

   $EmailTo = $ADUser.Mail
   $EmailCC = $Manager.Mail
   $EmailBody = "You have $($User.VacationDays) unused vacation days."

Send-MailMessage -From $EmailFrom -To $EmailTo -Cc $EmailCC -Subject $EmailSubject -SmtpServer $SMTPServer -Body $EmailBody

}

Now I’m going to do the same thing, but using hash tables to pass my parameters to send-mailmessage:

Param (
$ImportFile     =  './user_vacation_list.csv',
$EmailFrom      =  'HR@mycompany.com',
$EmailSubject   =  'Unused vacation reminder',
$SMTPServer     =  'mail.mycompany.com'
)

$AllEmailParams =
        @{
          From       = $EmailFrom
          Subject    = $EmailSubject
          SMTPServer = $SMTPServer
          }

Foreach ($User in inport-csv $ImportFile)
 {
   $ADUser = Get-ADUser $User.Name -Propertes Mail,Manager
   $Manager = Get-ADUser $User.Manager -Properties Mail

   $ThisEmailParams = 
        @{
          To   = $ADUser.Mail
          Cc   = $Manager.Mail
          Body = "You have $($User.VacationDays) unused vacation days."
         }

 Send-MailMessage @AllEmailParams @ThisEmailParams
 }

The “fixed” parameters that are being passed as script parameters can be in one hash table, close to the param block where it’s easy to associate the script parameters being passed with the send-mail message parameters they will map to.

Another hash table inside the loop is used to hold and set the parameters that will unique to each email, and then both are splatted to the command when it’s time to send the email. I think this produces much cleaner and more intuitive code.