How far back do your message tracking logs REALLY go?

Exchange lets you set both a history depth, or number of days, to keep message tracking logs.  It also lets you set a limit on how much disk space to allocate to message trackinglogs.  What happens when it reaches the disk space limit?  You start losing log history. 

When you enable message tracking logging the default settings are to keep 30 days of history, and not use more than 1GB of log space, per hub transport server.  If you’ve increased the history depth, but haven’t increased the amout of disk space to allocate to the logs, or if you have a really busy HT server you may not have the log depth you think you do.

Here’s a little function to go find all your Hub Transport servers, check their message tracking log settings, and find out how much disk space the logs are using, and what the actual history depth is on each one.

function mt_status {
$hts = get-exchangeserver |
? {$_.serverrole -match “hubtransport”}

$proplist = @’
Name
MessageTrackingLogEnabled
MessageTrackingLogMaxAge
MessageTrackingLogMaxDirectorySize
MessageTrackingLogMaxFileSize
MessageTrackingLogPath
MessageTrackingLogSubjectLoggingEnabled
NewestLog
OldestLog
CurrentLogCount
CurrentLogSize
CurrentLogHistDepth
‘@
$props = $proplist -split “`n” |% {$_.trim()}

$hts |% {get-transportserver $_.name | select $props} |% {
if ($_.messagetrackinglogenabled){
$log_unc = “\\$($_.name)\$($_.messagetrackinglogpath -replace “:”,”$”)”
$logs = gci $log_unc\*.log | sort lastwritetime
$newest_log = $logs | select -last 1
$oldest_log = $logs | select -first 1
$_.newestlog = $newest_log.name
$_.oldestlog = $oldest_log.name
$_.currentlogcount = $logs.count
$_.currentlogsize = “$([int](($logs | measure length -sum).sum / 1MB)) MB”
$_.currentloghistdepth = “$([int](($newest_log.LastWriteTime – $oldest_log.creationtime).totaldays)) days”
$_
}
}
}

Invoke-ScritptAsync V2

Added the option to take the scriptblock arguments as either a collection of argument lists or a collection of parameter hashes to be splatted to the script block.

The argument lists or hash tables can be passed to the function either as an argument ($InputObject) or from the pipeline. The -InputType parameter specifies what type (argumentlist or parameters) the input objects are.

The default is ‘ArgumentList’, which is used for relatively simple script block that use $args or will use an argument list passed to positional parameters. More complex scenarios can be handled using the ‘Parameters’ input type, and passing hash tables of named parameters and values to the script block



function Invoke-ScriptAsync {

  [cmdletbinding()]

  param(
          #Script block to execute
          [Parameter(Mandatory)]
          [ScriptBlock] $ScriptBlock,
 
          #Array of argument lists or parameter hashes for the script block
          [Parameter(ValueFromPipeline)]
          [Object[]]$InputObject,

          #Type of input objects (argument lists or parameter hashes)
          [Parameter()]
          [ValidateSet('ArgumentList','Parameters')]
          [string]$InputType = 'ArgumentList',
      
          #Maximum number of threads to run
          [Parameter()]
          [int]$MaxThreads = 10,
 
          #Thread timeout (in seconds)
          [Parameter()]
          [int]$Timeout = 15,
       
          #Update refresh interval (in seconds)
          [Parameter()]
          [int]$RefreshInterval = 1,

          #Hash table reference to use for storing result objects
          [Parameter()]
          [HashTable]$ResultHash
      )

 Begin{ 

  $MaxRunTime = New-TimeSpan -Seconds $Timeout
 
  $RunspacePool = [RunspaceFactory ]::CreateRunspacePool(1, $MaxThreads)
  $RunspacePool.Open()

  $JobScript =
 @"
    &{ 
      `$DebugPreference = 'Continue'
      Write-Debug "Start(Ticks) = `$((get-date).Ticks)"
    }
 
    & { $ScriptBlock } @args
 
    &{
        `$DebugPreference = 'Continue' 
        Write-Debug "End(Ticks) = `$((get-date).Ticks)"
     }
"@
 
  $SaveJobData = 
   {
     #Save Pipeline Streams and information to ResultHash

     $JobData = $ResultHash[$Job.Pipe.InstanceID]

     $JobData.Started   = $Job.Started
     $JobData.Ended     = $Job.Ended
     $JobData.HadErrors = $Job.Pipe.HadErrors
     $JobData.State     = $Job.State
     $JobData.Verbose   = $Job.Pipe.Streams.Verbose.Readall() | Out-String

     $JobData.Error     = $Job.Pipe.Streams.Error | foreach { $_.Exception.Message }
     $JobData.Warning   = $Job.Pipe.Streams.Warning.Readall() | Out-String

     $Debug = $Job.Pipe.Streams.Debug
     if ($Debug.count -gt 2)
     { $JobData.Debug   = $Debug[1..($Debug.count - 2)] | Out-String }

     $JobData.Duration  = '{0:f2} ms' -f ($Job.Ended - $Job.Started).totalmilliseconds

    if ($Job.State -ne 'Timeout')
      { $JobData.OutputCount = $Job.Pipe.EndInvoke($Job.Result).count }
                
   } #End Save Job data

  if ($PSBoundParameters.ContainsKey('InputObject') )
    { $ScriptArgs = [Collections.ArrayList]@($InputObject) }

   else {$ScriptArgs = [Collections.ArrayList]@() }

 } #End Begin block

 Process {

 if ($_) 
   { [void]$ScriptArgs.Add($_) }

 } #End Process Block

 End {

  $Sequence = 0 

  $Jobs = 
    Foreach ($ScriptArg in $ScriptArgs)
     { 
       $Sequence++

       #Using argument list
       if ( $InputType -eq 'ArgumentList' )
         { 
          $Job = [powershell]::Create().
                 AddScript($JobScript).
                 AddArgument($ScriptArg)
          $Job.RunspacePool = $RunspacePool
         }

       #Using parameter hash
       if ( $InputType -eq 'Parameters' )
         { 
           $Job = [powershell]::Create().
                  AddScript($JobScript).
                  AddParameters($ScriptArg)
           $Job.RunspacePool = $RunspacePool
         }

      [PSCustomObject]@{
         Pipe     = $Job
         Result   = $Job.BeginInvoke()
         Started  = $null
         Ended    = $null
         State    = $null
         Sequence = $Sequence
        }

     if ( $PSBoundParameters.ContainsKey('ResultHash') )
       { 
        $ResultHash[$Job.InstanceID] =        
          [PSCustomObject]@{
            Sequence    = $Sequence
            Args        = $ScriptArg | Out-String
            Started     = $null
            Ended       = $null
            Duration    = $null
            State       = $null
            HadErrors   = $null
            Verbose     = $null
            Warning     = $null
            Error       = $null
            Debug       = $null
            OutPutCount = $null
           }
        }
  }# End Job creation (foreach script argument)


  $Waiting = $Jobs 

  While ( $Waiting )
    {
     foreach ($Job in  $Waiting )
      {
        #New job started - record start time
        if (
             ($Job.started -eq $null) -and
             ($job.pipe.Streams.Debug[0].Message -match 'Start')
            )
             {
               $StartTicks = 
                $Job.pipe.Streams.Debug[0].Message -replace 'Start\(Ticks\) = (\d+)','$1'
               $Job.Started = [Datetime]::MinValue + [TimeSpan]::FromTicks($StartTicks)
             }
        
        #Job completed - record end time and job data
        if ($Job.Result.IsCompleted)
          {
            $EndTicks = 
             $Job.pipe.Streams.Debug[-1].Message -replace 'End\(Ticks\) = (\d+)','$1'
            $Job.Ended = [Datetime]::MinValue + [TimeSpan]::FromTicks($EndTicks)
            $Job.State = $Job.pipe.InvocationStateInfo.State
            if ( $PSBoundParameters.ContainsKey('ResultHash') )
              { .$SaveJobData }
            $Job.Pipe.EndInvoke($Job.Result)
            $Job.Pipe.Dispose()
          } 
        
        #Job running, exceeded max run time. Record job data and stop thread.
        if ( ($Job.Started) -and
             (-not ($Job.Result.IsCompleted) ) -and
             (get-date) -gt ($Job.Started + $MaxRunTime))
          {
            Write-Warning "Job sequence number $($job.sequence) timed out."
            $Job.Ended = (Get-Date)
            $Job.Pipe.Stop()
            $Job.State = 'Timeout'
            if ( $PSBoundParameters.ContainsKey('ResultHash') )
              { .$SaveJobData }
            $Job.Pipe.Dispose()
           }
        } # end foreach waiting job 
          
          #Done checking waiting jobs. Get new waiting list and write progress

          #Jobs left to finish or time out
          $Waiting = $Jobs | Where { $_.Ended -eq $null }

          $Progress = 
            @{
               Activity = 'Running Scripts'
               Status   = "Completed $($Jobs.count - $Waiting.count) of $($Jobs.count)"
               PercentComplete = ($Jobs.count - $Waiting.count)  / $Jobs.count * 100
             }

          Write-Progress @Progress

          Start-Sleep -Seconds $RefreshInterval

     } # End while jobs waiting
    
    #All jobs finished.  Clean up runspace pool

    $RunSpacePool.Close()
    $RunSpacePool.Dispose()

 } # End End block
 
} # End function
 

Invoke-ScriptAsynch

My foray into the subject of Powershell runspaces and runspace pools continues, and at this point I’ve gotten past the Proof of Concept stage and have something that kind of resembles a usable function.

There are a few other runspace handling functions out there, and I learned much from studying them, but I really needed to sit down and write one as part of the learning process, and this is what I ended up with.

I wanted to take advantage of runspace pools, but they have the drawback of not providing a way to know when a particular instance has begun executing. This makes it difficult to specify and impose a timeout on stalled instances, and they can eventually bottle up your runspace pool if you can’t keep them cleaned out.

I solved this by doing a minor hijack of the script’s Debug stream to get a timestamp. Before it’s attached to a Powershell object, the script is wrapped in code to set the $DebugPreference to ‘Continue’, and write a debug message that contains the current time. This is done in it’s own scope, so the preference setting won’t override whatever might be set in the script. This will be the first thing the script does when it starts and the script streams can be interrogated while the script is running, so I periodically cycle through the instance objects, looking for that timestamp in the Debug stream, and then use that to determine if the instance has been running too long.

Another feature is the ability to save a collection of PS objects that have some detailed information about the instance execution and the content of it’s Verbose, Warning, Error, and Debug streams that can be used for debugging or reporting after the script blocks have finished.

It’s relatively simplistic in terms of the arguments it accepts compared to some of the other functions out there. It takes two arguments – a script block, and an array of argument lists. For each argument list in the array, it creates a Powershell object using the script block and that argument list, and hands it over to the runspace pool.

The result objects that contain the run details and stream information are kept in a hash table in the current scope (so you still have them after the function exits). To save those objects, create an empty hash table, and the assign it to the ResultHash parameter.

Thanks to Boe Prox and Dave Wyatt for their help on this, and I look forward to their (and anyone else’s) feedback, suggestions, and constructive criticism.


function Invoke-ScriptAsync {
    [cmdletbinding()]
    param(
        [Parameter(Mandatory=$true,position=0)]
        [ScriptBlock] $ScriptBlock,
 
        [Parameter(Mandatory=$true)]
        [array]$ArgumentList,
 
        [Parameter()]
        [int]$MaxThreads = 10,
 
        [Parameter()]
        [int]$Timeout = 15,
 
        [Parameter()]
        [int]$RefreshInterval = 1,

        [Parameter()]
        [HashTable]$ResultHash
  )
 
$MaxRunTime = New-TimeSpan -Seconds $Timeout
 
$RunspacePool = [RunspaceFactory ]::CreateRunspacePool(1, $MaxThreads)
$RunspacePool.Open()

$JobScript =
 @"
    &{ 
      `$DebugPreference = 'Continue'
      Write-Debug "Start(Ticks) = `$((get-date).Ticks)"
    }
 
    & { $ScriptBlock } @args
 
    &{
        `$DebugPreference = 'Continue' 
        Write-Debug "End(Ticks) = `$((get-date).Ticks)"
     }
"@
 
$SaveJobData = 
 {
   #Save Pipeline Streams and information to ResultHash

   $JobData = $ResultHash[$Job.Pipe.InstanceID]
   $JobData.Started = $Job.Started
   $JobData.Ended = $Job.Ended
   $JobData.HadErrors = $Job.Pipe.HadErrors
   $JobData.State = $Job.State
   $JobData.Error = $Job.Pipe.Streams.Error |
      foreach { $_.Exception.Message }
   $JobData.Warning = $Job.Pipe.Streams.Warning.Readall() | Out-String
             
   $Debug = $Job.Pipe.Streams.Debug
   if ($Debug.count -gt 2)
     { $JobData.Debug = $Debug[1..($Debug.count - 2)] | Out-String }
 
   $JobData.Verbose = $Job.Pipe.Streams.Verbose.Readall() | Out-String
 
   $JobData.Duration = '{0:f2} ms' -f ($Job.Ended - $Job.Started).totalmilliseconds

  if ($Job.State -ne 'Timeout')
    { $JobData.OutputCount = $Job.Pipe.EndInvoke($Job.Result).count }
                
 } #End Save Job data

 
$Sequence = 1 
 
$Jobs = 
  Foreach ($Argument in $ArgumentList)
   { 
     $Job = [powershell]::Create().
                          AddScript($JobScript).
                          AddArgument($argument)
 
     $Job.RunspacePool = $RunspacePool
 
     New-Object PSObject -Property @{
               Pipe     = $Job
               Result   = $Job.BeginInvoke()
               Args     = $argument
               Started  = $null
               Ended    = $null
               State    = $null
              }

     if ( $PSBoundParameters.ContainsKey('ResultHash') )
       { 
         $ResultHash[$Job.InstanceID] =        
           New-Object PSObject -Property @{
               Sequence    = $Sequence++
               Args        = $argument
               Started     = $null
               Ended       = $null
               Duration    = $null
               State       = $null
               HadErrors   = $null
               Verbose     = $null
               Warning     = $null
               Error       = $null
               Debug       = $null
               OutPutCount = $null
              }
        }                                                                          
  }
 
$Waiting = $Jobs 

While ( $Waiting )
  {
    foreach ($Job in  $Waiting )
      {
        if (
             ($Job.started -eq $null) -and
             ($job.pipe.Streams.Debug[0].Message -match 'Start')
            )
             {
               $StartTicks = $Job.pipe.Streams.Debug[0].Message -replace 'Start\(Ticks\) = (\d+)','$1'
               $Job.Started = [Datetime]::MinValue + [TimeSpan]::FromTicks($StartTicks)
             }
 
        if ($Job.Result.IsCompleted)
          {
            $EndTicks = $Job.pipe.Streams.Debug[-1].Message -replace 'End\(Ticks\) = (\d+)','$1'
            $Job.Ended = [Datetime]::MinValue + [TimeSpan]::FromTicks($EndTicks)
            $Job.State = $Job.pipe.InvocationStateInfo.State
            if ( $PSBoundParameters.ContainsKey('ResultHash') )
              { .$SaveJobData }
            $Job.Pipe.EndInvoke($Job.Result)
            $Job.Pipe.Dispose()
          } 
 
        if ( ($Job.Started) -and
             (get-date) -gt ($Job.Started + $MaxRunTime))
          {
            Write-Warning "Job $($job.Pipe.InstanceId) using argument $($job.args) timed out."
            $Job.Ended = (Get-Date)
            $Job.Pipe.Stop()
            $Job.State = 'Timeout'
            if ( $PSBoundParameters.ContainsKey('ResultHash') )
              { .$SaveJobData }
            $Job.Pipe.Dispose()
           }
        }
 
          $Waiting = $Jobs | Where { $_.Ended -eq $null }

          $Progress = 
            @{
               Activity = 'Running Scripts'
               Status   = "Completed $($Jobs.count - $Waiting.count) of $($Jobs.count)"
               PercentComplete = ($Jobs.count - $Waiting.count)  / $Jobs.count * 100
             }

          Write-Progress @Progress

          Start-Sleep -Seconds $RefreshInterval
 }
    
 
  $RunSpacePool.Close()
  $RunSpacePool.Dispose()
 
  } # End function
 
 

Chained AddScript methods breaks pipeline


#Chained .AddScript()

$Scriptblock = 
{
  $VerbosePreference = 'Continue'
  $DebugPreference   = 'Continue'

  Write-Verbose 'Verbose Message'
  Write-Debug 'Debug Message'
  Write-Output 'Pipeline Output'
}


$LogStart =
 {
   &{ 
      $DebugPreference = 'Continue'
      Write-Debug "Start(Ticks) = $((get-date).Ticks)"
    }
 }

$LogEnd =
 {
   &{
      $DebugPreference = 'Continue' 
      Write-Debug "End(Ticks) = $((get-date).Ticks)"
    }
 }


$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,1)
$RunspacePool.Open()

     $Job = [powershell]::Create().
              AddScript($LogStart).
              AddScript($ScriptBlock).
              AddScript($LogEnd)

     $Job.RunspacePool = $RunspacePool
     $job =
     New-Object PSObject -Property @{
                                     Pipe = $Job
                                     Result = $Job.BeginInvoke()
                                    }
     
 
 
  Do {} Until ($Job.Result.IsCompleted) 

  $Job.Pipe.Streams
  $Job.Pipe.EndInvoke($Job.Result)
  $Job.Pipe.Dispose()
  $RunspacePool.Close()
  $RunspacePool.Dispose()

''
'#'*20
''

#Single AddScript

  $Scriptblock = 
{
  $VerbosePreference = 'Continue'
  $DebugPreference   = 'Continue'

  Write-Verbose 'Verbose Message'
  Write-Debug 'Debug Message'
  Write-Output 'Pipeline Output'
}

$LogStart =
@'
   &{ 
      $DebugPreference = 'Continue'
      Write-Debug "Start(Ticks) = $((get-date).Ticks)"
    }
'@

$LogEnd =
@'
   &{
      $DebugPreference = 'Continue' 
      Write-Debug "End(Ticks) = $((get-date).Ticks)"
    }
'@

$JobScript = 
[Scriptblock]::Create($LogStart + $Scriptblock + $LogEnd)

$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,1)
$RunspacePool.Open()

     $Job = [powershell]::Create().AddScript($JobScript)

     $Job.RunspacePool = $RunspacePool
     $job =
     New-Object PSObject -Property @{
                                     Pipe = $Job
                                     Result = $Job.BeginInvoke()
                                    }
     
 
 
  Do {} Until ($Job.Result.IsCompleted) 

  $Job.Pipe.Streams
  $Job.Pipe.EndInvoke($Job.Result)
  $Job.Pipe.Dispose()
  $RunspacePool.Close()
  $RunspacePool.Dispose()

Accurate timeout for threads in runspace pool

Just a POC right now, but it seems to work:

function Invoke-ScriptAsync {
    [cmdletbinding()]
    param(
        [Parameter(Mandatory=$true,position=0)]
        [ScriptBlock] $ScriptBlock,

        [Parameter(Mandatory=$true)]
        [array]$ArgumentList,

        [Parameter()]
        [int]$MaxThreads = 5,

        [Parameter()]
        [int]$Timeout = 15,

        [Parameter()]
        [int]$RefreshInterval = 1
  )

$MaxRunTime = New-TimeSpan -Seconds $Timeout

$RunspacePool = [RunspaceFactory ]::CreateRunspacePool(1, $MaxThreads)
$RunspacePool.Open()

$LogStart =
@'
   &{ 
      $DebugPreference = 'Continue'
      Write-Debug "Start(Ticks) = $((get-date).Ticks)"
    }
'@

$LogEnd =
@'
   &{
      $DebugPreference = 'Continue' 
      Write-Debug "End(Ticks) = $((get-date).Ticks)"
    }
'@

$JobScript = 
[Scriptblock]::Create($LogStart + $Scriptblock + $LogEnd)

$SaveJobData = 
          {
            #Save Pipeline Streams and information

             $Job.HadErrors = $Job.Pipe.HadErrors

             $Job.Error = $Job.Pipe.Streams.Error |
                  foreach { $_.Exception.Message }

            $Job.Warning = $Job.Pipe.Streams.Warning.Readall() | Out-String
            
            $Debug = $Job.Pipe.Streams.Debug
            if ($Debug.count -gt 2)
              { $Job.Debug = $Debug[1..($Debug.count - 2)] | Out-String }

            $Job.Verbose = $Job.Pipe.Streams.Verbose.Readall() | Out-String

            $Job.Duration = '{0:f2} ms' -f ($Job.Ended - $Job.Started).totalmilliseconds

               
          } #End Save Job data

$Sequence = 1 
$Jobs = @()


Foreach ($Argument in $ArgumentList)
 { 
  $Job = [powershell ]::Create().
                        AddScript($JobScript).
                        AddArgument($argument)

  $Job.RunspacePool = $RunspacePool

  $Jobs += New-Object PSObject -Property @{
                                           Pipe        = $Job
                                           Result      = $Job.BeginInvoke()
                                           Sequence    = $Sequence++
                                           Args        = $argument 
                                           Started     = $null
                                           Ended       = $null
                                           Duration    = $null
                                           Finalized   = $False
                                           State       = $null
                                           HadErrors   = $null
                                           Verbose     = $null
                                           Warning     = $null
                                           Error       = $null
                                           Debug       = $null
                                          }                                          

 }


While ($Jobs | Where { $_.Finalized -eq $false })
  {
    foreach ($Job in ($Jobs |  Where { $_.Finalized -eq $False }))
      {
        if (
             ($Job.started -eq $null) -and
             ($job.pipe.Streams.Debug[0].Message -match 'Start')
            )
             {
               $StartTicks = $Job.pipe.Streams.Debug[0].Message -replace '[^0-9]'
               $Job.Started = [Datetime]::MinValue + [TimeSpan]::FromTicks($StartTicks)
             }

        if ($Job.Result.IsCompleted)
          {
            $EndTicks = $Job.pipe.Streams.Debug[-1].Message -replace '[^0-9]'
            $Job.Ended = [Datetime]::MinValue + [TimeSpan]::FromTicks($EndTicks)
            $Job.State = $Job.pipe.InvocationStateInfo.State
            $Job.Finalized = $true
            $Job.HadErrors = $Job.Instance.HadErrors
            .$SaveJobData
            $Job.Pipe.EndInvoke($Job.Result)
            $Job.Pipe.Dispose()
          } 

        if ( ($Job.Started) -and 
             (get-date) -gt ($Job.Started + $MaxRunTime))
          {
            Write-Warning "Job $($job.Pipe.InstanceId) using argument $($job.args) time out."
            $Job.Ended = (Get-Date)
            $Job.Pipe.Stop()
            $Job.State = 'Timeout'
            $Job.Finalized = $True
            .$SaveJobData
            $Job.Pipe.Dispose()
           }

        }

          Start-Sleep -Seconds $RefreshInterval
 }
   

  $RunSpacePool.Close()
  $RunSpacePool.Dispose()

  $Jobs | Select Sequence,Args,Started,Ended,Duration,State,HadErrors,Error,Warning,Debug,Verbose

  } # End function


  #Test Script 

  $ScriptBlock = 
  {
   $VerbosePreference = 'Continue'
   Write-Verbose "Processing $($args[0])"

   If ($args[0] -eq 13){ 
                        Write-Error 'Triskaidekaphobia exception' 
                        Write-Output 'Shaka, when the walls fell.'
                        Return
                       }

   If ($args[0] -eq 1) { Write-Warning 'Warning - one is the loneliest number.' }

   $DebugPreference = 'Continue'
   Write-Debug 'Debug Message'

   Write-Verbose "Sleeping for $($args[0]) seconds at $(Get-Date)"
   Start-Sleep -Seconds $args[0]
   Write-Output "I'm awake! I'm awake! The answer is $($args[0])!"
  }

Invoke-ScriptAsync -ScriptBlock $ScriptBlock -ArgumentList (1..15 | get-random -Count 15) -Timeout 10

Backing up Exchange Configurations, Pt. 2

Continuing the work on backing up Exchange configurations, I decided to simplify the script to back up the configurations to this:


$su = get-credential

$Target_Directory = 'C:\testfiles\Ex_ConfigExport'
$ExchangeServer = '006Exch-Srv1'

$SessionParams = 
   @{
     ConfigurationName = 'MicroSoft.Exchange'
     ConnectionURI     = "http://$ExchangeServer/powershell/"
     Authentication    = 'Kerberos'
     ErrorAction       = 'Stop'
     Credential        = $su
    }

$ExSession = New-PSSession @SessionParams

$Ex_Object_Folders = 
   @{
     Server =    @(
                    'MailboxServer',
                    'TransportServer',
                    'ClientAccessServer',
                    'UMServer'
                   )

     UM =        @(
                    'UMDialPlan'
                    'UMAutoAttendant'
                    'UMIPGateway'
                    'UMHuntGroup'
                   )

     Transport = @(
                    'TransportRule',
                    'JournalRule',
                    'SendConnector',
                    'ReceiveConnector',
                    'ForeignConnector',
                    'RoutingGroupConnector',
                    'AcceptedDomain',
                    'RemoteDomain'
                   )

     Database =   @(
                     'MailboxDatabase',
                     'PublicFolderDatabase',
                     'DatabaseAvailabilityGroup'
                   )

     Policy =     @(
                     'ActiveSyncMailboxPolicy',
                     'AddressBookPolicy',
                     'EmailAddressPolicy',
                     'ManagedFolderMailboxPolicy',
                     'OwaMailboxPolicy',
                     'RetentionPolicy',
                     'RetentionPolicyTag',
                     'RoleAssignmentPolicy',
                     'SharingPolicy',
                     'ThrottlingPolicy',
                     'UMMailboxPolicy'
                    )
    }
     
#Export set folder creation

$DirParams = @{
                ItemType    =  'Directory'
                Verbose     =  $true
                ErrorAction =  'Stop'
              }

Write-Verbose "Checking target root folder"
if ( -not ( Test-Path $Target_Directory ) )
  { New-Item  @DirParams -Path $Target_Directory }

$Export_Set = (get-date).tostring('yyyy-MM-dd_HH.mm.ss')

$Export_Set_Path = "$Target_Directory\$Export_Set"
Write-Verbose "Creating folder for this export set ($Export_Set)"
New-Item  @DirParams -Path $Export_Set_Path 

Write-Verbose "Exporting Exchange configuration objects"

foreach ($Ex_Object_Folder in $Ex_Object_Folders.keys)
   {
    $Ex_Object_Folder_Path = "$Export_Set_Path\$Ex_Object_Folder"
    New-Item @DirParams -Path $Ex_Object_Folder_Path

    foreach ( $Ex_Object_Type in $Ex_Object_Folders.$Ex_Object_Folder )
       {
        $Ex_Object_Type_Path = "$Ex_Object_Folder_Path\$Ex_Object_Type"
        New-Item @DirParams -Path $Ex_Object_Type_Path

        $SB = [ScriptBlock]::Create("Get-$Ex_Object_Type")
        Invoke-Command -ScriptBlock $SB -Session $ExSession |
          foreach {
             $_ | Export-Clixml "$Ex_Object_Type_Path\$($_.guid).clixml" -Verbose
           }
       }
   }
  Remove-PSSession $ExSession         
        

Again, right now this is in a testing phase and you’ll need to modify the server name and target directory, and you can remove the credential prompt and session parameter if it’s not necessary in your environment.

Running will create a set of exported configuration objects each time it’s run. Now I want to be able to compare one set to another to see what’s changed. This script will compare two of those configuration sets, and create objects that detail the objects and properties where it finds differences.



 <#
 .Synopsis
    Compare two export object sets
 .DESCRIPTION
    Compares deserialized objects from two directories of .clixml files.
 
 #>
     [CmdletBinding()]

     Param
     (
         # First export object set directory
         [Parameter(Mandatory=$true)]
         $ExportSetPath1,
 
         # Second export object set directory
         [Parameter(Mandatory=$true)]
         $ExportSetPath2
     )
 
     Begin
     {
       
      $CompareSetTimestamps = Get-Item $ExportSetPath1,$ExportSetPath2 |
       Select FullName,LastWriteTime |
       Sort LastWriteTime

      $ReferenceSetPath  = $CompareSetTimestamps[0].FullName
      $DifferenceSetPath = $CompareSetTimestamps[1].FullName

      $ExcludedProperties = @(
                              'PSShowComputerName',
                              'RunspaceId',
                              'OriginatingServer',
                              'WhenChanged'
                              )

      $ExcludeRegex = [regex]($ExcludedProperties -join '|')

     }
     Process
     {
     }
     End
     {
      $ReferenceSetFolders =
         Get-ChildItem $ReferenceSetPath -Directory

      Write-Verbose "Found $($ReferenceSetFolders.count) Export set folders"

      foreach ( $ReferenceSetFolder in $ReferenceSetFolders )
        { 
          Write-Verbose "***** Comparing $ReferenceSetFolder Objects*****`n"
 
          $ReferenceObjectTypeFolders =
           Get-ChildItem $ReferenceSetFolder.FullName -Directory

          foreach ( $ReferenceObjectTypeFolder in  $ReferenceObjectTypeFolders )
            {
              Write-Verbose "     *****Comparing  $ReferenceObjectTypeFolder Objects*****`n"
            
              $ReferenceObjectFiles = 
                Get-ChildItem $ReferenceObjectTypeFolder.FullName -File

              $DifferenceObjectTypeFolderPath = 
                       @(
                         $DifferenceSetPath,
                         $ReferenceSetFolder,
                         $ReferenceObjectTypeFolder
                         ) -join '\'

              $DifferenceObjectFiles = 
                Get-ChildItem $ReferenceObjectTypeFolder.FullName -File

              $AddedObjectFiles = 
                $DifferenceObjectFiles |
                Where { $ReferenceObjectFiles.name -notcontains $_.name }

              $DeletedObjectFiles = 
                $ReferenceObjectFiles |
                Where { $DifferenceObjectFiles.name -notcontains $_.name }

              $ComparedObjectFiles = 
              $ReferenceObjectFiles |
                Where { $DifferenceObjectFiles.name -Contains $_.name }

              foreach ( $ComparedObjectFile in $ComparedObjectFiles )
                {  
 
                  $DifferenceObjectFilePath = 
                    "$DifferenceObjectTypeFolderPath\$($ComparedObjectFile.Name)"

                  $DifferenceObject = Import-Clixml $DifferenceObjectFilePath

                  if ($DifferenceObject.WhenChanged -le $ComparedObjectFile.LastWriteTime)
                    { Continue }

                  Write-Verbose "   Change detected in $($ReferenceObjectTypeFolder.Name) $($DifferenceObject.Identity)"

                  $ReferenceObject = Import-Clixml $ComparedObjectFile.FullName

                  
                  Write-Debug "  Comparing properties of $($ReferenceObject.Identity)"
                  $Properties = 
                    $ReferenceObject.psobject.properties.name -notmatch $ExcludeRegex

                  foreach ( $Property in $Properties )
                    {
                      if ([string]$ReferenceObject.$Property -ne [string]$DifferenceObject.$Property)
                        {

                         Write-Verbose "    Found change in property $Property of $($ReferenceObjectTypeFolder.Name) $($ReferenceObject.Identity)`n"
                         Write-verbose "`n`nOld value = $($ReferenceObject.$Property)`n`nNew value = $($DifferenceObject.$Property)`n" 

                         [PSCustomObject]@{
                             Guid = $ReferenceObject.Guid
                             RefExportObjectPath = $ComparedObjectFile.FullName
                             DiffExportObjectPath = $DifferenceObjectFilePath
                             ObjectClass = $ReferenceObject.ObjectClass
                             Identity = $ReferenceObject.Identity
                             Property = $Property
                             RefWhenChanged = $ReferenceObject.WhenChanged
                             RefPropertyValue = $ReferenceObject.$Property
                             DiffWhenChanged = $DifferenceObject.WhenChanged
                             DiffPropertyValue = $DifferenceObject.$Property
                            }
                         
                        }

                  }#end property loop

              }#end object loop

           }#end  object type folder loop

      }#end export set folder loop

    }#end End block

For testing, I’m using this script to select the export sets to compare:


$Exportsets = Get-ChildItem C:\testfiles\Ex_ConfigExport -directory |
  select -expand fullname | 
  Out-GridView -OutputMode Multiple -Title "Select two export sets to compare."

If ($Exportsets.count -ne 2)
  {
    Write-Warning "Select two (and only two) export sets to compare"
    Return
  }

 $changes =  C:\scripts\Ex_Backup\Compare-ExportSets.ps1 $Exportsets[0] $Exportsets[1]

 $changes

I still need to address added and deleted objects between sets, but that’s relatively trivial compared to finding any changes in all the existing objects. This is not intended to provide a complete report, but to create a collection of objects containing the information necessary to produce it and provide enough information to easily investigate further. It’s also useful for doing ad-hoc comparisons of exports sets from arbitrary days.

In my environment it takes about 5.5 MB to save an export set to disk. These are plain-text files, and using NTFS compression on the folder will drop it down to about 3 MB, so for about 100MB of disk space I can keep about a month’s worth.

This is still obviously work in progress, and I welcome an comments or suggestions.

Backing up Exchange configuration

This is currently a work-in -progress. A recent outage caused by a configuration change in Exchange has made me realize we need some sort of configuration change control. To that end, I’ve got this script for backing up the configuration objects in Exchange. At this point is useful for backup up the objects as clixml files, which can be used to compare to current objects to see what’s changed, or help recreate configuration objects if necessary.

Starting with a root backup folder, it creates a set of subfolders (Server, UM, Transport, Database, and Policy), and then subfolders within those for each object type. All of this is laid out in the $Ex_Object_Folders, so it can be re-organized by changing that hash table. The object types are specified as the noun-part of whatever get-* command will return that object, and it uses the objet’s GUID as the file name.

On the first run it will create the necessary folders and get and export the object types listed for each one. On subsequent runs it will check the .whenchanged property of the object in Exchange, and compare it to the lastwritetime of the backup file it has for that object. If it’s changed since it was last exported the export is refreshed.

I’m hoping to get this set up as a Subversion repository and letting that provide the versioning. Welcome any comments from anyone who’s done anything similar, or has suggestions on a better way to do this.

In any case you’re welcome to use it, or any bits of it that look useful. You’ll need to specify a path to the root folder where you want your backups to be, and the name of an Exchange server to use for the session, since it’s written to not require an EMS shell.



$Target_Directory = '<Exchange backup directory root path>'
$ExchangeServer = '<Exchange server to use for session>'

$VerbosePreference = 'Continue'

$SessionParams = 
   @{
     ConfigurationName = 'MicroSoft.Exchange'
     ConnectionURI     = "http://$ExchangeServer/powershell/"
     Authentication    = 'Kerberos'
     ErrorAction       = 'Stop'
    }

$ExSession = New-PSSession @SessionParams

$Ex_Object_Folders = 
   @{
     Server =    @(
                   'MailboxServer',
                   'TransportServer',
                   'ClientAccessServer',
                   'UMServer'
                  )

     UM =        @(
                   'UMDialPlan'
                   'UMAutoAttendant'
                   'UMIPGateway'
                   'UMHuntGroup'
                  )

     Transport = @(
                    'TransportRule',
                    'JournalRule',
                    'SendConnector',
                    'ReceiveConnector',
                    'ForeignConnector',
                    'RoutingGroupConnector',
                    'AcceptedDomain',
                    'RemoteDomain'
                   )

     Database =   @(
                    'MailboxDatabase',
                    'PublicFolderDatabase',
                    'DatabaseAvailabilityGroup'
                   )

     Policy =     @(
                    'ActiveSyncMailboxPolicy',
                    'AddressBookPolicy',
                    'EmailAddressPolicy',
                    'ManagedFolderMailboxPolicy',
                    'OwaMailboxPolicy',
                    'RetentionPolicy',
                    'RetentionPolicyTag',
                    'RoleAssignmentPolicy',
                    'SharingPolicy',
                    'ThrottlingPolicy',
                    'UMMailboxPolicy'
                   )
    }
     
#Folder maintenance:

$DirParams = @{
                ItemType    =  'Directory'
                Verbose     =  $true
                ErrorAction =  'Stop'
              }

Write-Verbose "Checking target root folder"
if ( -not ( Test-Path $Target_Directory ) )
  { New-Item  @DirParams -Path $Target_Directory }

Write-Verbose "Checking Object folders"
foreach ($Ex_Object_Folder in $Ex_Object_Folders.keys)
   {
    if ( -not ( test-path "$Target_Directory\$Ex_Object_Folder\" ) )
      { New-Item @DirParams -Path $Target_Directory\$Ex_Object_Folder }

    foreach ( $Ex_Object_Type in $Ex_Object_Folders.$Ex_Object_Folder )
       {
        if ( -not ( test-path "$Target_Directory\$Ex_Object_Folder\$Ex_Object_Type" ) )
          { New-Item @DirParams -Path $Target_Directory\$Ex_Object_Folder\$Ex_Object_Type }
       }
    }

#Update backups of changed objects

foreach ($Ex_Object_Folder in $Ex_Object_Folders.keys)
 {
   foreach ( $Ex_Object_Type in $Ex_Object_Folders.$Ex_Object_Folder )
     {
       Write-Verbose "******************************************************`n"
      
      Write-Verbose "Checking for $Ex_Object_Type backups."
      $LastBackups = @{}
      $Config_Export_FIles = Get-ChildItem "$Target_Directory\$Ex_Object_Folder\$Ex_Object_Type\*.clixml"
      Foreach ($Config_Export_FIle in $Config_Export_FIles)
       {
         $LastBackups[$Config_Export_FIle.basename] = $Config_Export_FIle.LastWriteTime 
       }
     
      Write-Verbose "Found $($Config_Export_FIles.count) $Ex_Object_Type backup files."

      Write-Verbose "Getting $Ex_Object_Type object configs from Exchange."

       $SB = [ScriptBlock]::Create("Get-$Ex_Object_Type ")

       $Ex_Object_Configs =
         @(Invoke-Command -ScriptBlock $SB -Session $ExSession)

       Write-Verbose "Found $($Ex_Object_Configs.Count) Exchange $Ex_Object_Type objects."

       #Check for updated configs in Exchange.
       foreach ($Ex_Object_Config in $Ex_Object_Configs)
         {
           if ( $Ex_Object_Config.WhenChanged -gt $LastBackups["$($Ex_Object_Config.guid)"] )
             { 
               Write-Verbose "$Ex_Object_Type $($Ex_Object_Config.Name) has been updated since last backup."
               Write-Verbose "Last change: $($Ex_Object_Config.WhenChanged) Last backup: $($LastBackups["$($Ex_Object_Config.guid)"])"
               $Ex_Object_Config | 
               Export-Clixml "$Target_Directory\$Ex_Object_Folder\$Ex_Object_Type\$($Ex_Object_Config.guid).clixml" -Verbose
             }
         }
      }
  }        
    
  Remove-PSSession $ExSession         
        

Compare-GridView



function Compare-GridView
{
    [CmdletBinding()]
     
    Param
    (
        # Reference Object
        [Parameter(Mandatory=$true,
                   Position=0)]
        [ValidateScript({
          if ($_.psbase.count)
            { Throw 'Reference and Difference Objects must be scalar (single object). Arrays or collections not allowed.' }
          else {$true}
          })]

        $ReferenceObject,
 
        # Difference Object
        [Parameter(Mandatory=$true,
                   Position=1)]
        [ValidateScript({
          if ($ReferenceObject.gettype() -ne $_.gettype())
            { Throw 'ReferenceObject and DifferenceObject type mismatch. Objects must be of the same type.' }
          else {$true}
          })]

        $DifferenceObject,
 
        # Show all properties
        [switch]
        $IncludeAllProperties
    )
 
   End
    {
      $grid = 
        foreach ($Property in ($ReferenceObject.psobject.properties).name)
          {
            if (
                 ( $PSBoundParameters.IncludeAllProperties) -or                
                 ( $ReferenceObject.$Property -ne $DifferenceObject.$Property )
                )
              {
                [PSCustomObject]@{
                    Property = $Property
                    ReferenceObject = $ReferenceObject.$Property
                    DifferenceObject = $DifferenceObject.$Property
                   }
              }
          }
 
      $grid | Out-GridView
 
    }
 }

Us a script cache to speed up repetitive directory lookups

In an Active Directory environment there will be many instances where you’ll be doing report scripting that will involve repetitive resolution of identity references. To demonstrate, I’ll bring back the unused vacation notification script from a previous post on splatting parameters:

Param (
$ImportFile     =  './user_vacation_list.csv',
$EmailFrom      =  'HR@mycompany.com',
$EmailSubject   =  'Unused vacation reminder',
$SMTPServer     =  'mail.mycompany.com'
)

$AllEmailParams =
        @{
          From       = $EmailFrom
          Subject    = $EmailSubject
          SMTPServer = $SMTPServer
          }

Foreach ($User in inport-csv $ImportFile)
 {
   $ADUser = Get-ADUser $User.Name -Propertes Mail,Manager
   $Manager = Get-ADUser $User.Manager -Properties Mail

   $ThisEmailParams = 
        @{
          To   = $ADUser.Mail
          Cc   = $Manager.Mail
          Body = "You have $($User.VacationDays) unused vacation days."
         }

 Send-MailMessage @AllEmailParams @ThisEmailParams
 }

Now, in most any organization this script can easily result in having to look up the email address for the same manager many times, since there will be relatively few managers compared to the number of employees, and multiple employees having the same manager. We can speed up our script and reduce the load on our DC's by creating an in-script cache of manager email addresses we've already looked up, so we don't have to keep doing it over and over again.


Param (
$ImportFile     =  './user_vacation_list.csv',
$EmailFrom      =  'HR@mycompany.com',
$EmailSubject   =  'Unused vacation reminder',
$SMTPServer     =  'mail.mycompany.com'
)
 
$AllEmailParams =
        @{
          From       = $EmailFrom
          Subject    = $EmailSubject
          SMTPServer = $SMTPServer
          }

#Create an empty hash table for the cache
$Manager = @{} 
 
Foreach ($User in inport-csv $ImportFile)
 {
   $ADUser = Get-ADUser $User.Name -Propertes Mail,Manager

   #Check to see if the entry we want is already in cache, if not add it.
   if ($Manager[$ADUser.Manager] -eq $null)
     {
       $Manager[$ADUser.Manager] = 
        (Get-ADUser $User.Manager -Properties Mail).Mail
     }
   
   #Use the identity reference to get the entry from cache.
   $ThisEmailParams = 
        @{
          To   = $ADUser.Mail
          Cc   = $Manager[$ADUser.Manager]
          Body = "You have $($User.VacationDays) unused vacation days."
         }
 
 Send-MailMessage @AllEmailParams @ThisEmailParams
 }

By using the hash table created ahead of entering the processing loop:


#Create an empty hash table for the cache
$Manager = @{} 

We can test that to see if we already have an entry cached for this manager. If we don’t have it yet, we’ll go ahead and do that lookup and add it to our cache:

if ($Manager[$ADUser.Manager] -eq $null)
     {
       $Manager[$ADUser.Manager] = 
        (Get-ADUser $User.Manager -Properties Mail).Mail
     }

Then use the cache reference to get our manager’s email address

$ThisEmailParams = 
        @{
          To   = $ADUser.Mail
          Cc   = $Manager[$ADUser.Manager]
          Body = "You have $($User.VacationDays) unused vacation days."
         }

Now we only go to our DC once for each manager the first time we encounter a user with that manager in our input data, and for every subsequent user with that same manager we just retrieve it from our hash table.

Reporting scripts will frequently be focused on a particular group or type of objects in the directory and associations between them are linked by an identity reference in AD (DN, SID, guid) you’ll find many instances where you will end up having to resolve the same identity references and retrieve the same information for that object over and over again, and using a cache of already resolved indentities and data can both speed up your script and reduce the impact on your critical infrastructure servers.

Random Password generator

SANS:

$length = 15
{
    If ($length -lt 4) { $length = 4 }   #Password must be at least 4 characters long in order to satisfy complexity requirements.

    Do {
        $password =     $null 
        $hasupper =     $false   #Has uppercase letter character flag.
        $haslower =     $false   #Has lowercase letter character flag.
        $hasnumber =    $false   #Has number character flag.
        $hasnonalpha =  $false   #Has non-alphanumeric character flag.
        $isstrong =     $false   #Assume password is not strong until tested otherwise.
        
        For ($i = $length; $i -gt 0; $i--)
        {
            $x = get-random -min 33 -max 126              #Random ASCII number for valid range of password characters.
                                                          #The range eliminates the space character, which causes problems in other scripts.        
            If ($x -eq 34) { $x-- }                       #Eliminates double-quote.  This is also how it is possible to get "!" in a password character.
            If ($x -eq 39) { $x-- }                       #Eliminates single-quote, causes problems in scripts.
            If ($x -eq 47) { $x-- }                       #Eliminates the forward slash, causes problems for net.exe.
            If ($x -eq 96) { $x-- }                       #Eliminates the backtick, causes problems for PowerShell.
            If ($x -eq 48 -or $x -eq 79) { $x++ }         #Eliminates zero and capital O, which causes problems for humans. 
            
            $password = $password + [System.Char] $x      #Convert number to an ASCII character, append to password string.

            If ($x -ge 65 -And $x -le 90)  { $hasupper = $true }
            If ($x -ge 97 -And $x -le 122) { $haslower = $true } 
            If ($x -ge 48 -And $x -le 57)  { $hasnumber = $true } 
            If (($x -ge 33 -And $x -le 47) -Or ($x -ge 58 -And $x -le 64) -Or ($x -ge 91 -And $x -le 96) -Or ($x -ge 123 -And $x -le 126)) { $hasnonalpha = $true } 
            If ($hasupper -And $haslower -And $hasnumber -And $hasnonalpha) { $isstrong = $true } 
        } 
    } While ($isstrong -eq $false)
    
    $password

Replaced with:


Length = 15

$PasswordCharCodes = {33..126}.invoke()

#Exclude ",',/,`,O,0
34,39,47,96,48,79 | foreach {[void]$PasswordCharCodes.Remove($_)}

$PasswordChars = [char[]]$PasswordCharCodes 

do { 
    $NewPassWord =  $(foreach ($i in 1..$length) 
     { Get-Random -InputObject $PassWordChars }) -join '' 
   }

 until (
         ( $NewPassword -cmatch '[A-Z]' ) -and
         ( $NewPassWord -cmatch '[a-z]' ) -and
         ( $NewPassWord -imatch '[0-9]' ) -and 
         ( $NewPassWord -imatch '[^A-Z0-9]' )
       ) 
        
 $NewPassword 

Testing V4’s Where method and the pipeline

Continuing with the topic of the where() method and return types, I decided to to some bench testing.

I can get the same object type return from the where() collection method that I do from using the Where-Object comdlet by piping the collection to Set-Variable. But did I just give up the performance advantage using the method by sticking another pipeline into it?

Here’s my test setup:

$counter = 1..100

"`n"
'Where-Object cmdlet'

  Measure-Command {

   foreach ($count in $counter)
    {$result = $counter | where {$_ -match '33'}}

  } | select TotalMilliseconds | fl 

"`n"
'Where method, pipe to Set-Variable'
Measure-Command {

foreach ($count in $counter)
 {$counter.where({$_ -match '33'}) | Set-Variable result}

} | select TotalMilliseconds | fl 


"`n"
'Where method, return collection'
Measure-Command {

foreach ($count in $counter)
{$result = $counter.Where({$_ -match '33'})}

} | select TotalMilliseconds | fl 

And the results:

Where-Object cmdlet
TotalMilliseconds : 704.2918

Where method, pipe to Set-Variable
TotalMilliseconds : 203.6046

Where method, return collection
TotalMilliseconds : 160.8534

It gave up some performance piping to Set-Variable, but not much.

The Where-Object pipeline had to handle every item in the array.
The where() pipeline only had to handle one.
Filter left.