Tag Archives: powershell

Powershell: scan a file with regex and write the output

So, let’s say you have a biiiiig log file. There’s some info in there, like URLs, that you need them in a list.

Copy-pasting? Hell no, Powershell to the rescue!

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, February 2025
#

# Change the regex to fit your purposes
# and of course the input file
$regEx = 'https?://[^\s/$.?#].[^\s]*'
$inputFile = "C:\logs\mybiglog.txt"

$outputFile = [System.IO.Path]::Combine([System.IO.Path]::GetDirectoryName($inputFile), "out_$([guid]::NewGuid().ToString().Split('-')[0]).txt")

$content = Get-Content -Path $inputFile -Raw
$matches = [regex]::Matches($content, $regEx)
$matches | ForEach-Object { $_.Value } | Out-File -FilePath $outputFile

Have fun coding!

Stop CI/CD pipeline if a Powershell script contains errors

Contrary to “normal” languages like C# or Java, Powershell is not a compiled language, but rather an interpreted one. This means that instead of using a compiler, the Powershell Scripting Runtime Environment reads and executes the code line-by-line during runtime.

That has well known advantages -for example, you can change code on the spot- and disadvantages -e.g. performance. But one major disadvantage is that there are no compiler errors. That means that if you forget to close a parenthesis or a bracket, nothing works. It’s the silliest of mistakes but still crashes everything.

With Powershell being used in non-interactive environments, like Azure Functions, it’s becoming all the more important to guard against such errors.

Fortunately, there is a solution for this. Microsoft has published the PSScriptAnalyzer module (link) which includes the Invoke-ScriptAnalyzer (link) command. Running this against your code, you get a list of warnings and errors:

The best things is, you can include this in your CI/CD pipelines, e.g. in Azure Devops or Github.

So here’s an example of an Azure Devops pipeline task that checks for ParseErrors (meaning, the script is not readable) and stops the build in case such an error is found:

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, October 2024
#
- task: PowerShell@2
  displayName: Check for Powershell parsing errors
  inputs:
    targetType: 'inline'
    errorActionPreference: 'stop'
    pwsh: true
    script: | 
      Install-Module -Name PSScriptAnalyzer -Scope CurrentUser -Force
      Write-Host 'Performing code analysis using Microsoft Invoke-ScriptAnalyzer'
      $findings = Invoke-ScriptAnalyzer -Path '$(System.DefaultWorkingDirectory)' -Recurse -Severity ParseError,Error
      $findings | Format-List
      if (($findings | Where-Object { $_.Severity -eq 'ParseError' }).Count -gt 0) { Write-Warning "Parse error(s) were found, review analyser results."; exit 1 }   

Enjoy 😊

How to get a backup of your Azure Devops repository including all branches

While Azure Devops is widely used, Microsoft’s backup solutions are surprisingly thin. With people depending on it, individuals and enterprises alike, you’d expect a bit more.

There are various tools around, but here’s my version in the form of a Powershell script. What it does is:

  • Connects to a specific Azure Devops project and repo.
  • Lists all branches, downloads them using git and zips them.
  • The zip, one for every branch, is named Backup_yyyy-MM-dd_branch.zip.

Prerequisites are not much, but:

  • You need git installed and
  • you need a PAT with read access to your code (instructions here).

So here’s the script:

#
# Source: DotJim blog (https://dandraka.com)
# Jim Andrakakis, October 2024
# Updated September 2025, fix for projects and orgs containing spaces
#

# BackupBranches.ps1

param (
    [string]$organization = "MYORG",
    [string]$project = "MYPROJECT",
    [string]$repository = "MYREPO",
    [string]$backupFolder = "C:\Temp\DevOpsBranches",    
	[string]$branchFilter = "" # leave empty for all branches
)

Clear-Host
$ErrorActionPreference='Stop'
 
$pat = Read-Host -MaskInput -Prompt "Enter Personal Access Token for $($env:USERNAME) and $($project)/$($repository)"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$pat"))
 
$tempFolder = Join-Path $backupFolder $repository
$repoNoSpace = $repository.Replace(' ','%20')
$projNoSpace = $project.Replace(' ','%20')
 
Write-Host "[$([datetime]::Now.ToString('yyyy-MM-dd HH:mm:ss'))] Starting, output directory is $tempFolder"
 
# Ensure temp folder exists
if (-not (Test-Path -Path $tempFolder)) {
    New-Item -Path $tempFolder -ItemType Directory | Out-Null
}
 
# API URL for branches
$branchesApiUrl = "https://dev.azure.com/$organization/$projNoSpace/_apis/git/repositories/$repoNoSpace/refs?filter=heads/&api-version=6.0"
 
# Get all branches from the repository
$response = Invoke-RestMethod -Uri $branchesApiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}
 
$branchList = $response.value | Sort-Object -Property name
 
# Iterate through each branch
foreach ($branch in $branchList) {
	try {
		$branchName = $branch.name -replace "refs/heads/", ""

		# branch filter, if any
		if (-not ([string]::IsNullOrWhiteSpace($branchFilter)) -and ($branchName -notlike "*$branchFilter*")) {
			continue
		}
	 
		# Define the folder for the branch
		$branchNameStrilized = "$($branchName.Replace('/','_').Replace(' ','_'))"
		$branchFolder = "$tempFolder\$branchNameStrilized"
		 
		# Remove the folder if it exists from previous runs
		if (Test-Path -Path $branchFolder) {
			Remove-Item -Recurse -Force $branchFolder
		}
	 
		# Clone the specific branch
		$gitUrl = "https://dev.azure.com/$organization/$projNoSpace/_git/$repoNoSpace"
		Write-Host "Cloning branch '$branchName' from $gitUrl to $branchFolder"
		$gitResp = [string] (& git clone --branch $branchName --single-branch $gitUrl $branchFolder 2>&1)
		if ($gitResp -like "*fatal*") {
			Write-Error "Error cloning branch '$branchName': $gitResp"
		}
	 
		# Zip the branch folder
		$backupDate = [datetime]::Now.ToString('yyyy-MM-dd')
		$zipFilePath = "$tempFolder\Backup_$($backupDate)_$($branchNameStrilized).zip"
		if (Test-Path $zipFilePath) {
			Remove-Item $zipFilePath
		}
		Compress-Archive -CompressionLevel Fastest -Path "$branchFolder\*" -DestinationPath $zipFilePath
	 
		Write-Host "Branch '$branchName' zipped to $zipFilePath"
	 
		# Clean up branch folder after zipping
		Remove-Item -Recurse -Force $branchFolder
	}
	catch {
		Write-Warning $_
	}
}
 
Write-Host "[$([datetime]::Now.ToString('yyyy-MM-dd HH:mm:ss'))] Finished, $($response.value.Count) branches processed."

Usage example:

BackupBranches.ps1 -organization 'BIGBANK' -project 'KYCAML' -repository 'KYCAMLapiV2' -backupFolder '\\backupfileserver\codebackups\'

Powershell: How to store secrets the right way

There are secrets that can be deadly.

Here we’re not going to talk about this kind 😊 But that doesn’t mean it’s not important.

It happens quite often that a script you need to run needs access to a resource, and for this you need to provide a secret. It might be a password, a token, whatever.

The easy way is obviously to have them in the script as variables. Is that a good solution?

If you did not answer NO THAT’S HORRIBLE… please change your answer until you do.

Ok so you don’t want to leave it lying around in a script. You can ask at runtime, like this:

$token = Read-Host -Prompt "Please enter the connection token:" -AsSecureString

That’s definitely not as bad. But the follow up problem is, the user needs to type (or, most probably, copy-paste) the secret every time they run the script. Where do the users store their secrets? Are you nudging them to store it in a notepad file for convenience?

In order to keep our systems safe, we need a way that is both secure and convenient.

That’s why using the Windows Credential Manager is a much, much better way. The users only have to recover the secret once, and then they have it stored in a safe way.

Here’s an example of how you can save the secret in Windows Credential manager. It uses the CredentialManager module.

# === DO NOT SAVE THIS SCRIPT ===

# How to save a secret

# PREREQUISITE: 
# Install-Module CredentialManager -Scope CurrentUser

$secretName = 'myAzureServiceBusToken' # or whatever

New-StoredCredential -Target $secretName -Username 'myusername' -Pass 'mysecret' -Persist LocalMachine

And here’s how you can recover and use it:

# How to use the secret

# PREREQUISITE: 
# Install-Module CredentialManager -Scope CurrentUser

$secretName = 'myAzureServiceBusToken' # or whatever

$cred=Get-StoredCredential -Target $secretName
$userName = $cred.UserName
$secret = $cred.GetNetworkCredential().Password

# do whatever you need with the secret

Just for completeness, here’s an example of how to call a REST API with this secret. I imagine that’s one of the most common use cases.

#
# Source: DotJim blog (https://dandraka.com)
# Jim Andrakakis, April 2024
#

# PREREQUISITES: 
# 1. Install-Module CredentialManager -Scope CurrentUser
# 2. New-StoredCredential -Target 'myRESTAPICredential' -Username 'myusername' -Pass 'mysecret' -Persist LocalMachine

# === Constants ===
$uri = 'https://myhost/myapi'
$credName = 'myRESTAPICredential'
$fileName = 'C:\somepath\data.json'
# === Constants ===

$cred=Get-StoredCredential -Target $credName
$pair="$($cred.UserName):$($cred.GetNetworkCredential().Password)"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($pair))
$basicAuthValue = "Basic $encodedCreds"

$headers = @{
    Authorization = $basicAuthValue;
    ContentType = 'application/json';
    Accept = 'application/json'
}

try {
    $resp = Invoke-WebRequest -UseBasicParsing -Uri $uri -Headers $headers -Method Post -InFile $fileName
}
catch {
    $errorMsg = "Error sending file '$fileName', exception in line $($_.InvocationInfo.ScriptLineNumber): $_.Exception.Message $_"
    Write-Warning $errorMsg     
}

Powershell: Get Active Directory group members (without the need to install the ActiveDirectory module)

Powershell offers a number of Active Directory (AD for short) commandlets to make an AD admin’s life a little easier. For example, if you need to get a list of members from an AD group, you can use something like:

Get-ADGroupMember -Identity 'Enterprise Admins' -Recursive

The problem is that this doesn’t work everywhere. The ActiveDirectory module is not a “normal” one you can install with Install-Module; instead, you need to install a Windows feature, either from Control Panel or by using the Add-WindowsCapability commandlet.

But you don’t have to use this module. You can use something that’s available everywhere, the adsiSearcher type accelerator.

So here are a couple of scripts I came up with (credits where they’re due). The first searches through all groups, finds all the ones that match a string and lists all their members.

#
# Source: DotJim blog (https://dandraka.com)
# Jim Andrakakis, January 2024
#
  
# ===== Parameters =====
  
param(
    [string]$searchString = 'accounting'
)
  
# ======================

Clear-Host
$ErrorActionPreference='Stop'

# === Get all groups ===
$objSearcher=[adsisearcher]'(&(objectCategory=group))'
$objSearcher.PageSize = 20000 # may need to adjust, though should be enough for most cases

# specify properties to include
$colProplist = "name"
foreach ($i in $colPropList) { $objSearcher.PropertiesToLoad.Add($i) | out-null } 
	
$colResults = $objSearcher.FindAll()

foreach ($objResult in $colResults)
{
    #group name
    $group = $objResult
    $groupname = ($objResult.Properties).name    

    if (-not ($groupname[0].ToLower().Contains($searchString.ToLower()))) {
        continue
    }

    Write-Host "Members of $groupname [$($group.Path)]"    

    $Group = [ADSI]$group.Path
    $Group.Member | ForEach-Object {
        $Searcher = [adsisearcher]"(distinguishedname=$_)"
        $member = $searcher.FindOne()
        $userName = $member.Properties.samaccountname
        $name = $member.Properties.displayname

        Write-Host "`t[$userName]`t$name"
    }
}

The second displays all details of all users whose name matches a substring.

#
# Source: DotJim blog (https://dandraka.com)
# Jim Andrakakis, January 2024
#
   
# ===== Parameters =====
   
param(
    [string]$searchString = 'Papadomanolakis'
)
   
# ======================
 
Clear-Host
$ErrorActionPreference='Stop'
 
# === Get all groups ===
$objSearcher=[adsisearcher]"(&(objectClass=user)(displayname=*$($searchString)*))"
$objSearcher.PageSize = 20000 # may need to adjust, though should be enough for most cases
#$objSearcher.FindOne().Properties.Keys
$objSearcher.FindAll() | % { $_.Properties }

And the third one is a brilliant one-liner by Jos Lieben that lists all groups of a user.

$userName = $env:USERNAME # change if different user needed
([ADSISEARCHER]"(member:1.2.840.113556.1.4.1941:=$(([ADSISEARCHER]"samaccountname=$userName").FindOne().Properties.distinguishedname))").FindAll().Properties.distinguishedname -replace '^CN=([^,]+).+$','$1'

Hope that helps. Enjoy! 😊

RabbitMQ: list queues and exchanges with Powershell

As I haven’t yet had the time to set up a proper devops deployment pipeline from my development RabbitMQ to UAT and then to production (don’t yell at me, I know, I’ll get to it… eventually), I find myself comparing instances in order not to forget adding a queue or an exchange.

So I wrote this script, that produces a diff-friendly text file that I can use to compare instances and see what’s missing:

#
# Source: DotJim blog (https://dandraka.com)
# Jim Andrakakis, April 2023
#
 
# ===== Parameters =====
 
param(
    [string]$rqServer = 'http://myServer:15672', # better use HTTPS though
    [string]$rqVhostName = 'myVhost',
    [string]$rqUsername = 'myUsername', # this user needs at least 'Management' permissions to post to the REST API
    [string]$rqPassword = 'myPassword',
    [string]$outDir = 'C:\temp'
)
 
# ======================
 
Clear-Host
$ErrorActionPreference = 'Stop'
$WarningPreference = 'Continue'

$plainCredentials = "$($rqUsername):$($rqPassword)"
$encodedCredentials = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($plainCredentials))
$authHeader = "Basic " + $encodedCredentials

[string]$rqUrlQueues = "$rqServer/api/queues/$rqVhostName/"
[string]$rqUrlExchanges = "$rqServer/api/exchanges/$rqVhostName/"

[string]$filename = [System.IO.Path]::Combine($outDir, [guid]::NewGuid().ToString().Split('-')[0] + ".txt")
Out-File -FilePath $filename -Encoding utf8 -Append -InputObject "Server : $rqServer"
Out-File -FilePath $filename -Encoding utf8 -Append -InputObject "VHost : $rqVhostName"
 
$respQueues = Invoke-WebRequest -Method Get -Uri $rqUrlQueues -Headers @{'Authorization'= $authHeader} 
$respExchanges = Invoke-WebRequest -Method Get -Uri $rqUrlExchanges -Headers @{'Authorization'= $authHeader} 

$queuesJson = ConvertFrom-Json $respQueues.Content
$queuesJson | Sort-Object -Property name | % { Out-File -FilePath $filename -Encoding utf8 -Append -InputObject  "Queue : $($_.name)" }

$exchangesJson = ConvertFrom-Json $respExchanges.Content
$exchangesJson | Sort-Object -Property name | % { Out-File -FilePath $filename -Encoding utf8 -Append -InputObject  "Exchange : $($_.name)" }

Write-Host "Finished, output written to $filename"

Jams Scheduler: Get info about the currently running job in Powershell

When writing jobs in JAMS Scheduler it’s very common to need info about the job that’s currently running. Logging is an obvious need but it’s not the only one.

You’d think the way to do this should be prominently displayed in their otherwise very good documentation, but for whatever reason it’s not. I couldn’t find this anywhere.

Fortunately their support is beyond excellent -it’s hands down the best support I’ve ever worked with- and they gave me the answer as soon as I asked:

Import-Module JAMS
$currentJobEntry = Get-JAMSEntry <<JAMS.JAMSEntry>>

That’s it! It’s really as simple as that. That’s a sample that you can use to see what properties you get:

Import-Module JAMS
$currentJobEntry = Get-JAMSEntry <<JAMS.JAMSEntry>>

# to print all info
$currentJobEntry | select *

# or to get individual info
$jobId = $currentJobEntry.JAMSEntry
$jobName = $currentJobEntry.Name
$jobError = $currentJobEntry.Error

Write-Host "Job $jobId [$jobName]"
Write-Host "Error (if any): $jobError"

# === all properties with example values ===
#Entry               : 21760
#ExtensionData       : System.Runtime.Serialization.ExtensionDataObject
#JAMSEntry           : 21760
#RON                 : 3565757
#AvgElapsedTime      : 
#TodaysDate          : 2/17/2023 12:00:00 AM
#ScheduledTime       : 2/17/2023 10:35:40 AM
#ScheduledTimeUTC    : 2/17/2023 9:35:40 AM
#OriginalHoldTime    : 2/17/2023 10:35:39 AM
#OriginalHoldTimeUTC : 2/17/2023 9:35:39 AM
#HoldTime            : 2/17/2023 10:35:39 AM
#HoldTimeUTC         : 2/17/2023 9:35:39 AM
#StartTime           : 2/17/2023 10:35:40 AM
#StartTimeUTC        : 2/17/2023 9:35:40 AM
#CompletionTime      : 1/1/0001 12:00:00 AM
#CompletionTimeUTC   : 1/1/0001 12:00:00 AM
#MethodId            : 16
#MethodName          : PowerShell
#ParentFolderID      : 96
#ParentFolderName    : \el\MyFolder\Test
#JobID               : 1747
#ExecutingAgentID    : 1
#ExecutingAgentName  : Agent JAMS App-Server
#InitiatorType       : ManualSubmit
#InitiatorID         : 0
#InitiatorUid        : 00000000-0000-0000-0000-000000000000
#ProcessID           : 5408
#SchedulingPriority  : 0
#ExecutionPriority   : 0
#FinalStatusCode     : 0
#FinalSeverity       : Success
#RetainOption        : Default
#RetainTime          : 
#RestartCount        : 0
#CurrentState        : Executing
#Tags                : 
#Debug               : False
#Held                : False
#Icon                : Default
#IconPermanent       : False
#IconMessage         : 
#LogFilename         : D:\JAMS\Logs\Logs\test-jobinfo_003668BD.log
#TempFilename        : D:\JAMS\Temp\test-jobinfo_003668BD.ps1
#SubmittedBy         : MYDOMAIN\myuser
#Name                : test-jobinfo
#JobName             : test-jobinfo
#DisplayName         : test-jobinfo
#FinalStatus         : 
#Note                : 
#JobStatus           : 
#ReconnectAgentName  : MYSERVER
#Source              : 
#JAMSId              : 939fb4c8-ce31-4e1a-8704-10258e85c003
#WFNextTimer         : 1/1/0001 12:00:00 AM
#WFState             : 0
#WFInstance          : 00000000-0000-0000-0000-000000000000
#AuditTrail          : {}
#WFTracking          : {}
#Parameters          : {[JAMSTraceLevel, MVPSI.JAMS.EntryParam], [PSExecutionPolicyPreference, MVPSI.JAMS.EntryParam], 
#                      [ErrorActionPreference, MVPSI.JAMS.EntryParam]}
#Elements            : {MaintenanceWindow, SendEMail, SendEMail}
#SourceElements      : {}
#Properties          : {ExecuteAs: JAMS, HomeDirectory: C:\JamsWorkingFolder, SingleInstanceAction: AllowMultiple, 
#                      NotifyEMail: ...}
#ExecuteAsName       : JAMS
#ExecuteAsID         : 1
#LoadedFrom          : Server: MYSERVER.mycompany.local/Default
#BatchQueue          : Queue: 
#BatchQueueName      : 
#BatchQueueID        : 0
#Calendar            : Calendar: 
#CalendarName        : 
#CalendarID          : 0
#Agent               : Agent: Agent JAMS App-Server
#AgentID             : 1
#AgentName           : Agent JAMS App-Server
#MinimumSeverity     : Warning
#Job                 : Job: test-jobinfo
#LogFile             : System.ServiceModel.Dispatcher.StreamFormatter+MessageBodyStream
#Modified            : False
#NewObject           : False
#Validated           : False
#InEdit              : False
#HasErrors           : False
#Error               : 

RabbitMQ: How to publish (upload) a file to a queue via Powershell using REST

As part of my job, this is something I use a lot. And the thing is, it’s quite easy, it’s just an Invoke-WebRequest. Here’s how I do it:

#
# Source: DotJim blog (https://dandraka.com)
# Jim Andrakakis, January 2023
#

# ===== Parameters =====

param(
    [string]$fileName = 'C:\temp\uploadinfo.json',
    [string]$rqServer = 'http://myServer:15672', # better use HTTPS though
    [string]$rqVhostName = 'myVhost',
    [string]$rqQueueName = 'myQueue',
    [string]$rqExchangeName = 'amq.default', # or your exchange name
    [string]$rqUsername = 'myUser', # this user needs at least 'Management' permissions to post to the REST API
    [string]$rqPassword = 'myPass',
	# RabbitMQ has a recommended message size limit of 128 MB
    # See https://www.cloudamqp.com/blog/what-is-the-message-size-limit-in-rabbitmq.html
    # But of course depending on your app you might want to set it lower
	[int]$rqMessageLimitMB = 128		
)

# ======================

Clear-Host
$ErrorActionPreference = 'Stop'
$WarningPreference = 'Continue'

[string]$rqUrl = "$rqServer/api/exchanges/$rqVhostName/$rqExchangeName/publish"

# Sanity check
if (-not (Test-Path $fileName)) {
    Write-Error "File $fileName was not found"
}

# Check RabbitMQ size limit
[int]$rqMessageLimit = $rqMessageLimitMB * 1024 * 1024 
[long]$fileSize = (Get-Item -Path $fileName).Length
if ($fileSize -gt $rqMessageLimit) {
    Write-Error "File $fileName is bigger that the maximum size allowed by RabbitMQ ($rqMessageLimitMB MB)"
}

$plainCredentials = "$($rqUsername):$($rqPassword)"
$encodedCredentials = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($plainCredentials))
$authHeader = "Basic " + $encodedCredentials

[string]$content = Get-Content -Path $fileName -Encoding UTF8
$msgBase64 = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($content))
$json = "{`"properties`":{`"content_type`":`"application/json`",`"delivery_mode`":2},`"routing_key`":`"$rqQueueName`",`"payload`":`"$msgBase64`",`"payload_encoding`":`"base64`"}"
$resp = Invoke-WebRequest -Method Post -Uri $rqUrl -Headers @{'Authorization'= $authHeader} -Body $json
if([math]::Floor($resp.StatusCode/100) -ne 2) {
    Write-Error "File $fileName could not be posted, error $($resp.BaseResponse)"
}

Write-Host "File $fileName was posted to $rqUrl"

My script cheat sheet

That’s not a post, at least in the classical sense 😊 Rather it’s a collection of small scripts, that I will keep updating, for me to find easily. No rocket science, just small everyday stuff that I find myself googling again and again.

Powershell: Basic CATCH block

$ErrorActionPreference='Stop'
Clear-Host

# PLEASE PLEASE PLEASE do not write logs where the program is
# See https://dandraka.com/2021/11/04/dont-write-logs-in-program-files/
$errorLogFolder = "C:\logs\"
$errorFile = Join-Path $errorLogFolder "$([System.IO.Path]::GetFileNameWithoutExtension($PSCommandPath))_errors_$([guid]::NewGuid().ToString().Split("-")[0]).txt" 

$errorList = New-Object -TypeName System.Collections.ArrayList
try {
    # say you're processing a file or whatever
    $fileName = "C:\data\lalala.txt"
    # throw an error to show what the log looks like
    $i = 3/0
}
catch {
    $errorMsg = "Error while processing file '$($fileName)'`nException $($_.Exception.GetType().FullName): $($_.Exception.Message)`nException location:`n$($_.InvocationInfo.ScriptName)`n$($_.InvocationInfo.ScriptLineNumber):$($_.InvocationInfo.Line)"
    # you can add the error(s) to a list and report at the end
    $errorList.Add($errorMsg) | Out-Null
    # or, alternatively, immediately write it into a file
    Out-File -FilePath $errorFile -InputObject $errorMsg
    Write-Warning $errorMsg      
}

The error log produced looks like this:

Error while processing file 'C:\data\lalala.txt'
Exception System.Management.Automation.RuntimeException: Attempted to divide by zero.
Exception location:
C:\code\MyDataProcessingScript.ps1
11:    $i = 3/0

An important and often overlooked point, and the reason I updated this cheat sheet, is $_.InvocationInfo.ScriptName. That’s important because sometimes the error doesn’t happen on your main script, but in a script or module that the main script calls. So if all you have is a line number you’re left scratching your head wondering, say, how can an IO error happen in a line that splits a string -because you’re looking at the wrong script.

Powershell: Get the first X bytes of a file.

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, September 2022
#

# ===== Change here =====
$path = 'C:\somepath\somehugelogfile.log'
$pathOut = 'C:\temp\sneakpeak.txt'
$numBytes = 10000
# =======================

$ErrorActionPreference='Stop'
Clear-Host

$bytes = Get-Content -Path $path -Encoding byte -TotalCount $numBytes
$str = [System.Text.Encoding]::UTF8.GetString($bytes)
Out-File -FilePath $pathOut -InputObject $str

Powershell: Find json files with a duplicate attribute (same value in multiple files)

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, January 2025
#

# Modify the path and the attribute (SomeAttribute) as needed in the highlighted lines

$path = 'C:\somepath'
$fileList = Get-ChildItem -Path $path -Filter '*.json'
$valueIndex = [System.Collections.Generic.Dictionary[string,string]]::new(); $fileList | % { $json = (Get-Content -Path $_.FullName | ConvertFrom-Json -AsHashtable); $valueIndex.Add($_.FullName, $json.SomeAttribute) }
$duplicateValueList = [System.Collections.Generic.List[string]]::new(); $valueIndex.Values | Group | Where{$_.Count -gt 1} | % { $duplicateValueList.Add($_.Name) }
$valueIndex.Keys | % { if ($duplicateValueList.Contains($valueIndex[$_])) { Write-Host "File = $($_) `t Value = $($valueIndex[$_])" } }

Powershell: Remove files older than X days recursively.

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, September 2022
#

# ===== Change here =====
$path = 'C:\somepath\'
$filter = '*.xml'
$numDays = 30 
# =======================

$ErrorActionPreference='Stop'
Clear-Host

Get-ChildItem -Path $path -Filter $filter -Recurse | Where-Object {($_.LastWriteTime -lt (Get-Date).AddDays($numDays * -1))} | Remove-Item

Powershell: Change information in XML files en masse.

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, September 2022
#

# ===== Change here =====
$dir = 'C:\somepath\'
# =======================

$ErrorActionPreference='Stop'
Clear-Host

$list = Get-ChildItem -Path $dir -Filter '*.xml'
foreach($file in $list) {
    [xml]$xml=Get-Content -Path $file.FullName -Encoding UTF8
    # customize the XML paths below
    $customerName = $xml.PrintJob.DocumentHeader.CustomerName 
    if ([string]::IsNullOrWhiteSpace($customerName )) {
        continue
    }
    $xml.PrintJob.DocumentBody.RecipientName = $customerName 
    $xml.Save($file.FullName)
}

Powershell: Change encoding of files.

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, October 2024
#

# !!! Prerequisite: Powershell 7 or later
# Does not work with Powershell 5.x

# ===== Change here =====
$dir = 'C:\somepath\'
$filter = '*.json'
$fromEncoding = 'utf8BOM'
$toEncoding = 'utf8'
# =======================

$ErrorActionPreference='Stop'
Clear-Host

Get-ChildItem -Path $dir -Filter $filter | % { $c = Get-Content -Path $_.FullName -Encoding $fromEncoding; $c | Set-Content -Encoding $toEncoding -Path $_.FullName }

Windows command line: Change permissions of all files and directories in a path recursively.

CD C:\somepath
FOR /D %o IN (*.*) DO echo Y| cacls %o /T /G "NT AUTHORITY\Authenticated Users":F

Here “NT AUTHORITY\Authenticated Users” stands for the authenticated users group of the local machine; “F” stands for Full Permissions.

Linux command line: copy file from one PC to another

# scp -r /path/to/file USERNAME@IP_OF_TARGET:/path/to/dir
scp -r /home/dimitris/Downloads/Win11.iso dimitris@192.168.0.5:/home/dimitris/Downloads

Powershell: Create and use a dictionary

# dictionary definition
$myIndex = New-Object -TypeName 'System.Collections.Generic.Dictionary[string,string]'

# add some values
$myIndex.Add([guid]::NewGuid().ToString(),'lalala')
$myIndex.Add([guid]::NewGuid().ToString(),'hohoho')

# iterate through keys and values
foreach($myId in $myIndex.Keys) {
	$myValue = $myIndex[$myId]
	Write-Host "Id = '$myId' Value = '$myValue'"
}

Powershell: Get info from multiple XML files and write into CSV

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, October 2022
#

# The file filter here is based on date from-to, 
# but obvs you can change Where-Object to match size or name or whatever.

# The CSV separator was chosen to be tab (`t) because this is 
# very Excel-friendly.

# ===== Change here =====
$path = 'E:\somepath'
$outPath = "C:\temp\out-$([guid]::NewGuid().ToString().Split('-')[0]).csv"

$strDateTimeFrom = "2022-09-29 18:00:00"
$strDateTimeTo = "2022-09-29 20:00:00"
# =======================

Clear-Host
$ErrorActionPreference='Stop'

[DateTime]$dateTimeFrom = [DateTime]::Parse($strDateTimeFrom)
[DateTime]$dateTimeTo = [DateTime]::Parse($strDateTimeTo)

$filesList = Get-ChildItem -Path $path -Recurse -Filter '*.xml' | Where-Object { ($_.LastWriteTime -gt $dateTimeFrom) -and ($_.LastWriteTime -lt $dateTimeTo) }

$dataList = New-Object -TypeName 'System.Collections.Generic.List[string]'
# change this to match the XML info below
$dataList.Add("CompanyName`tInvoiceId`tTemplateId`tDocumentId`tFileName")
foreach($file in $filesList) {
    $fileFullName = $file.FullName
    $fileName = $file.Name
    [xml]$xml = Get-Content -Path $fileFullName
    # that's the part where you specify what info you need from the XML
    # my XMLs have multiple Document nodes per file, that's why I need a loop
    foreach($document in $xml.PrintJob.Documents.Document) {
        $documentId = $document.DocumentHeader.DocumentId
        $templateId = $document.DocumentHeader.TemplateId
        $invoiceId = $document.ArchiveAttributes.InvoiceId
        $custName = $document.DocumentHeader.Addresses.Recipient.CompanyName

        $dataList.Add("$custName`t$invoiceId`t$templateId`t$documentId`t$fileName")
    }
}

Out-File -FilePath $outPath -Encoding utf8 -InputObject $dataList
Write-Host "Finished, $($filesList.Count) files processed"

Powershell: Copy XML files depending on their data plus data from a database

Clear-Host
$ErrorActionPreference='Stop'

# === Parameters ===
$path = 'D:\dataFilesPath'
$copyPath = 'D:\copyFilesPath'
$dbServer = 'mySqlServer'
$dbName = 'Customers'
$sql = 'SELECT CustomerName FROM Customers WHERE Active=1'
# === Parameters ===

$res = Invoke-Sqlcmd -ServerInstance $dbServer -Database $dbName -Query $sql

$exclusionList = New-Object -TypeName 'System.Collections.Generic.List[string]'

# if a different field of the query is needed, change [0]
$res | % { $exclusionList.Add($_[0].ToLowerInvariant().Trim()) }

if (-not (Test-Path -Path $copyPath)) { New-Item -ItemType Directory -Path $copyPath} $list = Get-ChildItem -Path $path -Filter '*.xml' -Recurse

$cnt=0
foreach($file in $list) {
    [xml]$contents = Get-Content -Path $file.FullName -Encoding UTF8
    # obvs you'll need to change the XPATH 
    # to match your XML structure
    $customerName = $contents.Data.CustomerData.Customer_Name.'#cdata-section'.ToLowerInvariant().Trim()

    if ($exclList.Contains($customerName)) {
        continue
    }
    Write-Host "Found $customerName"

    Copy-Item -Path $file.FullName -Destination $copyPath
    $cnt++
}

Write-Host "Finished, $cnt files copied to $copyPath"

Chocolatey: My dev machine install list

choco install notepadplusplus
choco install winmerge -y 
choco install vscode -y 
choco install vscode-powershell -y 
choco install vscode-csharp -y 
choco install vscode-gitlens -y 
choco install git -y 
choco install tortoisegit -y 
choco install svn -y 
choco install tortoisesvn -y 
choco install postman -y 
choco install soapui -y 
choco install sql-server-management-studio -y

choco install intellijidea-community
choco install openjdk8

choco install visualstudio2019professional --package-parameters " --add Microsoft.VisualStudio.Workload.Azure --add Microsoft.VisualStudio.Workload.ManagedDesktop --add Microsoft.VisualStudio.Workload.NetCoreTools --add Microsoft.VisualStudio.Workload.NetWeb --add Microsoft.VisualStudio.Workload.Universal --includeRecommended --includeOptional --passive --locale en-US" -y
choco install visualstudio2022professional --package-parameters " --add Microsoft.VisualStudio.Workload.Azure --add Microsoft.VisualStudio.Workload.ManagedDesktop --add Microsoft.VisualStudio.Workload.NetCoreTools --add Microsoft.VisualStudio.Workload.NetWeb --add Microsoft.VisualStudio.Workload.Universal --includeRecommended --includeOptional --passive --locale en-US" -y
choco install dotnet-5.0-sdk -y
choco install dotnet-6.0-sdk -y

choco install ServiceBusExplorer -y

Install-Package \\fileserver\share\JamsScheduler\SetupClientx64.msi

Powershell: Keep the lights on

Clear-Host
Add-Type -AssemblyName System.Windows.Forms
Write-Host 'Starting...'
$WShell = New-Object -com "Wscript.Shell"
while ($true)
{
  $WShell.sendkeys("{F16}")
  Start-Sleep -Seconds 180
}

Powershell: Quickly format (“pretty print”) XML files

cd C:\mydata
gci *.xml | % { [xml]$x=get-content $_ ; $x.Save($_.FullName) }

Powershell: Archive files (zip + delete)

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, November 2022
#

# Prerequisite: 7zip is installed in the system.

# =================================
# This script zips everything found in $archiveName that 
# has a ModifiedDate after $dateFromStr and matches $filter.
# Inside $archivePath, it creates one dir per run and 
# inside this, one 7zip file per month plus one txt file
# that has the 7zip contents.
# E.g.
# C:\OldLogFiles
#     Run-LogFileArchive-20221122-112015
#         Archive-LogFileArchive-20200301-20200401.7z.001
#         Archive-LogFileArchive-20200301-20200401.txt
#         Archive-LogFileArchive-20200401-20200501.7z.001
#         Archive-LogFileArchive-20200401-20200501.txt
#         (etc etc)
# - The names of the run dirs (Run-LogFileArchive-20221122-112015) are
#   always Run-[archiveName]-[current date-time].
# - The names of the archives are Archive-[archiveName]-[From date]-[To date].7z.
# - Obviously the script will only generate 7z files for the months 
#   where it finds files.
# =================================

Clear-Host
$ErrorActionPreference = 'Stop'

try
{
    # ===== Parameters - Change here =====

    # A short description for the archive. This is added to the filenames.
    $archiveName = "LogFileArchive"

    # Directory to create the archive in
    $archivePath = "C:\OldLogFiles"

    # Directory to archive files from
    $path = "C:\logs"
    
    # Filter for files to archive, for example *.*, *.log or *.pdf
    $filter = "*.log"

    # How many months of files to keep (i.e. not archive), for example 12 (1 year).
    $monthsToKeep = 1

    # From-date to archive, e.g. '2020-12-31'
    # If $deleteFiles = $true you don't need to change this ever.
    $dateFromStr = "1900-01-01"

    # Delete files and empty folders after archiving?
    $deleteFiles = $true

	# Path of 7zip command line
	$zip = "C:\Program Files\7-Zip\7z.exe"

    # ===== Parameters =====

    if ([string]::IsNullOrWhitespace($filter)) 
    { 
    	$filter = "*.*"
    }   
            
    if ($monthsToKeep -le 0)
    {
    	throw "Months to keep cannot be 0 or negative"
    }

    if ([string]::IsNullOrWhitespace($dateFromStr))
    { 
    	throw "Date From cannot be empty"
    }     
        
    $dateToStr = [datetime]::Today.AddMonths($monthsToKeep * -1).ToString("yyyy-MM-01")

	Write-Host "Delete files set to $deleteFiles"	

    # ===== Sanity checks =====
    if ([string]::IsNullOrWhitespace($archiveName)) { throw "Parameter archiveName cannot be empty" }
    if ([string]::IsNullOrWhitespace($archivePath)) { throw "Parameter archivePath cannot be empty" }
    if ([string]::IsNullOrWhitespace($path)) { throw "Parameter path cannot be empty" }
    if ([string]::IsNullOrWhitespace($zip)) { throw "Parameter sevenZipPath cannot be empty" }
    
    if (-not(Test-Path -Path $archivePath)) { throw "Archive path $archivePath does not exist" }
    if (-not(Test-Path -Path $path)) { throw "Root path $path does not exist" }
    if (-not(Test-Path -Path $zip)) { throw "7zip not found in $zip" }
    
    $archivePath = [System.IO.Path]::Combine($archivePath, "Run-$archiveName-$([datetime]::Now.ToString("yyyyMMdd-HHmmss"))")

    # Loop through months
    $dateFrom = [datetime]::Parse($dateFromStr)
    $dateTo = [datetime]::Parse($dateToStr)

    $dateFromLoop = $dateFrom
    $loop = $true

    $haveArchivedFiles = $false

    if (Test-Path -Path $archivePath)
    {
        throw "Directory $archivePath already exists, stopping out of precaution"
    }
    New-Item -ItemType Directory -Path $archivePath | Out-Null
    
    $fullList = Get-ChildItem -Path $path -Filter $filter -File -Recurse `
    	| Where-Object { ($_.LastWriteTime -ge $dateFrom) -and ($_.LastWriteTime -lt $dateTo) }    

    while($loop)
    {    
        $dateToLoop = $dateFromLoop.AddMonths(1)
        if ($dateToLoop -gt $dateTo)
        {
            $dateToLoop = $dateTo
            $loop = $false
        }

        $archiveFile = [System.IO.Path]::Combine($archivePath, "Archive-$archiveName-$($dateFromLoop.ToString("yyyyMMdd"))-$($dateToLoop.ToString("yyyyMMdd")).7z")
        #Write-Host $archiveFile
        $archiveList = $archiveFile.Replace(".7z", ".txt")
    
        $list = $fullList | Where-Object { ($_.LastWriteTime -ge $dateFromLoop) -and ($_.LastWriteTime -lt $dateToLoop) }

        if ($list.Count -gt 0)
        {
            $haveArchivedFiles = $true
            $list | % { Out-File -FilePath $archiveList -Encoding utf8 -Append -InputObject "$($_.FullName)" }

            $cmd = $zip
            Write-Host "================ Archiving files from $path to $archiveFile ================"
            $params = "a $archiveFile -spf -ssw -stl -v2g -mx2 @$archiveList"
            & "$cmd" $params.Split(" ")
            
            # 7z parameter -sdel instructs 7zip to delete files after archiving
            # $params = "a $archiveFile -sdel -spf -ssw -stl -v2g -mx2 @$archiveList"
            # BUUUUUUUUT there's an open 7z bug which is that -sdel doesn't work
            # with file lists (which we need here)
            # that's why we need to delete the files with powershell after 7z
            if ($deleteFiles) {
				$list | % { Remove-Item -Force -ErrorAction Continue -Path "$($_.FullName)" }
                Write-Host "Deleted $($list.Count) files"
            }         
        }

        $dateFromLoop = $dateToLoop
    }

    if (-not $haveArchivedFiles)
    {
        Write-Host "================ No files found to archive ================"
    }
}
catch
{
    Write-Host "================ An error occured ================"
    Write-Error $_
}

Git: Add existing code to repo

# See also https://docs.github.com/en/migrations/importing-source-code/using-the-command-line-to-import-source-code/adding-locally-hosted-code-to-github
cd C:\mysourcecode
git init -b main
git add . 
git commit -m "initial commit"
git remote add origin https://mycodeprovider/myrepo
git push -u origin --all

Powershell: 1-liner to create azure devops project and repos from zip files

Prerequisite: git config and az login have been completed.

cls; $proj=(Split-Path -Path $pwd -Leaf); az devops project create --name $proj; gci *.zip | % { Expand-Archive $_.FullName }; gci -Directory | % { cd $_.FullName; $dir=(Split-Path -Path $pwd -Leaf); az repos create --project $proj --name $dir; git init -b main ; git add . ; git commit -m "initial commit" ; git remote add origin https://MYUSERNAME@dev.azure.com/MYUSERNAME/$proj/_git/$dir ; git push -u origin --all }

Powershell: Zip directory and generate hash

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, April 2023
#

Clear-Host
$ErrorActionPreference = 'Stop'

# customize here
$releaseId = [guid]::NewGuid().ToString().Split('-')[0]
$SourceDir = "$PSScriptRoot\source\" 
$OutputFile = 'mypackage.zip'
$DestinationPath = "$PSScriptRoot\deploy\$($releaseId)"
$tempPath = "$($env:TEMP)\$releaseId"

New-Item -ItemType Directory -Path $DestinationPath
New-Item -ItemType Directory -Path $tempPath

# to avoid file lock issues
Copy-Item -Recurse -Path $SourceDir -Destination $tempPath

$compressParams = @{
    Path = "$tempPath\source\*"
    CompressionLevel = 'Optimal'
    DestinationPath = Join-Path $DestinationPath $OutputFile
    Force = $true
}

Compress-Archive @compressParams
(Get-FileHash $compressParams.DestinationPath).Hash | Out-File $(Join-Path $DestinationPath $OutputFile.Replace('.zip','_hash.txt'))

Write-Host "Finished"
Start-Sleep -Seconds 30

Git: how to avoid checking in secrets (using a Powershell pre-commit hook)

Who among us hasn’t found him- or herself in this very awkward position: committing a config or code file with secrets (such as passwords or API keys) and then semi-panicked googling how to delete it from source control.

Been there and let me tell you the easiest way to delete it: copy all the code on disk, delete the repository completely and then re-create it.

(if this is not an option, well, there’s still a way but with much more work and risk, so do keep that code backup around!)

But you know what’s even better? That’s right, avoid this in the first place! That’s why Git hooks are so useful: they work without you neededing to remember to check your config files every time.

So here’s my solution to this:

  1. In the repository, go to .git/hooks and rename pre-commit.sample to pre-commit (i.e. remove the extension)
  2. Open pre-commit with a text editor and replace its contents with the following:
#!/bin/sh
C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe -ExecutionPolicy Bypass -Command '.\hooks\pre-commit.ps1'
  1. Add a new directory on the root of the repository named hooks.
  2. Inside this, add a text file named pre-commit.ps1 with the following code:
#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, July 2022
#
Clear-Host
$ErrorActionPreference='Stop'

# ===== Change here =====
$listOfExtensions=@('*.xml','*.config')
$listOfSecretNodes=@('username','password','clientid','secret','connectionstring')
$acceptableString='lalala'
# ===== Change here =====

$codePath = (Get-Item -Path $PSScriptRoot).Parent.Parent.FullName

$errorList=New-Object -TypeName 'System.Collections.ArrayList'

foreach($ext in $listOfExtensions) {
    $list = Get-ChildItem -Path $codePath -Recurse -Filter $ext

    foreach($file in $list) {
        $fileName = $file.FullName
        if ($fileName.Contains('\bin\')) {
            continue
        }
        Write-Host "Checking $fileName for secrets"
        [xml]$xml=[xml]((Get-Content -Path $fileName).ToLowerInvariant())
        foreach($secretName in $listOfSecretNodes) {
            $nodes = $xml.SelectNodes("//*[contains(local-name(), '$secretName')]")
            foreach($node in $nodes) {
                if ($node.InnerText.ToLowerInvariant() -ne $acceptableString) {
                    $str = "[$fileName] $($node.Name) contains text other than '$acceptableString', please replace this with $acceptableString before commiting."
                    $errorList.Add($str) | Out-Null
                    Write-Warning $str
                }
            }
        }
    }
}

if ($errorList.Count -gt 0) {
    Write-Error 'Commit cancelled, please correct before commiting.'
}

So there you have it. I’m getting automatically stopped every time I tried to commit any .xml or .config file that contains a node with a name that contains username, password, clientid, secret or connectionstring, whenever the value of it is not ‘lalala’.

Obviously the extensions, node names and acceptable string can be changed at the top of the script. You can also change this quite easily to check JSON files as well.

Also note that this works on Windows (because of the Powershell path in the pre-commit hook) but with a minor change in the pre-commit bash script, you should be able to make it work cross-platform with Powershell core. I haven’t tested it but it should be:

#!/usr/bin/env pwsh -File '.\hooks\pre-commit.ps1'

Have fun coding!