Tag Archives: git

Software, Tutorials and guides

How to get a backup of your Azure Devops repository including all branches

Leave a comment

While Azure Devops is widely used, Microsoft’s backup solutions are surprisingly thin. With people depending on it, individuals and enterprises alike, you’d expect a bit more.

There are various tools around, but here’s my version in the form of a Powershell script. What it does is:

  • Connects to a specific Azure Devops project and repo.
  • Lists all branches, downloads them using git and zips them.
  • The zip, one for every branch, is named Backup_yyyy-MM-dd_branch.zip.

Prerequisites are not much, but:

  • You need git installed and
  • you need a PAT with read access to your code (instructions here).

So here’s the script:

#
# Source: DotJim blog (https://dandraka.com)
# Jim Andrakakis, October 2024
# Updated September 2025, fix for projects and orgs containing spaces
#

# BackupBranches.ps1

param (
    [string]$organization = "MYORG",
    [string]$project = "MYPROJECT",
    [string]$repository = "MYREPO",
    [string]$backupFolder = "C:\Temp\DevOpsBranches",    
	[string]$branchFilter = "" # leave empty for all branches
)

Clear-Host
$ErrorActionPreference='Stop'
 
$pat = Read-Host -MaskInput -Prompt "Enter Personal Access Token for $($env:USERNAME) and $($project)/$($repository)"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$pat"))
 
$tempFolder = Join-Path $backupFolder $repository
$repoNoSpace = $repository.Replace(' ','%20')
$projNoSpace = $project.Replace(' ','%20')
 
Write-Host "[$([datetime]::Now.ToString('yyyy-MM-dd HH:mm:ss'))] Starting, output directory is $tempFolder"
 
# Ensure temp folder exists
if (-not (Test-Path -Path $tempFolder)) {
    New-Item -Path $tempFolder -ItemType Directory | Out-Null
}
 
# API URL for branches
$branchesApiUrl = "https://dev.azure.com/$organization/$projNoSpace/_apis/git/repositories/$repoNoSpace/refs?filter=heads/&api-version=6.0"
 
# Get all branches from the repository
$response = Invoke-RestMethod -Uri $branchesApiUrl -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}
 
$branchList = $response.value | Sort-Object -Property name
 
# Iterate through each branch
foreach ($branch in $branchList) {
	try {
		$branchName = $branch.name -replace "refs/heads/", ""

		# branch filter, if any
		if (-not ([string]::IsNullOrWhiteSpace($branchFilter)) -and ($branchName -notlike "*$branchFilter*")) {
			continue
		}
	 
		# Define the folder for the branch
		$branchNameStrilized = "$($branchName.Replace('/','_').Replace(' ','_'))"
		$branchFolder = "$tempFolder\$branchNameStrilized"
		 
		# Remove the folder if it exists from previous runs
		if (Test-Path -Path $branchFolder) {
			Remove-Item -Recurse -Force $branchFolder
		}
	 
		# Clone the specific branch
		$gitUrl = "https://dev.azure.com/$organization/$projNoSpace/_git/$repoNoSpace"
		Write-Host "Cloning branch '$branchName' from $gitUrl to $branchFolder"
		$gitResp = [string] (& git clone --branch $branchName --single-branch $gitUrl $branchFolder 2>&1)
		if ($gitResp -like "*fatal*") {
			Write-Error "Error cloning branch '$branchName': $gitResp"
		}
	 
		# Zip the branch folder
		$backupDate = [datetime]::Now.ToString('yyyy-MM-dd')
		$zipFilePath = "$tempFolder\Backup_$($backupDate)_$($branchNameStrilized).zip"
		if (Test-Path $zipFilePath) {
			Remove-Item $zipFilePath
		}
		Compress-Archive -CompressionLevel Fastest -Path "$branchFolder\*" -DestinationPath $zipFilePath
	 
		Write-Host "Branch '$branchName' zipped to $zipFilePath"
	 
		# Clean up branch folder after zipping
		Remove-Item -Recurse -Force $branchFolder
	}
	catch {
		Write-Warning $_
	}
}
 
Write-Host "[$([datetime]::Now.ToString('yyyy-MM-dd HH:mm:ss'))] Finished, $($response.value.Count) branches processed."

Usage example:

BackupBranches.ps1 -organization 'BIGBANK' -project 'KYCAML' -repository 'KYCAMLapiV2' -backupFolder '\\backupfileserver\codebackups\'
Security, Software, Tutorials and guides

Git: how to avoid checking in secrets (using a Powershell pre-commit hook)

Leave a comment

Who among us hasn’t found him- or herself in this very awkward position: committing a config or code file with secrets (such as passwords or API keys) and then semi-panicked googling how to delete it from source control.

Been there and let me tell you the easiest way to delete it: copy all the code on disk, delete the repository completely and then re-create it.

(if this is not an option, well, there’s still a way but with much more work and risk, so do keep that code backup around!)

But you know what’s even better? That’s right, avoid this in the first place! That’s why Git hooks are so useful: they work without you neededing to remember to check your config files every time.

So here’s my solution to this:

  1. In the repository, go to .git/hooks and rename pre-commit.sample to pre-commit (i.e. remove the extension)
  2. Open pre-commit with a text editor and replace its contents with the following:
#!/bin/sh
C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe -ExecutionPolicy Bypass -Command '.\hooks\pre-commit.ps1'
  1. Add a new directory on the root of the repository named hooks.
  2. Inside this, add a text file named pre-commit.ps1 with the following code:
#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, July 2022
#
Clear-Host
$ErrorActionPreference='Stop'

# ===== Change here =====
$listOfExtensions=@('*.xml','*.config')
$listOfSecretNodes=@('username','password','clientid','secret','connectionstring')
$acceptableString='lalala'
# ===== Change here =====

$codePath = (Get-Item -Path $PSScriptRoot).Parent.Parent.FullName

$errorList=New-Object -TypeName 'System.Collections.ArrayList'

foreach($ext in $listOfExtensions) {
    $list = Get-ChildItem -Path $codePath -Recurse -Filter $ext

    foreach($file in $list) {
        $fileName = $file.FullName
        if ($fileName.Contains('\bin\')) {
            continue
        }
        Write-Host "Checking $fileName for secrets"
        [xml]$xml=[xml]((Get-Content -Path $fileName).ToLowerInvariant())
        foreach($secretName in $listOfSecretNodes) {
            $nodes = $xml.SelectNodes("//*[contains(local-name(), '$secretName')]")
            foreach($node in $nodes) {
                if ($node.InnerText.ToLowerInvariant() -ne $acceptableString) {
                    $str = "[$fileName] $($node.Name) contains text other than '$acceptableString', please replace this with $acceptableString before commiting."
                    $errorList.Add($str) | Out-Null
                    Write-Warning $str
                }
            }
        }
    }
}

if ($errorList.Count -gt 0) {
    Write-Error 'Commit cancelled, please correct before commiting.'
}

So there you have it. I’m getting automatically stopped every time I tried to commit any .xml or .config file that contains a node with a name that contains username, password, clientid, secret or connectionstring, whenever the value of it is not ‘lalala’.

Obviously the extensions, node names and acceptable string can be changed at the top of the script. You can also change this quite easily to check JSON files as well.

Also note that this works on Windows (because of the Powershell path in the pre-commit hook) but with a minor change in the pre-commit bash script, you should be able to make it work cross-platform with Powershell core. I haven’t tested it but it should be:

#!/usr/bin/env pwsh -File '.\hooks\pre-commit.ps1'

Have fun coding!