Anyone who’s ever developed anything connected to a database knows about transactions. Using a transaction we can group data operations so that they happen “all or nothing”, i.e. either all of them succeed or no one does. One example is a transfer from one bank account to another: the complete transaction requires subtracting the amount to be transferred from one account and adding that same amount to the other. We wouldn’t want one to happen without the other, would we?
(yes, I know that’s not a complete description of what transactions are or do; that’s not my purpose here)
But what happens when we need the same from our filesystem?
Let’s take this scenario (which is a simplified version of a true story): we are receiving CSV files from solar panels (via SFTP) and we want to do some preprocessing and then store the data to a database. When processing them we have to generate a lot of intermediate files. After that, we want to move them to a different dir. But if something happens, say if the database is down, we want the whole thing to be cancelled so that, when we retry, we can start over.
Obviously a simple solution would be as follows:
try {
# do a lot of hard work
# store the data in the db
# clean up intermediate files
# move the CSV file to an "archive" dir
}
catch {
# clean up intermediate files, potentially clean up any db records etc
}
That’s… well it can work but it’s not great. For example, the cleanup process itself -inside the catch block- might fail.
A much better process would be like that:
try {
# start a transaction
# do a lot of hard work
# store the data in the db
# clean up intermediate files
# move the CSV file to an "archive" dir
# commit the transaction
}
catch {
# rollback everything: files, db changes, the whole thing
}
That’s much cleaner! But is it possible to group db changes and filesystem changes (file create, move, delete & append, dir create & delete etc) all in one transaction?
Unix geeks are allowed to feel smug here: some flavors like HP-UX (though not Linux as far as I know) have this baked in from the get go. Your code doesn’t have to do anything special; the OS takes care of this on user request.
But as a matter of fact yes, it is also available on Windows, and it has been for some time now. The requirement is that you’re working on a file system that supports this, like NTFS.
But there’s a problem for the average .NET/Powershell coder: the standard methods, the ones inside System.IO, do not support any of this. So you have to go on a lower level, using the Windows API. Which for .NET coders, there’s no other way to put this, sucks. That’s also the reason why the Powershell implementation of file transactions (e.g. New-Item -ItemType File -UseTransaction) doesn’t work -it relies on System.IO.
I’m pretty sure that this is what crossed the minds of the developers that wrote AlphaFS which is just wonderful. It’s exactly what you’d expect but never got from Microsoft: a .NET implementation of most of System.IO classes that support NTFS’s advanced features that are not available in, say, ye olde FAT32. Chief among them is support for file system transactions.
So the example below shows how to do exactly that. I tried to keep it as simple as possible to highlight the interesting bits, but of course a real world example would be much more complicated, e.g. there would be a combined file system and database transaction, which would commit (or rollback) everything at the same time.
Note that there’s no need for an explicit rollback. As soon as the transaction scope object is disposed without calling Complete(), all changes are rolled back.
#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, July 2019
#
# Prerequisite:
# 1) internet connectivity
# 2) nuget command line must be installed
# (see https://www.nuget.org/downloads).
# If nuget is not in %path%, you need to change
# the installation (see below) to call it with
# its full path.
# Stop on error
$ErrorActionPreference = "Stop"
if ($psISE)
{
$binPath = Split-Path -Path $PSISE.CurrentFile.FullPath
}
else
{
$binPath = $PSScriptRoot
}
$alphaFSver = "2.2.6"
$libPath = "$binPath\AlphaFS.$alphaFSver\lib\net40\AlphaFS.dll"
$basePath = "$binPath\..\alphatest"
# ====== installation ======
if (-not [System.IO.File]::Exists($libPath)) {
Out-File -FilePath "$binPath\packages.config" `
-Force `
-Encoding utf8 `
-InputObject ("<?xml version=`"1.0`" encoding=`"utf-8`"?><packages>" + `
"<package id=`"AlphaFS`" version=`"$alphaFSver`" targetFramework=`"net46`" />" + `
"</packages>")
cd $binPath
& nuget.exe restore -PackagesDirectory "$binPath"
}
# ==========================
# Make sure the path matches the version from step 2
Import-Module -Name $libPath
if (-not (Test-Path $basePath)) {
New-Item -ItemType Directory -Path $basePath
}
# Check if the filesystem we're writing to supports transactions.
# On a FAT32 drive you're out of luck.
$driveRoot = [System.IO.Path]::GetPathRoot($basePath)
$driveInfo = [Alphaleonis.Win32.Filesystem.DriveInfo]($driveRoot)
if (-not $driveInfo.VolumeInfo.SupportsTransactions) {
Write-Error ("Your $driveRoot volume $($driveInfo.DosDeviceName) " + `
"[$($driveInfo.VolumeLabel)] does not support transactions, exiting")
}
# That's some example data to play with.
# In reality you'll probably get data from a DB, a REST service etc.
$list = @{1="Jim"; 2="Stef"; 3="Elena"; 4="Eva"}
try {
# Transaction starts here
$transactionScope = [System.Transactions.TransactionScope]::new([System.Transactions.TransactionScopeOption]::RequiresNew)
$fileTransaction = [Alphaleonis.Win32.Filesystem.KernelTransaction]::new([System.Transactions.Transaction]::Current)
# Here we're doing random stuff with our files and dirs,
# just to show how this works.
# The important thing to remember is that for the transaction
# to work correctly, ALL methods you use have to be -transacted.
# I.e. you must not use AppendText() but AppendTextTransacted(),
# not CreateDirectory() but CreateDirectoryTransacted() etc etc.
$logfileStream = [Alphaleonis.Win32.Filesystem.File]::AppendTextTransacted($fileTransaction, "$basePath\list.txt")
foreach($key in $list.Keys) {
$value = $list.$key
$filename = "$([guid]::NewGuid()).txt"
$dir = "$basePath\$key"
Write-Host "Processing item $key $value"
if (-not [Alphaleonis.Win32.Filesystem.Directory]::ExistsTransacted($fileTransaction, $dir)) {
[Alphaleonis.Win32.Filesystem.Directory]::CreateDirectoryTransacted($fileTransaction, $dir)
}
[Alphaleonis.Win32.Filesystem.File]::WriteAllTextTransacted($fileTransaction, "$basePath\$key\$filename", $value)
$logfileStream.WriteLine("$filename;$key;$value")
}
$logfileStream.Close()
# to simulate an error and subsequent rollback:
# Write-Error "Something not great, not terrible happened"
# Commit transaction
$transactionScope.Complete()
Write-Host "Transaction committed, all modifications written to disk"
}
catch {
Write-Host "An error occured and the transaction was rolled back: '$($_.Exception.Message)'"
throw $_.Exception
}
finally {
if ($null -ne $logfileStream -and $logfileStream -is [System.IDisposable]) {
$logfileStream.Dispose()
$logfileStream = $null
}
if ($null -ne $transactionScope -and $transactionScope -is [System.IDisposable]) {
$transactionScope.Dispose()
$transactionScope = $null
}
}
Have fun coding!
Dot Jim 