Category Archives: Software

My script cheat sheet

That’s not a post, at least in the classical sense 😊 Rather it’s a collection of small scripts, that I will keep updating, for me to find easily. No rocket science, just small everyday stuff that I find myself googling again and again.

Powershell: Get the first X bytes of a file.

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, September 2022
#

# ===== Change here =====
$path = 'C:\somepath\somehugelogfile.log'
$pathOut = 'C:\temp\sneakpeak.txt'
$numBytes = 10000
# =======================

$ErrorActionPreference='Stop'
Clear-Host

$bytes = Get-Content -Path $path -Encoding byte -TotalCount $numBytes
$str = [System.Text.Encoding]::UTF8.GetString($bytes)
Out-File -FilePath $pathOut -InputObject $str

Powershell: Remove files older than X days recursively.

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, September 2022
#

# ===== Change here =====
$path = 'C:\somepath\'
$filter = '*.xml'
$numDays = 30 
# =======================

$ErrorActionPreference='Stop'
Clear-Host

Get-ChildItem -Path $path -Filter $filter -Recurse | Where-Object {($_.LastWriteTime -lt (Get-Date).AddDays($numDays * -1))} | Remove-Item

Powershell: Change information in XML files en masse.

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, September 2022
#

# ===== Change here =====
$dir = 'C:\somepath\'
# =======================

$ErrorActionPreference='Stop'
Clear-Host

$list = Get-ChildItem -Path $dir -Filter '*.xml'
foreach($file in $list) {
    [xml]$xml=Get-Content -Path $file.FullName -Encoding UTF8
    # customize the XML paths below
    $customerName = $xml.PrintJob.DocumentHeader.CustomerName 
    if ([string]::IsNullOrWhiteSpace($customerName )) {
        continue
    }
    $xml.PrintJob.DocumentBody.RecipientName = $customerName 
    $xml.Save($file.FullName)
}

Windows command line: Change permissions of all files and directories in a path recursively.

CD C:\somepath
FOR /D %o IN (*.*) DO echo Y| cacls %o /T /G "NT AUTHORITY\Authenticated Users":F

Here “NT AUTHORITY\Authenticated Users” stands for the authenticated users group of the local machine; “F” stands for Full Permissions.

Linux command line: copy file from one PC to another

# scp -r /path/to/file USERNAME@IP_OF_TARGET:/path/to/dir
scp -r /home/dimitris/Downloads/Win11.iso dimitris@192.168.0.5:/home/dimitris/Downloads

Powershell: Get info from multiple XML files and write into CSV

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, October 2022
#

# The file filter here is based on date from-to, 
# but obvs you can change Where-Object to match size or name or whatever.

# The CSV separator was chosen to be tab (`t) because this is 
# very Excel-friendly.

# ===== Change here =====
$path = 'E:\somepath'
$outPath = "C:\temp\out-$([guid]::NewGuid().ToString().Split('-')[0]).csv"

$strDateTimeFrom = "2022-09-29 18:00:00"
$strDateTimeTo = "2022-09-29 20:00:00"
# =======================

Clear-Host
$ErrorActionPreference='Stop'

[DateTime]$dateTimeFrom = [DateTime]::Parse($strDateTimeFrom)
[DateTime]$dateTimeTo = [DateTime]::Parse($strDateTimeTo)

$filesList = Get-ChildItem -Path $path -Recurse -Filter '*.xml' | Where-Object { ($_.LastWriteTime -gt $dateTimeFrom) -and ($_.LastWriteTime -lt $dateTimeTo) }

$dataList = New-Object -TypeName 'System.Collections.Generic.List[string]'
# change this to match the XML info below
$dataList.Add("CompanyName`tInvoiceId`tTemplateId`tDocumentId`tFileName")
foreach($file in $filesList) {
    $fileFullName = $file.FullName
    $fileName = $file.Name
    [xml]$xml = Get-Content -Path $fileFullName
    # that's the part where you specify what info you need from the XML
    # my XMLs have multiple Document nodes per file, that's why I need a loop
    foreach($document in $xml.PrintJob.Documents.Document) {
        $documentId = $document.DocumentHeader.DocumentId
        $templateId = $document.DocumentHeader.TemplateId
        $invoiceId = $document.ArchiveAttributes.InvoiceId
        $custName = $document.DocumentHeader.Addresses.Recipient.CompanyName

        $dataList.Add("$custName`t$invoiceId`t$templateId`t$documentId`t$fileName")
    }
}

Out-File -FilePath $outPath -Encoding utf8 -InputObject $dataList
Write-Host "Finished, $($filesList.Count) files processed"

Chocolatey: My dev machine install list

choco install notepadplusplus
choco install winmerge -y 
choco install vscode -y 
choco install vscode-powershell -y 
choco install vscode-csharp -y 
choco install vscode-gitlens -y 
choco install git -y 
choco install tortoisegit -y 
choco install svn -y 
choco install tortoisesvn -y 
choco install postman -y 
choco install soapui -y 
choco install sql-server-management-studio -y

choco install intellijidea-community
choco install openjdk8

choco install visualstudio2019professional --package-parameters " --add Microsoft.VisualStudio.Workload.Azure --add Microsoft.VisualStudio.Workload.ManagedDesktop --add Microsoft.VisualStudio.Workload.NetCoreTools --add Microsoft.VisualStudio.Workload.NetWeb --add Microsoft.VisualStudio.Workload.Universal --includeRecommended --includeOptional --passive --locale en-US" -y
choco install visualstudio2022professional --package-parameters " --add Microsoft.VisualStudio.Workload.Azure --add Microsoft.VisualStudio.Workload.ManagedDesktop --add Microsoft.VisualStudio.Workload.NetCoreTools --add Microsoft.VisualStudio.Workload.NetWeb --add Microsoft.VisualStudio.Workload.Universal --includeRecommended --includeOptional --passive --locale en-US" -y
choco install dotnet-5.0-sdk -y
choco install dotnet-6.0-sdk -y

choco install ServiceBusExplorer -y

Install-Package \\fileserver\share\JamsScheduler\SetupClientx64.msi

Powershell: Archive files (zip + delete)


#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, November 2022
#

# Prerequisite: 7zip is installed in the system.

# =================================
# This script zips everything found in $archiveName that 
# has a ModifiedDate after $dateFromStr and matches $filter.
# Inside $archivePath, it creates one dir per run and 
# inside this, one 7zip file per month plus one txt file
# that has the 7zip contents.
# E.g.
# C:\OldLogFiles
#     Run-LogFileArchive-20221122-112015
#         Archive-LogFileArchive-20200301-20200401.7z.001
#         Archive-LogFileArchive-20200301-20200401.txt
#         Archive-LogFileArchive-20200401-20200501.7z.001
#         Archive-LogFileArchive-20200401-20200501.txt
#         (etc etc)
# - The names of the run dirs (Run-LogFileArchive-20221122-112015) are
#   always Run-[archiveName]-[current date-time].
# - The names of the archives are Archive-[archiveName]-[From date]-[To date].7z.
# - Obviously the script will only generate 7z files for the months 
#   where it finds files.
# =================================

Clear-Host
$ErrorActionPreference = 'Stop'

try
{
    # ===== Parameters - Change here =====

    # A short description for the archive. This is added to the filenames.
    $archiveName = "LogFileArchive"

    # Directory to create the archive in
    $archivePath = "C:\OldLogFiles"

    # Directory to archive files from
    $path = "C:\logs"
    
    # Filter for files to archive, for example *.*, *.log or *.pdf
    $filter = "*.log"

    # How many months of files to keep (i.e. not archive), for example 12 (1 year).
    $monthsToKeep = 1

    # From-date to archive, e.g. '2020-12-31'
    # If $deleteFiles = $true you don't need to change this ever.
    $dateFromStr = "1900-01-01"

    # Delete files and empty folders after archiving?
    $deleteFiles = $true

	# Path of 7zip command line
	$zip = "C:\Program Files\7-Zip\7z.exe"

    # ===== Parameters =====

    if ([string]::IsNullOrWhitespace($filter)) 
    { 
    	$filter = "*.*"
    }   
            
    if ($monthsToKeep -le 0)
    {
    	throw "Months to keep cannot be 0 or negative"
    }

    if ([string]::IsNullOrWhitespace($dateFromStr))
    { 
    	throw "Date From cannot be empty"
    }     
        
    $dateToStr = [datetime]::Today.AddMonths($monthsToKeep * -1).ToString("yyyy-MM-01")

	Write-Host "Delete files set to $deleteFiles"	

    # ===== Sanity checks =====
    if ([string]::IsNullOrWhitespace($archiveName)) { throw "Parameter archiveName cannot be empty" }
    if ([string]::IsNullOrWhitespace($archivePath)) { throw "Parameter archivePath cannot be empty" }
    if ([string]::IsNullOrWhitespace($path)) { throw "Parameter path cannot be empty" }
    if ([string]::IsNullOrWhitespace($zip)) { throw "Parameter sevenZipPath cannot be empty" }
    
    if (-not(Test-Path -Path $archivePath)) { throw "Archive path $archivePath does not exist" }
    if (-not(Test-Path -Path $path)) { throw "Root path $path does not exist" }
    if (-not(Test-Path -Path $zip)) { throw "7zip not found in $zip" }
    
    $archivePath = [System.IO.Path]::Combine($archivePath, "Run-$archiveName-$([datetime]::Now.ToString("yyyyMMdd-HHmmss"))")

    # Loop through months
    $dateFrom = [datetime]::Parse($dateFromStr)
    $dateTo = [datetime]::Parse($dateToStr)

    $dateFromLoop = $dateFrom
    $loop = $true

    $haveArchivedFiles = $false

    if (Test-Path -Path $archivePath)
    {
        throw "Directory $archivePath already exists, stopping out of precaution"
    }
    New-Item -ItemType Directory -Path $archivePath | Out-Null
    
    $fullList = Get-ChildItem -Path $path -Filter $filter -File -Recurse `
    	| Where-Object { ($_.LastWriteTime -ge $dateFrom) -and ($_.LastWriteTime -lt $dateTo) }    

    while($loop)
    {    
        $dateToLoop = $dateFromLoop.AddMonths(1)
        if ($dateToLoop -gt $dateTo)
        {
            $dateToLoop = $dateTo
            $loop = $false
        }

        $archiveFile = [System.IO.Path]::Combine($archivePath, "Archive-$archiveName-$($dateFromLoop.ToString("yyyyMMdd"))-$($dateToLoop.ToString("yyyyMMdd")).7z")
        #Write-Host $archiveFile
        $archiveList = $archiveFile.Replace(".7z", ".txt")
    
        $list = $fullList | Where-Object { ($_.LastWriteTime -ge $dateFromLoop) -and ($_.LastWriteTime -lt $dateToLoop) }

        if ($list.Count -gt 0)
        {
            $haveArchivedFiles = $true
            $list | % { Out-File -FilePath $archiveList -Encoding utf8 -Append -InputObject "$($_.FullName)" }

            $cmd = $zip
            Write-Host "================ Archiving files from $path to $archiveFile ================"
            $params = "a $archiveFile -spf -ssw -stl -v2g -mx2 @$archiveList"
            & "$cmd" $params.Split(" ")
            
            # 7z parameter -sdel instructs 7zip to delete files after archiving
            # $params = "a $archiveFile -sdel -spf -ssw -stl -v2g -mx2 @$archiveList"
            # BUUUUUUUUT there's an open 7z bug which is that -sdel doesn't work
            # with file lists (which we need here)
            # that's why we need to delete the files with powershell after 7z
            if ($deleteFiles) {
				$list | % { Remove-Item -Force -ErrorAction Continue -Path "$($_.FullName)" }
                Write-Host "Deleted $($list.Count) files"
            }         
        }

        $dateFromLoop = $dateToLoop
    }

    if (-not $haveArchivedFiles)
    {
        Write-Host "================ No files found to archive ================"
    }
}
catch
{
    Write-Host "================ An error occured ================"
    Write-Error $_
}

Stories from the Field, #2: The customer is not always right

(note: all “Stories from the Field” are true, thinly anonymized to protect the -usually- guilty)

Project Manager: Hi Jim and team, we need a new version of our product with this and that new feature.

Team: Sure, but it’ll need a bit more memory on our customers’ computers.

PM: No worries, our customers have more than adequate computers already.

Team: Hmmmokey

Team: (codes)

Team: (tests)

Team: It’s ready and has passed our internals tests successfully.

Team: Now it’s time to go and test it on a few customers just to be safe, like we did last time.

PM: Yeah about that.

Team: WHAT NOW THIS IS A GOOD THING

PM: No no no don’t get me wrong, our customers loved it.

PM: In fact they loved it so much that they have been asking for it.

PM: Jim can you talk to Sales? They’ll tell you some great customers to test with.

Jim: (makes the rookie mistake and calls Sales)

Sales: Oh hi so happy to hear you.

Jim: Really? You usually complain about how our product is a piece of shit.

Sales: AAAAHAHAHAFUNNY no really I’m totally happy to hear you.

Sales: I’ve got a totally great customer you can test the new version with.

Sales: Who is totally not a cheapskate bloodsucking asshole.

Sales: You should totally call him.

Jim: I have a bad feeling about this.

Jim: (calls customer)

Totally Not A Cheapskate Customer: Oh hi happy to hear you, you’re supposed to give me free product to resell to my customers right?

Jim: NO NOT AT ALL we need to test our product and therefore 1) yes you get something for free but 2) be aware that your computers need to have more memory and 3) it’s still being tested which means DO NOT CHARGE YOUR CUSTOMERS FOR IT AND PLEASE TELL THEM THIS MIGHT NOT WORK.

Totally Not A Cheapskate Customer: Yada yada yada so I get free stuff great see you on Thursday bye.

Jim: (has a very bad feeling about this)

Jim and a Teammate: (show up early)

Jim and a Teammate: (start installing the new version on the Totally Not a Cheapskate Customer’s computers)

Jim and a Teammate: Wait this doesn’t work you don’t have enough memory here.

Jim and a Teammate: This isn’t even enough memory for the previous version.

Totally Not A Cheapskate Customer: SO WHAT I DON’T CARE ARE YOU TRYING TO STEAL THE FOOD OF MY CHILDREN? (* actual quote)

Jim and a Teammate: No wait we made very very clear that…

Totally Not A Cheapskate Customer: YOU ARE WORSE THAN THIEVES (* actual quote)

Totally Not A Cheapskate Customer’s Customers: WHY DON’T YOU GIVE US CHEAP STUFF AREN’T WE CITIZENS OF THIS COUNTRY??? (* actual quote)

Totally Not A Cheapskate Customer: I’LL CALL YOUR BOSS RIGHT NOW

PM: Hi Jim what’s going on there?

Jim: This and that, the Totally Not A Cheapskate Customer turned out to be a Totally Cheapskate Customer.

PM: Hmmm so Sales wasn’t 100% sincere.

PM: Who would’ve thought.

PM: Can you remove the installation restriction just for now?

Jim: I can but some things might work, some might not.

PM: Do your best.

Team: (does their best and has a special version ready within 15 minutes)

Jim and a Teammate: (install the software)

Jim and a Teammate: See it kind of works but it has issues BECAUSE IT NEEDS MORE MEMORY.

Totally Not A Cheapskate Customer: SEE MY LOYAL CUSTOMERS I HAVE SLAIN THE THIEVING EVIL CORPORATE DRAGON AND GAVE YOU CHEAP STUFF.

Jim and a Teammate: Wait you actually charged them for this we specifically asked you not to.

Totally Not A Cheapskate Customer: BEGONE YOU FOUL DEMON.

Jim: (silently curses in languages he doesn’t even speak)

Git: how to avoid checking in secrets (using a Powershell pre-commit hook)

Who among us hasn’t found him- or herself in this very awkward position: committing a config or code file with secrets (such as passwords or API keys) and then semi-panicked googling how to delete it from source control.

Been there and let me tell you the easiest way to delete it: copy all the code on disk, delete the repository completely and then re-create it.

(if this is not an option, well, there’s still a way but with much more work and risk, so do keep that code backup around!)

But you know what’s even better? That’s right, avoid this in the first place! That’s why Git hooks are so useful: they work without you neededing to remember to check your config files every time.

So here’s my solution to this:

  1. In the repository, go to .git/hooks and rename pre-commit.sample to pre-commit (i.e. remove the extension)
  2. Open pre-commit with a text editor and replace its contents with the following:
#!/bin/sh
C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe -ExecutionPolicy Bypass -Command '.\hooks\pre-commit.ps1'
  1. Add a new directory on the root of the repository named hooks.
  2. Inside this, add a text file named pre-commit.ps1 with the following code:
#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, July 2022
#
Clear-Host
$ErrorActionPreference='Stop'

# ===== Change here =====
$listOfExtensions=@('*.xml','*.config')
$listOfSecretNodes=@('username','password','clientid','secret','connectionstring')
$acceptableString='lalala'
# ===== Change here =====

$codePath = (Get-Item -Path $PSScriptRoot).Parent.Parent.FullName

$errorList=New-Object -TypeName 'System.Collections.ArrayList'

foreach($ext in $listOfExtensions) {
    $list = Get-ChildItem -Path $codePath -Recurse -Filter $ext

    foreach($file in $list) {
        $fileName = $file.FullName
        if ($fileName.Contains('\bin\')) {
            continue
        }
        Write-Host "Checking $fileName for secrets"
        [xml]$xml=[xml]((Get-Content -Path $fileName).ToLowerInvariant())
        foreach($secretName in $listOfSecretNodes) {
            $nodes = $xml.SelectNodes("//*[contains(local-name(), '$secretName')]")
            foreach($node in $nodes) {
                if ($node.InnerText.ToLowerInvariant() -ne $acceptableString) {
                    $str = "[$fileName] $($node.Name) contains text other than '$acceptableString', please replace this with $acceptableString before commiting."
                    $errorList.Add($str) | Out-Null
                    Write-Warning $str
                }
            }
        }
    }
}

if ($errorList.Count -gt 0) {
    Write-Error 'Commit cancelled, please correct before commiting.'
}

So there you have it. I’m getting automatically stopped every time I tried to commit any .xml or .config file that contains a node with a name that contains username, password, clientid, secret or connectionstring, whenever the value of it is not ‘lalala’.

Obviously the extensions, node names and acceptable string can be changed at the top of the script. You can also change this quite easily to check JSON files as well.

Also note that this works on Windows (because of the Powershell path in the pre-commit hook) but with a minor change in the pre-commit bash script, you should be able to make it work cross-platform with Powershell core. I haven’t tested it but it should be:

#!/usr/bin/env pwsh -File '.\hooks\pre-commit.ps1'

Have fun coding!

Stories from the Field, #1: Learn how to push back

(note: all “Stories from the Field” are true, thinly anonymized to protect the -usually- guilty)

Teammate: (goes to a big and important customer)

Customer: I want a software like this and that

Customer: And I want it yesterday

Project Manager: They want it yesterday

Teammate: But I need a bit more time in order to implement some UI checks, so that users don’t make mistakes

Customer: Our users don’t make mistakes, they are permanent employees for so many years, they know their job

Teammate: Hmmmokey

Teammate: (implements software in just a few days)

Teammate: (delivers)

Customer: (installs)

Users: (use the software)

Users: (literally fuck up everything that is possible and some things that are not)

Customer: WHY ARE THE DATA WRONG

Teammate: …but you said…

Customer: you’re not good I want another one

Project Manager: Jim you’re assigned to this

Jim: I will rewrite it from zero and I will implement these UI checks plus many many more

Customer: I WANT IT YESTERDAY

Project Manager: THEY WANT IT YESTERDAY

Jim: (doesn’t give a shit)

Jim: (writes code anywhere, anytime, day, night, while eating, while getting the baby to sleep, while helping his wife with breastfeeding etc etc)

Jim: (delivers)

Customer: Why is this 40MB this is bigger than the previous one I don’t like this

Jim: (loses his shit and starts screaming)

Customer: Jeez why are you so nervous you need to calm down

Customer: (installs)

Users: (use the software)

Users: OH HEY THIS WORKS

Users: IT HAS HELPFUL COLOURS TOO

Users: AND IT HAS EXPLANATIONS FOR EACH FIELD

Users: THIS IS GREAT

Customer: great job Jim, see I told you the first guy was not good

Jim: (silently curses in languages he doesn’t even speak)

Note: to be fair, the “40MB” complaint wasn’t as irrational as it sounds. The software had to be copied to many client computers, some of them in remote parts of the country with slow lines; this was still the days of ISDN. Still, the refactoring was worth it. The added volume was caused by a reporting library (Crystal Reports for .Net) which solved many problems by itself. I now understand the frustration of the customer’s IT as someone had to stay up all night copying. But the pressure from management was so much that at this point the poor guy just said the wrong thing at the wrong time to the wrong person. Elias if you ever read this, please accept my apologies 😊

New version of Zoro: 2.0

I just published a new version of my open source C# Zoro library in Github and Nuget.org.

Zoro is a data masking/anonymization utility. It fetches data from a database or a CSV file, masks (i.e. anonymizes) them according to the configuration provided and uses the masked data to create a CSV file or run SQL statements such as INSERTs or UPDATEs.

The new version, 2.0, has been converted to DotNet Standard 2.1 to take advantage of some useful DotNet features. The command line utility and the test project are written with DotNet Core 5.0.

The issue from 1.0.2, where the Nuget package did not contain the executables, has been corrected. The package now contains both a Win64 and a Linux64 executable. Since they are self-contained programs, no prior installation of DotNet is needed.

But the most important new feature is a new MaskType, “Query”. With this, the library can retrieve values from a database and pick a random one. In previous versions this was only possible with lists that were fixed in the XML (MaskType=List).

For example, let’s say you are masking the following data:

IDNameCityCountry
1ABBBadenCH
2FAGEAthensGR
3IKEADelftNL
Table “customers”

In the database you might also have a table with cities and countries:

CityNameCountryCode
ZürichCH
GenevaCH
BernCH
RethimnoGR
ChaniaGR
KalamataGR
GoudaNL
GeldropNL
Table “cities”

In order to anonymize the above data, your config could look like this:

<?xml version="1.0"?>
<MaskConfig xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <FieldMasks>
    <FieldMask>
      <FieldName>ID</FieldName>
      <MaskType>None</MaskType>
    </FieldMask>  
    <FieldMask>
      <FieldName>Name</FieldName>
      <MaskType>Similar</MaskType>
    </FieldMask>      
    <FieldMask>
      <FieldName>Country</FieldName>
      <MaskType>None</MaskType>
    </FieldMask>  
    <FieldMask>
      <FieldName>City</FieldName>
      <MaskType>Query</MaskType>
      <QueryReplacement
      	SelectorField="Country" 
      	GroupField="Countrycode" 
      	ValueField="Cityname" 
      	Query="SELECT cityname, countrycode FROM cities" />
    </FieldMask>  
  </FieldMasks>
  <DataSource>Database</DataSource>
  <DataDestination>Database</DataDestination>
  <ConnectionString>
Server=DBSRV1;Database=appdb;Trusted_Connection=yes;
  </ConnectionString>
  <ConnectionType>
System.Data.SqlClient
  </ConnectionType>
  <SqlSelect>
SELECT * FROM customers
  </SqlSelect>
  <SqlCommand>
INSERT INTO customers_anonymous
(ID, Name, City, Country)
VALUES
($ID, $Name, $City, $Country)
  </SqlCommand>
</MaskConfig>

This will result in a table looking like this:

IDNameCityCountry
1EGTBernCH
2SOLEChaniaGR
3UFOEGeldropNL
Table “customers_anonymous”

If you have any questions, please write in the comments.

Enjoy!

Please don’t write logs inside Program Files (here’s how to do it right)

So the other day I’m troubleshooting a Windows Service that keeps failing on a server, part of a product we’re using in the company. Long story short, that’s what the problem was:

Access to the path 'C:\Program Files\whatever\whatever.log is denied'

I mean, dear programmer, look. You want to write your application’s logs as simple text files. I get it. Text files are simple, reliable (if the file system doesn’t work, you have bigger problems than logging) and they’re shown in virtually every coding tutorial in every programming language. Depending on the case, there might be better ways to do that such as syslog, eventlog and others.

But sure, let’s go with text files. Take the following example somewhere in the middle of a Python tutorial. Look at line 3:

import logging

logging.basicConfig(filename='app.log', filemode='w', format='%(name)s - %(levelname)s - %(message)s')
logging.warning('This will get logged to a file')

Did you notice? This code writes the log in the same place as the binary. It’s not explicitly mentioned and usually you wouldn’t give it a second thought, right?

To be clear, I don’t want to be hard on the writers of this or any other tutorial; it’s just a basic tutorial, and as such it should highlight the core concept. A professional developer writing an enterprise product should know a bit more!

But the thing is, these examples are everywhere. Take another Java tutorial and look at line 16:

package com.javacodegeeks.snippets.core;

import java.util.logging.Logger;
import java.util.logging.FileHandler;
import java.util.logging.SimpleFormatter;
import java.io.IOException;

public class SequencedLogFile {

    public static final int FILE_SIZE = 1024;
    public static void main(String[] args) {

        Logger logger = Logger.getLogger(SequencedLogFile.class.getName());
        try {
            // Create an instance of FileHandler with 5 logging files sequences.
            FileHandler handler = new FileHandler("sample.log", FILE_SIZE, 5, true);
            handler.setFormatter(new SimpleFormatter());
            logger.addHandler(handler);
            logger.setUseParentHandlers(false);
        } catch (IOException e) {
            logger.warning("Failed to initialize logger handler.");
        }
        logger.info("Logging info message.");
        logger.warning("Logging warn message.");
    }
}

Or this Dot Net tutorial, which explains how to set up Log4Net (which is great!) and gives this configuration example. Let’s see if you can spot this one. Which line is the problem?

<log4net>
  <root>
    <level value="ALL" />
    <appender-ref ref="LogFileAppender" />
  </root>
  <appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender">
    <file value="proper.log" />
    <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
    <appendToFile value="true" />
    <rollingStyle value="Size" />
    <maxSizeRollBackups value="2" />
    <maximumFileSize value="1MB" />
    <staticLogFileName value="true" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%d [%t] %-5p %c %m%n" />
    </layout>
  </appender>
</log4net>

If you answered “7”, congrats, you’re starting to get it. Not using a path -this should be obvious, I know, but it’s easy to forget nevertheless- means writing in the current path, which by default is wherever the binary is.

So this works fine while you’re developing. It works fine when you do your unit tests. It probably works when your testers do the user acceptance testing or whatever QA process you have.

But when your customers install the software, the exe usually goes to C:\Program Files (that’s in Windows; in Linux there are different possibilities as explained here, but let’s say /usr/bin). Normal users do not have permission to write there; an administrator can grant this, but they really really really shouldn’t. You’re not supposed to tamper with the executables! Unless you’re doing some maintenance or an upgrade of course.

So how do you do this correctly?

First of all, it’s a good idea to not reinvent the wheel. There are many, many, MANY libraries to choose from, some of them very mature, like log4net for Dot Net or log4j for Java.

But if you want to keep it very simple, fine. There are basically two ways to do it.

If it’s a UI-based software, that your users will use interactively:

Create a directory under %localappdata% (by default C:\Users\SOMEUSER\AppData\Local) with the brand name of your company and/or product, and write in there.

You can get the localappdata path using the following line in Dot Net:

string localAppDataPath = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData);

Take for example the screen-capturing software called Greenshot. These guys do it right:

If it’s a non-interactive software, like a Windows Service:

You can do the same as above, but instead of Environment.SpecialFolder.LocalApplicationData use Environment.SpecialFolder.CommonApplicationData, which by default is C:\ProgramData. So your logs will be in C:\ProgramData\MyAmazingCompany\myamazingproduct.log.

Or -not recommended, but not as horrible as writing in Program Files- you can create something custom like C:\MyAmazingCompany\logs. I’ll be honest with you, it’s ugly, but it works.

But in any case, be careful to consider your environment. Is your software supposed to run on Windows, Linux, Mac, everything? A good place to start is here, for Dot Net, but the concept is the same in every language.

And, also important, make your logging configurable! The location should be changeable via a config file. Different systems have different requirements. Someone will need the logs somewhere special for their own reasons.

But whatever you do, PLEASE PLEASE PLEASE DON’T WRITE WHERE THE BINARY IS. DON’T WRITE IN C:\PROGRAM FILES. IT. DOES. NOT. WORK.

New version of Zoro: 1.0.2

I just published a new version of my open source C# Zoro(*) library in Github and Nuget.org.

Zoro is a data masking/anonymization utility. It fetches data from a database or a CSV file, and creates a CSV file with masked data.

The new version, 1.0.2, has been converted to DotNet Standard 2.0. The command line utility and the test project have been converted to Dotnet Core 5.0.

There is a known issue, not with the code but with the Nuget package. The description claims, as was intended, that the package contains not only the library but also the exe, which can be used as a standalone command line utility. But due to some wrong path in the Github Action, it doesn’t.

I’ll try to get that fixed in the next weeks. Until then, if you need the exe, please checkout the code and build with Visual Studio or Visual Studio Code.

(*) YES NOW I KNOW THIS IS MISSPELLED AND THE CORRECT SPELLING IS ZORRO, I DIDN’T WHEN I STARTED THE LIBRARY, SORRY!

How to ask for a certificate the right way: CSR via Windows or Keytool with Subject Alternative Names (SANs)

Sooo you’re working in an enterprise and have to maintain an internal server. The security audit asks you to ensure all HTTP communications are encrypted, so you need to change to HTTPS. Boy is this SO not obvious. You’d think this should be quite easy by now, but there are A LOT of pitfalls in your way.

If you want the TL;DR version, to skip the explanation and go directly to the instructions, scroll directly to the Mandalorian below. No hard feelings, honest 😊

Mistake #1: Use a self-signed certificate

Many, many, MANY tutorials you’ll find online are written with a developer in mind, leaving the maintainer/admin as an afterthought -if that. So what they care about is having some certificate, any certificate, as long as it works on the developer’s PC.

But what this certificate says is basically “I’m Jim because I say so”.

Do I need to say that it won’t work for other PCs? Yes? Well surprise, it won’t.

Mistake #2: Get a certificate from your PC’s certificate authority

I don’t know how some people don’t understand that this, while being a bit more complex, it’s basically the same as #1. What this certificate says is “I’m Jim because someone else who is also Jim says so”.

Yeah, no, it won’t work.

Mistake #3: Get a certificate from a trusted certificate authority using only a server name (or an alias).

Now we’re getting more serious.

Getting a certificate from a trusted certificate authority (CA for short) is the right thing to do. The certificate you get then says “I’m Jim because someone else who you already trust says so”.

So if you get a certificate that verifies you’re, say, server web-ch-zh.xyz123.com or mysite.xyz123.com is good enough. Right?

Ummm…

IT DEPENDS.

If you run a website (e.g. https://www.xyz123.com) and want your HTTPS URL to work without giving a certificate warning that’s fine. You don’t need to do anything else. That’s why most tutorials that avoid the self-signed certificate mine stop here.

But remember, our scenario is that we’re working for an enterprise (a big company) and we’re maintaining an internal server. What that usually -not always, but a lot of the time- means is that communication to our server happens using different hostnames.

Let me give you my own example:

  • I run a service called Joint Information Module or JIM for short -that’s a totally real service name [1].
  • The server name is ch-zh-jim-01.mycompany.local.
  • The users use the web interface of the service by navigating to https://jim.mycompany.com.
  • Another application uses the REST API of the service using the server name (ch-zh-jim-01) without the domain name (mycompany.local).
  • The service uses a queuing software that is installed on the same server. We want to use the same certificate for this as well. The JIM service accesses the queues via https://localhost (and a port number).

Now, if the certificate you got says “ch-zh-jim-01.mycompany.local ” and you try to access the server via https://ch-zh-jim-01, https://jim.mycompany.com, https://localhost or https://127.0.0.1, you’ll get a certificate error much like the following:

certificate error chrome

Also, the REST API won’t work. The caller will throw an exception, e.g. java.security.cert.CertPathValidatorException in Java or System.Security.Authentication.AuthenticationException in DotNet. You can avoid this by forcing your code to not care about invalid certificates but this is a) lazy b) bad c) reaaaaaaaaaaly bad, seriously man, don’t do this unless the API you’re connecting to is completely out of your control (e.g. it belongs to a government).

The correct way

So you need a certificate that is trusted and valid for all the names that will be used to communicate with your server. How do you do that? SIMPLEZ!

  1. Generate a CSR (a certificate signing request, which is a small file you send to the CA) with the alternative names (SANs) you need. That’s what I’ll cover here.
  2. Send it to a trusted CA
    1. either the one your own company operates or
    2. a commercial one (which you have to pay), say Digicert.
  3. Get the signed certificate and install it on your software.

Important note: the CA you send the CSR to must support SANs. Not every CA supports this, for their own reasons. Make sure you read their FAQ or ask their helpdesk. Let’s Encrypt, a free and very popular CA, supports them.

Here I’ll show how you can generate a CSR, both in the “Microsoft World” (i.e. on a Windows machine) and in the “Java World” (i.e. on any machine that has Java installed).

A. Using Windows

Note that this is the GUI way to do this. There’s also a command line tool for this, certreq. I won’t cover it here as this post is already quite long, but you can read a nice guide here and Microsoft’s reference here. One thing to note though is that it’s a bit cumbersome to include SANs with this method.

  1. Open C:\windows\System32\certlm.msc (“Local Computer Certificates”).
  2. Expand “Personal” and right click on “Certificates”. Select “All tasks” > “Advanced Operations” > “Create Custom Request”.
  3. In the “Before you begin” page, click Next.
  4. In the “Select Certificate Enrollment Policy” page, click “Proceed without enrollment policy” and then Next.
  5. In the “Custom Request” page, leave the defaults (CNG key / PKCS #10) and click Next.
  6. In the “Certificate Information” page, click on Details, then on Properties.
  7. In the “General” tab:
    1. In the “Friendly Name” field write a short name for your certificate (that has nothing to do with the server). E.g. cert-jim-05-2021.
    2. In the “Description” field, write a description, duh 😊
  8. In the “Subject” tab:
    1. Under “Subject Name” make sure the “Type” is set to “Full DN” and in the Value field paste the following (without the quotes): “CN=ch-zh-jim-01.mycompany.local, OU=IT, O=mycompany, L=Zurich, ST=ZH, C=CH” and click “Add”. Here:
      • Instead of “ch-zh-jim-01.mycompany.local” enter your full server name, complete with domain name. You can get it by typing ipconfig /all in a command prompt (combine Host Name and Primary Dns Suffix).
      • Instead of “IT” and “mycompany” enter your department and company name respectively.
      • Instead of “Zurich”, “ZH” and “CH” enter the city, state (or Kanton or Bundesland or region or whatever) and country respectively.
    2. Under “Alternative Name”:
      1. Change the type to “IP Address (v4)” and in the Value field type “127.0.0.1”. Click “Add”.
      2. Change the type to “DNS” and in the Value field type the following, clicking “Add” every time:
        • localhost
        • ch-zh-jim-01 (i.e. the server name without the default domain)
        • jim.mycompany.com (i.e. the alias that will be normally used)
        • (add as many names as needed)

Important note: all names you enter there must be resolvable (i.e. there’s a DNS entry for the name) by the CA that will generate your certificate. Otherwise there’s no way they can confirm you’re telling the truth and the request will most likely be rejected.

It should end up looking like this:

  1. In the “Extensions” tab, expand “Extended Key Usage (application policies)”. Select “Server Authentication” and “Client Authentication” and click “Add”.
  2. In the “Private Key” tab, expand “Key Options”.
    1. Set the “Key Size” to 2048 (recommended) or higher.
    2. Check the “Mark private key exportable” check box.
    3. (optional, but HIGHLY recommended) Check the “Strong private key protection” check box. This will make the process ask for a certificate password. Avoid only if your software doesn’t support this (although if it does, you really should question if you should be using it!).

At the end, click OK, then Next. Provide a password (make sure you keep it somewhere safe NOT ON A TEXT FILE ON YOUR DESKTOP, YOU KNOW THAT RIGHT???) and save the CSR file. That’s what you have to send to your CA, according to their instuctions.

B. Using Java

Here the process is sooo much simpler:

  1. Open a command prompt (I’m assuming your Java/bin is in the system path; if not, cd to the bin directory of your Java installation). You should have enough permissions to write to your Java security dir; in Windows, that means that you need an administrative command prompt.
  2. Create the certificate. Type the following, in one line, but given here splitted for clarity. Replace as explained below.
keytool
-genkey
-noprompt
-cacerts
-alias cert-jim-05-2021 
-dname "CN=ch-zh-jim-01.mycompany.local, OU=IT, O=mycompany, L=Zurich, ST=ZH, C=CH" 
-keyalg RSA
-keysize 2048
-storepass changeit
-keypass MYSUPERSECRETPASSWORD
  1. Create the certificate signing request (CSR). Type the following, in one line, but given here splitted for clarity. Replace as explained below.
keytool 
-certreq 
-file c:\temp\cert-jim-05-2021.csr 
-cacerts 
-alias cert-jim-05-2021 
-dname "CN=ch-zh-jim-01.mycompany.local, OU=IT, O=mycompany, L=Zurich, ST=ZH, C=CH" 
-ext "SAN=IP:127.0.0.1,DNS:localhost,DNS:ch-zh-jim-01,DNS:jim.mycompany.com" 
-ext "EKU=serverAuth,clientAuth"
-storepass changeit 
-keypass MYSUPERSECRETPASSWORD

In the steps above, you need to replace:

  • “cert-jim-05-2021”, both in the filename and the alias, with your certificate name (which is the short name for your certificate; this has nothing to do with the server itself).
  • “ch-zh-jim-01.mycompany.local” with the full DNS name of your server.
  • “IT” and “mycompany” with your department and company name respectively.
  • “Zurich”, “ZH” and “CH” with your city, state (or Kanton or Bundesland or region or whatever) and country respectively.
  • “ch-zh-jim-01” with your server name (without the domain name).
  • “jim.mycompany.com” with the DNS alias you’re using. You can add as many as needed, e.g. “DNS:jim.mycompany.com,DNS:jim-server.mycompany.com,DNS:jim.mycompany.gr,DNS:jim.mycompany.ch”

Important note: all names you enter there must be resolvable (i.e. there’s a DNS entry for the name) by the CA that will generate your certificate. Otherwise there’s no way they can confirm you’re telling the truth and the request will most likely be rejected.

  • “changeit” is the default password of the Java certificate store (JAVA_HOME/jre/lib/security/cacerts). It should be replaced by the actual password of the certificate store you’re using. But 99.999% of all java installations never get this changed 😊 so if you don’t know otherwise, leave it as it is.
  • “MYSUPERSECRETPASSWORD” is a password for the certificate. Make sure you keep it somewhere safe NOT ON A TEXT FILE ON YOUR DESKTOP, YOU KNOW THAT RIGHT???

That’s it. The CSR is saved in the path you specified (in the “-file” option). That’s what you have to send to your CA, according to their instuctions.

Enjoy!

[1] no it’s not, c’mon

RabbitMQ: How to move configuration, data and log directories on Windows

A good part of my job has to do with enterprise messaging. When a piece of data -a message- needs to be sent from, say, an invoicing system to an accounting system and then to a customer relationship system and then to the customer portal… it has to navigate treacherous waters.

Avast ye bilge-sucking scurvy dogs! A JSON message from accounting says they hornswaggled 1000 doubloons! Aarrr!!!

So we need to make sure that whatever happens, say if a system is overloaded while receiving the message, the message will not be lost.

A key component in this is message queues (MQ), like RabbitMQ. An MQ plays the middleman; it receives a message from a system and stores it reliably until the next system has confirmed that it picked it up.

My daily duties includes setting up, configuring and maintaining a few RabbitMQ instances. It works great! Honestly, so far -for loads up to a couple of 100s of messages per second- I haven’t even had the need to do any serious tuning.

But one thing that annoys me on Windows is that, after installation, the location of everything except the binaries -configuration, data, logs- is under the profile dir of the user (C:\Users\USERNAME\AppData\Roaming\RabbitMQ) that did the installation, even if the service runs as LocalSystem. Not very good, is it?

Therefore I’ve created this script to help me. The easiest way to use it is run it before you install RabbitMQ. Change the directories in this part and run it from an admin powershell:

# ========== Customize here ==========
$BaseLocation = "C:\mqroot\conf"
$DbLocation = "C:\mqroot\db"
$LogLocation = "C:\mqroot\log"
# ====================================

Then just reboot and run the installation normally; when it starts, RabbitMQ will use the directories you specified.

You can also do it after installation, if you have a running instance and want to move it. In this case do the following (you can find these steps also in the script):

  1. Stop the RabbitMQ service.
  2. From Task Manager, kill the epmd.exe process if present.
  3. Go to the existing base dir (usually C:\Users\USERNAME\AppData\Roaming\RabbitMQ)
    and move it somewhere else (say, C:\temp).
  4. Run this script (don’t forget to change the paths).
  5. Reboot the machine
  6. Run the “RabbitMQ Service (re)install” (from Start Menu).
  7. Copy the contents of the old log dir to $LogLocation.
  8. Copy the contents of the old db dir to $DbLocation.
  9. Copy the files on the root of the old base dir (e.g. advanced.config, enabled_plugins) to $BaseLocation.
  10. Start the RabbitMQ service.

Here’s the script. Have fun 🙂

#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, March 2021
#

# What this script does is:
#   1. Creates the directories where the configuration, queue data and logs will be stored.
#   2. Downloads a sample configuration file (it's necessary to have one).
#   3. Sets the necessary environment variables.

# If you're doing this before installation: 
# Just run it, reboot and then install RabbitMQ.

# If you're doing this after installation, i.e. if you have a 
# running service and want to move its files:
#   1. Stop the RabbitMQ service
#   2. From Task Manager, kill the epmd.exe process if present
#   3. Go to the existing base dir (usually C:\Users\USERNAME\AppData\Roaming\RabbitMQ)
#      and move it somewhere else (say, C:\temp).
#   4. Run this script.
#   5. Reboot the machine
#   6. Run the "RabbitMQ Service (re)install" (from Start Menu)
#   7. Copy the contents of the old log dir to $LogLocation.
#   8. Copy the contents of the old db dir to $DbLocation.
#   9. Copy the files on the root of the old base dir (e.g. advanced.config, enabled_plugins) 
#      to $BaseLocation.
#   10. Start the RabbitMQ service.

# ========== Customize here ==========

$BaseLocation = "C:\mqroot\conf"
$DbLocation = "C:\mqroot\db"
$LogLocation = "C:\mqroot\log"

# ====================================

$exampleConfUrl = "https://raw.githubusercontent.com/rabbitmq/rabbitmq-server/master/deps/rabbit/docs/rabbitmq.conf.example"

Clear-Host
$ErrorActionPreference = "Stop"

$dirList = @($BaseLocation, $DbLocation, $LogLocation)
foreach($dir in $dirList) {
    if (-not (Test-Path -Path $dir)) {
        New-Item -ItemType Directory -Path $dir
    }
}

# If this fails (e.g. because there's a firewall) you have to download the file 
# from $exampleConfUrl manually and copy it to $BaseLocation\rabbitmq.conf
try {
    Invoke-WebRequest -Uri $exampleConfUrl -OutFile ([System.IO.Path]::Combine($BaseLocation, "rabbitmq.conf"))
}
catch {
    Write-Host "(!) Download of conf file failed. Please download the file manually and copy it to $BaseLocation\rabbitmq.conf"
    Write-Host "(!) Url: $exampleConfUrl"
}

&setx /M RABBITMQ_BASE $BaseLocation
&setx /M RABBITMQ_CONFIG_FILE "$BaseLocation\rabbitmq"
&setx /M RABBITMQ_MNESIA_BASE $DbLocation
&setx /M RABBITMQ_LOG_BASE $LogLocation

Write-Host "Finished. Now you can install RabbitMQ."

New version of XMLSlurper: 1.3.0

I just published a new version of my open source C# XmlSlurper library in Github and Nuget.org.

The new version, 1.3.0, contains two major bug fixes:

  1. In previous versions, when the xml contained CDATA nodes, an error was thrown (“Type System.Xml.XmlCDataSection is not supported”). This has been fixed, so now the following works:
<CustomAttributes>
    <Title><![CDATA[DOCUMENTO N. 1234-9876]]></Title>
</CustomAttributes>

This xml can be used as follows:

var cdata = XmlSlurper.ParseText(getFile("CData.xml"));
Console.WriteLine(cdata.Title);
// produces 'DOCUMENTO N. 1234-9876'
  1. In previous versions, when the xml contained xml comments, an error was thrown (“Type System.Xml.XmlComment is not supported”). This has been fixed; the xml comments are now ignored.

Separately, there are a few more changes that don’t impact the users of the library:

  1. A Github action was added that, when the package version changes, automatically builds and tests the project, creates a Nuget package and publishes it to Nuget.org. That will save me quite some time come next version 🙂
  2. The test project was migrated from DotNet Core 2.2 to 3.1.
  3. The tests were migrated from MSTest to xUnit, to make the project able to be developed both in Windows and in Linux -my personal laptop runs Ubuntu.

The new version is backwards compatible with all previous versions. So if you use it, updating your projects is effortless and strongly recommended.