As part of my job, this is something I use a lot. And the thing is, it’s quite easy, it’s just an Invoke-WebRequest. Here’s how I do it:
#
# Source: DotJim blog (https://dandraka.com)
# Jim Andrakakis, January 2023
#
# ===== Parameters =====
param(
[string]$fileName = 'C:\temp\uploadinfo.json',
[string]$rqServer = 'http://myServer:15672', # better use HTTPS though
[string]$rqVhostName = 'myVhost',
[string]$rqQueueName = 'myQueue',
[string]$rqExchangeName = 'amq.default', # or your exchange name
[string]$rqUsername = 'myUser', # this user needs at least 'Management' permissions to post to the REST API
[string]$rqPassword = 'myPass',
# RabbitMQ has a recommended message size limit of 128 MB
# See https://www.cloudamqp.com/blog/what-is-the-message-size-limit-in-rabbitmq.html
# But of course depending on your app you might want to set it lower
[int]$rqMessageLimitMB = 128
)
# ======================
Clear-Host
$ErrorActionPreference = 'Stop'
$WarningPreference = 'Continue'
[string]$rqUrl = "$rqServer/api/exchanges/$rqVhostName/$rqExchangeName/publish"
# Sanity check
if (-not (Test-Path $fileName)) {
Write-Error "File $fileName was not found"
}
# Check RabbitMQ size limit
[int]$rqMessageLimit = $rqMessageLimitMB * 1024 * 1024
[long]$fileSize = (Get-Item -Path $fileName).Length
if ($fileSize -gt $rqMessageLimit) {
Write-Error "File $fileName is bigger that the maximum size allowed by RabbitMQ ($rqMessageLimitMB MB)"
}
$plainCredentials = "$($rqUsername):$($rqPassword)"
$encodedCredentials = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($plainCredentials))
$authHeader = "Basic " + $encodedCredentials
[string]$content = Get-Content -Path $fileName -Encoding UTF8
$msgBase64 = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($content))
$json = "{`"properties`":{`"content_type`":`"application/json`",`"delivery_mode`":2},`"routing_key`":`"$rqQueueName`",`"payload`":`"$msgBase64`",`"payload_encoding`":`"base64`"}"
$resp = Invoke-WebRequest -Method Post -Uri $rqUrl -Headers @{'Authorization'= $authHeader} -Body $json
if([math]::Floor($resp.StatusCode/100) -ne 2) {
Write-Error "File $fileName could not be posted, error $($resp.BaseResponse)"
}
Write-Host "File $fileName was posted to $rqUrl"
Who among us hasn’t found him- or herself in this very awkward position: committing a config or code file with secrets (such as passwords or API keys) and then semi-panicked googling how to delete it from source control.
Been there and let me tell you the easiest way to delete it: copy all the code on disk, delete the repository completely and then re-create it.
(if this is not an option, well, there’s still a way but with much more work and risk, so do keep that code backup around!)
But you know what’s even better? That’s right, avoid this in the first place! That’s why Git hooks are so useful: they work without you neededing to remember to check your config files every time.
So here’s my solution to this:
In the repository, go to .git/hooks and rename pre-commit.sample to pre-commit (i.e. remove the extension)
Open pre-commit with a text editor and replace its contents with the following:
Add a new directory on the root of the repository named hooks.
Inside this, add a text file named pre-commit.ps1 with the following code:
#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, July 2022
#
Clear-Host
$ErrorActionPreference='Stop'
# ===== Change here =====
$listOfExtensions=@('*.xml','*.config')
$listOfSecretNodes=@('username','password','clientid','secret','connectionstring')
$acceptableString='lalala'
# ===== Change here =====
$codePath = (Get-Item -Path $PSScriptRoot).Parent.Parent.FullName
$errorList=New-Object -TypeName 'System.Collections.ArrayList'
foreach($ext in $listOfExtensions) {
$list = Get-ChildItem -Path $codePath -Recurse -Filter $ext
foreach($file in $list) {
$fileName = $file.FullName
if ($fileName.Contains('\bin\')) {
continue
}
Write-Host "Checking $fileName for secrets"
[xml]$xml=[xml]((Get-Content -Path $fileName).ToLowerInvariant())
foreach($secretName in $listOfSecretNodes) {
$nodes = $xml.SelectNodes("//*[contains(local-name(), '$secretName')]")
foreach($node in $nodes) {
if ($node.InnerText.ToLowerInvariant() -ne $acceptableString) {
$str = "[$fileName] $($node.Name) contains text other than '$acceptableString', please replace this with $acceptableString before commiting."
$errorList.Add($str) | Out-Null
Write-Warning $str
}
}
}
}
}
if ($errorList.Count -gt 0) {
Write-Error 'Commit cancelled, please correct before commiting.'
}
So there you have it. I’m getting automatically stopped every time I tried to commit any .xml or .config file that contains a node with a name that contains username, password, clientid, secret or connectionstring, whenever the value of it is not ‘lalala’.
Obviously the extensions, node names and acceptable string can be changed at the top of the script. You can also change this quite easily to check JSON files as well.
Also note that this works on Windows (because of the Powershell path in the pre-commit hook) but with a minor change in the pre-commit bash script, you should be able to make it work cross-platform with Powershell core. I haven’t tested it but it should be:
Normally I start every post with a small introduction. This one I want to keep as short as possible so I’ll just say this: It’s 2021. You need a password manager.
Let’s start from the very beginning. First, I’ll explain a few things you’ll hear often. A lot of these words can seem daunting but actually are quite simple. Then we get down to the nitty gritty.
I DON’T WANT TO DO THIS WHY DO I NEED TO DO THIS???!??!
Because there are some things that you 1) want to be able to do on the internet but 2) don’t want other people to be able do (at least not without you knowing).
You don’t want other people to move money from your bank account. Or buy things with your credit card. You get the idea.
But but but I already have a password!
Yes, you do. But there are some problems.
If you’re, well, human, you can remember some things but not many and not very well (read this if you don’t believe me). And it’s 2021, if you don’t live under a rock you have at the very least 10-20 accounts in different services, like your bank, your email etc etc. Try to count them and write in the comments how many you found 😊
The other problem is: criminals steal data from these services. A lot. Like, in the billions. Estee Lauder had a breach on February 2020 where 440 million records -data about people- were stolen. MGM Resorts, which you know from the casino in “Ocean’s 11”, had personal information about more than 10 million guests stolen. And these are just 2 of the around 3000 data breaches that were reported in 2020 in the US alone.
What this means is that your password will get stolen and there’s nothing you can do about it. Well, almost nothing. You can and should do 3 things:
Have a unique password per service. This way, when your H&M password is stolen, it cannot be used to pay from your PayPal.
Use random passwords. For crying out loud, do not use your phone number. You think that adding a few letters here and there makes it safe. It does not. A computer with a program you can download for free can crack your “safe” password in like an hour. The password must be long and random, something like g5D9C467YxeEfAmqL. You get the idea.
Use 2-factor authentication. Since this post is already long, I’ll get to this in a later one.
What does “authentication” mean? And what are these “credentials” I keep hearing about?
Credentials just means whatever you need to give to a service, like a web site, so that it checks it’s really you. Some of it is secret, some of it is not. Usually it’s a username and a password but it might be more, like your fingerprint or a code that you receive in your phone.
Authentication is just the process that checks the credentials and lets you in (or not).
What’s a password manager?
It’s a program that stores your credentials and helps you use them. Because your passwords must be long, it’s tedious to have to type them yourself. So the password manager for example can auto-fill them, or you can copy-paste them, in your e-banking web site.
Ok, ok, I’ll do it, but which one should I use?
There are many good password managers you can use like 1Password, LastPass, Devolutions, NordPass and others. Here I’ll use my favourite one which is Bitwarden, because it’s arguably the best free one and in my humble opinion the easiest to use.
Obviously this is just one way to do it; it works and it’s secure, but of course you can change things, for example use a different program. The main things to consider if you decide to use another one is:
It should have both a computer as well as a smartphone application.
It should be able to synchronize your credentials between them.
It should be as simple to use as possible.
And how much time will it take?
Realistically, assuming you’re an average computer and smartphone user, for 5-10 web sites you’ll need around a couple of hours from start to finish. Obviously if you have dozens it will take more -not proportionally- but it’s also worth more. If you get stuck, write me in the comments and I’ll do my best to help.
UPDATE: some friends suggested that instead of doing all your sites at once, it makes the effort more manageable to do the most important ones first -e-banking, email etc. The rest you can do when you come across them in everyday use.
Now I’ll explain how you do it in your computer and smartphone. Ready, set, go!
We’ll start from your computer because usually it’s easier to create the account there. Then we’ll continue to your smartphone. But the very first thing you need to do is grab a piece of old fashioned paper.
Step 1: Write a password and a 6 digit code.
Get a paper. Yes the traditional one!
Not necessarily a post-it, but this will do as well
Write 20 or more random numbers and letters, both lower and capital. Something like 6xTzHx41jKQ3yg48FeR9sAb. This will be your password.
You don’t need to remember this.
In the same piece of paper write 6 random numbers. DO NOT USE ANYTHING REAL OR EVEN CLOSE TO IT LIKE YOUR BIRTHDAY OR YOUR POSTCODE OR YOUR PHONE, NOT EVEN CHANGED. This will be your unlock code.
This code will be the one and only thing you need to learn by heart.
Keep this paper safe in your desk at home but NOT in your computer -don’t take a photo of it or write it in a Word file.
Step 2: Create your Bitwarden account
On your computer, go to bitwarden.com and click “Get started”.
Fill in the form, it’s really simple. Use the password you wrote on the paper.
Step 3: Install the browser extension
Still on your computer, open your favourite browser -Firefox, Chrome, Edge, Opera, whatever- go to the bitwarden extension and install it.
In case you’re using anything else, just google “bitwarden <browser name>” and you’ll find it.
NOTE: As you’ll see, about the only annoying thing with Bitwarden is that if you click outside of it before you save your changes, it closes and loses your input. There’s a solution for this: you can click the “Pop out” button” and then it opens as a separate window. The “Pop up” button is this one:
When the extension is installed, you’ll get the Bitwarden shield icon on the top right corner of your browser. Click it and fill in your email and password.
Once you log in you see your list of passwords. This a called your “vault”. For now, it’s obviously empty.
Click “Settings”, then “Unlock with pin”. Enter the 6 numbers you wrote on the paper and uncheck the “lock with master password…” check box.
Step 4: Store your credentials
If you’ve done so far, great job! Now it’s the time to start storing your passwords, one by one.
Click the shield icon of Bitwarden, then the plus icon on the top right corner.
Start with your email. Enter the name, username and password -the ones you have already. Add also the URL you use to access the site. Then click “Save”.
One by one, add all the sites and other services you have. This will probably take some time; my list has more than 400 entries 😊
Step 5: Try it
So all of this is supposed to help you right? Here’s how it helps you login. Say you want to log in to your email for example.
Click the shield icon of Bitwarden, click “My vault” and click the little arrow of the site. You’ll see that it takes you there.
In your email site, click “Sign in” or “Login” or whatever it has. Right click in the username or password and select Bitwarden > Auto-fill > your site name. Then click Next or Login or whatever it has.
If for whatever reason right click doesn’t find the site, there’s another way that’s not as easy but works every time. From “My vault” click the head icon to copy the username, then paste it in the site, then click the key icon to copy the password, then paste it in the site.
After doing it a few times, you’ll get the hang of it; it will feel very easy very quickly.
Step 6: Change your passwords
Until now you’ve done great, but we’re still using our old passwords. Now it’s the time to make them big and hard 😉
The exact process differs slightly for every site, obviously, but not much. In this example, I’ll use a popular e-shop, Zara UK.
Go to your profile and go to change password:
In the bitwarden “My vault” click the key icon of the site (see above) to copy the existing password. Paste it in the “Current password” box of the web site.
Then go in the bitwarden “My vault” again and click somewhere in the middle of the site name. This will open the entry. Click Edit on the top right corner.
Click the double arrow next to the password and click “yes” in the “overwrite password” question. Slide the length of the password to something over 17, click “regenerate” and then “select”.
Click “Save” to save the new password.
Now go to “My vault” again, click the key icon to copy the new password, go to the web site and paste it twice. Then click “Update password” or whatever button is there.
The first time you do it will be cumbersome, but after the first 2-3 sites, it will feel really easy.
If you’ve reached this far, congratulations 🥳🎉👏 You’ve done the hard work! The last thing to do is install the app on your smartphone so you can use it there too. Let’s go!
Here we get to the fun part -well, if not fun, certainly the easiest and most useful. I’ll give screenshots for iPhone, because that’s what I have, but for Android it’s almost the same.
Step 1: Install the Bitwarden App
Go to your App Store (or Play Store for Android), find Bitwarden and install it.
Step 2: Login
Open the app, click Log In and fill in the email and password (the one you wrote on the paper).
Go to Settings and press “Unlock with PIN code”. Enter the 6 digit number you wrote on the paper and select “No”.
We’re ready to use it!
Step 3: Use it to login to sites
Let’s try to use the browser in our smartphone to login to Zara UK. Navigate to the web site and click Login, or My Account or whatever it has:
Now switch to Bitwarden (you might need to unlock it with your 6 digit code), find the site, press the 3 dots and click Copy Username.
Switch to the browser, tap in the username box and paste the username.
Repeat the same steps for the password and click Log In.
Ta da! We’re in!
That’s all folks
This was what you have to do to get started and work with Bitwarden. It’s not an exhaustive guide, mind you, there are more to it. But it covers the most important part: securely creating, storing and using unique passwords that are impossible to guess.
I hope this works for you. If you have any questions or suggestions, I’ll be more than happy to discuss in the comments!
This is one I’m going to need every holiday season 🏖️☀️🏊😊
Every time I go for summer holidays I end up, one way or another, with a bunch of photos. You know, actual photos. The paper type. And lovely -in an old-timey way- as they may be, one cannot easily display it on one’s laptop, phone, tablet etc etc.
Sure, there are ways, but…
So what I do is scan them and upload them to the cloud. But then I have another problem: the date of the photos is not correct, and there’s no GPS location info (the so-called metadata). My cloud photo app doesn’t display it on the right “On this day” day. I also cannot find it by searching for e.g. “Cyprus” (where the photo above was taken, a looong time ago) and it won’t get shown when I create a map of where I’ve been.
First world problems, I know, but still. I googled around and found the solution using exiftool and touch (on linux) or powershell (on windows).
As a prerequisite, you need:
For Linux: install exiftool using sudo apt install exiftool
For Windows: download and install exiftool from here.
Next step is that, every time I have a new batch of scanned photos, I have to decide when and where they were taken. For old photos, this takes some guesswork but that’s ok -it doesn’t have to be 100% accurate.
Then I go to Google Maps and find the place they were taken in. Again, I don’t need it to be 100% correct. In the case of the photo above “somewhere in Larnaca” is good enough -that’s where we went for our customary end-of-highschool 5-day trip. After I find a suitable place I click on it and copy the coordinates.
Note that the these coordinates need an addition. You need to specify if you’re on the northern or southern hemisphere (i.e. above or below the equator) and west or east of the prime meridian (i.e. left or right of Greenwich, England).
In my case, Cyprus is on the northern hemisphere (N) and east of the prime meridian (E), so the correct way to write these coordinates is 34.914722N, 33.638000E. You can paste it in Google Maps to verify if it’s correct:
Now that we have our date (say, 15th of April 1993) and place (say, 34.914722N, 33.638000E) we’re ready. The process is quite simple:
Note that the Northern/Southern and Western/Eastern (N and E in this case) are given separately in the GPSLatitudeRef and GPSLongitudeRef parameters respectively. Also note that, for completeness, I’ve added a height (GPSAltitude) of 10 meters.
Obviously you might change the filter (in my case I tagged all the jpg photos in the folder) but otherwise, that’s it. This is what they look like in my cloud photo app (I use amazon photos):
Sooo you’re working in an enterprise and have to maintain an internal server. The security audit asks you to ensure all HTTP communications are encrypted, so you need to change to HTTPS. Boy is this SO not obvious. You’d think this should be quite easy by now, but there are A LOT of pitfalls in your way.
If you want the TL;DR version, to skip the explanation and go directly to the instructions, scroll directly to the Mandalorian below. No hard feelings, honest 😊
Mistake #1: Use a self-signed certificate
Many, many, MANY tutorials you’ll find online are written with a developer in mind, leaving the maintainer/admin as an afterthought -if that. So what they care about is having some certificate, any certificate, as long as it works on the developer’s PC.
But what this certificate says is basically “I’m Jim because I say so”.
Do I need to say that it won’t work for other PCs? Yes? Well surprise, it won’t.
Mistake #2: Get a certificate from your PC’s certificate authority
I don’t know how some people don’t understand that this, while being a bit more complex, it’s basically the same as #1. What this certificate says is “I’m Jim because someone else who is also Jim says so”.
Yeah, no, it won’t work.
Mistake #3: Get a certificate from a trusted certificate authority using only a server name (or an alias).
Now we’re getting more serious.
Getting a certificate from a trusted certificate authority (CA for short) is the right thing to do. The certificate you get then says “I’m Jim because someone else who you already trust says so”.
So if you get a certificate that verifies you’re, say, server web-ch-zh.xyz123.com or mysite.xyz123.com is good enough. Right?
Ummm…
IT DEPENDS.
If you run a website (e.g. https://www.xyz123.com) and want your HTTPS URL to work without giving a certificate warning that’s fine. You don’t need to do anything else. That’s why most tutorials that avoid the self-signed certificate mine stop here.
But remember, our scenario is that we’re working for an enterprise (a big company) and we’re maintaining an internal server. What that usually -not always, but a lot of the time- means is that communication to our server happens using different hostnames.
Let me give you my own example:
I run a service called Joint Information Module or JIM for short -that’s a totallyreal service name [1].
Another application uses the REST API of the service using the server name (ch-zh-jim-01) without the domain name (mycompany.local).
The service uses a queuing software that is installed on the same server. We want to use the same certificate for this as well. The JIM service accesses the queues via https://localhost (and a port number).
Now, if the certificate you got says “ch-zh-jim-01.mycompany.local ” and you try to access the server via https://ch-zh-jim-01, https://jim.mycompany.com, https://localhost or https://127.0.0.1, you’ll get a certificate error much like the following:
Also, the REST API won’t work. The caller will throw an exception, e.g. java.security.cert.CertPathValidatorException in Java or System.Security.Authentication.AuthenticationException in DotNet. You can avoid this by forcing your code to not care about invalid certificates but this is a) lazy b) bad c) reaaaaaaaaaaly bad, seriously man, don’t do this unless the API you’re connecting to is completely out of your control (e.g. it belongs to a government).
The correct way
So you need a certificate that is trusted and valid for all the names that will be used to communicate with your server. How do you do that? SIMPLEZ!
Generate a CSR (a certificate signing request, which is a small file you send to the CA) with the alternative names (SANs) you need. That’s what I’ll cover here.
Send it to a trusted CA
either the one your own company operates or
a commercial one (which you have to pay), say Digicert.
Get the signed certificate and install it on your software.
Important note: the CA you send the CSR to must support SANs. Not every CA supports this, for their own reasons. Make sure you read their FAQ or ask their helpdesk. Let’s Encrypt, a free and very popular CA, supports them.
Here I’ll show how you can generate a CSR, both in the “Microsoft World” (i.e. on a Windows machine) and in the “Java World” (i.e. on any machine that has Java installed).
A. Using Windows
Note that this is the GUI way to do this. There’s also a command line tool for this, certreq. I won’t cover it here as this post is already quite long, but you can read a nice guide here and Microsoft’s reference here. One thing to note though is that it’s a bit cumbersome to include SANs with this method.
Open C:\windows\System32\certlm.msc (“Local Computer Certificates”).
Expand “Personal” and right click on “Certificates”. Select “All tasks” > “Advanced Operations” > “Create Custom Request”.
In the “Before you begin” page, click Next.
In the “Select Certificate Enrollment Policy” page, click “Proceed without enrollment policy” and then Next.
In the “Custom Request” page, leave the defaults (CNG key / PKCS #10) and click Next.
In the “Certificate Information” page, click on Details, then on Properties.
In the “General” tab:
In the “Friendly Name” field write a short name for your certificate (that has nothing to do with the server). E.g. cert-jim-05-2021.
In the “Description” field, write a description, duh 😊
In the “Subject” tab:
Under “Subject Name” make sure the “Type” is set to “Full DN” and in the Value field paste the following (without the quotes): “CN=ch-zh-jim-01.mycompany.local, OU=IT, O=mycompany, L=Zurich, ST=ZH, C=CH” and click “Add”. Here:
Instead of “ch-zh-jim-01.mycompany.local” enter your full server name, complete with domain name. You can get it by typing ipconfig /all in a command prompt (combine Host Name and Primary Dns Suffix).
Instead of “IT” and “mycompany” enter your department and company name respectively.
Instead of “Zurich”, “ZH” and “CH” enter the city, state (or Kanton or Bundesland or region or whatever) and country respectively.
Under “Alternative Name”:
Change the type to “IP Address (v4)” and in the Value field type “127.0.0.1”. Click “Add”.
Change the type to “DNS” and in the Value field type the following, clicking “Add” every time:
localhost
ch-zh-jim-01 (i.e. the server name without the default domain)
jim.mycompany.com (i.e. the alias that will be normally used)
(add as many names as needed)
Important note: all names you enter there must be resolvable (i.e. there’s a DNS entry for the name) by the CA that will generate your certificate. Otherwise there’s no way they can confirm you’re telling the truth and the request will most likely be rejected.
It should end up looking like this:
In the “Extensions” tab, expand “Extended Key Usage (application policies)”. Select “Server Authentication” and “Client Authentication” and click “Add”.
In the “Private Key” tab, expand “Key Options”.
Set the “Key Size” to 2048 (recommended) or higher.
Check the “Mark private key exportable” check box.
(optional, but HIGHLY recommended) Check the “Strong private key protection” check box. This will make the process ask for a certificate password. Avoid only if your software doesn’t support this (although if it does, you really should question if you should be using it!).
At the end, click OK, then Next. Provide a password (make sure you keep it somewhere safe NOT ON A TEXT FILE ON YOUR DESKTOP, YOU KNOW THAT RIGHT???) and save the CSR file. That’s what you have to send to your CA, according to their instuctions.
B. Using Java
Here the process is sooo much simpler:
Open a command prompt (I’m assuming your Java/bin is in the system path; if not, cd to the bin directory of your Java installation). You should have enough permissions to write to your Java security dir; in Windows, that means that you need an administrative command prompt.
Create the certificate. Type the following, in one line, but given here splitted for clarity. Replace as explained below.
“cert-jim-05-2021”, both in the filename and the alias, with your certificate name (which is the short name for your certificate; this has nothing to do with the server itself).
“ch-zh-jim-01.mycompany.local” with the full DNS name of your server.
“IT” and “mycompany” with your department and company name respectively.
“Zurich”, “ZH” and “CH” with your city, state (or Kanton or Bundesland or region or whatever) and country respectively.
“ch-zh-jim-01” with your server name (without the domain name).
“jim.mycompany.com” with the DNS alias you’re using. You can add as many as needed, e.g. “DNS:jim.mycompany.com,DNS:jim-server.mycompany.com,DNS:jim.mycompany.gr,DNS:jim.mycompany.ch”
Important note: all names you enter there must be resolvable (i.e. there’s a DNS entry for the name) by the CA that will generate your certificate. Otherwise there’s no way they can confirm you’re telling the truth and the request will most likely be rejected.
“changeit” is the default password of the Java certificate store (JAVA_HOME/jre/lib/security/cacerts). It should be replaced by the actual password of the certificate store you’re using. But 99.999% of all java installations never get this changed 😊 so if you don’t know otherwise, leave it as it is.
“MYSUPERSECRETPASSWORD” is a password for the certificate. Make sure you keep it somewhere safe NOT ON A TEXT FILE ON YOUR DESKTOP, YOU KNOW THAT RIGHT???
That’s it. The CSR is saved in the path you specified (in the “-file” option). That’s what you have to send to your CA, according to their instuctions.
A good part of my job has to do with enterprise messaging. When a piece of data -a message- needs to be sent from, say, an invoicing system to an accounting system and then to a customer relationship system and then to the customer portal… it has to navigate treacherous waters.
Avast ye bilge-sucking scurvy dogs! A JSON message from accounting says they hornswaggled 1000 doubloons! Aarrr!!!
So we need to make sure that whatever happens, say if a system is overloaded while receiving the message, the message will not be lost.
A key component in this is message queues (MQ), like RabbitMQ. An MQ plays the middleman; it receives a message from a system and stores it reliably until the next system has confirmed that it picked it up.
My daily duties includes setting up, configuring and maintaining a few RabbitMQ instances. It works great! Honestly, so far -for loads up to a couple of 100s of messages per second- I haven’t even had the need to do any serious tuning.
But one thing that annoys me on Windows is that, after installation, the location of everything except the binaries -configuration, data, logs- is under the profile dir of the user (C:\Users\USERNAME\AppData\Roaming\RabbitMQ) that did the installation, even if the service runs as LocalSystem. Not very good, is it?
Therefore I’ve created this script to help me. The easiest way to use it is run it before you install RabbitMQ. Change the directories in this part and run it from an admin powershell:
Then just reboot and run the installation normally; when it starts, RabbitMQ will use the directories you specified.
You can also do it after installation, if you have a running instance and want to move it. In this case do the following (you can find these steps also in the script):
Stop the RabbitMQ service.
From Task Manager, kill the epmd.exe process if present.
Go to the existing base dir (usually C:\Users\USERNAME\AppData\Roaming\RabbitMQ) and move it somewhere else (say, C:\temp).
Run this script (don’t forget to change the paths).
Reboot the machine
Run the “RabbitMQ Service (re)install” (from Start Menu).
Copy the contents of the old log dir to $LogLocation.
Copy the contents of the old db dir to $DbLocation.
Copy the files on the root of the old base dir (e.g. advanced.config, enabled_plugins) to $BaseLocation.
Start the RabbitMQ service.
Here’s the script. Have fun 🙂
#
# Source: DotJim blog (http://dandraka.com)
# Jim Andrakakis, March 2021
#
# What this script does is:
# 1. Creates the directories where the configuration, queue data and logs will be stored.
# 2. Downloads a sample configuration file (it's necessary to have one).
# 3. Sets the necessary environment variables.
# If you're doing this before installation:
# Just run it, reboot and then install RabbitMQ.
# If you're doing this after installation, i.e. if you have a
# running service and want to move its files:
# 1. Stop the RabbitMQ service
# 2. From Task Manager, kill the epmd.exe process if present
# 3. Go to the existing base dir (usually C:\Users\USERNAME\AppData\Roaming\RabbitMQ)
# and move it somewhere else (say, C:\temp).
# 4. Run this script.
# 5. Reboot the machine
# 6. Run the "RabbitMQ Service (re)install" (from Start Menu)
# 7. Copy the contents of the old log dir to $LogLocation.
# 8. Copy the contents of the old db dir to $DbLocation.
# 9. Copy the files on the root of the old base dir (e.g. advanced.config, enabled_plugins)
# to $BaseLocation.
# 10. Start the RabbitMQ service.
# ========== Customize here ==========
$BaseLocation = "C:\mqroot\conf"
$DbLocation = "C:\mqroot\db"
$LogLocation = "C:\mqroot\log"
# ====================================
$exampleConfUrl = "https://raw.githubusercontent.com/rabbitmq/rabbitmq-server/master/deps/rabbit/docs/rabbitmq.conf.example"
Clear-Host
$ErrorActionPreference = "Stop"
$dirList = @($BaseLocation, $DbLocation, $LogLocation)
foreach($dir in $dirList) {
if (-not (Test-Path -Path $dir)) {
New-Item -ItemType Directory -Path $dir
}
}
# If this fails (e.g. because there's a firewall) you have to download the file
# from $exampleConfUrl manually and copy it to $BaseLocation\rabbitmq.conf
try {
Invoke-WebRequest -Uri $exampleConfUrl -OutFile ([System.IO.Path]::Combine($BaseLocation, "rabbitmq.conf"))
}
catch {
Write-Host "(!) Download of conf file failed. Please download the file manually and copy it to $BaseLocation\rabbitmq.conf"
Write-Host "(!) Url: $exampleConfUrl"
}
&setx /M RABBITMQ_BASE $BaseLocation
&setx /M RABBITMQ_CONFIG_FILE "$BaseLocation\rabbitmq"
&setx /M RABBITMQ_MNESIA_BASE $DbLocation
&setx /M RABBITMQ_LOG_BASE $LogLocation
Write-Host "Finished. Now you can install RabbitMQ."
Some time ago, a friend of mine (the one of “How I fought off a Facebook hacker” fame) had problems with his Windows laptop, basically the machine became next to useless. Sadly, while I generally like Windows (there are exceptions) this is something that happens all too often. So I solved it by installing Ubuntu, and even though he’s not technically proficient he’s very happy -the machine isn’t exactly lightning fast, but it works and it’s stable.
But a small mistake I made was installing the latest-greatest Ubuntu version available at the time, 19.04. Now for those who don’t know, Ubuntu has some releases that are supported for a long time, called LTS for Long Term Support, and the ones in between that are… not. Full list here.
So as of January 2020, 19.04 went into End-Of-Life status, meaning you can’t download and install updates the normal way (apt upgrade) any more. And without updates, you can’t upgrade to a newer release (do-release-upgrade) as well. The first symptom is that, while trying to install updates, he was getting errors similar to the following:
E: Unable to locate package XXX
An additional problem is that we’re in different countries, so I couldn’t just do the usual routine backup-format-reinstall everything 🙂
But as usual, Google is your friend! That’s how I solved it from the command line:
sudo sed -i -re 's/([a-z]{2}\.)?archive.ubuntu.com|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list
sudo apt update
sudo apt upgrade
# ...wait for like 30min, then restart...
sudo do-release-upgrade
# ...wait for a couple of hours, restart
What does this do? Well everything except the first line is the standard procedure to upgrade: update (i.e. refresh info for) the software repositories, upgrade (i.e. download and install the updates), restart and then do-release-upgrade which upgrades the complete Ubuntu system -always to the latest LTS release.
But the “magic” is in the first line (and let’s give credit where it’s due). This changes the list that keeps the repositories location (/etc/apt/sources.list) from the normal locations (under archive.ubuntu.com or security.ubuntu.com) to the “historic” servers, old-releases.ubuntu.com. For more info, see “Update sources.list” here.
So after that is done, apt upgrade can now install whatever updates are available and then do-release-upgrade can do its job.
We already saw that the health page works. But it’s time to check if our objective was met.
Remember, the goal is to let developers outside our network use the mock service to help them with their implementation. To see if this works as intended, we can use the swagger file we created and the online Swagger editor.
So open the CustomerTrust.yaml file with a text editor, copy all its contents, navigate your browser to https://editor.swagger.io, delete the default content and paste ours. You’ll get this:
Select the mock service from the drop down, click on one of the services, click “Try it out” and then “Execute“. After a few seconds you… will get an error, something like “NetworkError when attempting to fetch resource.“
Why? Well it’s the browser preventing us from doing do. If you press F12 and watch the Console, you’ll see something like “Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at xxxxxx”. More info here, but in short, it’s a security measure. You can either disable it in your browser’s settings (WHICH IS A BAD IDEA) or use the curl utility, which you can download here for your operating system.
[EDIT] or I could not be lazy, go back to the wiremock config and set the CORS-related HTTP response headers properly, as explained here, here and specifically for wiremock here.
So after you install curl, you can get the command line from the Swagger Editor:
For GET, the command should be:
curl -X GET "https://graubfinancemock.azurewebsites.net/api/1.0/CustomerTrust/CHE-123.456.789" -H "accept: application/json"
Which ever method you pick -GET or POST- we’ll add to this a -v at the end (for more info). So run it and at the end you’ll get this:
401 Unauthorized* Connection #0 to host graubfinancemock.azurewebsites.net left intact
Makes sense right? The mock service expects an authorization token which we haven’t provided. Let’s add this:
A small note on Windows: if you try this in Powershell, it seems that the json escaping is acting funny. If you try it through cmd, it works just fine.
That’s all folks
So now our kind-of-fictional-but-actually-quite-real developers can access the service and test their code against it. And wherever we make a change and push it, the service is updated automatically. Not bad, isn’t it? 🙂
That concludes this guide and its introductory journey in the world of Devops (or, as a friend of mine more accurately calls the field, SRE -short for “Site Reliability Engineering”).
I hope you enjoyed it as much as I did writing it -I really did. I’m sure you’ll have many, many questions which I can try to answer -but no promises 🙂 You can ask here in the comments (better) or in my twitter profile @JimAndrakakis.
Resources
I’ve put all the code in a github repository, here; the only change is that I moved the pipeline yaml in the devops folder and removed my name. You can also find the docker image in docker hub, here.
Have fun coding!
Software, Greece, Switzerland. And coffee. LOTS of coffee !
Dot Jim 