All posts by Jim

Software engineer from Crete living in Switzerland; C# & Azure paladin; economics hobbyist; firearm enthusiast; perpetually tormented by 3 beautiful women :-)

Stories from the Field, #1: Learn how to push back

(note: all “Stories from the Field” are true, thinly anonymized to protect the -usually- guilty)

Teammate: (goes to a big and important customer)

Customer: I want a software like this and that

Customer: And I want it yesterday

Project Manager: They want it yesterday

Teammate: But I need a bit more time in order to implement some UI checks, so that users don’t make mistakes

Customer: Our users don’t make mistakes, they are permanent employees for so many years, they know their job

Teammate: Hmmmokey

Teammate: (implements software in just a few days)

Teammate: (delivers)

Customer: (installs)

Users: (use the software)

Users: (literally fuck up everything that is possible and some things that are not)

Customer: WHY ARE THE DATA WRONG

Teammate: …but you said…

Customer: you’re not good I want another one

Project Manager: Jim you’re assigned to this

Jim: I will rewrite it from zero and I will implement these UI checks plus many many more

Customer: I WANT IT YESTERDAY

Project Manager: THEY WANT IT YESTERDAY

Jim: (doesn’t give a shit)

Jim: (writes code anywhere, anytime, day, night, while eating, while getting the baby to sleep, while helping his wife with breastfeeding etc etc)

Jim: (delivers)

Customer: Why is this 40MB this is bigger than the previous one I don’t like this

Jim: (loses his shit and starts screaming)

Customer: Jeez why are you so nervous you need to calm down

Customer: (installs)

Users: (use the software)

Users: OH HEY THIS WORKS

Users: IT HAS HELPFUL COLOURS TOO

Users: AND IT HAS EXPLANATIONS FOR EACH FIELD

Users: THIS IS GREAT

Customer: great job Jim, see I told you the first guy was not good

Jim: (silently curses in languages he doesn’t even speak)

Note: to be fair, the “40MB” complaint wasn’t as irrational as it sounds. The software had to be copied to many client computers, some of them in remote parts of the country with slow lines; this was still the days of ISDN. Still, the refactoring was worth it. The added volume was caused by a reporting library (Crystal Reports for .Net) which solved many problems by itself. I now understand the frustration of the customer’s IT as someone had to stay up all night copying. But the pressure from management was so much that at this point the poor guy just said the wrong thing at the wrong time to the wrong person. Elias if you ever read this, please accept my apologies 😊

New version of Zoro: 2.0

I just published a new version of my open source C# Zoro library in Github and Nuget.org.

Zoro is a data masking/anonymization utility. It fetches data from a database or a CSV file, masks (i.e. anonymizes) them according to the configuration provided and uses the masked data to create a CSV file or run SQL statements such as INSERTs or UPDATEs.

The new version, 2.0, has been converted to DotNet Standard 2.1 to take advantage of some useful DotNet features. The command line utility and the test project are written with DotNet Core 5.0.

The issue from 1.0.2, where the Nuget package did not contain the executables, has been corrected. The package now contains both a Win64 and a Linux64 executable. Since they are self-contained programs, no prior installation of DotNet is needed.

But the most important new feature is a new MaskType, “Query”. With this, the library can retrieve values from a database and pick a random one. In previous versions this was only possible with lists that were fixed in the XML (MaskType=List).

For example, let’s say you are masking the following data:

IDNameCityCountry
1ABBBadenCH
2FAGEAthensGR
3IKEADelftNL
Table “customers”

In the database you might also have a table with cities and countries:

CityNameCountryCode
ZürichCH
GenevaCH
BernCH
RethimnoGR
ChaniaGR
KalamataGR
GoudaNL
GeldropNL
Table “cities”

In order to anonymize the above data, your config could look like this:

<?xml version="1.0"?>
<MaskConfig xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <FieldMasks>
    <FieldMask>
      <FieldName>ID</FieldName>
      <MaskType>None</MaskType>
    </FieldMask>  
    <FieldMask>
      <FieldName>Name</FieldName>
      <MaskType>Similar</MaskType>
    </FieldMask>      
    <FieldMask>
      <FieldName>Country</FieldName>
      <MaskType>None</MaskType>
    </FieldMask>  
    <FieldMask>
      <FieldName>City</FieldName>
      <MaskType>Query</MaskType>
      <QueryReplacement
      	SelectorField="Country" 
      	GroupField="Countrycode" 
      	ValueField="Cityname" 
      	Query="SELECT cityname, countrycode FROM cities" />
    </FieldMask>  
  </FieldMasks>
  <DataSource>Database</DataSource>
  <DataDestination>Database</DataDestination>
  <ConnectionString>
Server=DBSRV1;Database=appdb;Trusted_Connection=yes;
  </ConnectionString>
  <ConnectionType>
System.Data.SqlClient
  </ConnectionType>
  <SqlSelect>
SELECT * FROM customers
  </SqlSelect>
  <SqlCommand>
INSERT INTO customers_anonymous
(ID, Name, City, Country)
VALUES
($ID, $Name, $City, $Country)
  </SqlCommand>
</MaskConfig>

This will result in a table looking like this:

IDNameCityCountry
1EGTBernCH
2SOLEChaniaGR
3UFOEGeldropNL
Table “customers_anonymous”

If you have any questions, please write in the comments.

Enjoy!

Signs that you need coffee, #6

You wake up on Monday morning, which is bad by itself because Monday.

You decide you’ll have tea instead of coffee so you boil some water, pour it in a mug and dip two bags of your favourite tea in it.

You go to your home office room to start your laptop, check emails etc.

Then you go back to the kitchen and spend the next 5 minutes wondering where your coffee is.

Password Manager For Dummies

Normally I start every post with a small introduction. This one I want to keep as short as possible so I’ll just say this: It’s 2021. You need a password manager.

Let’s start from the very beginning. First, I’ll explain a few things you’ll hear often. A lot of these words can seem daunting but actually are quite simple. Then we get down to the nitty gritty.

I DON’T WANT TO DO THIS WHY DO I NEED TO DO THIS???!??!

Because there are some things that you 1) want to be able to do on the internet but 2) don’t want other people to be able do (at least not without you knowing).

You don’t want other people to move money from your bank account. Or buy things with your credit card. You get the idea.

But but but I already have a password!

Yes, you do. But there are some problems.

If you’re, well, human, you can remember some things but not many and not very well (read this if you don’t believe me). And it’s 2021, if you don’t live under a rock you have at the very least 10-20 accounts in different services, like your bank, your email etc etc. Try to count them and write in the comments how many you found 😊

The other problem is: criminals steal data from these services. A lot. Like, in the billions. Estee Lauder had a breach on February 2020 where 440 million records -data about people- were stolen. MGM Resorts, which you know from the casino in “Ocean’s 11”, had personal information about more than 10 million guests stolen. And these are just 2 of the around 3000 data breaches that were reported in 2020 in the US alone.

What this means is that your password will get stolen and there’s nothing you can do about it. Well, almost nothing. You can and should do 3 things:

  • Have a unique password per service. This way, when your H&M password is stolen, it cannot be used to pay from your PayPal.
  • Use random passwords. For crying out loud, do not use your phone number. You think that adding a few letters here and there makes it safe. It does not. A computer with a program you can download for free can crack your “safe” password in like an hour. The password must be long and random, something like g5D9C467YxeEfAmqL. You get the idea.
  • Use 2-factor authentication. Since this post is already long, I’ll get to this in a later one.

What does “authentication” mean? And what are these “credentials” I keep hearing about?

Credentials just means whatever you need to give to a service, like a web site, so that it checks it’s really you. Some of it is secret, some of it is not. Usually it’s a username and a password but it might be more, like your fingerprint or a code that you receive in your phone.

Authentication is just the process that checks the credentials and lets you in (or not).

What’s a password manager?

It’s a program that stores your credentials and helps you use them. Because your passwords must be long, it’s tedious to have to type them yourself. So the password manager for example can auto-fill them, or you can copy-paste them, in your e-banking web site.

Ok, ok, I’ll do it, but which one should I use?

There are many good password managers you can use like 1Password, LastPass, Devolutions, NordPass and others. Here I’ll use my favourite one which is Bitwarden, because it’s arguably the best free one and in my humble opinion the easiest to use.

Obviously this is just one way to do it; it works and it’s secure, but of course you can change things, for example use a different program. The main things to consider if you decide to use another one is:

  • It should have both a computer as well as a smartphone application.
  • It should be able to synchronize your credentials between them.
  • It should be as simple to use as possible.

And how much time will it take?

Realistically, assuming you’re an average computer and smartphone user, for 5-10 web sites you’ll need around a couple of hours from start to finish. Obviously if you have dozens it will take more -not proportionally- but it’s also worth more. If you get stuck, write me in the comments and I’ll do my best to help.

UPDATE: some friends suggested that instead of doing all your sites at once, it makes the effort more manageable to do the most important ones first -e-banking, email etc. The rest you can do when you come across them in everyday use.

Now I’ll explain how you do it in your computer and smartphone. Ready, set, go!

Password Manager For Dummies: Store your passwords

Part 1: Introduction
Part 2: Store your passwords
Part 3: Now on your phone

We’ll start from your computer because usually it’s easier to create the account there. Then we’ll continue to your smartphone. But the very first thing you need to do is grab a piece of old fashioned paper.

Step 1: Write a password and a 6 digit code.

Get a paper. Yes the traditional one!

Not necessarily a post-it, but this will do as well

Write 20 or more random numbers and letters, both lower and capital. Something like 6xTzHx41jKQ3yg48FeR9sAb. This will be your password.

You don’t need to remember this.

In the same piece of paper write 6 random numbers. DO NOT USE ANYTHING REAL OR EVEN CLOSE TO IT LIKE YOUR BIRTHDAY OR YOUR POSTCODE OR YOUR PHONE, NOT EVEN CHANGED. This will be your unlock code.

This code will be the one and only thing you need to learn by heart.

Keep this paper safe in your desk at home but NOT in your computer -don’t take a photo of it or write it in a Word file.

Step 2: Create your Bitwarden account

On your computer, go to bitwarden.com and click “Get started”.

Fill in the form, it’s really simple. Use the password you wrote on the paper.

Step 3: Install the browser extension

Still on your computer, open your favourite browser -Firefox, Chrome, Edge, Opera, whatever- go to the bitwarden extension and install it.

Here it is for Firefox

Here for Chrome

Here for Microsoft Edge (you’re not still using Internet Explorer, are you?)

And here for Opera.

In case you’re using anything else, just google “bitwarden <browser name>” and you’ll find it.

NOTE: As you’ll see, about the only annoying thing with Bitwarden is that if you click outside of it before you save your changes, it closes and loses your input. There’s a solution for this: you can click the “Pop out” button” and then it opens as a separate window. The “Pop up” button is this one:

When the extension is installed, you’ll get the Bitwarden shield icon on the top right corner of your browser. Click it and fill in your email and password.

Once you log in you see your list of passwords. This a called your “vault”. For now, it’s obviously empty.

Click “Settings”, then “Unlock with pin”. Enter the 6 numbers you wrote on the paper and uncheck the “lock with master password…” check box.

Step 4: Store your credentials

If you’ve done so far, great job! Now it’s the time to start storing your passwords, one by one.

Click the shield icon of Bitwarden, then the plus icon on the top right corner.

Start with your email. Enter the name, username and password -the ones you have already. Add also the URL you use to access the site. Then click “Save”.

One by one, add all the sites and other services you have. This will probably take some time; my list has more than 400 entries 😊

Step 5: Try it

So all of this is supposed to help you right? Here’s how it helps you login. Say you want to log in to your email for example.

Click the shield icon of Bitwarden, click “My vault” and click the little arrow of the site. You’ll see that it takes you there.

In your email site, click “Sign in” or “Login” or whatever it has. Right click in the username or password and select Bitwarden > Auto-fill > your site name. Then click Next or Login or whatever it has.

If for whatever reason right click doesn’t find the site, there’s another way that’s not as easy but works every time. From “My vault” click the head icon to copy the username, then paste it in the site, then click the key icon to copy the password, then paste it in the site.

After doing it a few times, you’ll get the hang of it; it will feel very easy very quickly.

Step 6: Change your passwords

Until now you’ve done great, but we’re still using our old passwords. Now it’s the time to make them big and hard 😉

The exact process differs slightly for every site, obviously, but not much. In this example, I’ll use a popular e-shop, Zara UK.

Go to your profile and go to change password:

In the bitwarden “My vault” click the key icon of the site (see above) to copy the existing password. Paste it in the “Current password” box of the web site.

Then go in the bitwarden “My vault” again and click somewhere in the middle of the site name. This will open the entry. Click Edit on the top right corner.

Click the double arrow next to the password and click “yes” in the “overwrite password” question. Slide the length of the password to something over 17, click “regenerate” and then “select”.

Click “Save” to save the new password.

Now go to “My vault” again, click the key icon to copy the new password, go to the web site and paste it twice. Then click “Update password” or whatever button is there.

The first time you do it will be cumbersome, but after the first 2-3 sites, it will feel really easy.

If you’ve reached this far, congratulations 🥳🎉👏 You’ve done the hard work! The last thing to do is install the app on your smartphone so you can use it there too. Let’s go!

Password Manager for Dummies: Now on your phone

Part 1: Introduction
Part 2: Store your passwords
Part 3: Now on your phone

Here we get to the fun part -well, if not fun, certainly the easiest and most useful. I’ll give screenshots for iPhone, because that’s what I have, but for Android it’s almost the same.

Step 1: Install the Bitwarden App

Go to your App Store (or Play Store for Android), find Bitwarden and install it.

Step 2: Login

Open the app, click Log In and fill in the email and password (the one you wrote on the paper).

Go to Settings and press “Unlock with PIN code”. Enter the 6 digit number you wrote on the paper and select “No”.

We’re ready to use it!

Step 3: Use it to login to sites

Let’s try to use the browser in our smartphone to login to Zara UK. Navigate to the web site and click Login, or My Account or whatever it has:

Now switch to Bitwarden (you might need to unlock it with your 6 digit code), find the site, press the 3 dots and click Copy Username.

Switch to the browser, tap in the username box and paste the username.

Repeat the same steps for the password and click Log In.

Ta da! We’re in!

That’s all folks

This was what you have to do to get started and work with Bitwarden. It’s not an exhaustive guide, mind you, there are more to it. But it covers the most important part: securely creating, storing and using unique passwords that are impossible to guess.

I hope this works for you. If you have any questions or suggestions, I’ll be more than happy to discuss in the comments!

Have fun 😊

Please don’t write logs inside Program Files (here’s how to do it right)

So the other day I’m troubleshooting a Windows Service that keeps failing on a server, part of a product we’re using in the company. Long story short, that’s what the problem was:

Access to the path 'C:\Program Files\whatever\whatever.log is denied'

I mean, dear programmer, look. You want to write your application’s logs as simple text files. I get it. Text files are simple, reliable (if the file system doesn’t work, you have bigger problems than logging) and they’re shown in virtually every coding tutorial in every programming language. Depending on the case, there might be better ways to do that such as syslog, eventlog and others.

But sure, let’s go with text files. Take the following example somewhere in the middle of a Python tutorial. Look at line 3:

import logging

logging.basicConfig(filename='app.log', filemode='w', format='%(name)s - %(levelname)s - %(message)s')
logging.warning('This will get logged to a file')

Did you notice? This code writes the log in the same place as the binary. It’s not explicitly mentioned and usually you wouldn’t give it a second thought, right?

To be clear, I don’t want to be hard on the writers of this or any other tutorial; it’s just a basic tutorial, and as such it should highlight the core concept. A professional developer writing an enterprise product should know a bit more!

But the thing is, these examples are everywhere. Take another Java tutorial and look at line 16:

package com.javacodegeeks.snippets.core;

import java.util.logging.Logger;
import java.util.logging.FileHandler;
import java.util.logging.SimpleFormatter;
import java.io.IOException;

public class SequencedLogFile {

    public static final int FILE_SIZE = 1024;
    public static void main(String[] args) {

        Logger logger = Logger.getLogger(SequencedLogFile.class.getName());
        try {
            // Create an instance of FileHandler with 5 logging files sequences.
            FileHandler handler = new FileHandler("sample.log", FILE_SIZE, 5, true);
            handler.setFormatter(new SimpleFormatter());
            logger.addHandler(handler);
            logger.setUseParentHandlers(false);
        } catch (IOException e) {
            logger.warning("Failed to initialize logger handler.");
        }
        logger.info("Logging info message.");
        logger.warning("Logging warn message.");
    }
}

Or this Dot Net tutorial, which explains how to set up Log4Net (which is great!) and gives this configuration example. Let’s see if you can spot this one. Which line is the problem?

<log4net>
  <root>
    <level value="ALL" />
    <appender-ref ref="LogFileAppender" />
  </root>
  <appender name="LogFileAppender" type="log4net.Appender.RollingFileAppender">
    <file value="proper.log" />
    <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />
    <appendToFile value="true" />
    <rollingStyle value="Size" />
    <maxSizeRollBackups value="2" />
    <maximumFileSize value="1MB" />
    <staticLogFileName value="true" />
    <layout type="log4net.Layout.PatternLayout">
      <conversionPattern value="%d [%t] %-5p %c %m%n" />
    </layout>
  </appender>
</log4net>

If you answered “7”, congrats, you’re starting to get it. Not using a path -this should be obvious, I know, but it’s easy to forget nevertheless- means writing in the current path, which by default is wherever the binary is.

So this works fine while you’re developing. It works fine when you do your unit tests. It probably works when your testers do the user acceptance testing or whatever QA process you have.

But when your customers install the software, the exe usually goes to C:\Program Files (that’s in Windows; in Linux there are different possibilities as explained here, but let’s say /usr/bin). Normal users do not have permission to write there; an administrator can grant this, but they really really really shouldn’t. You’re not supposed to tamper with the executables! Unless you’re doing some maintenance or an upgrade of course.

So how do you do this correctly?

First of all, it’s a good idea to not reinvent the wheel. There are many, many, MANY libraries to choose from, some of them very mature, like log4net for Dot Net or log4j for Java.

But if you want to keep it very simple, fine. There are basically two ways to do it.

If it’s a UI-based software, that your users will use interactively:

Create a directory under %localappdata% (by default C:\Users\SOMEUSER\AppData\Local) with the brand name of your company and/or product, and write in there.

You can get the localappdata path using the following line in Dot Net:

string localAppDataPath = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData);

Take for example the screen-capturing software called Greenshot. These guys do it right:

If it’s a non-interactive software, like a Windows Service:

You can do the same as above, but instead of Environment.SpecialFolder.LocalApplicationData use Environment.SpecialFolder.CommonApplicationData, which by default is C:\ProgramData. So your logs will be in C:\ProgramData\MyAmazingCompany\myamazingproduct.log.

Or -not recommended, but not as horrible as writing in Program Files- you can create something custom like C:\MyAmazingCompany\logs. I’ll be honest with you, it’s ugly, but it works.

But in any case, be careful to consider your environment. Is your software supposed to run on Windows, Linux, Mac, everything? A good place to start is here, for Dot Net, but the concept is the same in every language.

And, also important, make your logging configurable! The location should be changeable via a config file. Different systems have different requirements. Someone will need the logs somewhere special for their own reasons.

But whatever you do, PLEASE PLEASE PLEASE DON’T WRITE WHERE THE BINARY IS. DON’T WRITE IN C:\PROGRAM FILES. IT. DOES. NOT. WORK.

New version of Zoro: 1.0.2

I just published a new version of my open source C# Zoro(*) library in Github and Nuget.org.

Zoro is a data masking/anonymization utility. It fetches data from a database or a CSV file, and creates a CSV file with masked data.

The new version, 1.0.2, has been converted to DotNet Standard 2.0. The command line utility and the test project have been converted to Dotnet Core 5.0.

There is a known issue, not with the code but with the Nuget package. The description claims, as was intended, that the package contains not only the library but also the exe, which can be used as a standalone command line utility. But due to some wrong path in the Github Action, it doesn’t.

I’ll try to get that fixed in the next weeks. Until then, if you need the exe, please checkout the code and build with Visual Studio or Visual Studio Code.

(*) YES NOW I KNOW THIS IS MISSPELLED AND THE CORRECT SPELLING IS ZORRO, I DIDN’T WHEN I STARTED THE LIBRARY, SORRY!

This is the great ideological battle of our time

It used to be capitalism vs. communism, free markets vs. central planning

When people talk about ideologies and political beliefs, the mind usually goes to the left/right political divide, commonly referred to as the “political spectrum“. On one side -and I’m obviously oversimplifying here for the sake of brevity- are the proponents of ideas like central planning, social welfare, big government and redistributive taxation. Names like Marx, Lenin, Rosa Luxemburg, Olof Palme and many others come to mind. On the other side one can find ideas like free markets (and their invisible hand), the law of supply and demand, laissez-faire, entrepreneurship and, generally speaking, as small a government as possible. People mention names like Adam Smith, Keynes, Hayek, Milton Friedman and at the extreme end, Ayn Rand.

In addition to the above, if you follow a more holistic school of thought like e.g. the political compass, you would note that there’s more to politics than the way the state handles people’s money. The social behaviours that are or are not allowed are at least as important, if not more. On one hand you have progressive, permissive societies, like Denmark or the Netherlands, and on the other conservative, authoritarian ones like China or Russia.

So clearly both the economic and the societal dimensions of politics are important. People have not only hotly debated the differences between different forms of government, they have literally given their lives for it.

Today, it’s… a bit different.

It’s not that these differences do not exist, or have been ironed out. There are still people that believe that “workers should own the means of production” (a Marxist thesis), others that “a man’s ego is the fountainhead of human progress” (an Ayn Rand thesis) and others of course everything in between.

But everyone that takes even a casual look at talk shows, or the social networks, actually any medium that hosts public discourse will note that the lines are increasingly being drawn in a different way: those that base their opinions on reason -or at least try- and those that advance or fall prey to conspiracy theories and other instances of unfounded beliefs.

I’ve first come to this realization some months ago, when I noted that, during discussions in a social network, I kept agreeing with a left-leaning doctor while being more of a free-market person myself (though I’m nowhere near to being an Ayn Rand fan). At the same time, the doctor kept disagreeing with people that belonged to the same party -or at least were clearly left-leaning. Why? Both of us were defending government policies that, no matter how much you liked or hated the current Greek government, were based on the medical research that was available at the time. The “others” were bashing the actions of the government… because they were followers of a different political party. Whether the actions of the government were right or wrong, it simply didn’t matter. For them, the opposing party will always be wrong.

I’ve since tried to follow this a bit more closely, and by now it’s clear to me. You see distinguished members of the US Republican Party distance themselves or even denounce Donald Trump. You have virologists that present solid, peer reviewed evidence on COVID-19, and others that rely on unscientific gobbledygook, even flat-out schoolboy math errors, just so they can advance that “COVID is mostly harmless” or “it’s no worse than the flu” (which it bloody isn’t).

Are there always two sides of the story?

Look, let’s be honest here: the last man described as “universalist”, an all-knowing polymath, was Henri Poincaré, and he died more than 100 years ago. Since then, all of us have to please our trust somewhere. We have to believe that someone, in certain areas, tells the truth -and build up on that.

But reducing the matter to just “I believe A, you believe B, why is your belief better than mine?” misses a very big point. At the very minimum, there are beliefs that are validated by reality and ones that are not. To take a simple example: many of us regularly use GPS to help us drive through our city. If you believe that earth is flat, it’s… a little strange to be able use the signal from satellites, isn’t it?

Seems legit

So the conclusion I’ve drawn from all this is the following: sometimes I find myself agreeing with people that normally don’t share my political opinions. Conversely, I might disagree with people that do. That’s fine. At the end, we share the most important political thesis of all: there exists an objective reality and the best way to discover it is to use the scientific method. And yes, reality imposes constraints on us, often unpleasant, and until we find a way to work around them, we need to accept them.

And no amount of YouTube videos can change that.