Mattia Natali's Blog

Six things I wish I knew before becoming an Engineering Manager

Mon, 08 Apr 2024 08:50:08 GMT

I have always worked as an individual contributor in a team. I see a continuous pattern: I start as an Engineer in a company, but sooner or later I end up leading the team, taking tough decisions that nobody wants to take, assuming accountability for the good and bad, and people outside the team see me as a go-to-person.

I start thinking: maybe I can contribute more if I manage people and not just code. Instead of just starting from scratch if I change jobs, why don’t I start directly as an Engineering Manager?

So here I am: I’ve been promoted as an Engineering Manager. Congratulations!

I have been an Engineering Manager for a while now, and I want to share something I wish I had known earlier.

Having extremely good technical skills and business acumen is not enough anymore

I think my biggest skill is to learn quickly: if you are good at it, you can solve any technical problems. Engineers are very lucky for this: there’s an abundance of documentation on the Internet. You can literally learn everything without spending any penny, you just want to learn: you don’t need to beg anyone to get information. If you are working in good companies, also the internal processes and knowledge are open.

So you can grow without relying too much on people. At least until Senior Software Engineer, then you have to sharpen your influential skill to become a Staff Software Engineer.

But if you are on the Manager path, your team is your most valuable asset for growth. Then you should “expand your influence” as soon as possible.

I learned it the hard way during the performance check-in. I spent a whole year keeping my team engaged and keeping up with deliverables even though we were understaffed. This means I also contributed as an Individual Contributor.

I was happy about my performance: I am an Engineering Manager that can be an Individual Contributor when the company needs it. I’m exceeding expectations, right?

But my hard work as an individual contributor was almost worthless during the performance check-in. And it definitely did not help me in my career path.

I was quite upset the next day, but I learned not to bother too much about something that I can’t change (the company game rules), it’s always better to focus on things I can change (my behaviors).

So don’t be like me: if you are a Manager, you need a team, and possibly more than one, and the more people who know you, the better it is.

You have to give up the most joyful thing in the job

If you were previously a Software Engineer for several years and you have survived, it likely means you love coding. Coding is hard and also stressful, especially when you have tight deadlines.

However, the dopamine you receive after finishing a task, solving a complex problem, or seeing your work used by customers is priceless! I love it, and I also love building stuff. The more experience you gain as a Software Engineer, the less you get stuck, so you also feel you can achieve anything. This boosts the self-esteem of the Engineer a lot.

Such joy and happiness are harder to find if you are a Manager.

Every task in the backlog should be done by your reports. And your reports are humans; humans are lazy by nature. If you are lucky, you find engineers who combat laziness by automating stuff and still doing the hard stuff because they enjoy it. But you usually have engineers who have any sort of hiccups: they spend time on something not required in the task, they have personal issues, they are not convinced about the task itself, they do not follow what the team agreed, etc.

So you need to deliver through people: you always have this level of indirection that makes things more complicated. And this is stressful; you won’t have the same dopamine level as you get as an individual contributor, for sure.

Now I’m trying to rewire my brain to find joy in something I paid little attention to before:

I’m happy when I have a meeting with some agreed action points.
I’m happy when I see the sprint working as expected and my reports finding solutions autonomously.
I’m happy when I have 1:1s and they are happy with how the team is working, despite the environment around us not being that good.
I’m happy when I see a graduate/reporter growing and taking me as a mentor, recognizing my effort.
I’m happy when I see different teams/clients having the same context and working together to build something very big.

As you can see, there are a lot of opportunities for happiness here. But you always need to engage people, and sometimes the outcome is not tangible. So you need to practice some “mindful” techniques to appreciate.

Sometimes I really need to speak to myself: “Hey! You have tons of things. I know you’re upset you spent 1 hour explaining a trivial thing to someone else. But now the other guy knows more, and he will be quicker next time. Enjoy the compound interest in the upcoming days.” This is what I’m doing to enjoy my managerial path. The more I’m on it, the less I need to speak to myself to feel the joy. As I said, this is needed to rewire your brain to enjoy your new job.

I’m sure there are people who naturally feel it. But if you don’t, just give “mindful” techniques a try.

To succeed, you need people

It’s very tempting to do the task by yourself, it’s certainly easier doing that: talking to someone, explain what to do, convince him it’s the right guy to do it, commit him doing that in a timely fashion takes a lot of time… So… Why do I need to take that path instead of doing by myself? Because it’s not scalable, you can’t do everything by yourself. The quicker you understand it, the lesser pain you get. I learned the hard way: I was complaining with my product owner about I can’t take everything by myself, things started to slip out of my hands, he interrupted me abruptly and he said: “you have a whole team at your disposal, make them work!“.

That was my a-ha moment! I was too focused on myself and I wasn’t exploiting my team as I should. From that day, I decided to spend more effort on others than myself. Was it easy? No, because the temptation of returning to the old bad habit was high… But I knew it couldn’t work in the old way.

In the end, I was able to manage my team effectively at its full potential. Then, we had an understaffing problem, but at least no one complained about me underutilizing my resources 😅.

There is no best technique to manage people

Managing people to craft software is completely different from crafting by yourself: writing software requires expertise, and this experience you gain is ubiquitous. If you write good TypeScript code, you will likely be able to write good code even in other languages. Best practices and debugging techniques are still the same and will remain the same over the years.

But people are a completely different beast; every person is unique. What works with one of your reports can completely put off another one. Mentoring a junior is different from mentoring a senior.

If you are working in an international environment, you’ll quickly discover that your managerial skills should adapt to different cultures.

For example, Nordic people don’t like to work if one of the team members is not fully convinced. People from Mediterranean areas like to be in a friendly environment and are more prone to take action based on friendship rather than the manager’s authority. They enjoy chit-chat. On the other hand, people from Germany hate chit-chat. Europeans are more prone to express when something is wrong, while Americans tend to say that everything is good even when it’s not. Indian people often say they understand what you said, even if they don’t.

There is a ton of information out there to learn about these multicultural dynamics. It gives you a good starting point, but then you face the reality where you quickly understand that it won’t work because every person is different and can differ completely from what you learned.

So what’s the solution? Just stay silent.

Listen to your reports in 1:1s. You can learn how they see their jobs, how they think. Watch them work, observe when and how they get angry, when they are happy. Understand what they care about and what they don’t. Then you can start to talk, assign them tasks, and involve them in the project based on what you have learned.

This is the best way to get things done with them. I’ll try to write other blog posts on this topic because I know there’s much more to say here.

You need to be empathetic to be a good manager

Being empathetic means “showing an ability to understand and share the feelings of another”, and I believe this is mandatory to be a good manager.

If some of my ex-colleagues are reading what I’ve just written, they are probably laughing at me: “C’mon, you are the least empathetic person I know!“. I’m sure this is what they are thinking. And they are right! But I don’t want to deny it, I’m fully aware that my worst managerial mistakes have all had the same root: lack of empathy.

I will write about the mistakes I made so you don’t have to make them in order to learn.

What can you do if you are an emotionless ice block like myself? You can still learn to be empathetic. Maybe you won’t feel the same emotions just when your interlocutor talks. But you can ask yourself, “What feelings did your report experience when they talked to you? Why is the person feeling that way? If I do X, can they feel better?“. If you are not an empathetic person, it is difficult to express empathy and derive empathetic action points directly from the conversation, but you don’t have to make them on the spot! You can just digest offline what you heard and then come back later.

It’s your responsibility to have difficult conversations as a manager

It is your duty to have difficult conversations that nobody wants to have. These conversations may involve addressing performance issues, providing constructive feedback, or resolving conflicts within the team.

While these conversations can be uncomfortable, they are crucial for the growth and development of both individuals and the team as a whole. By addressing these issues directly and honestly, you can create a more productive and harmonious work environment, fostering personal and professional growth for everyone involved.

Unfortunately, you can’t delegate those to someone else. So, if you are not comfortable with having difficult conversations, you need to work on improving your skills in this area.

Conclusion

I hope this post has given you some insights into the challenges and rewards of transitioning from an individual contributor to a manager. While the transition can be challenging, it can also be incredibly rewarding. By focusing on building strong relationships with your team, embracing new ways of working, and developing your leadership skills, you can become an effective and successful manager.

If you are considering making the switch, I encourage you to take the time to reflect on your strengths and weaknesses, and consider how you can leverage your skills and experience to excel in your new role.

Remember, it’s okay to make mistakes along the way, as long as you are willing to learn from them and grow as a leader.

See you in the next post!

How to store secrets in public repositories

Sun, 27 Nov 2022 08:04:12 GMT

We all feel this pain sooner or later: you are working hard on your git repository. For whatever reason, you change the PC but you still want to work on your project. So you download the repo using git clone. You’re ready to go, but then you’re stuck because you don’t remember the secrets that you put in your .gitignore file.

Luckily now there’s a solution for that. And it’s called git-crypt.

Git-crypt will encrypt your secrets before pushing your changes on your remote branch, and it decrypts them when you download from it. So after you set it up, everything is completely transparent. You will forget to have secrets in your repo!

Bootstrap

To start with it, you need to install it. On Mac you can run the following command in the terminal after you installed the package manager brew

brew install git-crypt

Then, you need to go to your public repository where you want to put secrets. And initialize git-crypt

cd <YOUR_REPO_PATH>
git-crypt init

Add secrets

Now you need to add the .gitattribute file in the repository’s root folder. In this file, you declare the files and folders you want to keep secret. This is an example of .gitattribute file

# Make the file .env secret
.env filter=git-crypt diff=git-crypt

# Make all files that ends with .key secret
*.key filter=git-crypt diff=git-crypt

# Make all the files contained in secretdir folder secret
secretdir/** filter=git-crypt diff=git-crypt

After you created the .gitattribute file, you can put the secrets in the place you have just declared. If you put the files before the .gitattribute, your secrets won’t be encrypted! Before committing anything, you can have a confirmation that files will be encrypted using git-crypt status command

# Show whether files are encrypted or not
git-crypt status

# Show encrypted files only
git-crypt status -e

# Show unencrypted files only
git-crypt status -u

If a file was previously there, it won’t be encrypted. You need to run the command git-crypt status -f. I repeat myself: if you’re using this command, it means the file was already in your repo in cleartext. So you need to rotate the secrets that are in that files. Otherwise, it’s very easy for an attacker to see the secrets just by looking at previous commits.

Now you can commit and push. You’ll see only some encrypted binary files in your public repository! You’re able to see them locally.

Moving the symmetric key to another device

So good so far… But if you clone the repository elsewhere, you will download the encrypted files. So how can we decrypt the files again? To answer this question we need to reveal some magic. git-crypt decrypt files using a symmetric key.

During the initialization with git-crypt init, it created the symmetric key and he put it in <YOUR_REPO_PATH>/.git-crypt/keys/default. You can easily export it with this command

git-crypt export-key /path/to/keyfile

with this key you are able to decrypt also on your other devices. So keep it secure. To decrypt a repository protected with git-crypt you can use the command

git-crypt unlock /path/to/keyfile

Using GPG keys

I know what you’re thinking: “Come on… I have 30+ repositories. I can’t store and move the keys for every repo! There should be an easier way to handle them.”. There is, indeed!

You can use your GPG key to unlock your repository. If you don’t have one, you can follow the Github guide to create the GPG key.

All you need to do is to allow your GPG key to decrypt the repository

# From now on, you're able to unlock using <YOUR_GPG_KEY>
git-crypt add-gpg-user <YOUR_GPG_KEY_ID>

To find your GPG key, you can use the following command. I’m assuming you have already installed GnuPG

gpg --list-secret-keys --keyid-format=long

Then, if you clone your repo elsewhere, you can unlock it with

git-crypt unlock

git-crypt will be smart enough to look into your machine for the GPG key. If it finds it, it will decrypt everything!

How the GPG mode works

… But how does it work? I assume you don’t like magic.

When you write

git-crypt add-gpg-user <YOUR_GPG_KEY_ID>

git-crypt will encrypt the symmetric key using your public GPG key, and it will commit into your repo in .git-crypt/keys/default/0 folder.

Then, when you write

git-crypt unlock

git-crypt will find your GPG key in your Mac, it will decrypt the symmetric key that it saved in .git-crypt/keys/default/0 using your private GPG key. Finally, it will decrypt the secrets using the symmetric key just decrypted.

I hope I helped you to find a good way to manage the secrets in your repositories. I’m extensively using it both in my hobby projects and at work. I’m so happy with it that I wanted to share it with you.

See you!

How to build an Air Quality Sensor using a Raspberry Pi

Sun, 17 Jan 2021 16:40:20 GMT

I live in Milan and I was wondering how much the air is polluted. So I started learning how to build an air quality sensor and how to integrate it in my Home Assistant. Then I regretted it because I understood the Milan air quality is very awful! It’s not breaking news, but it’s different having your air quality sensor that analyzes the air you’re breathing instead of hearing the news on TV that says: “Milan had stopped the traffic to lower the emissions because it’s above the limits”. Then I moved the sensor to Serina, a little town lost in the mountains, where the air quality is much better!

Now I want to share my learning: how to create the same air quality sensor I built, and how to read this data into Home Assistant just like this

The hardware we need

To reach this goal we need:

2 Raspberry Pis and related power adapter and SD Cards: one will be the air quality sensor (Rpi Sensor), the other one will be our receiver with Home assistant (Rpi Hub). You also need a WiFi USB dongle if the air quality Raspberry Pi doesn’t have built-in.
The air quality sensor SDS011: this hardware is commonly shipped with the USB dongle, so it’s easy to attach to the Raspberry Pi.

The software we use

Mosquitto: open source message broker that implements MQTT protocol. MQTT protocol is widely used in IoT system because it performs very well under extreme conditions like weak WiFi signal, and very limited WiFi bandwidth.
Home Assistant: open-source home automation system.

I’m currently using a Mac, so all the commands you see will work on Mac or Linux terminal. If you are using Windows you have two options:

Install WSL.
Use a virtual machine with Linux installed using VirtualBox.

What we learn

Someone can say: “Hey! You are a liar! You’re not building an air quality sensor, you’ve just bought that!“. Maybe he is right, but I think the sensor is just the tip of the iceberg. We have a lot of work to extract the data from the sensor and make them useful.

The overall architecture we want to build is shown in the next figure

So through this tutorial, we will learn:

How to create a program that extracts data from the sensor. We will write it in Go. To simplify the development we will learn how to cross-compile the binary for ARM architecture using our PC/Mac that has a different architecture than the Raspberry Pi.
How to automatically connect the Raspberry Pi with the sensor to the WiFi with the strongest signal, how to automatically reconnect if we have some outage.
How to start automatically the Go program if there are some issues.
How to pass data using MQTT protocol and how to digest those inside Home Assistant.

Now that we have the overall picture in our minds, let’s get started!

Build the Raspberry Pi with the air quality sensor

Let’s start with the Raspberry Pi with the air quality sensor. The first thing we need is install the Raspberry Pi OS Lite. We just need the lite because we just use the terminal and we will connect using SSH.

Now we are ready to attach the SD card to the Raspberry, attach the ethernet port to connect it to our home network for the first setup, attach the SDS011 sensor using the USB. The Raspberry Pi should automatically detect it.

Let’s check it out: connect to this Raspberry Pi using via SSH using ssh pi@<RPI_SENSOR_IP>, with password raspberry. Change <RPI_SENSOR_IP> with the Raspberry Pi sensor IP address. I usually find its IP address through my router control panel, or just type on the terminal ping raspberrypi.local. You should find something like this

PING raspberrypi.local (192.168.1.123): 56 data bytes
64 bytes from 192.168.1.123: icmp_seq=0 ttl=64 time=13.452 ms
64 bytes from 192.168.1.123: icmp_seq=1 ttl=64 time=17.451 ms
--- raspberrypi.local ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 13.452/15.451/17.451/2.000 ms

The IP you see there, it’s the IP you’re looking for.

Now that we are connected via SSH, we type lsusb to check the Raspberry Pi sees the sensor

pi@edge-rpi:~ $ lsusb
Bus 001 Device 005: ID 1a86:7523 QinHeng Electronics HL-340 USB-Serial adapter
Bus 001 Device 004: ID 2001:330d D-Link Corp.
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp. SMC9512/9514 USB Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

You should see some Chinese names like “QinHeng Electronics HL-340”. This is our sensor. If we type ls /dev/ttyUSB* we should see /dev/ttyUSB0 as response: this is our channel to communicate with the sensor.

Compile the Go program to fetch data

Now we should create a program that fetches data from /dev/ttyUSB0 created by the sensor and publishes those on an MQTT topic.

If you don’t know MQTT, don’t worry: Wikipedia explains briefly what it is. This is my explanation: it’s a lightweight protocol that relies on topics. Someone publish something on a topic, like the air quality sensor data. Someone else can subscribe to that topic, in that way he’s notified whenever there’s new data on that topic. Maybe you are more comfortable with the HTTP protocol, but I think that MQTT is best suited for this kind of application.

I made this application for you in Go. You can find it on GitHub. We should change the configuration that is in cmd/sensor/main.go, the file should be like this

package main

import "mattianatali.it/sds011-mqtt/internal/sensor"

func main() {
  c := sensor.Config{
    Topic: "home/serina/edge-rpi/air-quality",
    SensorPortPath: "/dev/ttyUSB0",
    CycleMinutes: 1,
    MqttBroker: "tcp://192.168.1.117:1883",
  }
  sensor.Start(c)
}

Change the Topic with a string of your choice. I had his convention home/<HOME_IDENTIFIER>/<DEVICE_IDENTIFIED>/<DATA>. The topic is just a string that refers to where we want to publish the data and then where we want to subscribe to fetch them.
SensorPortPath: should be the same, it’s where we fetch the sensor data.
CycleMinutes: Change it if 1 sample/minute is too much for you. You can’t put less than 1 minute.
MqttBroker: You should put the IP address of the Raspberry Pi where we put Mosquitto, the message broker that handles our MQTT topics. I suggest you use the same Raspberry Pi where you install also Home Assistant.

If you are curious, you can see the implementation in internal/sensor/main.go.

Now it’s time to cross-compile it for our Raspberry Pi! I’m expecting that you have downloaded this source code on your Mac/PC and you have Go installed. Then, we need to know what type of architecture your Raspberry Pi has.

Launch this command on the Raspberry Pi: cat /proc/cpuinfo You should see something like this

cat /proc/cpuinfo

processor : 0
model name : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS : 697.95
Features : half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xb76
CPU revision : 7

Hardware : BCM2835
Revision : 0002
Serial : 00000000cdea9a84
Model : Raspberry Pi Model B Rev 1

My raspberry has ARMv6, so I need to compile for that type. If you have a Raspberry Pi 2 you should see ARMv7. I created a Makefile to simplify the cross-compilation. Makefile is very common on Linux, it’s used to simplify common tasks. If you don’t know about it I suggest reading Learn make in Y minutes.

If you have ARMv6, just type make build-arm6 in the folder where you downloaded my GitHub Project.
If you have ARMv7, type make build-arm.

Now we should have a binary file located in dist/pm-sensor. We should transfer to our Raspberry Pi sensor. We will use the command scp.

scp dist/pm-sensor pi@<RPI_SENSOR_IP>:

As usual, change <RPI_SENSOR_IP> with the real IP, but leave the colon at the end. This means we’re going to place the pm-sensor in the home folder of our user pi inside the Raspberry Pi.

Now we can enter via SSH in the Raspberry Pi, we should see our new binary file called pm-sensor. If we try to run, it fails

# Add execute permission to the script
sudo chmod +x ./pm-sensor
# Start it
./pm-sensor
# This is the program output
panic: Network Error : dial tcp 192.168.1.118:1883: connect: connection refused

goroutine 1 [running]:
mattianatali.it/sds011-mqtt/internal/sensor.Start(0x12d3bcd, 0x20, 0x12cc844, 0xc, 0x1, 0x12d0ec0, 0x18)
 /Users/mattia/Repositories/sds011-mqtt/internal/sensor/main.go:28 +0x7e5
main.main()
 /Users/mattia/Repositories/sds011-mqtt/cmd/sensor/main.go:12 +0xac

it’s complaining because it can’t connect to the MQTT broker to publish the sensor data. It is right, we don’t have prepared the other Raspberry Pi.

Now we leave the sensor for a moment, and we’ll work on the Raspberry Pi that is inside our home, we call it Hub for simplicity.

Add mosquitto to the Raspberry Pi Hub

Now it’s time to set up the “receiver” part of the architecture. So we need the other Raspberry Pi with a clean Raspberry OS installed.

Then we install Mosquitto

wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-buster.list
apt-get update
sudo apt-get install mosquitto

The new version of Mosquitto added the authentication by default for very good reasons. I started this project when Mosquitto didn’t have authentication by default, so my script doesn’t support it, yet. To enable support for anonymous users we should add these two lines in the file /etc/mosquitto/mosquitto.conf

listener 1883
allow_anonymous true

The first one enables the incoming messages from different hosts to the port 1883. This is required because our sensor is not on this Rpi Hub. The second one enables anonymous users.

We make sure that Mosquitto is working, it takes the new config file, and it’s starting on Raspberry boot.

# Restart mosquitto to get the new config file
sudo systemctl restart mosquitto

# Make sure mosquitto is enabled on startup
sudo systemctl enable mosquitto

Now we have the MQTT Broker up and running! It means that our sensor can send the data to this broker. Let’s test if our sensor can send data to this broker.

# Connect to Raspberry Pi sensor and type the password
ssh pi@<RPI_SENSOR_IP>
# Start the Go program to send air quality data
./pm-sensor
# This is the expected output (1 line every one minute)
2021-01-06T05:58:15Z,8.4,21
2021-01-06T05:59:15Z,10.5,60.9
2021-01-06T06:00:15Z,8.5,48.9
2021-01-06T06:01:15Z,12.3,99.5
2021-01-06T06:02:15Z,20.1,214.3
2021-01-06T06:03:14Z,17.6,183.2
2021-01-06T06:04:14Z,16.5,173.7
2021-01-06T06:05:14Z,10.4,52.2

This is a really good sign! It means we can fetch data from our quality sensor and we can send them to our MQTT broker that it’s installed on our Raspberry Pi Hub!

To double-check it, we leave the program active and we subscribe on the topic we are sending data to

# We connect to Raspberry Pi Hub
ssh pi@<RPI_HUB_IP>
# We subscribe to our topic
mosquitto_sub -t 'home/serina/edge-rpi/air-quality'
# We should see an output like this
{"PM25":14.8,"PM10":18.6,"Timestamp":"2021-01-06T13:42:08.33019206Z"}
{"PM25":14.5,"PM10":16.2,"Timestamp":"2021-01-06T13:43:08.192798603Z"}
{"PM25":8.7,"PM10":9.7,"Timestamp":"2021-01-06T13:44:08.057092356Z"}

You should change <RPI_HUB_IP> with the IP of your Hub and replace home/serina/edge-rpi/air-quality with the topic you have inserted in the Go configuration inside the property Topic.

Now we know what our Go program publish on the topic: it sends a JSON file with three properties:

PM25: contains the PM 2.5 expressed in µg/m^3.
PM10: contains the PM 10 expressed in µg/m^3.
Timestamp: contains the time in ISO 8601 when it fetched the data.

Recap

Congratulation! Now we have a sensor that fetches air quality data and sends them to an MQTT broker, every application that wants this data should subscribe to the chosen topic and that’s it! Are we finished?

Well… Not exactly.

We want to put the sensor outside, so we should use the WiFi and detach the Ethernet port. We don’t want to connect to the sensor to start the program. If we put the sensor far away from our router, the sensor should be able to reconnect every time automatically, it should do the same if we restart the router. We have all these issues to fix!

So let’s pimp up our sensor to make it wireless and smart enough to keep the connection on automatically.

Pimp the sensor!

Make the program start automatically

The program doesn’t start automatically. We have to connect via SSH and start it with the command ./pm-sensor. This is very frustrating.

To reach this goal, we take leverage of systemd. Systemd is the system and service manager for the Raspberry Pi. Did you notice we have already used systemd in this tutorial? When we started mosquitto we wrote

sudo systemctl restart mosquitto
sudo systemctl enable mosquitto

using the command systemctl we can start/stop/restart the services on a Linux machine. And we can also start automatically a service on boot if we write sudo systemctl enable <SERVICE>. So we need to create a service for our pm-sensor program.

Luckily it’s very easy: the custom services are stored in the folder /etc/systemd/system. So we can add the new service, if we add the file called pm-sensor.service in that folder. The content of that file is the following

[Unit]
Description=PM Air Quality Sensor
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=pi
Group=pi
ExecStart=/home/pi/pm-sensor
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

As you can see, the service declaration consists of three sections: [Unit], [Service] and [Install].

In the [Unit] section we put a little description of our service and we declare that we want the network up and running before starting it.
[Service] declare the type of the service, the name, and the group of the user that should start the script, where the program is located and we define how systemd should behave if the service panics. We want the program to always restart.
[Install] defines that this service does not need the graphical user interface.

Systemd is a very big topic and it deserves a complete book on it. I know this explanation scratches only the surface, but it’s enough for our purposes. But I know you’re very curious! So if you want to dig deeper you can type man systemd.unit and man systemd.service on your Linux machine. In linux, if you want to know something more just type man <PROGRAM_YOU_WANT_TO_LEARN_MORE>.

Now that we have created the service, we need to reload the systemd configuration, start the program, and enable it during startup.

sudo systemctl daemon-reload
sudo systemctl start pm-sensor
sudo systemctl enable pm-sensor

We can check if everything is started correctly with the following command

sudo systemctl status pm-sensor
# This is the output we got
● pm-sensor.service - PM Air Quality Sensor
 Loaded: loaded (/etc/systemd/system/pm-sensor.service; enabled; vendor preset: enabled)
 Active: active (running) since Wed 2021-01-06 15:01:26 GMT; 1h 43min ago
 Main PID: 1182 (pm-sensor)
 Tasks: 5 (limit: 272)
 CGroup: /system.slice/pm-sensor.service
 └─1182 /home/pi/pm-sensor

Jan 06 16:36:01 edge-rpi pm-sensor[1182]: 2021-01-06T16:36:01Z,21.2,33.9
Jan 06 16:37:00 edge-rpi pm-sensor[1182]: 2021-01-06T16:37:00Z,61.1,303.4

Everything is ok, we should also see the logs of our program.

We don’t need to start manually the program anymore! But we are still connected via Ethernet, we can’t put our sensor outside our home. Let’s fix it!

Add WiFi connection

The best way to add the WiFi in the Raspberry Sensor is by reading the official documentation, it’s useless to copy-paste. I suggest adding all your WiFi SSID that you have in your home. In this way, you set up the sensor just one time. If you want to move it to another place, it tries to connect to the nearest WiFi access point.

Be aware that now the sensor IP address could be changed. So remember to get the new address.

Keep WiFi persistent

Now we can put our sensor outside! It automatically connects to our WiFi and starts our program that fetches the sensor. We should be proud of our work until now… But we realize soon that there’s a problem. We realize it when we reboot our router or we put our sensor too far from it: the Raspberry Pi doesn’t try to reconnect the WiFi if it loses the connection!

Let’s fix it. I found a little script on the web and I adapted it for our purposes. We create the file check-wifi.sh

# Connect to sensor Raspberry Pi
ssh pi@<RPI_SENSOR_IP>

# Create the new file
touch check-wifi.sh

# Add execution permission on it
chmod +x check-wifi.sh

we put this content inside it

#!/bin/bash

##################################################################
# Settings
# Which Interface do you want to check/fix
wlan='wlan0'
# Which address do you want to ping to see if the network interface is alive?
pingip='192.168.1.1'
##################################################################

echo "Performing Network check for $wlan"
/bin/ping -c 1 -I $wlan $pingip > /dev/null 2> /dev/null
if [ $? -ge 1 ] ; then
  echo "Network connection down! Attempting reconnection."
  sudo ip link set $wlan down
  sleep 5
  sudo ip link set $wlan up
else
  echo "Network is Okay"
fi

The script should be self-explanatory. We have two variables at the top: wlan that you should not change and pingip where you put your router IP address.

The script tries to ping the router, if it reaches the router, everything is fine. If it can’t do it, it disables the WiFi, and then it enables again. It should trigger the reconnection.

We need someone that calls this script every minute… cron to the rescue! This is the common way to do such things on Linux.

To add a scheduled job that triggers our script every minute, we need to write crontab -e in the terminal. It will open a text editor. On that page write this stuff

* * * * * /home/pi/check-wifi.sh

This strange syntax can be scary but it’s very easy. The first part, the * * * * * indicate when the script should trigger. The last part refers to the script we want to launch. The scheduled time is always represented by five values. The order of them changes the meaning of those asterisks. From left to right they refer to:

Minute (0 - 59)
Hour (0 - 23)
Day of month (1 - 31)
Month (1 - 12)
Day of the week (0 - 6): Sunday to Saturday

But we put *… Not numbers. We added asterisks because it has a special meaning. * means “any”. So we wrote: “Please Linux, run the script /home/pi/check-wifi.sh every minute of every hour of every day of the month of every day of the week”. Linux is smart enough to understand he should trigger the script every minute from now on.

We have a very robust air quality sensor now! We can put it outside our home and no matter what happens, it should keep always the connection as long as it reaches our router WiFi signal.

Using mosquitto_sub is the only way to see the data from the sensor. This is ok only for debugging purposes, we should have a better way to see those data. Home Assistant will solve this problem.

Add sensor to Home Assistant

Install and learn Home Assistant

Home Assistant is a great piece of software. If you are a tech enthusiast that loves automating home, I think you have already installed it.

If you don’t know yet: you can attach smart devices to it and you can create rules to automate things. You can also attach sensors to easily see the data. In other words, if you decide to install it, Home Assistant can become the control center of your home.

Seeing the sensor data in the Home Assistant is our goal here.

As the first step, we need to install it on the Raspberry Pi Hub. Then, I suggest you read the configuration manual. Home Assistant heavily uses YAML files to install custom sensors like ours.

Maybe you’re thinking that Home Assistant is overkill for seeing just sensor data. But I think this is only the starting point: I’m sure that in the end, you will have more than 20 devices with a total of 200 sensors attached and dozens of automation that orchestrate everything!

So it’s not overkill, it’s only our starting point, trust me 😊.

Now that we have installed Home Assistant and learned about where its configuration is installed, it’s time to add our sensor.

Add MQTT integration

We need to add the MQTT integration to Home Assistant. Go to the Home Assistant portal, it should be exposed on port 8123. So open the browser and type the URL http://<RPI_HUB_IP>:8123 where <RPI_HUB_IP> is the IP of our Raspberry Pi Hub. Click the cog icon (Configuration) and click on “Add integration”. We can see a huge list of integrations, we are interested in MQTT. So search for that and then it will ask for two mandatory fields:

Broker: you need to add the Raspberry Pi Hub IP. It’s where we installed Mosquitto, our MQTT broker.
Port: it’s 1883. We put that in the mosquitto configuration.

Click submit and now we have linked Home Assistant to our MQTT broker Mosquitto.

We should see the brand new integration in Configuration -> Integration Tab. We click on configure

and then we can listen to our MQTT topic, just to check if Home Assistant can see our data.

In the section “Listen to a topic”, write down our chosen MQTT topic, click “start listening” and wait at least 1 minute. The wait depends on the configuration you set on our Go program. If you don’t change my value, it should publish one datum per minute.

If you see the data like the figure below, you have successfully attached Home Assistant to our MQTT Broker!

Add air quality sensors through the MQTT platform

We know that the values are published on an MQTT topic (you chose the name, mine is home/serina/edge-rpi/air-quality). We should create two sensors that subscribe to that topic and extract data. These sensors will be

Air Quality PM 10.
Air Quality PM 2.5.

To achieve this goal, we edit the Home assistant configuration.yaml file. We need to add these two sensors inside the sensor: section.

sensor:
  - platform: mqtt
    state_topic: "home/serina/edge-rpi/air-quality"
    unit_of_measurement: "μg/m³"
    icon: "mdi:weather-fog"
    value_template: '{{value_json.PM10 | float}}'
    name: 'Air Quality PM 10'
    unique_id: 'rpi-edge:air-quality-pm-10'
  - platform: mqtt
    state_topic: "home/serina/edge-rpi/air-quality"
    unit_of_measurement: "μg/m³"
    icon: "mdi:weather-fog"
    value_template: '{{value_json.PM25 | float}}'
    name: 'Air Quality PM 2.5'
    unique_id: 'rpi-edge:air-quality-pm-2.5'

Luckily, the YAML file is very readable. We are using MQTT sensor platform. We restart the Home Assistant.

You should change state_topic with yours. You can also change the unique_id with any value, it just need to be unique, of course.

value_template is the field where we extract the value from JSON in the topic with the keyword value_json and then we take the interesting values PM10 and PM25. Finally, we say that we are interested in a float value. All this stuff is surrounded by {{ }}: when you see this syntax means we are exploiting the template system that is powered by Jinja2.

If you want to change the icon of the sensor, I usually use this website. For example, I search for a keyword, fog, and finally I hover the mouse on the chosen icon. You should prepend the name it appears with mdi:. In our example is mdi:weather-fog.

Add sensors to Home Assistant UI

Now we should have two new entities that show air quality. To see them, we should go to configuration -> entities and search for “Air quality”

as you can see, it’s quite uncomfortable. It’s better to put those entities on the overview/dashboard page.

In the Home Assistant, click on overview on the left bar, the three dots on the right, edit dashboard, and finally “Add card” on the bottom. A modal should appear, click on the “Entities” choice.

Write the title that you want; in the “entities” section, search for the entities we saw earlier. You should see a preview of the new card at the bottom. When you’re satisfied click “Save” and we’re done!

Now we should have this nice card every time we open Home Assistant! If we click on that, we should see also a graph for the last day.

Conclusions

We finally reach the end of this journey. I hope you’ve learned something new and appreciate the beauty of making an air quality sensor by yourself.

I would like to tell you more about it: for example, I haven’t explained how to see this data on Grafana

or how to see the WiFi SSID and its signal strength directly on Home Assistant.

I’ll talk about them in other blog posts because this is already very long.

If you want to be updated, leave your email below and I’ll notify you. Or simply visit my blog from time to time.

See you!

JWT authentication with Angular

Sun, 15 Dec 2019 16:40:20 GMT

At work, I had to create a new web portal that should communicate with our backend. Our frontend framework choice is Angular and our backend uses the JWT (JSON Web Token) as the authentication system.

So my goal was to integrate that authentication system in Angular to communicate with the backend. Since this was an Angular project started from scratch, I had the chance to think about how to implement it.

Objectives

My acceptance criteria were the following ones:

To get the token, the frontend should call the endpoint POST auth/login with email and password. The backend will reply with accessToken and refreshToken.
The frontend should persist both tokens, so the user will be still authenticated even if he closes the browser.
It should send the accessToken for every authenticated API calls. It sends the token inside the header X-AUTH-TOKEN.
If the backend replies 401 - Unauthorized, it means that the token is expired. So the frontend should call the endpoint POST auth/refresh passing refreshToken. If the refreshToken is valid, the backend replies with new accessToken and refreshToken in the response body.
- After retrieving the new accessToken, it should retry the failed call with the refreshed token.
- If the backend reply again with 401 - Unauthorized during refresh, it means that the refreshToken is not valid anymore. We need to logout the user.
We should handle all this stuff transparently, the developer should not worry too much about that when he implements some class to communicate with the backend.

This flow should be pretty standard for a JWT Auth system. Now let’s think about how to architecture it!

Prerequisites

I’m expecting from you some knowledge about these topics:

Angular and Angular interceptors.
RxJS that is built-in in Angular HTTP Client.

Architecture

I built this JWT integration using Test Driven Development (TDD). So I ended up with easily testable classes that have one single responsibility. This is the beauty of TDD: to simplify testing you have to write good code!

I built the following classes:

AuthClient: it makes the API calls to the backend. In our example, we will implement auth/login and auth/refresh.
AuthStore: it persists the tokens and keeps the state of the users. We can know whether the users are authenticated or not.
AuthService: it manages the business logic of our authentication system, it exploits both AuthClient and AuthStore.
AuthInterceptor: it injects the token in every authenticated API calls, also it handles the retry logic when the accessToken is expired.

We can achieve our acceptance criteria with these classes, now let’s implement them!

Implementation

Interfaces

Building a single page application is complex. It handles a lot of different objects and it’s very difficult to keep track of them without a type system. TypeScript adds types to JavaScript and I exploit this as much as I can. So I always create a folder /model that keeps track of all the objects that I create or receive from the backend. In this tutorial, we will face with these interfaces:

export interface ILogin {
    email: string;
    password: string;
}

export interface ILoginResponse {
    accessToken: string;
    refreshToken: string;
}

export interface IRefreshRequest {
  refreshToken: string;
}

AuthClient

AuthClient is very simple to implement. It just makes the API calls to our backend

import { Injectable } from '@angular/core';
import { ILogin } from './model/Login';
import { ILoginResponse } from './model/LoginResponse';
import { IRefreshRequest } from './model/RefreshRequest';
import { Observable } from 'rxjs';
import { HttpClient } from '@angular/common/http';
import { environment } from 'src/environments/environment';
import { INTERCEPTOR_NO_AUTH_HEADER } from './constants';

@Injectable({
  providedIn: 'root'
})
export class AuthClient {

  constructor(
    private http: HttpClient,
  ) { }

  login(request: ILogin): Observable<ILoginResponse> {
    return this.http
      .post<ILoginResponse>(`${environment.portalBaseUrl}/auth/login`,
        request,
        {
          headers: { [INTERCEPTOR_NO_AUTH_HEADER]: 'true' }
        }
      );
  }

  refreshToken(request: IRefreshRequest) {
    return this.http
      .post<ILoginResponse>(`${environment.portalBaseUrl}/refresh`,
        request,
        {
          headers: { [INTERCEPTOR_NO_AUTH_HEADER]: 'true' }
        }
      );
  }
}

You can see AuthClient is quite simple. The only weird thing is the header INTERCEPTOR_NO_AUTH_HEADER that has a special meaning. It’s my way to communicate with AuthInterceptor, I’m saying: “Hey! This is a public API call, you don’t need to inject the token”. I talk about it in more detail later.

AuthStore

It stores and persists the user credentials. My implementation is heavily inspired by the Business Logic Component (BLoC) pattern + RxJS. This is a new pattern that raises in Dart + Flutter world. I like it and I ported the concept in Angular. I don’t want to talk too much about BLoC pattern since there are many articles about it.

In a nutshell: I exploit RxJS to create a reactive credential store, anyone is interested in some auth data should subscribe to it. If someone wants to change auth data, he can call the auth methods. Very simple.

Generic Store

The first thing to do is creating the generic store that handles the state of type T. This is agnostic about the state, so this is an abstract class. This store is simply called Store.

import { Observable, BehaviorSubject } from 'rxjs';

export class Store<T> {
    state$: Observable<T>;
    private stateSubject$: BehaviorSubject<T>;

    protected constructor(initialState: T) {
        this.stateSubject$ = new BehaviorSubject(initialState);
        this.state$ = this.stateSubject$.asObservable();
    }

    get state(): T {
        return this.stateSubject$.getValue();
    }

    setState(nextState: T): void {
        this.stateSubject$.next(nextState);
    }
}

As you can see:

It accepts the initial state in the constructor.
You can change the state calling setState method.
You can subscribe to state changes by subscribing to state$ observable.
You can get the current state one-shot by state getter.

The engine of my store is the private stateSubject$ BehaviorSubject. If this is the first time that you see a BehaviorSubject, think about a stream that has also a “sink”, where you can add a new element in the stream by calling next method. BehaviorSubject has also the initial state that you should provide in the constructor.

Another cool thing about it, you can keep the “sink” private and only expose its observable. I’m exposing only the state$ variable that is the Observable created from stateSubject$. In this way, no one can add an element in this observable except calling setState method. In other words, you can only see the changes subscribing state$ observable, and you can change the store data by setState method. The implementation of the store is managed by stateSubject$ that is not exposed externally.

Concrete implementation

Now we can create our AuthStore, the first concrete implementation of the Store.

import { IAuthState } from './auth-store.service';
import { map } from 'rxjs/operators';
import { Injectable } from '@angular/core';
import { Store } from '../shared/store';
import { Observable } from 'rxjs';
import { ILoginResponse } from './model/LoginResponse';

export interface IAuthState {
  accessToken: string;
  refreshToken: string;
}

const ACCESS_TOKEN_LOCAL_STORAGE_KEY = 'auth.accessToken';
const REFRESH_TOKEN_LOCAL_STORAGE_KEY = 'auth.refreshToken';

@Injectable({
  providedIn: 'root'
})
export class AuthStoreService extends Store<IAuthState> {
  constructor() {
    const initialState: IAuthState = {
      accessToken: localStorage.getItem(ACCESS_TOKEN_LOCAL_STORAGE_KEY),
      refreshToken: localStorage.getItem(REFRESH_TOKEN_LOCAL_STORAGE_KEY),
    };
    super(initialState);
  }

  setState(nextState: IAuthState): void {
    if (nextState.accessToken) {
      localStorage.setItem(ACCESS_TOKEN_LOCAL_STORAGE_KEY, nextState.accessToken);
    } else {
      localStorage.removeItem(ACCESS_TOKEN_LOCAL_STORAGE_KEY);
    }
    if (nextState.refreshToken) {
      localStorage.setItem(REFRESH_TOKEN_LOCAL_STORAGE_KEY, nextState.refreshToken);
    } else {
      localStorage.removeItem(REFRESH_TOKEN_LOCAL_STORAGE_KEY);
    }
    super.setState(nextState);
  }

  get isLoggedIn(): boolean {
    return !!this.state.accessToken;
  }

  get isLoggedIn$(): Observable<boolean> {
    return this.state$.pipe(map((state) => !!state.accessToken));
  }

  get accessToken(): string {
    return this.state.accessToken;
  }

  setAccessToken(accessToken: string) {
    this.setState({ ...this.state, accessToken });
  }

  get refreshToken(): string {
    return this.state.refreshToken;
  }

  login(loginResponse: ILoginResponse): void {
    this.setState({ ...this.state, ...loginResponse});
  }

  refresh(newTokens: ILoginResponse): void {
    this.setState({ ...this.state, ...newTokens });
  }

  logout(): void {
    this.setState({
      ...this.state,
      accessToken: null,
      refreshToken: null,
    });
  }
}

There are a lot of helper methods to simplify the class usage. These are the key points of this class:

It’s a Store subclass, it has IAuthState interface as state. IAuthState is very simple, it contains only accessToken and refreshToken.
Every time someone calls setState, it persists data in the local storage. In this way, if the user closes or reloads the page, the credentials are not lost. If setState is called with falsy credentials (e.g. null or false), it removes from local storage.
In the constructor, it provides the initial state for the superclass State. It tries to get the credential from the local storage.
A lot of getters method: both reactive (ends with $), and standard (classic variables).
Some helper methods to change the state of the store (setAccessToken, login, refresh).

Be aware: storing accessToken and refreshToken inside the local storage can expose your application to security issues. Any JavaScript running on your web app can read it. So if your web app is vulnerable to cross-site scripting (XSS), then someone can steal them. Web frameworks, like Angular, are aware of this issue: they adopt high-security standards to avoid XSS vulnerabilities. I recommend you to spend some time to read about cross-site scripting (XSS). If you are scared about that, just don’t save the tokens inside the local storage, you’ll be safe… But if your user closes the page, it has to login again.

Now our AuthStore is ready to go! Let’s go to the class that wires everything together: the AuthService!

AuthService

AuthService is the class that contains the business logic to make authentication works. Let’s see the code:

import { Injectable } from '@angular/core';
import { Observable } from 'rxjs';
import { map, switchMap, tap } from 'rxjs/operators';
import { AuthClient, IRefreshRequest } from './auth-client.service';
import { AuthStoreService } from './auth-store.service';
import { ILogin } from './model/Login';
import { ILoginResponse } from './model/LoginResponse';
import { Router } from '@angular/router';

@Injectable({
  providedIn: 'root'
})
export class AuthService {
  constructor(
    private authClient: AuthClient,
    private authStore: AuthStoreService,
    private router: Router,
  ) { }

  login(request: ILogin): Observable<ILoginResponse> {
    return this.authClient.login(request)
      .pipe(
        tap(loginResponse => this.authStore.login(loginResponse)),
      );
  }

  refresh(refreshTokenRequest: IRefreshRequest): Observable<ILoginResponse> {
    return this.authClient.refreshToken(refreshTokenRequest)
      .pipe(
        tap(refreshResponse => this.authStore.refresh(refreshResponse))
      );
  }

  logout(): void {
    this.authStore.logout();
    this.router.navigateByUrl('/auth/login');
  }
}

This class uses both the AuthClient, to make API calls, and AuthStore, to persist credentials. There are three methods that we can call:

login: we call it when we want the user to login. It calls login API and then persists the credential in AuthStore.
refresh: used to refresh the token. It makes API calls and then persists them.
logout: it clears the AuthStore and then navigates users to login page.

This is a simplified version of my implementation. During login, I made also an API calls to an endpoint to get user data (also called auth/me). I didn’t add it because its implementation depends on your architecture and it’s out of the blog post’s scope. The key point is this: you should add all your authentication business logic here, delegating the persistence to AuthStore and the API calls to AuthClient.

AuthInterceptor

We are in the last part of this blog post: we can authenticate the user and persist credentials in the AuthStore. We can refresh the token when it expires. We can logout the user.

Now we focus on these goals:

Not passing the token when we call a public API.
Passing the JWT token during authenticated API calls.
Refreshing the token when the accessToken expires. We know the token expires only when we make some API calls: so when an API call fails, because the token is expired, we need to call refresh API call to get the new token, then we need to make again the first call with the refreshed token.

All this stuff is managed by the AuthInterceptor. It is a subclass of Angular HTTP Interceptor. Briefly speaking, an HTTP Interceptor is a layer that we can add between the API call we make and the real HTTP Request made by the browser. In this way, we can transparently inject all these auth behavior without changing our Client classes.

Understanding HTTP Interceptor

First of all, we need to understand how to implement an HTTP Interceptor: AuthInterceptor should implement HttpInterceptor interface. So we have to implement the method

intercept(request: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>>

The first parameter is the HTTP request that Angular is going to do. At the end of our logic, we need to call next.handle(request) to continue the HTTP request through the normal flow. In other words, let’s assume we create an HTTP interceptor that does absolutely nothing. The minimum code we have to write is this

@Injectable()
export class NoOpInterceptor implements HttpInterceptor {

    constructor() { }

    intercept(request: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {
      return next.handle(request);
    }
}

AuthInterceptor implementation

Now we do the same thing for our AuthInterceptor, but this time we have to add all the authentication logic that we said before.

Let’s have a look at its implementation:

import { AuthStoreService } from './auth-store.service';
import { HttpInterceptor, HttpRequest, HttpHandler, HttpEvent, HttpErrorResponse } from '@angular/common/http';
import { Observable } from 'rxjs';
import { Injectable } from '@angular/core';
import { tap, mergeMap, catchError } from 'rxjs/operators';
import { AuthService } from './auth.service';
import { INTERCEPTOR_NO_AUTH_HEADER } from './constants';

@Injectable()
export class AuthInterceptor implements HttpInterceptor {

    constructor(
        private authStore: AuthStoreService,
        private authService: AuthService,
    ) { }

    intercept(request: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {
        /* If the incoming request has the special INTERCEPTOR_NO_AUTH_HEADER,
           it means that we don't have to add the JWT token
        */
        if (request.headers.get(INTERCEPTOR_NO_AUTH_HEADER)) {
            /* So just send to the next handler the same request
               removing the special header INTERCEPTOR_NO_AUTH_HEADER
            */
            return next.handle(request.clone({
                headers: request.headers.delete(INTERCEPTOR_NO_AUTH_HEADER)
            }));
        }
        // If the user is logged in
        if (this.authStore.isLoggedIn) {
            return next
                // Then add the token and continue the API call.
                .handle(this.addTokenToRequest(request))
                .pipe(
                    // But if the call has failed, try to recover it using retryWhenExpiredToken method
                    catchError(this.retryWhenExpiredToken(request, next)),
                    tap({
                        error: (err: HttpErrorResponse) => {
                            /*
                              If the call, after our refresh method, is still failing with
                              401 UNAUTHORIZED status code
                            */
                            if (err.status === 401) {
                                // Then logout the user, nothing else we can do.
                                this.authService.logout();
                            }
                        }
                    }),
                );
        }
        /*
          If the user is not logged in, just do nothing.
          We don't have any JWT token to attach to the request
        */
        return next.handle(request);
    }

    // This method is called only during an error response
    private retryWhenExpiredToken(req: HttpRequest<any>, next: HttpHandler): (response: any) => Observable<any> {
        return response => {
            const refreshToken = this.authStore.refreshToken;
            /* If the status code is 401 UNAUTHORIZED and we don't have the refresh token,
               or the status code is different from 401 UNAUTHORIZED
            */
            if (response.status === 401 && !refreshToken || response.status !== 401) {
                // We can do nothing, just re-raise the error and continue.
                throw response;
            }

            // Otherwise, call the backend to refresh our token
            return this.authService
                .refresh({ refreshToken })
                .pipe(
                    // Then re-call the failed request with the brand new token
                    mergeMap(() => {
                        return next.handle(this.addTokenToRequest(req));
                    }),
                );
        };
    }

    // It just clones the request, adding the JWT token to the request
    private addTokenToRequest(request: HttpRequest<any>): HttpRequest<any> {
        return request
            .clone({
                setHeaders: {
                    'X-JWT-TOKEN': this.authStore.accessToken,
                }
            });
    }
}

Take your time to read it… We see that the intercept method is more complex. I added tons of comments inside the code, so it’s easier to understand what’s happening.

Let’s see whether we reached all our prefixed goals:

Not passing the token when we call a public API: done by INTERCEPTOR_NO_AUTH_HEADER special header.
Passing the JWT token during API calls: done thanks to addTokenToRequest method.
Refreshing the token when the accessToken expires: achieved by retryWhenExpiredToken method.

The code is not so long, it has more or less 50 lines. But I think is pretty full of tricky concepts from RxJS like mergeMap, tap, catchError.

I used closure to make retryWhenExpiredToken works. I created a function that returns a function. The first call gets the original request and the next handler and returns the function that we feed the method catchError.

Writing this code without bugs in one-shot is very difficult, even if it’s not long. Two things helped me reaching it:

Heavy usage of TypeScript typing system: TypeScript add types to JS, thank god! Adding types to everything is the best way to avoid nasty bugs. The transpiler can warn you as soon as possible if you’re returning something wrong. If you put any everywhere, or you don’t add types for your expected parameters, you’re alone! The transpiler can’t help you, soon or later you make a mistake. The only any you see in this code is because the intercept signature is coming from Angular and I can’t do anything about it.
Use Test Driven Development (TDD) for the crucial part of your system, or use TDD as much as you can. Testing manually this authentication system takes much longer than writing some unit tests. And there’s nothing worse than someone calls you saying users can’t sign in. I made this using TDD and the result is these tiny classes that are easy to test thanks to the Angular testing suite.

The perfect developer, that makes no bugs, doesn’t exist. Some developers are better than others not because are perfect, but because they take more countermeasures to avoid bugs, so they write less buggy code. TypeScript and TDD are the most effective countermeasure that I know for front-end development. So use them.

If you are interested in TDD, I wrote two articles about that. Unfortunately, they are written in Italian. If you are interested in the translation just let me know: ping me on LinkedIn and subscribe to my newsletter to be notified about these topics. Here are the links:

Adding AuthInterceptor to Angular app

We must remember to add the AuthInterceptor to our application. This is very easy: we need to declare it inside the root Angular module of the app

import { HTTP_INTERCEPTORS } from '@angular/common/http';
import { AuthInterceptor } from './auth/auth-interceptor';

@NgModule({
  declarations: [...],
  imports: [
    AuthModule,
    ...
  ],
  bootstrap: [AppComponent],
  providers: [
    { provide: HTTP_INTERCEPTORS, useClass: AuthInterceptor, multi: true },
    ...
  ]
})
export class AppModule { }

And that’s it! Now our app handles the JWT authentication of our back-end!

Conclusion

I hope you enjoyed this journey about JWT authentication + Angular. This is my implementation, and it satisfied all my goals. I hope this work inspires you to reach yours too!

See you!

Let's learn to automate with Ansible · DevOps

Sun, 25 Aug 2019 07:03:47 GMT

Introduction

You’re at work and you’ve been tasked with configuring 20 virtual machines all the same way? You’re at home and your Raspberry Pi unfortunately corrupted the SD card and so you have to reconfigure it for the umpteenth time? Then Ansible could solve many of your problems!

If you’re like me I’m sure you don’t like spending the day performing repetitive tasks that give practically zero added value to yourself: performing repetitive tasks isn’t a job for us humans, we can easily delegate it to machines so we can dedicate ourselves to things that really matter.

How do we automate these tasks? A certainly effective answer is Ansible, the configuration tool I prefer.

Goal

If you’ve read the official Ansible documentation, I’m sure you’ve noticed that it’s as boring as the tool itself is powerful. It’s easy to fall asleep reading it even before understanding Ansible’s real potential.

That’s why I decided to write this mini guide hoping it’s a bit more effective. I hope to achieve this goal through a concrete example that happened to me and that I still use: we’re going to configure a VPN Server on a Raspberry Pi.

Thanks to this Ansible playbook, we can create a VPN Server on a Raspberry Pi by running a single command ansible-playbook -vv -i hosts vpn.yml. The functions it performs behind the scenes are many:

It downloads the certificate via https://letsencrypt.org/ for the domain pointing to our home, for example myhome.mattianatali.it.
It sets up automatic renewal of the certificate itself.
It opens the correct ports on the Raspberry Pi to make the VPN work.
It installs all the libraries and applications needed to install the VPN Server.
It correctly configures the settings we decide for the VPN, for example the user to access it.
It sets up automatic certificate updates based on the renewal of the Let’s Encrypt certificate.

Now imagine if we didn’t have Ansible doing all these things with one command. Surely 2 days after running these commands manually I would have already forgotten everything, not to mention the number of attempts I would have had to make even the second time to reconfigure everything from scratch. Precisely for this reason, once I understood how I could do it, I created the Ansible script. Now I have the versioned, ready-to-use playbook in a public repository.

In this guide we won’t dwell too much on the details of the VPN: we’re more interested in the reasoning that led to this result and which Ansible features we used. Creating the VPN is just a concrete example that covers many Ansible features.

Requirements

The requirements to follow this guide are the following:

We have a minimum knowledge of Linux. The commands sudo, apt-get install, cd, chmod, ssh should be familiar to us.
We’ve installed Ansible on your laptop.
Our Raspberry Pi must have SSH enabled.

Glossary

The first obstacle you encounter with Ansible is learning and remembering the various terms it uses. Here’s a small summary that will help us throughout the guide:

Modules: modules are small programs that Ansible offers us to complete tasks. For example: we want to install a package like vim? We can use the apt module; we want to add a cronjob? There’s the cron module. There really are an infinite number of them.
Task: for example a task is “install the nginx program”. We can see them as the smallest unit we have in Ansible; basically it’s calling a module with the various parameters. An example of a task is the following
```
- name: Install nginx
    apt: 
      name:
        - nginx
```
Role: Ansible explains the role using a football metaphor. An example of a role could be “striker”, “defender”, “midfielder”. In our example we have two roles: “certificate installer”, “VPN installer”. In short it’s a set of tasks that have a very specific function, a role indeed.
Playbook: it’s the largest unit we have in Ansible. A playbook organizes the various roles to complete our task. For example in our case it’s the set of the two roles explained above. My repository about creating the VPN is indeed a playbook. In our case we’ll have a playbook that will carry out our task, that is, installing the VPN with the various certificates.
Inventory: it’s a file that contains groups that in turn contain machines. These machines are the ones that will be affected by our playbook. Ansible is very powerful; with a single command you can install the VPN on 100 machines. We’ll have a group called vpns that will have our target machine named rpi (the Raspberry Pi) defined inside it.

Now we have a common terminology; it’s not necessary to understand everything now, surely it’s all still very foggy. But later, after seeing a concrete example, everything will be clearer.

Let’s perform the task manually

Before automating we must be clear about what we have to do; the first time maybe we don’t have clear ideas at all, we make many attempts and some will lead nowhere. So the first thing to do is try manually without bothering Ansible so as to be faster in the “construction” phase of the various tasks to reach the goal. In our case we first educate ourselves a bit about the VPN, read how strongswan works, which libraries it needs to work, how to configure it, etc. Then we educate ourselves about Let’s Encrypt to issue a valid certificate for our domain myhome.mattianatali.it that points to our home, so as to have the access point to our VPN.

I’ll spare you this whole phase; the thing we must keep in mind is this: during the manual phase let’s keep in mind every step we take to complete our task, every single configuration file we modified, anything we must keep track of because soon we’ll automate it.

Let’s define the roles

Once the goal is reached manually, it’s time to organize our ideas. At the end of our analysis our Ansible playbook must perform the following tasks:

It downloads the certificate via https://letsencrypt.org/ for the domain pointing to our home, for example myhome.mattianatali.it.
It sets up automatic renewal of the certificate itself.
It opens the correct ports on the Raspberry Pi to make the VPN work.
It installs all the libraries and applications needed to install the VPN Server.
It correctly configures the settings we decide for the VPN, for example the user to access it.
It sets up automatic certificate updates based on the renewal of the Let’s Encrypt certificate.

Of all these tasks we can find two well-distinct roles that can be separated.

The generation and maintenance of certificates for myhome.mattianatali.it using Let’s Encrypt.
Installation of the VPN which uses the previously created certificates.

What advantages does separating the tasks found into these two roles bring?

The more self-contained and specific the roles are, the simpler they are to maintain.
The more self-contained they are, the easier they are to reuse for other playbooks.
If the defined role is standard enough, it’s likely someone has already implemented it!

In fact Ansible has a site to search for these roles: it’s called Ansible Galaxy. Generating certificates via Let’s Encrypt is a fairly standard thing, and indeed we’re lucky! We have a role that already does what we need; we can find it here; the best way to learn how to use it is anyway to look at its repository on GitHub.

To install the role, so we can use it shortly, let’s type in the console:

ansible-galaxy install geerlingguy.certbot

Searching again on that site I didn’t find a role that creates the VPN, so for that we have to fend for ourselves.

Let’s define the Playbook

We said that modules and tasks are the atomic part of Ansible; these are grouped into roles that perform a certain function (in our case there are two), the entity that orchestrates these roles is the playbook. If we wanted to depict it in a diagram:

playbook
  roles
    tasks

To create our VPN configuration we adopt a top-down approach. In other words we start working on the most “external” and general part of the problem, then go deeper and deeper into detail down to the various tasks. The part that contains all the others is the playbook. Our playbook must start in sequence:

The role that generates the certificates. Already implemented thanks to the library we found on Ansible Galaxy
The role that installs the VPN. We must implement it.

Ansible relies heavily on conventions; let’s start right away using the best practices provided by Ansible.

We can use my repository with the finalized playbook as a reference. The structure to create our playbook is the following:

group_vars/
  vpns.yml           # All the group-level variables go here (our group is called vpns)
   
host_vars/
   rpi.yml           # Here we assign the variables for individual machines (our only machine is called rpi)

vpn.yml              # This is our playbook
hosts                # This is our inventory, we define the groups (vpns) and the machines (rpi) on which we want to run the playbook.

roles/               # This folder contains all the roles we'll define
    vpn-install/     # This folder contains our role to install the VPN.
      ...            # We'll see the subfolder structure later.

Let’s start writing our playbook in the vpn.yml file

---
- name: Deploy IKEv2 Road Warrior VPN
  hosts: vpns

  roles:
    - role: geerlingguy.certbot
      become: yes
    - role: vpn-install
      become: yes

The format the file is written in is YAML; it’s a very readable format and is now a de facto standard in applications that cater to DevOps. If you’ve never seen it I recommend reading this guide, which explains the format quickly.

Our Playbook will have to do only one thing: create a VPN Server. In Ansible jargon, a playbook is made up of multiple plays. We have only one play in our playbook.

How do you figure out how and when to create a play? Usually if we have multiple roles that affect a single group of machines, then this is a good candidate to be a play. If instead we had some roles targeting the Raspberry and other roles that had to configure routers, then our playbook would be made up of two plays. Because one play worked on the Raspberry Pis, the other on the routers.

Let’s explain the various commands inside a play:

name: we can insert a short description of what our play will do.
hosts: we define which group will be used as the target of our play. In our case it’s the vpn group, which will contain our Raspberry Pi.
roles: we define the roles in chronological order that will be started on our hosts.
role: geerlingguy.certbot: the first role to start will be the one we found on Ansible Galaxy; it will install and keep the certificates for the VPN up to date.
become: yes: it means the script will be started enabling privilege escalation. In practice we’ll start the role as root. Both the role to download the certificates and the VPN installation need to be started as root. So in both we’ll put become: yes.
role: vpn-install: we start the VPN installation after the certificates role. We still have to figure out how to define this role.

We’ve finished the playbook. In practice our playbook is saying this: “Install the certificates and then install the VPN on the machines that are part of the vpns group”.

Let’s define the inventory

We said the playbook will work on the group of machines called vpns. But where is this group defined? This is where the inventory comes in. The inventory object is where we define the groups of machines and the machines that make up this group. The file containing this inventory is the hosts file. The name is arbitrary; whatever name we give it will then have to be referenced when we invoke the command to start the playbook.

Our hosts file will have this content.

[vpns]
rpi

Simply with these two lines we’re saying that the vpns group contains the rpi machine which is the name we’re giving to our Raspberry Pi. It’s easy to guess that if we had multiple Raspberry Pis, we could configure them at the same time by defining them in this file in the vpns group.

The inventory syntax is simple and it’s this

[group1]
machine1
machine2

[group2]
machine3
machine4

...

Let’s define the group and host variables

We’ve defined the playbook that works on the vpns group, we’ve declared the vpns group saying that it contains our Raspberry Pi named rpi through the inventory. But how does Ansible know how to connect to our Raspberry Pi? How do we give it the IP, user, password to connect with SSH? How do we tell it which domain we want to get the certificates for? For now it only knows there’s a machine called rpi.

All these things are variables; they vary based on the machine or group of machines we want to act on. Ansible has two folders dedicated to this purpose: a folder called group_vars will contain the variables that are common to all machines belonging to a group, the host_vars folder will contain all the variables specific to a single machine.

In our project we can insert these two folders; these folders will have YAML files named after our group in group_vars and named rpi.yml in host_vars.

So the structure of our folders, as far as variables are concerned, is the following:

group_vars/
  vpns.yml           # Contains all the variables common to all machines belonging to the vpns group
   
host_vars/
   rpi.yml           # Contains all the variables belonging to the rpi machine

Inside the rpi.yml file defined in the host_vars folder, Ansible will be able to find all the information it needs to complete the playbook. The rpi.yml file contains the following variables:

---
ansible_host: YOUR-SSH-HOST
ansible_user: pi
ansible_password: YOUR-SSH-PASSWORD
domain: myhome.mattianatali.it
# You can add more than one Road Warrior Client.
vpn_users:
  - username: CLIENT-WARRIOR-USERNAME
    password: CLIENT-WARRIOR-PASSWORD
# Change it if you want to restrict the local hosts the Road Warrior Client should see.
vpn_local_ts: "0.0.0.0/0"

# ansible-role-certbot variables
# More info here https://github.com/geerlingguy/ansible-role-certbot
certbot_auto_renew: true
certbot_create_if_missing: true
certbot_create_method: standalone
certbot_auto_renew_user: root
certbot_auto_renew_hour: 3
certbot_auto_renew_minute: 30
certbot_auto_renew_options: "--quiet --no-self-upgrade"
certbot_create_standalone_stop_services: []
certbot_certs:
  - email: YOUR-EMAIL-HERE
    domains:
      - "{{domain}}"

Through the ansible_host, ansible_user, ansible_password variables we can provide Ansible with all the information to connect to SSH, to run our playbook. We can avoid inserting username and password by using the SSH configurations. In this guide, in the SSH section, I explain how to do it.

The following fields are the ones we’ll use when we create the role to install the VPN. The domain field will contain the name of our domain that points to our home; in our case it’s myhome.mattianatali.it. In vpn_users we define the username and password of the clients that can connect to our VPN. vpn_local_ts is a variable that allows us to limit access to clients connecting to the VPN; for example at my home I limited it to my subnet 192.168.1.0/24.

All the variables starting with certbot_ are part of the role we downloaded with Ansible Galaxy; they allow us to configure anything about certificate renewal for the myhome.mattianatali.it domain. Finding the variables that make up the role we downloaded is simple; just read the documentation of the role we used.

We can notice that the last line of our file has a particular variable {{domain}}. The double braces allow us to reference another already-defined variable. This way we can define only the domain variable in the fourth line, and automatically the {{domain}} variable will have the same value. This way we have “a single source of truth” and we avoid gross errors like updating only one variable and not the other.

To insert values in the double braces, Ansible uses a very famous template engine in Python: Jinja2. Jinja2 is very powerful; for now we just need to know that when we write {{value}} it means we’re inserting the content of the variable we find in the braces.

Let’s implement the role to install the VPN

We’ve defined the playbook, the inventory, the variables and the role to install the certificates we found on Ansible Galaxy; now we just need to implement the role that installs the VPN server on the Raspberry Pi.

Roles are defined in the roles/ folder; let’s create a folder inside it that will take the name of the role (vpn-install), the same name we defined in the playbook in vpn.yml. The structure of our folders will be defined like this:

...
roles/    
    vpn-install/
        files/        # Contains the files to copy to the Raspberry Pi
            ...
        tasks/        # Contains the instructions to execute
            ...
        templates/    # Contains the source templates to then upload to the Raspberry Pi
            ...

In this case too we’re using Ansible’s conventions, we’re not making anything up:

files: contains all the files we need to upload to the Raspberry Pi.
tasks: contains the tasks to execute on the Raspberry Pi to install the VPN.
templates: it’s similar to the files folder, but in this case we won’t have the files to copy, but a template that we must render before we can copy it to the Raspberry Pi. Wait a moment and then its use will be clear.

Let’s define the tasks

Now let’s focus on the tasks folder; as we said here we’ll have all the commands to execute on the Raspberry Pi to install the VPN. The first file Ansible looks for in the tasks folder is main.yml. In this file we write all the tasks.

How do we know which tasks to execute? Here we must retrieve the commands we executed manually at the beginning of the article. Once retrieved we must translate them into tasks, using the modules Ansible provides us.

These are the “commands” we need to translate into Ansible tasks; take them for granted, I don’t want to bore you too much with the VPN question:

Install the following packages using apt-get: strongswan, strongswan-swanctl, libstrongswan-standard-plugins, libstrongswan-extra-plugins, libcharon-extra-plugins.
Enable IP Forwarding.
Add the file with the VPN configuration in /etc/swanctl/swanctl.conf.
Add the file with the various Strongswan plugins in /etc/strongswan.conf.
Create the x509, x509ca, private folders in /etc/swanctl/; here we’ll add the various certificates created by the previous role we found on Ansible Galaxy.
Copy the certificates created by the role that creates them from the /etc/letsencrypt/live/myhome.mattianatali.it/ folder into the correct folders created in the previous point.
Add a script that repeats the previous point every time the certificates are renewed. Let’s Encrypt certificates only last 3 months.
Add a cronjob that runs the previous point right after the certificates are generated.
Enable IP Masquerading, changing the Raspberry Pi firewall rules.
Reboot the Raspberry Pi.

These are all the operations we need to automate to have our VPN working on the Raspberry Pi. There are many operations, some complex, boring to do manually, the chance of error is high, the commands are practically impossible to remember by heart for a future configuration. Maybe now you understand why I immediately thought of Ansible when I tackled this endeavor!

With Ansible we can do all this with a few lines of code and disarming clarity. The final result of our main.yml file, which includes all our commands, is the following. I added comments in the file so as to have the right correspondence with the points written just above:

---
# Install the following packages using `apt-get`: `strongswan`, `strongswan-swanctl`, `libstrongswan-standard-plugins`, `libstrongswan-extra-plugins`, `libcharon-extra-plugins`.
- name: Install strongswan dependencies
  apt: 
    name:
      - strongswan
      - strongswan-swanctl
      - libstrongswan-standard-plugins
      - libstrongswan-extra-plugins
      - libcharon-extra-plugins

# Enable IP Forwarding.
- name: Add Client Traffic Forwarding
  sysctl:
    name: "{{ item }}"
    value: 1
  with_items:
    - net.ipv4.ip_forward
    - net.ipv6.conf.all.forwarding

# Add the file with the VPN configuration in /etc/swanctl/swanctl.conf
- name: Add VPN configuration
  template:
    src: templates/swanctl.conf.j2
    dest: /etc/swanctl/swanctl.conf

# Add the file with the various Strongswan plugins in /etc/strongswan.conf
- name: Add strongswan.conf with correct plugins
  copy:
    src: files/strongswan.conf
    dest: /etc/strongswan.conf

# Create the x509, x509ca, private folders in /etc/swanctl/
- name: Create folders to store certificates
  file:
    path: /etc/swanctl/{{ item }}
    state: directory
    mode: 0755
  with_items:
    - x509
    - x509ca
    - private

# Copy the certificates created by the role that creates them from the /etc/letsencrypt/live/myhome.mattianatali.it/ folder into the correct folders created in the previous point
# Note the use of the domain variable to insert myhome.mattianatali.it
- name: Copy certificates from let's encrypt to strongswan
  shell: |      # In YAML "|" means "take everything I write after this as a variable and preserve line breaks"
    cp /etc/letsencrypt/live/{{ domain }}/chain.pem /etc/swanctl/x509ca/
    cp /etc/letsencrypt/live/{{ domain }}/privkey.pem /etc/swanctl/private/
    cp /etc/letsencrypt/live/{{ domain }}/fullchain.pem /etc/swanctl/x509/

# Add a script that repeats the previous point every time the certificates are renewed
- name: Add script to copy new certificate
  template:
    src: templates/renew-certs.sh.j2
    dest: /root/renew-certs.sh
    mode: u=rwx,g=r,o=r

# Add a cronjob that runs the previous point right after the certificates are generated
- name: Add cronjob to copy new certs
  cron:
    name: Copy certs to strongswan
    hour: "{{ certbot_auto_renew_hour }}"
    minute: "{{ certbot_auto_renew_minute + 15 }}"
    job: "/root/renew-certs.sh"

# Enable IP Masquerading]
- name: Add iptable rule permanently
  lineinfile:
    path: /etc/rc.local
    line: iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
    insertbefore: ^exit 0

# Reboot the Raspberry Pi
- name: Finished! Rebooting the system...
  reboot:

We can immediately notice that the tasks all have the following structure

- name: TASK DESCRIPTION
  MODULE_NAME:
    OPTION_1: OPTION_1_VALUE
    OPTION_2: OPTION_2_VALUE

The modules provided by Ansible are many.

The modules I used in this role are the following:

apt: to install the packages using the apt-get command.
sysctl: to modify the sysctl.conf file and therefore modify the system kernel attributes. In our case it was used to enable IP Forwarding.
template: to process a file using the Jinja2 template engine, and copy the result to the Raspberry Pi. We need Jinja2 to insert a file that’s dynamic based on the host and group variables we’re working on.
copy: to copy a file from a local folder to our Raspberry Pi; in this case we don’t need Jinja2 to insert variables.
file: to work with files and folders on the Raspberry Pi, for example change permissions, create or delete folders, etc.
shell: to execute classic commands via the CLI on the target machine.
cron: to modify the crontab file on the machine.
lineinfile: to verify that a specific line exists in a given file.
reboot: to reboot the Raspberry Pi at the end of everything.

The options of the various modules are quite self-explanatory; in any case if any aren’t clear you can read the documentation of the individual modules. The module names in the previous list all point to the reference guide.

There are some though that deserve a more in-depth description: for example the task

# Create the x509, x509ca, private folders in /etc/swanctl/
- name: Create folders to store certificates
  file:
    path: /etc/swanctl/{{ item }}
    state: directory
    mode: 0755
  with_items:
    - x509
    - x509ca
    - private

are three tasks in one; thanks to with_items you can define a list of variables and then, through {{ item }}, it’s possible to execute the module for each element in the list. So in the previous case it’s as if we had written:

# Create the x509, x509ca, private folders in /etc/swanctl/
- name: Create folder x509
  file:
    path: /etc/swanctl/x509
    state: directory
    mode: 0755
- name: Create folder x509ca
  file:
    path: /etc/swanctl/x509ca
    state: directory
    mode: 0755
- name: Create folder private
  file:
    path: /etc/swanctl/private
    state: directory
    mode: 0755

Seeing the {{ item }} tag we can guess that our main.yml file is processed with Jinja2 before being executed. In fact in our file to define the tasks we can refer to all the variables of the previously declared hosts and groups. For example the task:

- name: Add cronjob to copy new certs
  cron:
    name: Copy certs to strongswan
    hour: "{{ certbot_auto_renew_hour }}"
    minute: "{{ certbot_auto_renew_minute + 15 }}"
    job: "/root/renew-certs.sh"

Uses the certbot_auto_renew_hour and certbot_auto_renew_minute variables. Jinja2 is very powerful and also allows us to do operations with variables. With this task we’re essentially saying “run the command /root/renew-certs.sh 15 minutes after the task that generates the certificates”. This is possible thanks to the operation {{ certbot_auto_renew_minute + 15 }}.

The template module

The template module deserves an additional explanation since it’s very powerful. We said the template module allows you to insert a file processed with Jinja2 into the target machine. For example we used it for the task:

- name: Add script to copy new certificate
  template:
    src: templates/renew-certs.sh.j2
    dest: /root/renew-certs.sh
    mode: u=rwx,g=r,o=r

In this case we take the templates/renew-certs.sh.j2 file, Jinja2 replaces the variables it finds using the ones we defined in the group_vars and host_vars folders and then inserts it into the /root/renew-certs.sh folder on the Raspberry Pi setting permissions to 744.

Let’s quickly see the content of the template:

#!/bin/sh -
cp /etc/letsencrypt/live/{{ domain }}/chain.pem /etc/swanctl/x509ca/
cp /etc/letsencrypt/live/{{ domain }}/privkey.pem /etc/swanctl/private/
cp /etc/letsencrypt/live/{{ domain }}/fullchain.pem /etc/swanctl/x509/
/usr/sbin/swanctl --load-all

in our case domain will be replaced by myhome.mattianatali.it before being copied to the Raspberry Pi.

An even more complex template can be found with the task

# Add the file with the VPN configuration in /etc/swanctl/swanctl.conf
- name: Add VPN configuration
  template:
    src: templates/swanctl.conf.j2
    dest: /etc/swanctl/swanctl.conf

In this case templates/swanctl.conf.j2 contains:

connections {
    ikev2-eap {
        version = 2
        proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-ecp521,aes192-sha256-modp3072,default
        rekey_time = 0s
        pools = primary-pool-ipv4
        fragmentation = yes
        dpd_delay = 30s
        send_cert = always
        # dpd_timeout doesn't do anything for IKEv2. The general IKEv2 packet timeouts are used.
        local-1 {
            auth = pubkey
            certs = fullchain.pem
            id = {{ domain }}
        }
        remote-1 {
            auth = eap-mschapv2
            # go ask the client for its eap identity.
            eap_id = %any
        }
        children {
            ikev2-eap {
                local_ts = {{ vpn_local_ts }},
                rekey_time = 0s
                dpd_action = clear
                esp_proposals = aes192gcm16-aes128gcm16-prfsha256-ecp256-modp3072,aes192-sha256-ecp256-modp3072,default
            }
        }
    }
}

pools {
    primary-pool-ipv4 {
        addrs = 172.16.252.0/24
        dns = 8.8.8.8, 8.8.4.4
        split_exclude = 172.16.0.0/12
    }
}

secrets {
  {% for user in vpn_users %}
   eap-{{ user.username }} {
      id = {{ user.username }}
      secret = "{{ user.password }}"
   }
  {% endfor %}
}

Here the domain and vpn_local_ts variables will be replaced. Thanks to the {% for user in vpn_users %} and {% endfor %} construct we insert the usernames and passwords of the users present in the vpn_users variable we defined in the host_vars/rpi.yml file.

Now let’s imagine not having a template engine and having to do this operation manually… It’s not trivial at all! Thanks to Ansible we can clearly separate the variables from the static content of the files; this way we can make custom configuration on multiple machines very simple. In fact the content of the role would always remain the same; what changes is the inventory, where we define the new machine to configure and its variables to use in host_vars.

Running the playbook

We’ve finally completed our Playbook to create a VPN on our Raspberry Pi! You can find the final result on my repository. We just need to run our playbook through the command

ansible-playbook -vv -i hosts vpn.yml

Thanks to this command, we start the playbook defined in vpn.yml verbosely (-vv) passing the inventory defined in the hosts file (-i hosts). We’ll see that Ansible will try to connect via SSH to our Raspberry Pi and execute our tasks point by point, using all the variables we defined in host_vars and group_vars. In less than 2 minutes it will have finished doing everything and we can connect to the VPN by setting as server address and id the domain we defined (in my case myhome.mattianatali.it) and as username and password those entered in the vpn_users variable in the host_vars/rpi.yml file.

At the end we still need to do the port forwarding of the VPN ports from the router to the Raspberry Pi; I didn’t automate this part with Ansible because routers are all different from each other. If you want the complete VPN I recommend reading the README of my repository.

Conclusions

I hope this guide has provided you with guidelines on how to automate even complex tasks using Ansible. I deliberately skipped quite a few points that were strictly related to creating the VPN, because the purpose of the guide is to show you the potential of Ansible with an example that isn’t the simple “Hello World”.

What must be clear is the modus operandi with which we arrived at the solution:

We try the configuration manually, recording the steps.
We define any roles of our procedure. We search Ansible Galaxy to see if some role has already been made.
We create the playbook to declare the complete procedure.
To define the target machines, we define the inventory and insert all the variables in group_vars and host_vars.
We define the tasks to execute in the role. The modules will help us in defining these tasks.

Ansible relies heavily on conventions and best practices. This is a huge advantage because if you’ve understood this example, you can practically understand any Ansible playbook because the structure is always the same! Try browsing the various examples Ansible provides; you’ll find the same concepts and folders in all of them.

Once you’ve assimilated these concepts, it doesn’t matter if we have to set up a VPN, a redundant MongoDB instance, configure NTP on 100 machines; we can configure everything we want with Ansible efficiently and clearly.

See you next time!

Welcome

Sun, 25 Aug 2019 07:03:47 GMT

Here we are, with a new shiny blog!

I decided to make another personal blog. But this is different from my previous one.

It’s in English.

I used to write in Italian since I’m Italian! But I realized that the preferred language is English for technical posts.

It isn’t an easy switch for me, because I love Italy and I wanted to spread some knowledge in my native tongue. But writing posts in Italian, I reach a niche of a niche… I don’t know if it’s so useful.

Another big reason is that all the references that I link in my blog posts were in English! So even an Italian reader should know English to understand the content properly.

It’s about my job

My previous blog was more personal, it started with no fixed idea on what I would write in it. In fact, it was more an exercise on how to deploy a Ghost blog and how to build a Gatsby website. The content inside it is more mixed: from photography, games and tech posts.

In this blog, I want to share with you my thoughts on my daily job, how I tackled some challenging task, or simply what I’m doing at work.

I want to share some content that is useful for other developers.

I hope that I’ll have some time to share this stuff and I hope you like them!

How to develop APIs in TDD with Spring Boot

Tue, 30 Jul 2019 09:03:47 GMT

In this article we saw what it means to develop in Test Driven Development (TDD) and the benefits that come from it. The example used might seem like a pointless little game, but my focus there was to explain TDD to those who had never heard of it at all.

In this guide instead let’s look together at a use case that will be much more familiar to us: the development of RESTful APIs. To do this I decided to use the Spring Boot framework; the chosen language is Java. Spring Boot is widely used in the enterprise world to create a microservices architecture. Netflix uses it too. Moreover Java is a programming language known to everyone by now, so I think it’s a good choice to see a practical example together.

As always we start completely from scratch, to leave nothing to chance. In addition to TDD, we’ll also see together how to structure the project correctly and other very handy libraries that I use every day to increase our productivity, such as the Lombok library.

Requirements

The reader I’m imagining is a developer who has already heard of Spring Boot, isn’t scared if we talk about RESTful APIs, wants to use TDD as the main development driver but can’t apply it effectively.

This article is a continuation of my first episode on TDD, so I strongly recommend reading “How to develop in TDD starting from scratch” first. This way you’ll have an initial grounding in TDD and we’ll be aligned on some concepts that I’ll take for granted here.

My guide is based on a UNIX-like operating system (Mac OS X), but it can easily be followed by Windows users too.

On our PC, Java Development Kit 12 must be installed.

Goal

Let’s imagine we need to create two endpoints:

GET - /api/v1/vehicles/{id}: to retrieve a vehicle by id.
POST - /api/v1/vehicles: to add a vehicle.

The vehicle is made up of these properties

{
  "id": 1,
  "brand": "Ferrari",
  "model": "488 GTB",
  "year": 2019
}

brand, model and year are mandatory; the vehicle’s year of production can’t be earlier than 1950 for business reasons.

For now we’re not interested in other endpoints.

You can find the implementation of this small project in this public repository of mine; this way, if you get lost, you can pick up the thread of the discussion by reading the code directly.

Let’s create the project

Now that we have a clear idea of what we need to do, let’s generate the project. The best way to do this is to go to the Spring Initializr website. From here we can decide which libraries to integrate, the Java version, etc.

The configuration I chose is the following:

Project: Gradle Project. You’ve probably already heard of Maven as a dependency manager and build tool for Java. If instead you develop on Android, with Gradle you’ll feel at home, since it’s the one used by default.
Language: Java.
Spring Boot: 2.1.6.
Project Metadata:
- Group: it.mattianatali.
- Artifact: tdd-spring-boot-api.
- Name: tdd-spring-boot-api.
- Description: Sample project to explain how to develop APIs in TDD with Spring Boot.
- Package Name: it.mattianatali.tdd-spring-boot-api.
- Packaging: Jar.
- Java: 12.
Dependencies:
- Spring Web Starter: provides the various annotations to create RESTful APIs simply and quickly.
- Lombok: it’s a Java library that lets us reduce the verbosity of our Java code.
- Spring Data JPA: provides everything needed for data persistence.
- H2 Database: provides an in-memory DB. In production it would be appropriate to use a PostgreSQL or MySQL DB, but to avoid getting too bogged down in DB configuration, I preferred to use H2.

Let’s press “Generate the project”. We end up with a nice zip file. Let’s extract it into a folder of our choosing. Let’s open the project; I use IntelliJ.

Let’s enable lombok and the annotation processor in IntelliJ

If it’s the first time you’re using Lombok, you need the “Lombok” Plugin. Open “Preferences”, go to “Plugins”, search for “Lombok” in the marketplace and install it.

Once the project is open in IntelliJ, we need to enable the annotation processor. To do this open “Preferences”, search for “annotation” and check the “Enable annotation processing” option.

Let’s load JUnit 5

Unfortunately the test dependency org.springframework.boot:spring-boot-starter-test, which we can find in the build.gradle file, still uses JUnit 4. Since later on we’ll use some advanced features, we need JUnit 5.

To achieve the goal, let’s add the JUnit 5 dependencies to the build.gradle file.

dependencies {
  ...
  testImplementation 'org.junit.jupiter:junit-jupiter-api'
  testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine'
  testImplementation 'org.junit.jupiter:junit-jupiter-params'
  testImplementation 'org.mockito:mockito-junit-jupiter'
}

Finally we need to “tell” Gradle to use JUnit 5 during the ./gradlew test command and to show the full logs. This makes it easier to debug if there are problems. Again in the build.gradle file let’s add at the bottom

test {
 useJUnitPlatform()
 testLogging {
  exceptionFormat = 'full'
 }
}

In our project there’s already a test file written in JUnit 4. The file is located at src/test/java/it/mattianatali/tddspringbootapi/TddSpringBootApiApplicationTests.java. This test simply tries to load the Spring context. It may seem like a trivial test, but as the project grows, it takes little to forget some annotation and therefore have Spring not load correctly.

Our first step is to migrate it to JUnit 5

It should be written like this:

package it.mattianatali.tddspringbootapi;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

@RunWith(SpringRunner.class)
@SpringBootTest
public class TddSpringBootApiApplicationTests {

	@Test
	public void contextLoads() {
	}

}

We modify it like this:

package it.mattianatali.tddspringbootapi;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit.jupiter.SpringExtension;

@ExtendWith(SpringExtension.class)
@SpringBootTest
class TddSpringBootApiApplicationTests {

	@Test
	void contextLoads() {
	}

}

Did you notice any differences? To migrate a test from JUnit 4 to JUnit 5 you need to be careful about these things:

Change the @RunWith(SpringRunner.class) annotation to @ExtendWith(SpringExtension.class).
Be sure to always have junit.jupiter in the import of all JUnit classes. In fact, if you read jupiter it means you’re using JUnit 5. We can notice that the @Test annotation changed package… Remember you must use junit.jupiter!
Test classes and methods don’t necessarily have to be public. If you use IntelliJ it will complain about this.

Now let’s try to see if our test suite is configured correctly. To do this let’s create a temporary test that will surely fail.

...
import static org.junit.jupiter.api.Assertions.fail;
...
class TddSpringBootApiApplicationTests {
  ...

  @Test
	public void shouldFail() {
        fail("You shall not pass!");
	}
}

The ellipses ... in the code mean there’s more in the file which I haven’t written to avoid distractions.

Let’s try running the tests: we use IntelliJ or Gradle with the ./gradlew test command. We expect one test to pass and the other to fail.

The test output is the following


> Task :test FAILED

it.mattianatali.tddspringbootapi.TddSpringBootApiApplicationTests > shouldFail() FAILED
    org.opentest4j.AssertionFailedError: You shall not pass!
        at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
        at org.junit.jupiter.api.Assertions.fail(Assertions.java:84)
        at it.mattianatali.tddspringbootapi.TddSpringBootApiApplicationTests.shouldFail(TddSpringBootApiApplicationTests.java:20)
2019-07-20 12:56:43.465  INFO 13515 --- [       Thread-4] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'
2019-07-20 12:56:43.466  INFO 13515 --- [       Thread-4] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2019-07-20 12:56:43.466  INFO 13515 --- [       Thread-4] .SchemaDropperImpl$DelayedDropActionImpl : HHH000477: Starting delayed evictData of schema as part of SessionFactory shut-down'
2019-07-20 12:56:43.468  INFO 13515 --- [       Thread-4] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2019-07-20 12:56:43.472  INFO 13515 --- [       Thread-4] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.

2 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///Users/mattia/Repositories/tdd-spring-boot-api/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 5s
4 actionable tasks: 2 executed, 2 up-to-date

That’s exactly what we expected! Now let’s remove the failing test and we can be sure that JUnit 5 is configured perfectly!

How to structure the backend

Before we start writing code we need to be aligned on how we’re going to architect our software. Usually, the bare minimum to create a backend that exposes RESTful APIs is to implement three types of classes:

Controller: its only purpose is to validate the input values, declare the endpoints we want to expose, invoke a service and provide the response given by the service class.
Service: inside it we have the real business logic that depends on the project; it does the dirty work and, if it needs to persist some data, it invokes the repositories.
Repository: it exists to persist data somewhere, usually in a DB.

Of course you can do everything in one class, but if we want to structure good, testable and reusable code, this is the minimum. We must always remember that the more a class has a well-defined purpose, the simpler it is to implement but above all the easier it is to reuse in other places.

Let’s implement the GET call

Let’s start! The GET - /api/v1/vehicles/{id} call, we said, must provide the vehicle details. To develop it I imagine being the HTTP request. The first class affected by the call is the controller; as we said, its function is to validate the input values and expose the endpoint we’re going to call. Let’s create the controller. Let’s create the file src/main/java/it/mattianatali/tddspringbootapi/vehicle/VehicleController.java.

package it.mattianatali.tddspringbootapi.vehicle;

public class VehicleController {
}

For now this is enough; we’re developing in TDD so our implementation must be driven by tests. With IntelliJ we can press cmd + shift + T, select “Create New Test…” and confirm. This way we find the test file created specifically for our controller.

Let’s now think about what our controller must do to achieve the goal:

Declare that the endpoint GET - /api/v1/vehicles/{id} exists.
Receive the vehicle id from the call path.
Call the VehicleService service, which doesn’t exist yet, with the id taken from the previous point.
If VehicleService returns a vehicle, then respond with a 200 - Ok as statusCode and the vehicle in the body, otherwise respond with a nice 404 - Not Found.

Simplified version

Since we’re new to TDD, it’s really hard to write such a complete test on the first try. So let’s try to proceed in small steps. Let’s test a simplified version.

Our first test will only verify this, then little by little we’ll add the remaining points:

Verify that the endpoint GET - /api/v1/vehicles/{id} exists.
It returns 200 - Ok as statusCode.

Our test is the following:

package it.mattianatali.tddspringbootapi.vehicle;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.test.web.servlet.MockMvc;

import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@ExtendWith(SpringExtension.class)
@WebMvcTest(VehicleController.class)
class VehicleControllerTest {

    @Autowired
    private MockMvc mvc;

    @Test
    void getDetail_shouldReturn200() throws Exception {
        var vehicleId = 1;
        mvc.perform(get("/api/v1/vehicles/" + vehicleId))
                .andExpect(status().isOk());
    }
}

Now let’s explain point by point the new things we’ve never seen:

@ExtendWith(SpringExtension.class) is used to connect the JUnit context with the Spring Boot framework. This way we can use the annotations we commonly use with Spring Boot plus other annotations that exist specifically to make life easier during testing, like the next one.
@WebMvcTest(VehicleController.class) is our way of telling JUnit 5 + Spring Boot: “I want to test only the part dedicated to the VehicleController controller, so avoid loading the whole Spring Boot context”. In other words our test will start faster because our test knows well that it doesn’t have to load for example Hibernate, load DB drivers, configurations, services, repositories or other things, because we don’t need them. We’re testing only that controller. If instead we want to test something by loading the whole Spring Boot context we can use the @SpringBootTest annotation. But it’s always better to avoid it so as not to slow down test loading and execution.
@Autowired private MockMvc mvc means “Spring Boot: provide me with a way to communicate with the controller I defined in the previous annotation (VehicleController), which lets me make HTTP calls”. @Autowired means that Spring Boot provides it to us; we don’t have to worry about how it instantiates it for us. MockMvc mvc is the instance provided by Spring Boot that we’ll use to make HTTP calls.

I usually adopt this convention for test names: methodName_whatWeAreTesting. Don’t worry about writing a name that’s too long; you’ll never rewrite the method name. Here it’s important to be very clear about what we’re testing and what we expect. Whenever you program, put yourself in the shoes of another developer who knows nothing about what you’re doing; try to make life easier for the next person: in this case, the clearer we are, the happier our colleagues will be with our work.

mvc.perform(get("api/v1/vehicles/" + vehicleId)) is our way of making a GET call to the endpoint we’re testing.
.andExpect(status().isOk()) is our assertion on the statusCode of our HTTP call; we expect a nice 200 - Ok.

Let’s run the test; we expect it to fail and to return a 404 - Not Found instead of the 200 statusCode, because the endpoint hasn’t been declared yet. With IntelliJ it’s very easy to test a single test; just click the green arrow next to the method name.

This is the result of our test:

2019-07-20 16:33:15.748  WARN 14736 --- [    Test worker] o.s.web.servlet.PageNotFound             : No mapping for GET api/v1/vehicles/1

...
Status expected:<200> but was:<404>
Expected :200
Actual   :404
<Click to see difference>

java.lang.AssertionError: Status expected:<200> but was:<404>
	...

Exactly as we expected.

Now let’s implement the bare minimum to make the test pass. Let’s write in VehicleController

package it.mattianatali.tddspringbootapi.vehicle;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class VehicleController {
    @GetMapping("/api/v1/vehicles/{id}")
    void getVehicleDetails(@PathVariable Long id) {

    }
}

@RestController is the Spring Boot annotation used to declare that we’re creating a Controller.
@GetMapping("/api/v1/vehicles/{id}") we declare the existence of our endpoint with the GET method and that it has a variable in the path called id.
@PathVariable Long id is used to extract the id from the path.

As we can notice, we don’t return anything from the endpoint… In fact we haven’t put any assertion in the test, so there’s no need to put anything else. For now we just want to test that it responds with statusCode 200. Which is the default value if we don’t insert anything.

Let’s run the test, wait a few seconds, and this time it’s green! We can easily guess that our test isn’t sufficient; now let’s improve it so as to test at least the complete “happy path”.

Complete version

Let’s modify the test to make it complete:

package it.mattianatali.tddspringbootapi.vehicle;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.test.web.servlet.MockMvc;

import java.util.Optional;

import static org.hamcrest.Matchers.is;
import static org.mockito.Mockito.when;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@ExtendWith(SpringExtension.class)
@WebMvcTest(VehicleController.class)
class VehicleControllerTest {

    @Autowired
    private MockMvc mvc;

    @MockBean
    private VehicleService vehicleService;

    @Test
    void getDetail_shouldReturnVehicleIfFound() throws Exception {
        var vehicleId = 1L;
        var returnedVehicle = Vehicle.builder()
                .id(vehicleId)
                .brand("Ferrari")
                .model("488 GTB")
                .year(2019)
                .build();

        when(vehicleService.get(vehicleId)).thenReturn(Optional.of(returnedVehicle));

        mvc.perform(get("/api/v1/vehicles/" + vehicleId))
                .andExpect(status().isOk())
                .andExpect(jsonPath("$.id", is(returnedVehicle.getId().intValue())))
                .andExpect(jsonPath("$.brand", is(returnedVehicle.getBrand())))
                .andExpect(jsonPath("$.model", is(returnedVehicle.getModel())))
                .andExpect(jsonPath("$.year", is(returnedVehicle.getYear())));
    }
}

Right now our compiler is in despair because it can’t find VehicleService nor the Vehicle class… In fact we haven’t created them yet!

In this case too let’s see what’s new:

@MockBean private VehicleService vehicleService we’re telling Spring Boot: “If some class asks you for the VehicleService service, don’t pass the real implementation, but provide a fake instance, where I can decide what to return when invoking methods”. In fact later on with the when method we can decide what to return as a value, without calling the real implementation. The topic of what it means to “Mock” classes has already been covered in my previous article; if it’s not clear to you, it’s better that you read the previous episode.
var returnedVehicle = Vehicle.builder()....build() we’re simply creating a vehicle model that will be returned as output by VehicleService and consequently by the controller. In this case we’ll use the builder that Lombok provides us; don’t worry for now, when we create the class it will be clearer.
when(vehicleService.get(vehicleId)).thenReturn(Optional.of(returnedVehicle)); we’re saying “if someone invokes vehicleService.get with the id vehicleId, then return an optional with our newly created vehicle”.
.andExpect(jsonPath("$.id", is(returnedVehicle.getId()))) means that we expect in the response body a JSON containing an id field, with the id of our created vehicle inside. The other lines are very similar; only the field we’re checking changes.

We can’t run the tests because the code doesn’t even compile. So let’s create the missing classes. With IntelliJ it’s very simple to create classes or methods that don’t exist yet; just move the cursor over the non-existent class, press alt + Enter, click on ”Create class …”, choose main and not test as the target directory, and confirm.

Let’s do this procedure with Vehicle. Then let’s implement it. The file should be located at /src/main/java/it/mattianatali/tddspringbootapi/vehicle/Vehicle.java

package it.mattianatali.tddspringbootapi.vehicle;

import lombok.Builder;
import lombok.Data;

@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder(toBuilder = true)
public class Vehicle {
    private Long id;
    private String brand;
    private String model;
    private Integer year;
}

Here we can see Lombok in action! Thanks to the @Data annotation we don’t have to write getters or setters or override equals or hash; it does everything! With @Builder(toBuilder = true) instead we get for free a handy builder that we already used in the test. toBuilder = true provides us with a handy method on the instance that lets us get a builder from the instance itself, very handy for creating for example a clone by modifying maybe one of its fields. @NoArgsConstructor and @AllArgsConstructor are two more Lombok annotations that provide us with both the constructor with all parameters and the empty constructor. They’re very useful for serializing instances to JSON. In fact if we don’t put these two annotations, Jackson, which is the library Spring Boot uses to create response JSONs, will complain because it can’t serialize the object. Less code to write means less code to manage and test!

Now we need to create VehicleService. In this case let’s write the bare minimum to be able to compile; we must remember that in the test we use the mocked version, so the real implementation doesn’t interest us now. We’ll worry about it when we develop in TDD on it. Here too we can let IntelliJ help us.

Once the class is created, we can tell IntelliJ to create the VehicleService.get method we invoked in the test line when(vehicleService.get(vehicleId)). alt + enter is our best friend in this case.

The file src/main/java/it/mattianatali/tddspringbootapi/vehicle/VehicleService.java will therefore contain

package it.mattianatali.tddspringbootapi.vehicle;

import org.springframework.stereotype.Service;

import java.util.Optional;

@Service
public class VehicleService {
    public Optional<Vehicle> get(long vehicleId) {
        return null;
    }
}

@Service is used to say that the class is a service and that it can be created by Spring Boot through Dependency Injection (DI) and that it can be injected into other classes when they use the @Autowired annotation.

The method is obviously wrong, but I repeat, now we only care that it compiles so as to run the tests.

Here there’s a very interesting thing to notice: we defined the method signature first before thinking about its actual implementation. In this we define classes from the point of view of the user and not of the creator. Usually this is a good thing because it creates method signatures that are easy to use. If we reverse the procedure we end up simplifying the implementation but making it hard to use the created classes because we don’t think about how a possible user could exploit these classes.

Running the test we get the following error

...
java.lang.AssertionError: No value at JSON path "$.id"
...
Caused by: java.lang.IllegalArgumentException: json can not be null or empty
...

Now it’s complaining that our endpoint doesn’t return any JSON and therefore it can’t test the various properties. Perfect, let’s implement it to make the test pass.

Let’s modify our VehicleController

package it.mattianatali.tddspringbootapi.vehicle;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class VehicleController {
    private VehicleService vehicleService;

    @Autowired
    public VehicleController(VehicleService vehicleService) {
        this.vehicleService = vehicleService;
    }

    @GetMapping("/api/v1/vehicles/{id}")
    Vehicle getVehicleDetails(@PathVariable Long id) {
        return vehicleService.get(id).get();
    }
}

We can see for the first time how we can exploit Spring Boot’s Dependency Injection. We can ask Spring Boot to provide us with instances of classes we need to perform our task (in this case retrieving the vehicle), without us having to actually worry about how these instances are created. In other words, through the @Autowired annotation we’re telling Spring Boot to provide us in the constructor with all the instances we need, delegating to the framework how to create them.

This way we can create classes that are decoupled from each other and above all in tests we can inject fake (mocked) instances. By injecting mocked classes in tests we can drive the execution flow without taking our eyes off the class under test. Because in fact the only real implementation tested is the controller; the rest is all fiction (the fiction is given by the @MockBean annotation and the when().thenReturn() method).

We must in fact imagine that during the test, the controller receives our mocked instance in the constructor, so it doesn’t touch the real implementation of VehicleService (which for now doesn’t exist yet… it always returns null!).

Maybe you’re wondering why there’s a double get in vehicleService.get(id).get(). The first is the invocation of our VehicleService method; the second is used to extract the value contained in Optional<Vehicle>. It’s strongly discouraged to call get on an optional because we’re not handling the case where the optional is empty, in other words when the vehicle doesn’t exist. But for now our goal is to make the test pass; then when we write the test that tests the case where the vehicle doesn’t exist, we’ll fix it.

Running the test we see that it actually works! We return the JSON with all the fields.

Let’s test the vehicle-absent case

We’ve tested when the vehicle is returned by VehicleService, but what happens when the vehicle with a certain id doesn’t exist? We expect the endpoint to return 404 - Not Found.

Let’s create the test that verifies this case; in VehicleControllerTest let’s write:

@Test
void getDetail_shouldReturn404IfVehicleNotFound() throws Exception {
    var vehicleId = 1L;

    when(vehicleService.get(vehicleId)).thenReturn(Optional.empty());

    mvc.perform(get("/api/v1/vehicles/" + vehicleId))
            .andExpect(status().isNotFound());
}

Running the test it explodes like this


org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.util.NoSuchElementException: No value present

	...
Caused by: java.util.NoSuchElementException: No value present
  ...
	... 57 more

In practice it’s our famous get on the optional that we put in the controller. We’re trying to retrieve data that doesn’t exist and therefore it throws an exception.

Let’s modify the controller so that it returns a 404 - Not Found status code when the service doesn’t return any vehicle. To respond 404 we need an exception that generates that status code. So let’s create the exception in src/main/java/it/mattianatali/tddspringbootapi/vehicle/errors/VehicleNotFoundException.java and write

package it.mattianatali.tddspringbootapi.vehicle.errors;

import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.ResponseStatus;

@ResponseStatus(HttpStatus.NOT_FOUND)
public class VehicleNotFoundException extends RuntimeException {
}

Thanks to @ResponseStatus(HttpStatus.NOT_FOUND) we’re sure that the status code of the call will be 404 - Not Found.

Now let’s modify the VehicleController.getVehicleDetails method using this new exception

@GetMapping("/api/v1/vehicles/{id}")
Vehicle getVehicleDetails(@PathVariable Long id) {
    return vehicleService
            .get(id)
            .orElseThrow(VehicleNotFoundException::new);
}

In practice we replaced the .get method with .orElseThrow(VehicleNotFoundException::new). This is the way to generate an exception in case of a missing value in the optional.

Let’s run the test and now everything works!

Let’s recap

Let’s stop for a second to take stock. The work to finish the GET endpoint isn’t done yet. But now it’s important to focus on what we’ve learned and done so far:

We created a new Spring Boot project using Spring Initializr.
We migrated the test suite to JUnit 5.
We learned to use some features of Lombok.
We developed the controller for the GET call in TDD.
We learned to exploit Spring Boot’s Dependency Injection and @Autowired annotation to test the controller in isolation, even before implementing the VehicleService service.
We learned to make fake HTTP calls using @WebMvcTest.

Let’s take a break; all the concepts just exposed need a moment to be assimilated. Then we’ll continue developing the two remaining components: the VehicleService service and the VehicleRepository repository.

Let’s implement the VehicleService service

Now that we’ve regenerated a bit, let’s do a fairly simple job: the implementation of VehicleService. All the business logic should go in this class, but the logic that the GET call has is really trivial: retrieve the vehicle given its id. Don’t worry, in real projects the business logic always becomes as complex as you like. We’re just lucky because we’re at the beginning. In other words, in our case we must test that VehicleRepository is invoked to retrieve the vehicle, and the return value must be exactly the same as we get from VehicleRepository. VehicleRepository doesn’t exist yet, so we need to think about the method that suits our needs. Let’s not reinvent anything, so let’s imagine using the VehicleRepository.findById(Long id) method that JpaRepository gives us for free.

Let’s write the tests in the new file src/test/java/it/mattianatali/tddspringbootapi/vehicle/VehicleServiceTest.java:

package it.mattianatali.tddspringbootapi.vehicle;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;

import java.util.Optional;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.when;

@ExtendWith(MockitoExtension.class)
class VehicleServiceTest {
    @Mock
    private VehicleRepository vehicleRepository;

    @InjectMocks
    private VehicleService vehicleService;

    @Test
    void get_shouldReturnVehicle() {
        var vehicleId = 1L;
        var returnedVehicle = Optional.of(
                Vehicle.builder()
                        .id(vehicleId)
                        .brand("Ferrari")
                        .model("488 GTB")
                        .year(2019)
                        .build()
        );
        when(vehicleRepository.findById(vehicleId))
                .thenReturn(returnedVehicle);

        assertEquals(
                returnedVehicle,
                vehicleService.get(vehicleId)
        );

    }

    @Test
    void get_shouldReturnOptionalEmptyIfNotFound() {
        var vehicleId = 1L;
        when(vehicleRepository.findById(vehicleId))
                .thenReturn(Optional.empty());

        assertTrue(vehicleService.get(vehicleId).isEmpty());
    }
}

But let’s notice some differences in the annotations used; we’re referring to

@ExtendWith(MockitoExtension.class) instead of @ExtendWith(SpringExtension.class).
@Mock instead of @MockBean.
@InjectMocks is new.

Why do we have these differences compared to our controller test? Because if we think about it carefully we don’t need to load any Spring Boot component (in the controller we needed Spring Boot’s MockMvc to simulate HTTP calls), so we can use only the Mockito library to test our service. This means having faster tests to start because we don’t have to load the Spring Boot context. The annotations for using Mockito are precisely

@ExtendWith(MockitoExtension.class) to start the Mockito context and its annotations during the test.
@Mock to create a mocked instance whose responses we can decide, through the when(...).thenReturn(...) method during tests.
@InjectMocks to inject the mocked instances into our service under test (in our case VehicleService).

The rest shouldn’t surprise you too much; very normal tests organized into setup, call, assertions.

Since the compiler is complaining that VehicleRepository doesn’t exist, let’s create it without implementing the method. This way we can run the VehicleService test. In src/main/java/it/mattianatali/tddspringbootapi/vehicle/VehicleRepository.java let’s write:

package it.mattianatali.tddspringbootapi.vehicle;

import org.springframework.stereotype.Repository;

import java.util.Optional;

@Repository
public class VehicleRepository {
    public Optional<Vehicle> findById(long vehicleId) {
        return null;
    }
}

Now it compiles! Let’s run our service tests and of course they fail. Let’s implement the service, which will be very simple; it does nothing but act as a “proxy” to our future VehicleRepository.

package it.mattianatali.tddspringbootapi.vehicle;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.Optional;

@Service
public class VehicleService {
    private VehicleRepository vehicleRepository;

    @Autowired
    public VehicleService(VehicleRepository vehicleRepository) {
        this.vehicleRepository = vehicleRepository;
    }

    public Optional<Vehicle> get(long vehicleId) {
        return vehicleRepository.findById(vehicleId);
    }
}

Here too nothing exceptional. We take VehicleRepository via DI in the constructor and use it to provide the vehicle in the get call. Let’s run the tests and they’re green!

Let’s implement the VehicleRepository

We’re almost at the end; we only need VehicleRepository to finish the GET call. Here we need to test the findById method to retrieve the vehicle; to do a complete test we must also be able to save a vehicle before retrieving it.

Let’s write the test in src/test/java/it/mattianatali/tddspringbootapi/vehicle/VehicleRepositoryTest.java

package it.mattianatali.tddspringbootapi.vehicle;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.orm.jpa.DataJpaTest;
import org.springframework.test.context.junit.jupiter.SpringExtension;

import java.util.Optional;

import static org.junit.jupiter.api.Assertions.assertEquals;

@ExtendWith(SpringExtension.class)
@DataJpaTest
class VehicleRepositoryTest {
    @Autowired
    private VehicleRepository vehicleRepository;

    @Test
    void findById_shouldGetVehicle() {
        var vehicleToSave =
                Vehicle.builder()
                        .brand("Ferrari")
                        .model("488 GTB")
                        .year(2019)
                        .build();
        var expectedVehicle = vehicleRepository.save(vehicleToSave);
        assertEquals(
                Optional.of(expectedVehicle),
                vehicleRepository.findById(expectedVehicle.getId())
        );
    }
}

With this test we save a vehicle, then retrieve it with the id returned to us in save, and then verify that findById returns the vehicle. But let’s look at the annotations we used: they’re practically the same as we used for the controller, but this time instead of @WebMvcTest we used @DataJpaTest, which is used to test data persistence. We need to use the Spring Boot context because JPA requires it; consequently we can’t use only Mockito as we did with VehicleService.

In this case too it doesn’t compile, because we don’t have the save method declared in VehicleRepository. Let’s fix the issue intelligently. Let’s transform VehicleRepository like this:

package it.mattianatali.tddspringbootapi.vehicle;

import org.springframework.data.repository.CrudRepository;
import org.springframework.stereotype.Repository;

@Repository
public interface VehicleRepository extends CrudRepository<Vehicle, Long> {
}

We transformed VehicleRepository into an interface that extends CrudRepository; in this case we use a bit of magic offered by JPA. CrudRepository gives us for free the save and findById methods we’re using.

Now it compiles! Let’s try starting the Repository test, but unfortunately it fails like this:

java.lang.IllegalStateException: Failed to load ApplicationContext
	...
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vehicleRepository': Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Not a managed type: class it.mattianatali.tddspringbootapi.vehicle.Vehicle
	...
Caused by: java.lang.IllegalArgumentException: Not a managed type: class it.mattianatali.tddspringbootapi.vehicle.Vehicle
	...

What Spring Boot is trying to tell us is that the Vehicle model is missing the annotations needed to use it together with CrudRepository; in fact, if we think about it, nobody said which is the id field of our vehicle and how to generate it. Poor JPA only knows that the Vehicle model exists and nothing else.

Let’s fix the problem by modifying our Vehicle file:

package it.mattianatali.tddspringbootapi.vehicle;

import lombok.Builder;
import lombok.Data;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder(toBuilder = true)
@Entity
public class Vehicle {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String brand;
    private String model;
    private Integer year;
}

We added three annotations:

@Entity is used by JPA to understand that the Vehicle class can be used as a model for data persistence in the DB.
@Id we’re indicating the Long id field as the id of our table in the DB.
@GeneratedValue(strategy = GenerationType.IDENTITY) we’re telling JPA to delegate id generation to the DB if it isn’t specified.

Let’s start the test and this time it works!

Test in case of missing vehicle

Let’s add a simple test to verify that an Optional.empty() is returned if the vehicle doesn’t exist.

@Test
void findById_shouldReturnEmptyIfVehicleNotFound() {
    assertEquals(
            Optional.empty(),
            vehicleRepository.findById(999999L)
    );
}

We can rest assured that during the test there’s no value with that id because the @DataJpaTest annotation guarantees us that every time we run the test the DB is clean. Running it we see that the test passes as we expected; we don’t need to implement anything else.

We must remember that DB setup is usually much more demanding: there are migrations, explicit table creation, calls to a remote DB, etc. We’re getting off easily with the H2 DB we inserted during project creation. Unfortunately we won’t cover these topics together so as not to lose focus on TDD.

Tests on repositories

In our case, because of this simplification, we’re testing on an in-memory DB created specifically for tests, all thanks to the @DataJpaTest annotation. Unfortunately, by doing so, tests on repositories lose most of their effectiveness. But now let’s try to imagine using this modus operandi on a real project.

At Pirelli we test repositories on a DB very similar to the real one. In this case the advantages of testing repos are many, even though the implementation is done “magically” by Spring. For example by testing a trivial auto-generated .save():

We can test that all the model’s properties are mapped to the DB.
We can verify that migrations truly reflect the code.
We can verify that there are no typos in the enums that are mapped in the DB, or that all enums are present on the DB side too.
We can verify that the JDBC driver is able to map all the properties present on the model (e.g. old drivers might not map Java 8 Date classes correctly).
We can verify that relationships are saved as we expect.

Then the repository can grow and we can create new methods auto-generated by Spring, but here too:

We can test that relationships are retrieved correctly.
We can verify that there are no logic errors in the auto-generated method (it takes a moment to confuse one id with another or forget a Not in the method).

Finally we mustn’t fool ourselves into thinking that the implementation is simple (the .save() method is auto-generated!) and therefore the test is useless: tests guide our development in TDD. So first we test that the functionality doesn’t exist, then we implement it, and finally the test verifies that it exists and remains so for all future developments: the .save() method can become complicated; maybe it will no longer be auto-generated because ad hoc logic has been added. Our test will always be there to give us this guarantee. If we don’t do it at the beginning, we miss this opportunity and we might have regrets later.

GET call completed!

Finally we have the GET call completely developed in TDD! We have 3 well-isolated components, easy to test, with 100% coverage. It seems like it was long and difficult, but only because we explained every single line of code and also because we explained every single step we took. The good news is that through this process we can create any endpoint we can think of: we can complicate the business logic as much as we want, but the TDD procedure will always remain the same.

Now let’s implement the POST call, but we’ll explain with the same level of detail only the controller, where I can share some tricks with you that I haven’t covered yet. The development of the methods in the service and the repository will be very similar to what we did for the GET call, so it would be of little use to repeat the same things.

Let’s implement the controller for the POST call

Let’s now move on to developing the POST call to create new vehicles. As always let’s start with the tests. The first thing we verify is that our vehicle passed in the body is correctly digested by Spring Boot and that it’s passed to the VehicleService.save(vehicle) method to save it.

...

@ExtendWith(SpringExtension.class)
@WebMvcTest(VehicleController.class)
class VehicleControllerTest {
    @Autowired
    private MockMvc mvc;

    @Autowired
    private ObjectMapper objectMapper;

    @MockBean
    private VehicleService vehicleService;

    ...

    @Test
    void create_shouldSaveTheVehicle() throws Exception {
        var vehicleId = 1L;
        var vehicleToSave = aValidVehicle();

        when(vehicleService.save(vehicleToSave))
                .thenReturn(
                        vehicleToSave
                                .toBuilder()
                                .id(vehicleId)
                                .build()
                );

        mvc.perform(
                post("/api/v1/vehicles")
                .contentType(MediaType.APPLICATION_JSON)
                .content(objectMapper.writeValueAsBytes(vehicleToSave))
        )
                .andExpect(status().isCreated())
                .andExpect(jsonPath("$.id", is((int) vehicleId)))
                .andExpect(jsonPath("$.brand", is(vehicleToSave.getBrand())))
                .andExpect(jsonPath("$.model", is(vehicleToSave.getModel())))
                .andExpect(jsonPath("$.year", is(vehicleToSave.getYear())));
    }

    private static Vehicle aValidVehicle() {
        return Vehicle.builder()
                .brand("Ferrari")
                .model("488 GTB")
                .year(2019)
                .build();
    }
}

As we can read, we’re emulating a POST call, we correctly set the contentType of the call via .contentType(MediaType.APPLICATION_JSON) and we insert our vehicle to save in the body via .content(objectMapper.writeValueAsBytes(vehicleToSave)).

With

when(vehicleService.save(vehicleToSave))
    .thenReturn(
            vehicleToSave
                    .toBuilder()
                    .id(vehicleId)
                    .build()
    );

we emulate the fact that our service returns the same object passed as input but with the id populated. For this case we used the toBuilder method kindly offered by the @Builder(toBuilder = true) annotation we had added to the Vehicle class.

Finally we test that the HTTP call responds with the 201 - Created statusCode and that in the body we have all the fields populated.

objectMapper is Spring Boot’s default serializer, Jackson. We ask Spring Boot for an instance of it via @Autowired.

The compiler is complaining that the save method doesn’t exist on VehicleService; let’s add it without implementing the correct logic.

@Service
public class VehicleService {
    private VehicleRepository vehicleRepository;

    @Autowired
    public VehicleService(VehicleRepository vehicleRepository) {
        this.vehicleRepository = vehicleRepository;
    }

    ...

    public Vehicle save(Vehicle vehicleToSave) {
        return null;
    }
}

Let’s start the test and it fails for this reason

...
java.lang.AssertionError: Status expected:<201> but was:<404>
Expected :201
Actual   :404
<Click to see difference>
...

Rightly the endpoint doesn’t exist yet, so let’s add it to VehicleController.

package it.mattianatali.tddspringbootapi.vehicle;

import it.mattianatali.tddspringbootapi.vehicle.errors.VehicleNotFoundException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.*;

@RestController
public class VehicleController {
    private VehicleService vehicleService;

    @Autowired
    public VehicleController(VehicleService vehicleService) {
        this.vehicleService = vehicleService;
    }

    ...

    @PostMapping("/api/v1/vehicles")
    @ResponseStatus(HttpStatus.CREATED)
    Vehicle saveVehicle(@RequestBody Vehicle vehicle) {
        return vehicleService.save(vehicle);
    }
}

@PostMapping is the annotation that lets us define a POST endpoint.
@ResponseStatus(HttpStatus.CREATED) we can define the response status code; by default it’s 200 - Ok, but we want 201 - Created.
@RequestBody is the annotation that lets us take the value passed in the body and deserialize it to Vehicle.

In this case we just need to invoke the vehicleService.save method and return its value.

Running the tests everything is OK!

We’re almost done, but we still need to implement the rule we were given when saving a vehicle. Let me restate it here:

brand, model and year are mandatory; the vehicle’s year of production can’t be earlier than 1950 for business reasons.

Let’s create the test that verifies these conditions.

package it.mattianatali.tddspringbootapi.vehicle;

import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest;
import org.springframework.boot.test.mock.mockito.MockBean;
import org.springframework.http.MediaType;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.test.web.servlet.MockMvc;

import java.util.Optional;
import java.util.stream.Stream;

import static org.hamcrest.Matchers.is;
import static org.mockito.Mockito.when;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.post;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@ExtendWith(SpringExtension.class)
@WebMvcTest(VehicleController.class)
class VehicleControllerTest {
    ...
    @ParameterizedTest
    @MethodSource("getInvalidVehicles")
    void create_shouldReturnBadRequestIfBodyIsInvalid(Vehicle invalidVehicle) throws Exception {
        mvc.perform(
                post("/api/v1/vehicles")
                        .contentType(MediaType.APPLICATION_JSON)
                        .content(objectMapper.writeValueAsBytes(invalidVehicle))
        )
                .andExpect(status().isBadRequest());
    }

    private static Stream<Vehicle> getInvalidVehicles() {
        var validVehicle = aValidVehicle();

        return Stream.of(
                validVehicle.toBuilder().brand(null).build(),
                validVehicle.toBuilder().brand("").build(),
                validVehicle.toBuilder().model(null).build(),
                validVehicle.toBuilder().model("").build(),
                validVehicle.toBuilder().year(null).build(),
                validVehicle.toBuilder().year(1949).build()
        );
    }
}

Here we can see a beautiful JUnit 5 feature: parameterized tests!

Through the @ParametrizedTest annotation we define a test that wants a parameter as input.
@MethodSource("getInvalidVehicles") we define the method that will provide us with the vehicles that are invalid and that we need to test.
The getInvalidVehicles() method returns a stream of invalid vehicles; to create this stream we start from a valid vehicle that we’ve already tested in the other test (aValidVehicle()), then from this we create an invalid vehicle by modifying the parameters. This way we create 6 tests that respectively test all the rules the business has imposed on us, namely:
- The brand parameter can’t be null or an empty string.
- The model parameter can’t be null or an empty string.
- The year parameter can’t be null.
- The year parameter can’t be less than 1950.

Let’s start the test and indeed we see that we have 6 failing tests.

java.lang.AssertionError: Status expected:<400> but was:<201>
Expected :400
Actual   :201

To implement vehicle validation we can exploit Spring Boot’s validation. So let’s add the validation rules to the Vehicle model.

package it.mattianatali.tddspringbootapi.vehicle;

...

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.validation.constraints.Min;
import javax.validation.constraints.NotEmpty;
import javax.validation.constraints.NotNull;

...
public class Vehicle {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    @NotNull
    @NotEmpty
    private String brand;
    @NotNull
    @NotEmpty
    private String model;
    @NotNull
    @Min(1950)
    private Integer year;
}

@NotNull we say that the field can’t be null.
@NotEmpty we say that the field can’t be an empty string.
@Min(1950) means that the field must be at least 1950.

Finally, to enable validation, we need to modify the controller by putting the @Valid annotation in VehicleController:

...
@PostMapping("/api/v1/vehicles")
@ResponseStatus(HttpStatus.CREATED)
Vehicle saveVehicle(@RequestBody @Valid Vehicle vehicle) {
    return vehicleService.save(vehicle);
}
...

Let’s run the tests and now they’re all green! This way we did 6 tests in one go without duplicating code.

Now that we’ve implemented the controller, the vehicleService.save method is still missing; I’ll leave this as homework since it wouldn’t add any value to this article: we practically have to do the same thing we did with the GET call. And the VehicleRepository.save method we’ve already tested when we implemented the GET call. In any case in my repository you can find the complete project.

Conclusions

It’s been a long chat; I really hope that after this reading you’ve learned a few features offered by JUnit 5, Spring Boot, Lombok, but above all that the philosophy of TDD has entered into you.

The thing that must be clear is this: first we worry about what we want to achieve, then we worry about how to achieve it. The what we want to achieve translates into tests, the how translates into the implementation.

This way we have well-tested code, well organized into isolated classes (if a class is easy to test, it means it’s well isolated from the rest of the components) and above all easy to maintain (refactors are simple because every single feature is tested, regressions are practically zero).

At first it can be difficult, but then once internalized it comes naturally and you have an added value that you can’t understand until you have a project developed in TDD in production. Try it to believe it!

See you next time!

How to create a Continuous Integration and Continuous Deployment (CI/CD) pipeline · DevOps

Sat, 16 Mar 2019 07:03:47 GMT

Introduction

One thing I learned at Pirelli is the implementation of pipelines to get Continuous Integration (CI) and Continuous Deployment (CD), from now on I’ll simply call it CI/CD. Many projects with a microservices architecture are always being born: time at work is always short and repetitive work bores me easily… What’s more repetitive than putting all these microservices into production? These automatic pipelines therefore become essential so as not to spend the day doing only this boring job.

I don’t want to bore you too much talking about the various advantages that CI/CD brings; if you’ve come to this post it means you already know, so I’ll start right away with a practical example.

Creating the pipeline for GasLow Backend

As an example I’ll use the backend of a project I created in my free time: GasLow. Basically it’s a backend that’s able to retrieve open data about fuel prices in Italy and provide this information based on your location. The Android app, based on Flutter, that uses this backend is available for free on the Play Store.

The important things for us are these:

The backend is written in Node.js + TypeScript, the database is MongoDB.
The configuration file with the data to connect to the DB is in the .env file, in the project root. Of course the file isn’t published on Git to avoid the whole world knowing the DB credentials. You can see an example of the data contained in this file by reading .env.example.
The “production” server is on Heroku on a free machine. In this guide we’ll be able to automate the deploy to Heroku.
I’m also implementing the AWS serverless version, where it’s deployed using CloudFormation. Unfortunately this part is a bit too much to explain in this introductory guide on CI/CD. Maybe I’ll make a dedicated guide on this topic in the future.

Prerequisites

To fully understand this guide we need these skills:

At least a basic knowledge of Git.
How a Node.js project is structured and the basic commands of its package manager npm. This part actually isn’t mandatory to understand the pipeline as a concept, it’s only needed to understand the pipeline of this specific example.

Let’s choose the CI/CD service

The first important step to automate everything is choosing the service we want to use. The choice is really wide. The ones I know and have used are these:

CircleCI: very powerful and flexible. It’s hosted by them and so we don’t need our own machine to run it. It offers a free plan of 1000 minutes/month for both your public and private projects; not bad for hobby use. For now it’s my first choice for my private projects and it was what I used at Pirelli until a month ago.
GitLab: this is also as powerful as CircleCI. You can use the version hosted by them or install it on your machine. It’s a bit different from the others because it offers a complete package for your project (it has Git integrated so you can use it as a repository too). For now it’s our first choice at Pirelli and in fact we’re doing a nice migration from CircleCI.
Travis CI: maybe it’s a tad behind the first two. For example it’s not yet possible to do a manual deploy, unless you come up with hacks like this. By manual I mean having a button that allows us to run a deploy only after a confirmation. This feature is very useful for me to have full control of when we want to deploy to production. Apart from this small flaw it has a huge advantage: it’s free for all public projects! Precisely for this reason I use it for my public projects (like GasLow). Our guide will be based on Travis CI.
Jenkins: it’s a milestone and the oldest of all. I’ve used it only a few times and honestly it never excited me that much, especially as far as UX is concerned. But this is my personal opinion and therefore quite subjective.

Goal of the guide

In this guide I’d like to experience with you the creation of a pipeline with an already started project that’s already in production. Why do it with an already started project? Because it’s a typical use case you encounter at work and it’s probably your current situation; finally because the web is already full of “hello world” for creating pipelines. But usually they are examples focused on learning the service that offers the pipelines, and not so much on learning the concepts and difficulties encountered in real life.

Let’s make clear what we want to achieve: let’s imagine we’re at work and as the task of the week we’ve been asked to create the CI/CD pipeline for the GasLow project. At the end of the guide we’ll have a pipeline made up of two phases:

Running the tests: a fundamental step to get Continuous Integration (CI).
Deploy to production on the Heroku machine: a step to get Continuous Deployment (CD).

This is the bare minimum to get a complete CI/CD pipeline and to not make the guide too long we’ll limit ourselves to these two steps. Once you’ve become familiar with the topic you can complicate the pipeline as you like. For example a typical pipeline I use at work is made up of the following steps:

Build and unit tests of the application.
Creation of the Docker image through the Dockerfile, push of the image to the AWS ECR repository.
Database migration and deploy of the Docker image in the DEV environment.
Integration tests on the DEV environment.
Database migration and deploy with manual confirmation in the QLT environment used by our QA Team.
Database migration and deploy with manual confirmation in the Production environment.

Signing up to Travis CI

This step is very simple; just go to their home page and press the “SignUp” button, enter your GitHub credentials, and at this point you’ve effectively connected Travis CI to your GitHub. Once you’ve done this you’ll enter the Travis CI main page.

on the left your GitHub projects that use Travis CI as a CI/CD service will appear. At the beginning it will obviously be empty. For now we don’t need to know anything else.

Forking the project

If you want to try it out yourself too, you can fork my project and throw away the .travis.yml file because we’ll recreate it together to form the pipeline we have in mind. The reference commit has the hash d09f1068a14ddcdb857f1ced90b532680b050917, so to be perfectly aligned with this guide we’ll need to check out this specific commit; I don’t rule out that the project may have deep changes over time.

git checkout d09f1068a14ddcdb857f1ced90b532680b050917

Once forked and the file removed we can go to the Travis CI settings (user menu at the top right -> Settings). On this page all our repos should appear, including this newly forked one. If you don’t see anything click on the Sync Account button to synchronize Travis CI. Then we can see the gaslow repo as in the figure; let’s enable the toggle. Now we have the forked repo, Travis CI ready and connected to it. Now the fun part begins: let’s set up the first step of our pipeline.

Implementing the Continuous Integration pipeline (testing phase)

Testing locally

Before going on to create the pipeline let’s ask ourselves this question: how do you run the tests locally? The best way to create an automatic pipeline is to think of all the various steps we have to do locally to achieve the same goal. Here I take for granted that we have a minimum of experience in Node.js and how the npm package manager works.

Someone who breathes Node.js from morning to night has already thought about how to do it: there’s the npm test command… But are we sure that’s enough? In this case we’re taking for granted a few things that for Travis CI won’t be at all.

To run the tests with npm test we’re taking for granted the following points:

Node.js v8 is installed on your machine.
On your machine we have MongoDB installed and active on the standard port :27017.
We’ve installed all the dependencies (through the npm install command) and therefore we have the node_modules folder correctly populated.

So the correct steps to run the tests on a clean machine are the following:

Install Node.js v8.
Install MongoDB and activate it on the default port :27017.
npm install to install the dependencies present in the package.json file and therefore populate the node_modules folder.
npm test to run the tests.

Testing using Travis CI

Now that we’ve understood the local steps, let’s try to translate them into steps that can be digested by Travis CI.

To make Travis CI run the tests automatically we need a means of communicating. The way to communicate with this service is the .travis.yml file, precisely the file I had you delete during the project fork. You may be unfamiliar with the file extension. yml is used for the YAML format which stands for (YAML Ain’t Markup Language). It has become the de-facto standard in the DevOps world because on one hand it’s “expressively” very powerful, on the other it’s very simple for a human to read. This format, however simple it may be to read, has many rules that take some time to master and know all of them. Anyway we don’t have to worry too much: we’ll only use the bare minimum and very basic concepts. I usually use Learn yaml in Y minutes as a cheatsheet for yaml: this site is very concise and at the same time rich in information; give it a look for other languages too, you won’t regret it.

Let’s start by creating the .travis.yml file in the project root. Inside it let’s insert this content:

language: node_js

node_js: '8'

services:
  - mongodb

install: npm install
script: npm test

This way, Travis CI is able to run our tests. Let’s try committing the file; indeed we see that Travis CI has started working and has run the tests.

Now let’s try to see the similarities with the steps we needed locally:

Did we tell Travis CI to install Node.js v8? Yes, through the directive
```
language: node_js

node_js: '8'
```
Did we tell Travis CI to install MongoDB and open Mongo’s standard port? Yes, through the directive
```
services:
  - mongodb
```
Did we install the project dependencies before running the tests? Yes, through running the npm install script during the installation phase of our job, through the directive
```
install: npm install
```
Did we run the tests? Yes, through running the npm test script during the main phase of the job, through the directive
```
script: npm test
```

So we can say we’ve executed the same steps we did locally using a language that Travis CI understands. I don’t want to dwell too much on the language Travis CI uses because it’s very specific to the service we’re using: CircleCI, GitLab would have used a slightly different language. What must come across here is this: Travis CI is like a blank sheet of paper; it’s up to us to instruct it how to run our tests. The best way to instruct it is to first think about the steps we do locally, then redo them in the language of the platform we use.

For those who want to know more about the specific Travis CI language, I’ll leave some links here that go deeper into the topics just covered:

To learn about the supported languages, specifically JavaScript.
To learn how to add new services like MongoDB.
To learn about the various phases of a job.

Deployment using Travis CI

Now that we’ve run the tests we want to deploy our code to Heroku. Heroku, for those who don’t know it, allows you to deploy your code on their PaaS platform.

Before creating the pipeline, we need to create an application on Heroku, fortunately they give you the possibility to host our project for free.

Let’s sign up and create a new app.

Now let’s go back to the .travis.yml file and add at the end:

before_deploy:
  - <MYSTERIOUS_STEP>
  - npm run build
deploy:
  provider: heroku
  api_key:
    secure: '<API_KEY>'
  app: <HEROKU_APP>
  on:
    branch: master
  env:
    - NODE_ENV=production
  skip_cleanup: true

A little explanation of what we’ve added:

before_deploy is the keyword to define what to do right before the deploy phase. The deploy phase is the step within a job that’s dedicated to deploying our code to Heroku. The job, for Travis CI, essentially consists of three phases: install, script and deploy. If you want to go deeper, you can read the various phases that make up a job directly from the Travis CI documentation.
<MYSTERIOUS_STEP> is an essential step for the deploy that I haven’t told you yet; I’ll talk about it later.
npm run build is the script we need to run before the deploy. This script is strictly tied to our project. If you’re curious you can go to the package.json file and see that the build script runs the command parcel build src/server.ts --target node. It does nothing but call Parcel.js to transpile the TypeScript code into JavaScript, bundle it, minify it and put it in the /dist folder. At the end of this project we therefore have the /dist/server.js file ready to be run in production.
deploy is the keyword to define what to do during the deploy phase.
With provider we define that we’re using Heroku as the platform.
api_key is used to define the credentials to actually deploy with Heroku. The next step will be to replace '<API_KEY>' with the real (encrypted) keys.
app: <HEROKU_APP> tells Travis CI which Heroku app we want to deploy to. We need to replace <HEROKU_APP> with the name of the Heroku application we created in the previous step.
With on: branch: master we can deploy only when we push to the master branch. Without this directive every single push to any git branch would trigger the automatic deploy.
With env: we can define the environment variables that our project will have available on Heroku. In our case we added the NODE_ENV=production variable which is a standard in the Node.js world to define that we’re in a production environment.
skip_cleanup: true allows us to avoid deleting the files generated in the previous step. During the test phase we ran the npm install command which populates the node_modules folder that’s essential for the correct functioning of the system. This way we can avoid redoing it and therefore save time in the deploy phase. We must imagine that the deploy phase is separate from the script phase, so we’d only have the source code without this directive.

If we want to know all the various parameters we can use to deploy to Heroku we can read the official documentation.

To recap there are still two things to do:

Understand what we need to do in the MYSTERIOUS_STEP: it has to do with the configuration of our project in production. Here’s a hint: go read the src/util/secrets.ts file.
Retrieve the encrypted Heroku API_KEY so that Travis CI can authenticate and run the deploy.

Understanding the mysterious step

There’s a part I’ve kept hidden from the start and now it’s time to talk about it. We said the project is based on MongoDB, but where does it get the credentials to connect? We ran tests earlier, yet it seems it connected without problems. Even Travis CI, during the tests, connected. We can read the retrieval of the connection information in the src/util/secrets.ts file which I’ll quote here for simplicity:

import logger from "./logger";
import dotenv from "dotenv";
import fs from "fs";

if (fs.existsSync(".env")) {
    logger.debug("Using .env file to supply config environment variables");
    dotenv.config({ path: ".env" });
} else {
    logger.debug("Using .env.example file to supply config environment variables");
    dotenv.config({ path: ".env.example" });  // you can delete this after you create your own .env file!
}
export const ENVIRONMENT = process.env.NODE_ENV;
const prod = ENVIRONMENT === "production"; // Anything else is treated as 'dev'

export const SESSION_SECRET = process.env["SESSION_SECRET"];
export const MONGODB_URI = prod ? process.env["MONGODB_URI"] : process.env["MONGODB_URI_LOCAL"];

if (!SESSION_SECRET) {
    logger.error("No client secret. Set SESSION_SECRET environment variable.");
    process.exit(1);
}

if (!MONGODB_URI) {
    logger.error("No mongo connection string. Set MONGODB_URI environment variable.");
    process.exit(1);
}

In practice the code behaves like this:

It retrieves the .env file; if it’s not there it falls back to the .env.example file.
It retrieves the SESSION_SECRET value from this file.
It retrieves the MONGODB_URI value if you’re in production, otherwise MONGODB_URI_LOCAL.

Finally we can understand how the tests managed to connect to MongoDB without us doing anything: in practice it fell back to the .env.example file which is versioned on Git, read the value MONGODB_URI_LOCAL=mongodb://localhost:27017 and therefore connected to our local MongoDB, while Travis CI connected to the MongoDB service we instantiated in the pipeline.

Our task is to find a way to load the .env file in production securely, versioned, but at the same time without having all the credentials to access our MongoDB written in clear text.

Loading the .env file securely

A solution to the previous problem is given to us directly by Travis CI. For this step we need the Travis CI Command Line Client which on Mac we can install like this:

sudo gem install travis
travis login

Thanks to it we can encrypt files securely and save the encrypted file in the repo without fear that someone can steal our credentials.

The first step is to create the .env configuration file ready for production; we must be careful not to upload it to Git for any reason, it’s not encrypted and if we version it anyone is able to access our DB! If you’re working on my project I’ve already taken care of it. In fact if we go to the .gitignore file we notice that the .env file is in the blacklist of files not to consider with Git.

So let’s create the .env file with the same data you find in the .env.example file

# Get this from https://mlab.com/home after you've logged in and created a database
MONGODB_URI=mongodb://<mlab_user>:<mlab_password>@<mlab_connection_url>

# This is standard running mongodb locally
MONGODB_URI_LOCAL=mongodb://localhost:27017

SESSION_SECRET=<session_secret>

We need to modify two parts:

In MONGODB_URI, replace <mlab_user>, <mlab_password>, <mlab_connection_url> with our DB credentials we want to use in production. I use the mLab service.
Replace <session_secret> with a random alphanumeric string.

Now let’s delete the .env.enc file, where my encrypted keys are, because now we’ll create our own encrypted file. With the values we just entered in the .env file. For this step we use the Travis CI command line.

travis encrypt-file .env

The terminal will tell you to insert a string like this in the .travis.yml file

openssl aes-256-cbc -K $encrypted_0a6446eb3ae3_key -iv $encrypted_0a6446eb3ae3_iv -in .env.enc -out .env -d

this is exactly what we need to write in the mysterious step! So let’s update the .travis.yml file; the before_deploy phase will be like this

before_deploy:
  - openssl aes-256-cbc -K $encrypted_0a6446eb3ae3_key -iv $encrypted_0a6446eb3ae3_iv -in .env.enc -out .env -d
  - npm run build

The command we just added does nothing but decode the encrypted file before deploying.

Getting the encrypted Heroku API key

We’re almost at the end; now we just need to insert the encrypted API key into our pipeline to allow Travis CI to deploy through Heroku.

For this step we need Heroku CLI installed on our PC. If you have a Mac just type in the terminal (if you installed brew)

brew tap heroku/brew && brew install heroku

We authenticate with Heroku using the command

heroku login

and finally we type this command to get the API Key

travis encrypt $(heroku auth:token)

The command will return the API Key; we need to insert it in place of '<API_KEY>' in the .travis.yml file. Once this is done we’ll have finished the work and can enjoy our Continuous Integration and Continuous Deployment pipeline!

Tips for creating pipelines

Now that we’ve seen a real use case I want to share some advice with you. I hope they’ll help you avoid a few headaches that I got instead:

The sooner we create it, the simpler it is

The earlier you create the CI/CD implementation, the more time you save, but above all the simpler it is! Ideally it’s better to do it during project creation. There are many reasons:

During project creation we’re already working on the project’s configuration and dependencies. So everything we’re doing locally to set up our local environment, we just need to do the equivalent in the pipeline. Moreover if we know we’re implementing a pipeline from the start, we avoid creating tests or configurations that are too tied to our local machine, we avoid writing strange configurations that complicate the pipeline precisely because it’s hard for us to replicate them in Travis CI (such as retrieving files that are in folders outside our Git project).
We don’t have to do Reverse Engineering on the project to understand how to configure it: nobody will warn us to find “the mysterious step” in real life and nobody will tell us to look at the src/util/secrets.ts file to get a hint. In this guide I did it to avoid wasting too much time. Without this help we would have had to bang our heads several times and do a minimum of Reverse Engineering of the project to actually understand how the credential retrieval worked. All this could have been avoided if whoever designed this credential retrieval had kept the pipeline up to date. And whoever is new to the project just needs to look at the .travis.yml file to understand exactly the procedure to get a working local version.
Human error in deploys is avoided: having automated it is an advantage because there are no gross errors in the deploy. Imagine you have to run numerous deploys a day… After a while you’ll lower your guard and concentration, the error from carelessness is around the corner and making mistakes in deploys can cost a lot!
You save time: banally, the sooner we create a pipeline the more man-hours we save. Nobody in the company is responsible for the deploy, anyone can do it. I’ve been in companies where the deploy was manual, and I saw a person who was fully occupied just with deploys during the most hectic moments. In my opinion that’s unacceptable. The tools to automate exist and are within everyone’s reach. Let’s use them!

Let’s think about setting up the pipeline locally

Trying directly on Travis CI can take a lot of time. For example if we have problems in the deploy, we have to push the new changes, wait for all the previous steps and then maybe find a nice error after 15 minutes. In this case we try to fix the problem by uploading the new changes and the cycle starts again… All this takes hours. The best way to avoid all this is to try the steps locally. A bit like we did in this guide during the creation of the Continuous Integration pipeline. To achieve this goal we need to create steps that are agnostic with respect to the CI/CD service we use. Usually I create a folder with all the scripts useful for the deploy. This way I can invoke these scripts locally too and test the correct execution faster. Moreover if we keep this point in mind it will also be easier to achieve the next one.

Let’s not focus too much on the single CI/CD service

In this guide we used Travis CI, in my previous job we used Jenkins. In my current job we used CircleCI, now instead we use GitLab CI. In short you’ll end up using all sorts of things, but they all solve the same problem: automating integration and deploy. So let’s try to focus on the basic concepts and not so much on how to do it on a specific service. This last part is simple if we’re clear about what we need to achieve: just read the guide of the specific service. This way of working gives us a lot of flexibility both at work and in our hobby projects. Finally this flexibility makes us free to use a service based on the advantages it brings: in this guide we used Travis CI precisely because it’s free for open source projects, but it’s certainly not the best on the market. This wouldn’t be possible if we used one and only one service.

Let’s read pipelines that are already made

We must remember that we’re not the only ones on the face of the earth thinking about creating a pipeline. So let’s not underestimate the possibility of looking at open source project pipelines! There are numerous projects on GitHub that show practical examples of pipelines that very likely solve all or part of our problems. This way we can also see different ways of reasoning which is always useful for improving our cultural baggage. Finally I remind you that Travis CI is free for open source projects, which is the main reason I decided to write this guide. So GitHub is full of pipelines made with Travis CI!

Let’s not look for perfection

Let’s remember that our job isn’t to spend days making pipelines. It’s always possible to optimize it to speed it up, but like all optimizations they take time. My advice is to reach the bare minimum to achieve the goal, even if maybe it’s not the most elegant solution possible. Then let’s remember that over time we learn new concepts, such as caching and the possibility of running multiple jobs in parallel, we know the system better and so it will be simpler for us to optimize it taking less time. So let’s always keep in mind that the pipeline is a tool that speeds up and automates our work… But whether it takes 5 minutes or 10 minutes doesn’t really change much on our side, so let’s not spend hours or days on these optimizations, especially in the early stages where the challenges are already many.

Conclusions

I hope that with this guide you’ve experienced first-hand what it means to implement a Continuous Integration and Continuous Delivery pipeline on an already existing project. The difficulties, as we’ve seen, are greater if the project is already started. So, the next time you start building an idea of yours, remember to think about the CI/CD pipeline right away; you won’t regret it. This way you can delegate these tasks to a machine and you can focus exclusively on what really brings value to your project: code development.

See you next time!

How to program in Test-Driven Development (TDD) starting from scratch

Fri, 28 Dec 2018 07:03:47 GMT

TDD here, TDD there, everyone talks about it. Many praise its merits, others say it’s a waste of time… But in the end, how do you actually develop in TDD?

In this guide you’ll see a real example developed in Test-Driven Development. I’ve been adopting this programming technique for over a year for the backends I develop in Java, Python and TypeScript. The benefits in the code I’ve written are visible and many. I hope I can convince you that this technique:

Improves code quality.
Decreases the time spent debugging and searching for possible software regressions.
Decreases the number of bugs that manage to reach production.
Simplifies the development of complex applications.
Is simpler than it seems.

How can I convince you of all these things? I hope to succeed through a practical example starting completely from scratch.

Goal

In this guide we’ll develop a class that takes as input a URL and a word to search for in the page we’ve entered. This search will answer us by telling us how many times it found the word.

We’ll build this simple application in Node.Js + TypeScript, entirely in TDD. During this development I’ll explain all the steps and make explicit all the advantages that TDD is introducing.

Prerequisites

To follow the guide you should have:

Node.js installed.
A minimum of experience with Node.js. TypeScript, if you’ve never used it, is JavaScript with types; for now that’s all you need to know.
For the second part you need to know the concept of asynchronous programming in JavaScript: we’ll use Promises and the async and await keywords.
A UNIX system (Linux or Mac). Or having enabled WSL on Windows.

If you happen to get lost, in this GitHub repo you can find the finished product. But I advise you not to “peek” at the solution because we’ll introduce many concepts throughout the article. Seeing the finished project without knowing how we got there brings no benefit to learning.

Creating the project

The first step is obviously to create the project. To do this we open the terminal in a folder we’ll work in. My folder will be called word-counter.

mkdir word-counter
cd word-counter

Let’s create the package.json file where we’ll insert the various Node.js dependencies and scripts.

npm init

Let’s enter the various data it asks for. For now we can just press Enter and keep the default values.

Adding dependencies

Now we can insert the bare minimum needed to start developing in TDD. The dependencies we need are:

TypeScript: to develop with this language.
TSLint: the TypeScript linter that will help us during development to write clean code. This is optional, but I usually always add it to have great help during development. It’s also integrated very well with Visual Studio Code, which is my favorite text editor for the Node.js world.
Jest: our testing platform.

To install all this we write

npm install --save-dev typescript tslint jest ts-jest @types/jest

We can verify that these five dependencies have been added in package.json under devDependencies, that is, they’re dependencies needed for development and not for the program itself. We can also notice that these dependencies have been downloaded into the node_modules folder.

The command should be clear; the only part that maybe needs explanation is the Jest libraries. In fact we installed three dependencies to make it all work.

jest is our testing platform.
ts-jest is the transpiler that transforms TypeScript into JavaScript before running the tests. Without it we’d have to transpile the code manually every time, something we absolutely want to avoid so as not to waste time.
@types/jest is needed by TypeScript to know the types inside the Jest library. Imagine that many times libraries are written in JavaScript, so TypeScript would have no idea what the functions accept as input or return as output. Fortunately there are developers who create libraries that map JavaScript functions to TypeScript. Usually to install these mappings you install the @types/<LIBRARY_TO_MAP> dependency. If you want to know more about these type declarations in TypeScript you can read directly from the DefinitelyTyped site, but for this guide there’s no need to go deeper.

Initialization

Now that we’ve downloaded the libraries, we need to configure them to make them work at their best. Let’s start with TypeScript

npx typescript --init

The command means:

npx: starts a program that you can find in the node_modules folder.
typescript --init: the TypeScript initialization command.

It should have created the tsconfig.json file with the various configuration parameters. Let’s open this file and we should see the // "lib": [], part commented out; let’s uncomment it and write:

"lib": [
  "es2015"
]

This way we can use the async and await keywords when we deal with Promises.

Now let’s move on to TSLint

npx tslint --init

We can actually see that a file named tslint.json has been created in the root. For now it’s fine as it is, let’s not touch it.

Now let’s initialize Jest, our test platform, more or less as we did with TSLint

npx jest --init

It should automatically recognize that TypeScript is present, create the jest.config.js file containing Jest’s default configuration, and finally it added the test script to package.json. This way when we write npm test in the console, Jest starts and runs all the tests.

Here we need to make a small change to the jest.config.js file: we need to rename the field from globals.ts-jest.tsConfigFile to globals.ts-jest.tsConfig to avoid a warning message during tests. So we’ll need to rename that configuration part like this:

globals: {
  "ts-jest": {
    "tsConfig": "tsconfig.json"
  }
}

To be sure we’ve done everything, let’s finally try writing a test: we’ll insert it in the src/__tests__ folder and call it hello-test.ts.

So let’s write in the src/__tests__/hello-test.ts file:

test("should work", () => {
  expect(true).toBe(true);
});

The test we just wrote does nothing but verify that true is actually true. What’s the point? In real life, none, but in our specific case it helps us understand that Jest is configured correctly.

Let’s start the tests with npm test and you should get a result similar to mine.

$ npm test

> word-counter@1.0.0 test /Users/mattia/Repositories/word-counter
> jest

 PASS  src/__tests__/hello-test.ts
  ✓ should work (5ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.553s
Ran all test suites.

Perfect! Our first test in TypeScript worked.

Project analysis

Now we get down to business: let’s go back to our goal and start thinking about how to create the architecture.

To achieve the goal we need three parts:

Collecting the input values (URL and word to search for).
Downloading the web page.
Counting the word in the web page we just downloaded.

Test-Driven Development: here we go!

Where do we start? Obviously with a test! After all, that’s why we’re here, right? To learn Test-Driven Development. How do you start with a test? You need to think about how to create an architecture that can be easily testable, where each test checks only a single aspect of the project.

We’ll use a bottom-up approach to create the project, that is, we’ll start from the simplest, self-contained part that doesn’t need any dependency to work. In this case it’s part 3.

WordCounter class

WordCounter is our class that handles “Counting the word in the web page we just downloaded”. We can also translate it this way: “Given an input string text and a word to search for wordToFind, it returns the number of times wordToFind is contained in text“.

So let’s start from the test: without having this WordCounter class in hand, which doesn’t exist yet, I imagine using it like this:

const text = "downloaded text";
const wordToFind = "text";

const wordCounter = new WordCounter();

wordCounter.count(text, wordToFind); // Output 1

Let’s try to translate this hypothetical usage into a test. In the new file src/__tests__/word-counter.ts we write:

import WordCounter from "../word-counter";

describe("WordCounter", () => {
  const wordCounter = new WordCounter();

  it("should count the word inside text", () => {
    expect(
      wordCounter.count("downloaded text", "text"),
    ).toBe(1);
  });
});

Here we need to know a minimum of how Jest works. In short, we’ll use the convention of grouping tests that refer to a class using the describe function, and then each test of the class will use the it function which is our actual test. The it function consists of a string that briefly explains what we’re testing and a function that contains the test logic. For more details I recommend their official documentation.

Now that we’ve written our first test, we can notice an advantage of writing in TDD: we only care about the interface and not the implementation. In the end, the final user of our classes only cares about how to invoke a certain function, and not how it was made. It often happens in programming that you dive headfirst into the implementation only to end up with a convoluted, unclear interface, precisely because in traditional programming the cognitive effort all goes into the implementation and the interface is secondary. Here instead everything is flipped.

A very important rule in TDD is to proceed in small steps. We shouldn’t write a thousand tests and then move on to implementation. You write a single test, it fails, and then our goal is to make the test pass by implementing the necessary code. Why all this? Essentially for two reasons:

We know perfectly well what our goal is: making the only failing test pass. This way the project can be as complex as we like, but by setting only a small goal everything becomes simple; divide et impera always works!
We write the bare minimum to make the test pass. How many of us have started writing mountains of code because “You never know”, or because our mind thought of a thousand cases that nobody ever asked for? In TDD this doesn’t happen; every single line of code was written to make a test pass. If we write anything else we’re doing it wrong.

Now that we’ve told ourselves we must proceed in small steps, let’s actually write the code that will make our test green. First of all let’s verify that it fails.

In the console we write npm test.

As output we have something like this.


> word-counter@1.0.0 test /Users/mattia/Repositories/word-counter
> jest

 PASS  src/__tests__/hello-test.ts
 FAIL  src/__tests__/word-counter.ts
  ● Test suite failed to run

    TypeScript diagnostics (customize using `[jest-config].globals.ts-jest.diagnostics` option):
    src/__tests__/word-counter.ts:1:25 - error TS2307: Cannot find module '../word-counter'.

    1 import WordCounter from "../word-counter";
                              ~~~~~~~~~~~~~~~~~

Test Suites: 1 failed, 1 passed, 2 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.557s
Ran all test suites.
npm ERR! Test failed.  See above for more details.

The TypeScript transpiler rightly complains that the word-counter class doesn’t exist.

Now let’s write the bare minimum to make the test pass. Let’s create the class and the method in src/word-counter.ts.

export default class WordCounter {
  public count(text: string, wordToFind: string): number {
    return 1;
  }
}

Let’s run the test with npm test. And this is the result:


> word-counter@1.0.0 test /Users/mattia/Repositories/word-counter
> jest

 PASS  src/__tests__/hello-test.ts
 PASS  src/__tests__/word-counter.ts

Test Suites: 2 passed, 2 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        1.669s
Ran all test suites.

We did it! But I imagine you noticed that I cheated. The method I wrote always returns 1, regardless of the text passed. But this isn’t wrong in TDD because our goal is to make the tests pass as soon as possible with the bare minimum. If I managed to cheat, it means the test isn’t thorough enough to verify the correct functioning of the program. Now let’s fix it by writing a test that forces us to consider the text input. In the src/__tests__/word-counter.ts file let’s add

it("should count the word inside text (multiple)", () => {
  expect(wordCounter.count("pippo pippo ciao pippo", "pippo")).toBe(3);
  expect(wordCounter.count("ciao ciao", "ciao")).toBe(2);
  expect(wordCounter.count("ciao", "nonesistequestaparola")).toBe(0);
});

Running the tests, this test indeed fails

> word-counter@1.0.0 test /Users/mattia/Repositories/word-counter
> jest

 FAIL  src/__tests__/word-counter.ts
  ● WordCounter › should count the word inside text (multiple)

    expect(received).toBe(expected) // Object.is equality

    Expected: 3
    Received: 1

      11 |
      12 |   it("should count the word inside text (multiple)", () => {
    > 13 |     expect(wordCounter.count("pippo pippo ciao pippo", "pippo")).toBe(3);
         |                                                                  ^
      14 |     expect(wordCounter.count("ciao ciao", "ciao")).toBe(2);
      15 |     expect(wordCounter.count("ciao", "nonesistequestaparola")).toBe(0);
      16 |   });

      at Object.<anonymous> (src/__tests__/word-counter.ts:13:66)

 PASS  src/__tests__/hello-test.ts

Test Suites: 1 failed, 1 passed, 2 total
Tests:       1 failed, 2 passed, 3 total
Snapshots:   0 total
Time:        1.612s
Ran all test suites.
npm ERR! Test failed.  See above for more details.

Now let’s change the implementation of WordCounter.count in src/word-counter.ts so as to make the test pass

public count(text: string, wordToFind: string): number {
  return text
    .split(" ")
    .filter((word) => word === wordToFind)
    .length;
}

Both tests are OK! …But there’s one thing we haven’t tested: what happens if the word to search for is all lowercase but the text is uppercase? We don’t know and we don’t even bother finding out, we just write a test to be sure!

Let’s add the test

it("should count the word case insensitive", () => {
  expect(wordCounter.count("Ciao ciao CIAO", "ciao")).toBe(3);
});

Running the tests it fails… So let’s fix it!

public count(text: string, wordToFind: string): number {
  const wordToFindLower = wordToFind.toLowerCase();
  return text.toLowerCase()
    .split(" ")
    .filter((word) => word === wordToFindLower)
    .length;
}

All good, again. But we haven’t considered text where the word ends with a comma or any punctuation mark. New test

it("should count word that have signs nearby", () => {
  expect(wordCounter.count("ciao, ciao! ", "ciao")).toBe(2);
});

New implementation to handle this failing test

public count(text: string, wordToFind: string): number {
    const wordToFindLower = wordToFind.toLowerCase();
    return text.toLowerCase()
      .split(/[^a-zA-Z\d]/)
      .filter((word) => word === wordToFindLower)
      .length;
  }

All green! In short, now it’s quite clear how to create a class in TDD. We’ve seen that it’s a continuous ping pong of failing tests and implementation that fixes the failure.

But this ping pong, in addition to the benefits we wrote about earlier, also brings others:

It iterates the development of complex apps through simple steps, incrementally: it’s obvious that we started with a very simple implementation, and then, through small steps, we arrived at having a method that handles numerous cases we might not even have thought of initially.
Previously written tests remain, the implementation changes continuously: maybe through this example it’s not very noticeable, but every time we added a new test and changed the implementation, we were certain that even the previously written tests continue to work. This certainty is fundamental when we have to do heavy refactors of methods; without TDD this certainty, most of the time, doesn’t exist. So people tend not to modify existing code; they add code “around” it to avoid touching something “that already works” and therefore avoid software regressions. This behavior works but only for a short period; the code easily loses readability, becomes confusing and obscure, and here’s the famous technical debt! I’m sure you’ve already had code like that in your hands and it’s not fun.
All the features we need are tested: this is free with TDD. We write the test first and then the implementation, so by necessity it’s true.

By now we feel satisfied with our WordCounter class. Actually there would be another case to test. For example

What happens if we pass null to the two parameters?

Should we add a test for this case too? The answer is it depends. If we know that our code doesn’t need to handle null, we can also skip it. Many might counter that this way the code isn’t robust, but actually our goal isn’t to handle every case in the world; we need to create a class that’s OK for our project. So we should try to stay focused on the initial goal; this way we don’t complicate the code with cases that never happen and we don’t waste time writing useless code and tests.

Now that we’re satisfied with our WordCounter, let’s move on to the next class.

TextFetcher class

This class solves part 2 of our goal. That is, it downloads a web page and retrieves the text from it. By now we know how it works and, loaded like a spring, we use the same procedure we used with WordCounter:

Let’s think about the TextFetcher interface: I imagine using it like this new TextFetcher().getContent(url) and it returns a Promise<string>, in short it returns a string asynchronously (we can’t have it synchronous because the HTTP call to retrieve the content from a URL is slow). Here a prerequisite I wrote earlier comes into play: asynchronous programming in JavaScript/TypeScript. If you want to fully understand the examples I’m going to write, I recommend the readings I included in the prerequisites section.
Let’s write the test to test the future implementation.

import TextFetcher from "../text-fetcher"

describe("TextFetcher", () => {
  const textFetcher = new TextFetcher();

  it("should get the content as text", async () => {
    expect(
      await textFetcher.getContent("https://raw.githubusercontent.com/matitalatina/word-counter/master/README.md")
    ).toContain("WordCounter");
  });
});

If we open the URL I put in the test we can see it’s the README.md of the repository I’m using to create this app.

Inside it there’s the word WordCounter. Our test verifies that the WordCounter instance actually downloads the README and returns its content. In the test I only look for the word WordCounter and not exactly the content because the README could change, but at least the project name I hope will always remain!

Now let’s try to make the test pass (without cheating so as not to weigh down the discussion too much). To make the HTTP call that downloads the file, I use a very famous library in Node.js which is Axios. For this reason we install it among our dependencies

npm install --save axios

Let’s write a possible implementation in src/text-fetcher.ts

import axios from "axios";

export default class TextFetcher {
  public async getContent(url: string): Promise<string> {
    return (await axios.get(url)).data;
  }
}

We’ve essentially created and exported the TextFetcher class, created the asynchronous method async getContent, in the implementation we use axios to download the content of the url, we asynchronously wait for it to download the content using the await keyword, and finally we take the content of the response using the data property.

Let’s run the tests with npm test and everything works!

But I don’t know if you smell a little “burning” too… If you noticed, I put a URL in the test that fetches a resource external to our program, which could change, could be inaccessible because our PC is offline. All this would influence our test, making it fail even though the code is actually correct.

Remember that unit tests must be fast and must not depend on external factors. The HTTP call can be slow due to the connection or even fail due to possible changes or disconnections.

How can we solve this problem? We’ll mock the HTTP call.

To do this we need another external library: axios-mock-adapter.

npm install --save-dev axios-mock-adapter

Let’s modify the test in src/__tests__/text-fetcher.ts

import axios from "axios";
import MockAdapter from "axios-mock-adapter";
import TextFetcher from "../text-fetcher";

describe("TextFetcher", () => {
  const textFetcher = new TextFetcher();

  const mock = new MockAdapter(axios);

  it("should get the content as text", async () => {
    const content = "This is the content of the web page";

    mock.onGet("/fake-url").reply(200, content);

    expect(
      await textFetcher.getContent("/fake-url"),
    ).toBe(content);
  });
});

Through MockAdapter we hook into the axios library, so as to create fake HTTP calls. With mock.onGet("/fake-url").reply(200, content); we create a fake HTTP response through the /fake-url address with the value of the content variable as content. Finally we test that the response is actually content.

Running the tests everything works and this time we can test offline too.

Mocking HTTP calls external to the various URLs or to external microservices is fundamental to having a good unit test suite. We can’t call real resources during unit tests, also because these resources probably only exist in the production environment and we don’t want to pollute the production content with tests.

CounterApp class

This is the last class we’re going to build. Its goal is to:

Invoke TextFetcher to retrieve the text from the URL.
Invoke WordCounter to count how many times the word to search for is contained in the downloaded text.
Return the number returned by WordCounter.

In the unit tests of a class we don’t want to test the implementation of its dependencies too. In our case we don’t want to test TextFetcher and WordCounter again. We’ve already tested these classes well. For this reason we’ll mock these dependencies a bit like we did for HTTP calls.

Mocking the external dependencies of a class brings the following advantages:

We avoid a failure in the dependency also failing the tests of the class we’re testing. Ideally an error in the code should make only one test fail, and not all the parts that depend on it too.

We therefore need a library called ts-mockito which is inspired by the famous Mock library for Java called Mockito.

It makes it easy to create special cases: imagine we need to correctly test the handling of an exception, and generating this exception requires a very particular situation. If we mock, we can inject a mock that generates this exception without having to recreate this special situation.

Let’s put what we’ve said into practice. The first step is to install ts-mockito.

npm install --save-dev ts-mockito

Let’s write the test in the new file src/__tests__/counter-app.ts

import { instance, mock, when } from "ts-mockito";
import CounterApp from "../counter-app";
import TextFetcher from "../text-fetcher";
import WordCounter from "../word-counter";

describe("CounterApp", () => {
  // Initialization
  const mockWordCounter: WordCounter = mock(WordCounter);
  const mockTextFetcher: TextFetcher = mock(TextFetcher);
  const counterApp = new CounterApp(instance(mockTextFetcher), instance(mockWordCounter));

  it("should get parameters and return the count", async () => {
    const providedUrl = "url";
    const providedWordToFind = "word";
    const fakeContent = "Retrieved text";
    const expectedCount = 1;

    when(mockTextFetcher.getContent(providedUrl))
      .thenReturn(Promise.resolve(fakeContent));
    when(mockWordCounter.count(fakeContent, providedWordToFind))
      .thenReturn(expectedCount);

    expect(await counterApp.countWordFromUrl(providedUrl, providedWordToFind))
      .toBe(expectedCount);
  });
});

In the initialization we create the mocks of our two classes WordCounter and TextFetcher; with the line new CounterApp(instance(mockTextFetcher), instance(mockWordCounter)) we insert the instances of our mocks into the constructor of the CounterApp class. This technique of “injecting” the dependencies used directly into its constructor pairs well with frameworks that use dependency injection. If this last sentence I wrote is pure gibberish, don’t worry. The message that must get across is this: it’s good practice to require in the constructor the external dependencies the class needs to work correctly. This is true for two reasons:

It clearly indicates to the programmer using your class which dependencies it needs to perform its function.
It makes testing the class in question easy, because in the constructor we can inject mocked instances, as in our case.

The part inside the test it("should get parameters and return the count", ...) should be fairly self-explanatory: I create the variables I expect to use and then I use ts-mockito to make the appropriate assertions. For example with when(mockTextFetcher.getContent(providedUrl)).thenReturn(Promise.resolve(fakeContent)); it means: “when the class calls textFetcher using the getContent method with the providedUrl parameter, return a Promise resolved with the value fakeContent”. The when(...).thenReturn(...) functionality helps us bypass the real implementation of the mocked classes and return a value we decide.

This way we’re testing:

That the CounterApp class calls the methods of the WordCounter and TextFetcher classes with the correct parameters, through the various when(...).thenReturn(...).
That the CounterApp class returns the correct number through expect(...).toBe(...).

Let’s run the tests with npm test. Of course it fails; now let’s implement the class in the new file src/counter-app.ts;

import TextFetcher from "./text-fetcher";
import WordCounter from "./word-counter";

export default class CounterApp {
  constructor(private textFetcher: TextFetcher, private wordCounter: WordCounter) {

  }

  public async countWordFromUrl(url: string, wordToFind: string): Promise<number> {
    const text = await this.textFetcher.getContent(url);
    return this.wordCounter.count(text, wordToFind);
  }
}

We’ve created a constructor that inserts the instances of TextFetcher and WordCounter into the private variables textFetcher and wordCounter (the constructor is empty; it seems to do nothing, but the dirty work is being done by the private keyword). Then we implemented the asynchronous method countWordFromUrl which does nothing but download the content from the URL and then count the number of times the word we’re looking for appears and return it.

Let’s try with npm test and everything is indeed OK!

Real test

Now that we’ve finally reached our goal, programming entirely in TDD, let’s try a real test.

Let’s create the src/index.ts file where I’ll insert a small script to try it all out.

import CounterApp from "./counter-app";
import TextFetcher from "./text-fetcher";
import WordCounter from "./word-counter";

async function testApp() {
  const url = "https://raw.githubusercontent.com/matitalatina/word-counter/master/README.md";
  const wordToFind = "WordCounter";

  const count = await new CounterApp(new TextFetcher(), new WordCounter())
    .countWordFromUrl(url, wordToFind);

  console.log(`The word "${wordToFind}", in ${url}, appears ${count} times`);
}

testApp();

I created a small testApp function that creates my CounterApp class with the necessary dependencies, and I try to count how many times the word WordCounter appears in the README.md page of my repository for this project. To start the program I need to transpile TypeScript into JavaScript, because as we know Node.js can’t read TypeScript directly. To solve this I’ll use the ts-node library.

npm install --save-dev ts-node

Now that it’s installed, let’s start our script like this

npx ts-node src/index.ts

The result is this:

The word "WordCounter", in https://raw.githubusercontent.com/matitalatina/word-counter/master/README.md, appears 1 times

It works!!!

Project maintenance

We’ve finished our small TDD project. But a real project worth its salt is never “finished”; it must be maintained and improved every day. Another advantage I appreciated after a few months of TDD development is the drastically reduced time in terms of debugging or production bugs. So the time spent writing tests is far outweighed by the time saved from debugging the product. Try it to believe it! But don’t think that bugs no longer exist using TDD. There are far fewer of them but they do happen from time to time. When they happen we must always remember to write the test that manifests the bug before fixing the bug. This way we keep using TDD, but above all we have the guarantee that the bug won’t manifest again even if we heavily modify the code in the future.

When to use TDD

Now that we’ve seen what TDD is, a question may naturally arise: when should you use TDD? Here it’s better to make a distinction: if I have to choose between TDD and classic unit tests, from personal experience I’d always choose TDD. The classic approach has never brought me any benefit over TDD. The only “advantage” I’ve seen is that the classic approach lets you not test everything, but honestly I don’t consider that an advantage precisely because it creates dangerous holes that nullify much of the benefits we found in this guide.

If instead the choice is between TDD and only the implementation without unit tests, the discussion changes. These are certainly the cases where TDD (or at least unit tests) should be used.

Backend systems: when you have clients (mobile apps, microservices etc.) that rely on our system, it’s fundamental that the backend works perfectly.
“Mission critical” parts: if we have components that must work and are crucial to delivering our service, we can’t afford to have bugs or go on trust.
Components that are difficult to test “live”: if it takes me 5 minutes just to test a special case once, better to spend them creating a test that at least I’ll always have and a npm test is enough to recheck.

When instead is TDD not needed and you can go with implementation only? It sounds like heresy but I’ve found some cases where implementation alone is enough and more:

When our implementation is easily testable “live”: ever heard of live-reloading? With web apps or even in hybrid app development with Flutter, Ionic or React Native, you can implement and immediately view the output. This is a very powerful tool that enormously speeds up development, but above all the time between implementation and the final result is zero. What’s the point of writing tests if I just save the code and immediately see the final result and can try it?
When we can afford some freedom, the specifications are still fuzzy and our client is in close contact with development: it has happened to me to develop literally with the client who could give me feedback almost in real time. If the environment is very informal/conversational, it’s possible to create an entire project without writing a single line of test and the client is happy anyway, even if maybe from time to time they call you to report a bug. The important thing is that you can fix it quickly so that the client can appreciate your speed in fixing it.
When the project is an experiment: sometimes it happens to create a project just to understand whether a certain thing can work. Or you implement a small variation for A/B testing, or you develop something temporary to work around an equally temporary problem. In these cases we’ll never appreciate the benefits of TDD, because the time spent on tests won’t be compensated by the time saved from debugging or fixing bugs.

The message that must get across is this: evaluate your case and act accordingly. TDD is the solution to many problems, but not to all of them.

Conclusions

First of all I apologize if while reading this guide maybe the concept of “it’s simpler than you’d think” didn’t come across much. But don’t forget that here you’ve seen numerous concepts which, if it’s the first time you’ve seen them, take a little time to digest. If maybe you missed the concepts you learned in this article, here’s a small summary for you:

Initialization of a project in Node.js, use of the npm package manager.
Adding TypeScript with an integrated linter.
Using Jest for testing.
Asynchronous programming with the async and await keywords introduced in ECMAScript 2017.
Using mocking libraries, when and why to use them.
Test Driven Development.

At first TDD seems like a brake, like all new things, but once you get comfortable, everything becomes faster and even more fun! It’s always nice to reach the end of the day and see your project growing together with the tests that increase and are all green… except one! Yes indeed: at the end of the day, if the project isn’t finished, we write a failing test before leaving. This is a piece of advice that’s very effective for resuming work. This way we perfectly remember what we were doing when we return to work. The failed test will tell us exactly what we left behind.

I hope this article has made you want to try this fascinating technique which, at first glance, may seem almost absurd, but once adopted, enormously increases the quality of the code and consequently also the quality of the final product.

If you want to see TDD in action with Spring Boot you can read my article that talks about how to develop RESTful APIs in TDD with Spring Boot.

See you next time!

11 lessons I learned working in startups

Sat, 21 Apr 2018 07:03:47 GMT

I’ve been working in startups for almost 4 years now. I’m convinced that a startup is a great choice for a student fresh out of university, because they are very dynamic environments: given their unstable nature you are “forced” to adapt, you learn something new every day and so it’s a great opportunity to grow personally together with the company.

Since I’m now changing work environment, it’s time to sum up. In this post I’ll tell you what I learned working in startups, in my case wavein.ch and Cloud Academy.

Your boss wants solutions, not problems

In startups, your superiors are often the same people who wrote the code before you, so the temptation to ask them for help is very high. But remember that their role has changed now, so they have a thousand other things on their minds. In fact they rely on you to complete the task you’ve been assigned. Instead of asking for help, you need to ask them what decision to make. The correct flow is that they ask you to do X, you analyze the problem and bring solutions with the pros and cons of each one, and finally the superior decides which solution to pursue and takes responsibility for it. This way everyone is happy: you know exactly what to do, the superior has spent little time and received only the information they cared about.

Keep track of what you’re doing

Transparency in startups is essential: you obviously know very well what you’re working on, which bug is driving you crazy… But all this isn’t so obvious to the rest of the world. I used YouTrack and Jira as issue tracking systems; there’s nothing worse than having fiery days at work and an empty board. This can be a problem for you because you’re unable to show others what you’re working on, but above all it’s a problem for the Product Manager who will never be able to schedule work correctly, won’t have any visibility on possible delays and finally can’t prioritize work correctly. The same goes for blocked tasks: always put in writing what you’re working on.

Each of us is different

We have a character, our own personality: for this reason we often behave the same way with everyone. But we forget that others are all different too: there are those who are more outgoing, those who get offended easily, those who are shy, those who get angry over nothing, etc. We can’t behave the same way with everyone indiscriminately. And worse still we can’t expect others to adapt to our character, it will never work!

In my opinion it’s our job to bring out the best in everyone by leveraging each person’s qualities:

The person loves helping others: let them help you! They can surely always teach you something.
They get offended easily: try not to make useless jokes, keep a more professional / detached profile.
The person gets fired up easily and hates “being wrong”: if you already have the solution to a problem in mind and just want confirmation, don’t explain the macro problem to them right away. Otherwise they’ll give you their solution and won’t want to discuss it anymore. Instead try to break it down into small problems where the solutions are obvious and are the ones you adopted. Finally all these solutions will lead to your definitive solution by logical consequence.

If this person starts to heat up, don’t start mimicking their tone… Otherwise it’s just an escalation of tones that will only end badly. Don’t be afraid to interrupt the discussion without a solution if you see it’s going nowhere.
They love making dirty jokes and like to “friendly insult” others: play along! Don’t take yourself too seriously. Very often if they openly make fun of you it’s just because they want to have a laugh with you. If you’re not used to it, it’s really hard to digest at first, but then you stop noticing and actually enjoy it. In the end you understand when someone does it just for fun or because they hold a grudge against you.
You don’t particularly like a person: the worst thing you can do is burn all bridges around them. You have to spend your daily life with them and so it doesn’t help anyone. Instead try to avoid the behavior that bothers them if someone tells you about it. Otherwise try to always behave well with them: help them and don’t stick too close, so that slowly they’ll understand they shouldn’t have anything against you. If it doesn’t work… Well, so be it; always try not to give them good reasons to hate you, it’s the only thing you can do in these cases.
They’re shy: they’ll never come to ask you for help or advice. Every now and then reach out, so that they open up too. This way you can both work.
They’re a buck-passer: “Sorry, could you do this for me? I know you worked on it recently.”, “I know nothing about this stuff, can you do it since I know you know about it?“. Helping is one thing, doing other people’s work for them is another. You have your own goals too, try to stay focused on yours.

In short, these are just small examples and behaving differently with different people doesn’t mean being fake. In my opinion it’s the only way to live peacefully with many people in your working daily life.

It’s okay to say No

Work is like water: it flows where it’s done. So you have to be careful about saying too many yeses, otherwise all the work ends up on you and then you have two very serious problems: you risk burnout because you don’t have a moment to breathe, and on top of that they accuse you of being the bottleneck! It’s important to help others, to do extra work, but it’s more important to finish your assigned tasks. For the company it’s important to respect its plan; if you fail to meet the original timelines, all this will have repercussions on the planning and this will have a negative repercussion on the whole company. The fewer people there are, the more visible the consequences. So you shouldn’t be afraid to say ‘No’ to those who ask you to do more extra work. First do your own, then, if there’s time left, accept extra work. Other important ‘No’s are the technical ones. It’s not rare for tasks to come down from above that, from a technical or usability point of view, make no sense at all. In these cases it’s our duty to say ‘No’ and give our reasons: it’s the developer who can understand the consequences of exotic requirements (technical debt, murky logic from the start, excessive UX complexity, etc.).

A failure is truly a failure only if you learn nothing from it

Now I’ll say something nobody likes to hear. Everyone fails at something: I’ve made mistakes and I’ll make more, it’s impossible to do everything perfectly in your career. Be wary of those who tell you “I’ve never made mistakes / I’ve never failed / Honestly I don’t remember any mistakes I made in the past”. They’re just making fun of you or lying to themselves, because we, especially Italians, hate failing and above all hate admitting it.

But making mistakes is an integral part of learning. Experience is nothing more than a collection of mistakes you’ve taken home and are ready not to repeat. That’s why those who say they’ve never made mistakes are lying; it means they’ve either never done anything or never had a truly challenging task.

The sooner you recognize a mistake, the sooner you can fix it. The sooner you fix it, the less damage you do. If a path is leading nowhere, or is becoming harder than expected, stop… Take a deep breath, analyze the situation and see if it’s worth continuing. Otherwise it’s time to ask a colleague for an opinion and then announce the problem to your superior (possibly with some solutions in hand). Then you make the necessary corrections.

You’ll see that by doing so nobody can tell you anything, they’ll appreciate your ability to foresee even greater damage, you take note of what went wrong so as not to repeat the same mistake in the future and you carry on happy to have avoided a possible disaster.

The alternative to all this? Tell everyone everything is fine, keep going your own way, try to make the best of a bad situation, buy time to try to keep something that’s dead alive, buy more time, despair, throw in the towel and say you’re working on something stupid that can’t work, blaming the cleaning lady just to avoid taking responsibility… Nobody will appreciate it; on the contrary they’ll understand they can’t rely on you, so avoid it.

Failing isn’t nice, but it happens. So if it has to happen, let it happen as soon as possible.

Asking for help is a weapon, not a weakness

Unfortunately for many, asking for help is almost a personal defeat, while others don’t do it to avoid taking up too much time of the person who makes themselves available to help. But asking for a hand from those who know more about a certain product, piece of code or technology is essential:

You reach the goal sooner. Asking a question and getting an answer takes a few minutes. Trying to understand a piece of code written by others, looking for an answer online for a very specific case takes hours.
You really understand how the platform works. Unfortunately in startups documentation leaves something to be desired, so very often you end up reading the code directly. Some parts can be obscure to understand simply because the real reason isn’t made explicit. It’s not rare to find horrible hacks in the middle of the code to solve specific cases that only one person knows about… And that person may no longer be at the company! If they’re there, I recommend one thing: go ask.
You avoid writing blunders: I’ve seen people who, just to avoid asking or understanding how a part of the product works, completely “bypassed” that part by writing a very similar one. This is evil, because you create two pieces of code that coexist, it’s not clear why, and of course it makes the code unmaintainable. Or others who solved a problem only on the surface because “hey, I don’t touch code written by others!“… This too is absolute evil: this way you can destroy the architecture in a few weeks. Rather, ask whoever wrote the code and then solve it.
You learn faster! Even from a personal point of view it’s much better to ask for help: you learn much faster and hearing how other developers reason is the best way to improve yourself…
You need help too… What? You’re thinking you’re the best and you have nothing to learn, especially from your colleagues? Well, everyone who thinks that way is the one who grows the least and falls behind, because they think they have nothing left to learn and stay stuck with their knowledge. Literally anyone can teach us something, even the most junior person in the company, because simply every person has their own, unique, personal background.

Nobody cares about perfect code

Here’s a lesson that can hurt those who come fresh out of university. In the academic world the most important thing is writing perfect code, having the most abstract code possible so that it easily adapts to any use case. Nobody looks at your code until the final exam and so you can work in complete tranquility. There’s just one small detail… In the university world you have a lot of time available, in startups you don’t. The feature you’ve been asked for is blocking other developers, stakeholders want your part ready and deployed yesterday. So it’s not the goodness of your code that’s evaluated, but your execution. The speed with which you complete your task.

So how do you make good code coexist with execution speed? Through iterations: write the bare minimum to reach the goal, don’t complicate things with strange constructs. Writing the bare minimum is very different from writing bad code: keep following best practices, keep the code as clean as possible, keep evaluating that the more shortcuts you take the more maintainability you have to sacrifice. But don’t worry about the problem before it happens, because maybe that problem will never come… And if it ever does, you iterate, but at least your original feature is already on the market and can be validated by the customer.

Reach your goals, period

Another lesson I learned the hard way, and that I received like a spinning kick to the gums. As soon as I joined Cloud Academy everything was a new world, I was eager to do things and I was really fired up. One of the first tasks assigned to me was automating the generation of static files on the frontend. While doing it I had the brilliant idea of also making an extra improvement that unfortunately I don’t remember now. I went to my superior to show him my progress: after telling him where I had got to, I also explained my improvement and what we could do more thanks to it. While I was explaining he abruptly interrupted me with a “Why the hell should I care?“… Total embarrassment, 5 seconds of silence… All I managed to say was an “Ah…“.

I was used to wavein.ch having much more room to maneuver, but simply because we were smaller and I was almost my own boss. So everything I did, good or bad, went in the direction I deemed right. But in a startup you have “well-defined” tasks. If you have assigned tasks you have to do those and do the minimum to reach that goal. Everything else isn’t looked upon favorably, it’s seen as wasted time. Remember that time is the scarcest resource in a startup. So be careful with unrequested features; even if they may be a great improvement for you, others might not care “one damn bit”.

A startup is not a family

A startup is not a family. You need to keep that well in mind. We like meritocracy, we want it everywhere, but we must also be able to accept the flip side of the coin.

Now I’ll explain what a working relationship is for me, in the most meritocratic and cynical version possible: I get paid to offer a certain service to the company. Through my service, the company acquires value. If the value generated by my service is higher than my salary, the company has every interest in keeping me. The higher this delta is, the more the company “loves me”; if this delta is too marked I can ask for a raise and the company should recognize it. If it doesn’t give it to me, I have every right to look around to see if someone else recognizes my value. If instead the delta is negative it means the company no longer benefits from paying me and so it has every right to let me go. Finally there’s one last option: the company no longer has funds, so it starts laying off the people with a less marked delta, that is, those who generate less value for the company.

Using this clean, meritocratic, cynical-as-you-want version, you can’t remain “burned” by a possible layoff. These are purely logical and rational things.

But we’re not machines and so there are a lot of other variables at play that can deeply affect people’s sensitivity: the whole delta story is fine, but is it better to lay off a young person with no dependent children or lay off a person with a family and 3 children even if they generate less value for the company? Is it better to keep working for the company I work for even if they pay me little, where I’ve made a lot of friends and I feel good, or change jobs and go where they pay me more?

I’ve met people who experienced a layoff very badly because they felt “betrayed” by the company. But in my opinion there’s little to get angry about: nobody likes laying people off, and the choice of who to lay off is something I think would distress anyone.

I experienced first-hand what it means to be laid off from one day to the next. Nevertheless I remained on excellent terms with those who laid me off because, being in a startup, I knew very well that money doesn’t grow on trees: I understood the real reasons and so I was able to effectively separate the working relationship from friendship. They are two things that coexist together, but they certainly aren’t the same thing.

With this I’m not saying you can’t make friends at work. On the contrary, at work you meet wonderful people and friendships are born that last for years even outside work. I’m just saying that I don’t have a job because I’m friends with someone, but because I’m generating value for the company and it is able to recognize this generated value. If you keep this aspect in mind, in my opinion it’s easier to accept a layoff.

Don’t exasperate the work environment

“Is it possible, Natali, that you never get angry?“. Actually I get angry too, but I simply try not to show it. We spend a good part of the day at work and so it’s important that we keep the environment as calm as possible, even if anger is always around the corner. Over these years I’ve fundamentally learned these two things:

Getting angry is useless. Unfortunately it happens to see people who claim to be right simply by shouting or visibly showing irritation. The important thing is to stay calm anyway, wait a few seconds before answering and, if the situation really doesn’t calm down, it’s better to end the conversation and postpone it when the tones are calmer… Also because the problem isn’t solved in these situations.
You get nothing with force: nobody will do you a favor/task under duress, you have to motivate people, not force them. I’ve been lucky enough to see three types of leadership: those who force you, those who “threaten” you and those who instead motivate you to do things. The best results of course are obtained with the last one. Exasperating the work environment to get something is never a solution; maybe it can work in the short term, but in the long term it can only give rise to a much more serious problem: sending turnover through the roof. The problem is that forcing or threatening a person is simple, motivating is not at all. And that’s why it’s important to choose the various leaders in the company well… Not everyone is able to do it.

Personal training is essential

Did you think that once you finished your studies you could say goodbye to studying? Well, in IT it’s only the beginning! IT is a continuous evolution; if you stand still even for just two years you’re automatically cut out. For many this is a condemnation, but for me it’s the most interesting part of the job. You never stop learning and you can tangibly see a continuous evolution in your work and in yourself… Repetitive work ends up being boring! To train yourself there’s only the embarrassment of choice: from e-learning platforms, small projects, to meetups organized by passionate people who really put their soul into what they do.

Conclusions

I hope this post can be helpful to those who are afraid to start a working career, especially in a startup. It’s certainly a good gym to learn quickly, but I’m sure it can be a bit scary for those who have never worked in one.

If you’re young I can’t recommend anything other than working in these environments: you have fun, you meet truly wonderful people who share your same passion and you’re never bored!

See you next time!

How to fix software regressions using git bisect

Mon, 15 Jan 2018 07:03:47 GMT

You added a feature to the software and it went to production. After a few weeks the Product Owner tells you it no longer works. Don’t panic. Using only Git you can fix everything in a few minutes!

Anyone can run into a software regression and it’s never pleasant. I’ve seen many developers who, in a case like this, start trying to reproduce the bug locally and furiously re-read the code hoping to fix the bug. If many hands work on the code, finding the bug can be anything but simple.

Using this naive approach we’re not exploiting all the information we have: we are certain that the feature used to work correctly. So something happened in the meantime that surely broke our part of the code. So why look for this change manually when Git can do the dirty work for us in very few steps? Git can tell us precisely what happened, when, who did it and why (if the developer who introduced the bug wrote a decent commit message).

git bisect

Well yes, Git offers exactly this feature. Not everyone takes advantage of it but in cases like this it’s extremely powerful. It’s the git bisect command.

To cut a long story short, you define two commits: the first is a “buggy” commit, that is, one that presents the problem, the second is a healthy commit, where the problem didn’t exist yet. Git then shows a commit that’s in the middle and it’s up to us to tell Git whether it works or not. Git at this point shows another commit that’s always in the middle between the buggy commit and the healthy one. You keep going like this until Git shows us the commit that introduced the bug.

It seems like a very long process but it isn’t: in fact, with each check we halve the number of commits between the good one and the buggy one. For example, if we have 1000 commits separating our two reference commits, we find the one that introduced the problem in log_2(1000) ~ 10 steps!

Practical example

As an example I used a small project of mine that I had created to learn some concepts about React. Let’s suppose we want to know who added the particle effect to the page. During the git bisect procedure we label as bad every time the project has the src/components/atoms/ParticlesBackground folder. While all the commits that don’t have it will be good.

Let’s start! First we clone the project and enter it with the terminal.

git clone https://github.com/matitalatina/8ball
cd 8ball

Now we start the search

git bisect start

Starting the program we notice that the master branch has the src/components/atoms/ParticlesBackground folder. So we label the commit as bad.

git bisect bad

Then we go back in Git’s history far enough to no longer find the folder in question. In our case going back 10 commits is enough.

git checkout HEAD~10

Ok! Now the folder is gone! So we can label it as good.

git bisect good

Now Git will change commit and move exactly to a point halfway between the commits labeled as good and bad. It notifies you with the following message.

Bisecting: 5 revisions left to test after this (roughly 3 steps)
[5e2a30980acb38d3d3958359c56fd0a8d0325136] Improve particles system

Keep typing git bisect good or git bisect bad until you’ve analyzed about 4 steps.

In our specific case it will be:

git bisect bad
git bisect good
git bisect bad
git bisect good

Finally, when Git has identified the commit we were looking for, it tells you like this:

b7b5d042a8cba637c56cc52401dcd61ec785d9b4 is the first bad commit
commit b7b5d042a8cba637c56cc52401dcd61ec785d9b4
Author: Mattia Natali <***@***.***>
Date:   Mon Nov 6 22:24:32 2017 +0100

    Add particles

:100644 100644 d93738e6743c2460699030cfe8df000e1bc7ecd7 570e0890ece45c8d4ef6640a3ea7a386fd63f7da M	package-lock.json
:100644 100644 994aebed4cf0ebfb0dd56e2bebf2a6e411e44e1c 0dc0290897d7082dadda14595739b80b07766603 M	package.json
:040000 040000 9f912726175c5d9794d00deffdb422136555846f a934e7f36f14c1ec38058bfbfbf403d3fb8043b3 M	src

The commit message alone gives us the intuition that Git hit the bullseye (“Add particles”)! But we don’t trust it, so let’s actually see what was done in this commit: take the commit id (b7b5d042a8cba637c56cc52401dcd61ec785d9b4) and type:

git show b7b5d042a8cba637c56cc52401dcd61ec785d9b4

scrolling through the commit we notice the part

diff --git a/src/components/atoms/ParticlesBackground/index.js b/src/components/atoms/ParticlesBackground/index.js
new file mode 100644
index 0000000..748f50d
--- /dev/null
+++ b/src/components/atoms/ParticlesBackground/index.js
@@ -0,0 +1,129 @@
+import Particles from 'react-particles-js'
+import styled from 'styled-components'
+import React from 'react'
+
+const Wrapper = styled.div`
+  pointer-events: none;
+  position: fixed;
+  top: 0;
+  bottom: 0;
+  left: 0;
+  right: 0;
+...
+```

which is exactly the part we were looking for! So ultimately now we know that Mattia Natali added this part on Monday, November 6, 2017 at 22:24.


## Conclusions

Imagine now doing this for a bug that's driving you crazy; it's very powerful because it tells you exactly who did it and what was changed. This way you can ask the commit author for clarification and know exactly the change that created the bug.

Maybe you won't need `git bisect` every day, but it's right to keep it in mind because it's certain that **sooner or later it will come in handy** and it will save you a lot of time.

See you next time!

How to be productive with the terminal

Fri, 06 Oct 2017 07:03:47 GMT

If you’re reading this blog post I imagine you’re more or less in the same situation as me. You’re a developer and you have 8 hours a day to work: in this span of time you have to review other people’s code, attend meetings, manage your millions of issues and maybe even other people’s, help those who ask you for help, think about how to reach your goals and finally program what you’ve thought of.

Doing all these things and staying within the set 8 daily hours is hard without disappointing anyone’s expectations. So, as good engineers/developers, you’re trying to optimize everything.

In this blog post I’ll explain how I tried to optimize my time spent in front of the terminal. I’ll focus especially on the Mac operating system, which is the one I currently work on.

In this guide I don’t want to explain how to use the terminal. That’s a prerequisite for me. My goal is to share with you many little tricks that will save you time in your working daily life so that you can focus more on the activity that actually brings value: programming.

The basics

Let’s speed up the cursor

Let’s start with a small optimization that’s as simple as it is effective, even before opening the terminal. Many times in the terminal or in your IDE you need to move the cursor. By holding down the arrow keys the cursor moves until you release the key. Well, nothing new so far. The exasperating thing is how slowly this cursor moves! Many people I see, to go fast, start pressing the arrows compulsively or simply wait precious seconds until the cursor moves to the desired position.

A simple optimization everyone should do is to decrease the time before key repetition starts and the interval between one repetition and the next.

To do it it’s very simple. On Mac go to Settings -> Keyboard -> Keyboard tab. Set Key Repeat to Fast and Delay Until Repeat to Short. Now try it and the difference will be huge. I know at first the cursor seems too fast, but then you get used to it and never go back!

Shortcuts

This advice applies in general to every program. If you often use the same function, try to learn its keyboard shortcut. It’s definitely faster not to lift your fingers from the keyboard. It happens that the shortcut can’t exist for your function; fortunately most of the time you can create it in the settings. I recommend spending some of your time creating these shortcuts: you’ll save precious time in the long run.

iTerm

The Mac terminal is good, but if you want even more you can use iTerm. Basically it’s like the native terminal but it has the following additional features that will increase your productivity:

Improved search: press cmd + F and open search mode. Type the keyword to search and it will immediately highlight your word in the terminal. Using cmd + G and cmd + shift + G you can search for the next or previous word.
Split panels: you can split the view both vertically (cmd + D) and horizontally (cmd + shift + D) so you can have more terminals on a single page. Very useful if you have multiple programs running and need to monitor them at the same time.
Customizable profiles: with a profile you can completely customize the terminal such as colors, shortcuts. If you spend a lot of time connected to remote machines it’s easy to get confused between the various open tabs. A great way to avoid making a mess is to use different profiles for machines with different purposes. Our good DevOp uses this technique daily: for the production machine he has a beautiful bright red background, so he clearly understands to be careful and not stay there too long. For the stage environment it’s blue, the rest black background.
If you’re curious about iTerm I recommend reading the other noteworthy features, but above all trying it!

Everyday business

Here’s a list of tricks I use daily in the terminal. I wrote all the ones that come to mind; some will be obvious to those who live all day in front of the terminal, others I hope not ?.

tab: Just a few letters of what you want to type are enough, then press tab for autocompletion. This is very useful for example when navigating between tabs.
sudo !!: how many times have you typed apt-get install without giving root permissions? Many, after reading the error message, press up arrow and then move the cursor to the beginning of the command and finally insert the word sudo to become root… What a waste of time! Just type sudo !! and boom! Done. Basically !! is replaced with the previous command and that’s exactly what we wanted.
cd: returns to your home. Why type cd ~ or worse cd ../../../../../ to go back home if cd is enough?
cd -: returns to the previous directory. Very similar is the git checkout - command: it checks out the branch we were working on previously.
lsof -i :<PORT>: Mac only. This command is used to identify the program that’s using port <PORT>. Very useful when a program gets stuck in the background and you desperately need the port free. Once you’ve run the program just take note of the program’s PID and then kill <PID>. Now the port is free!
Concatenating commands: why stare at a command until it finishes and then run the next one? Using && and ; you can concatenate multiple commands. && runs the second command only if the first finished successfully, while ; runs it regardless.
ctrl + c: used to close applications but also to quickly delete what we’re typing.
ctrl + a and ctrl + e: move the cursor to the beginning and end of the command we’re typing respectively.
ctrl+l: clears the terminal, much faster than typing clear!
!$: takes the last argument of the previous command. Very useful in cases like these mkdir -p i/am/creating/a/new/folder/ && cd !$ As you may have guessed I’m creating a folder, and once created I enter it without having to type the whole path again.
man <APPLICATION>: how many times has it happened that you didn’t remember the options of a command and went online to look for the documentation? No need to go online! Just type man followed by the application and you’ll have all the documentation at hand without googling.

cmd + t and cmd + w: open and close a new terminal tab respectively. If you notice these commands are “universal”, that is, they also work with browser tabs, finder tabs, your favorite text editor, etc.
open .: opens the folder we’re in in Finder.

Navigating commands and auditing

last: shows who is connected to the machine with the terminal or who connected previously.
w: very similar to last. Shows the people currently connected to the machine.
up arrow: inserts the previously executed command. Very useful to review the command or modify it in case of a typo. If you keep pressing up arrow you go back in the history.
history: shows all executed commands. Together with the grep command it’s very easy to search for a command. For example, had you started an app using runserver but don’t remember all the options by heart? Just type a piece of the command, in our case we type history | grep runserver.
ctrl + r <TEXT_SEARCH>: it’s called reverse-i-search and it means “search the past for a command similar to this”. In short it’s like the previous command only even faster. For example we can press ctrl + r and then type runserver to have the command we were looking for in the terminal. If the command you were looking for isn’t the last one but a previous one, keep pressing ctrl + r until you find the desired command again!

Let’s go further!

Fuck!

Everyone happens to mistype some command: when we make a typo and see the terminal answer us with a no, the first thing we think is “fuck”. With thefuck we can vent our anger productively! This app is as simple as it is brilliant. If we mistype a command, we just type fuck in the terminal; this way this app proposes the previous command written correctly. See it as an automatic corrector for the terminal.

Installation is very simple: having installed Homebrew we just type

brew install thefuck
eval $(thefuck --alias)

Good! Now if you mistype some command get angry and type fuck and you’ll be more productive!

Zsh

We often find bash as the default installed shell. If you want that extra gear I recommend using Zsh together with the Oh My Zsh! plugin. This combo will allow you to be more productive for the following reasons:

Improved folder autocompletion: better navigation between the proposed choices.
Autocompletion for the most used programs: Git, Docker, npm, Django are the Oh My Zsh plugins I use, but there are many others!
Fully customizable themes: there are many ready-to-use themes, you’ll surely find one that satisfies your personal taste. My favorite is eastwood.
Greater Git integration: as soon as we enter a folder containing a project versioned with Git, we can notice that the branch we’re on is shown and also a small alert when we haven’t committed and pushed all our code. Thanks to these little aids we’ll save many git status.
Improved search of previous commands: type only the beginning of a command, then pressing up arrow shows you the previously used command that starts exactly the same way.

Installation

If you want to install Zsh with Oh My Zsh! these are the steps to follow:

brew install zsh zsh-completions
sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

Now restart the shell; to use Zsh just type zsh.

If you want Zsh to become your default shell then type

chsh -s $(which zsh)

Changing the theme

Zsh configuration files are in the ~/.zshrc file. With your favorite editor modify the ZSH_THEME variable. If you don’t know which theme to choose, you can use random mode: this way every time you open a shell you’ll have a different theme. When you find one you like, take the name of the theme that appears at startup and write it in the aforementioned variable.

As I said before, I’m using eastwood. It has very relaxing colors and doesn’t distract too much compared to other themes.

Changing the plugins

Also in the ~/.zshrc file we can find the plugins variable. You can see plugins as improvements in autocompletion for applications you often use in the terminal.

I currently use this configuration.

plugins=(django npm docker)

I remind you to separate the plugins with a space and not with a comma. Of course Zsh offers many other plugins, choose the bare minimum because they could slow down the terminal startup.

Aliases, aliases everywhere!

In the terminal you can add aliases to speed up the execution of one or more commands. If you notice that you always type the same command (or even a series of commands) then it means it’s time to add an alias.

Adding an alias is simple: you need to add them in the ~/.bash_profile file if you use bash or ~/.zshrc if you use Zsh. The syntax to add one is the following:

alias <ALIAS>="<COMMAND>"

Here are some of my aliases as an example:

alias gbd="git branch | grep -v '*' | xargs git branch -D": running gbd (Git Branch Delete) will delete all local Git branches of the project contained in the folder. Only run it if you’re sure of what you’re doing!
alias pycc='find . -name "*.pyc" -exec rm -f {} \;': running pycc (pyc clean) will recursively delete all compiled Python files present in the folder. Very useful when refactoring in Python and we fear that the compiled files no longer match the source code.
alias yda='youtube-dl -x --audio-format m4a --audio-quality 160k': running yda <YOUTUBE_URL> (YouTube Download Audio) will download the audio file of a YouTube video. This command only works if you followed my guide on how to download a video from YouTube.
alias ddev='docker-compose -f <ABSOLUTE_PATH_DOCKER_YAML_FILE>': running ddev <DOCKER_COMPOSE_COMMAND> (Docker dev) you can give any command to your Docker Compose without worrying about the folder you’re in. Without this command I was always forced to go to my Docker Compose project folder cd <PATH_TO_DOCKER_COMPOSE_PROJECT> and then type docker-compose -f <DOCKER_YAML_FILE> <DOCKER_COMPOSE_COMMAND>… All wasted time!
alias c<PROJECT_NAME>='cd <PROJECT_NAME_ABSOLUTE_PATH>': if you have a project you work on daily I recommend creating an alias to reach it quickly. As a convention I use the initial letter c which recalls the cd command and then the project name.

Git

Git also supports its own internal alias mechanism. Open or create the ~/.gitconfig file. Inside it let’s add

[alias]
    st = status -sbu
    lg = log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit

As you can guess we added the alias section and added two new git aliases. Once we’ve restarted the terminal we can use:

git st: a modified version of git status.
git lg: shows the commit tree.

Of course these are two of my commands but nobody forbids you from creating your own!

SSH

Access to a remote machine can also be sped up. I imagine the machines you connect to remotely are always the same, and typing ssh <USERNAME>@<REMOTE_ADDRESS> -i <PRIVATE_KEY_PATH> in the terminal every time is long, tedious and hard to remember. It’s a whole other story if we could connect simply by typing ssh <ALIAS> where <ALIAS> is the name we gave to the remote machine. All this is possible thanks to the ~/.ssh/config configuration file. If the file isn’t present, you can create it by typing touch ~/.ssh/config. Through this file we can save remote connections and recall them via an alias, without necessarily rewriting all the parameters every time.

The ~/.ssh/config file has the following syntax:

Host <ALIAS>
    HostName <REMOTE_ADDRESS>
    Port <PORT>
    User <USERNAME>
    IdentityFile <PRIVATE_KEY_FILE>

Replace the variables written between angle brackets with your parameters: this way instead of typing

ssh <USERNAME>@<REMOTE_ADDRESS>:<PORT> -i <PRIVATE_KEY_FILE>

now you just type

ssh <ALIAS>

much, much faster and more immediate. In the configuration file the HostName and User fields are mandatory, the rest are optional. You can add more Hosts if you have different machines you access remotely.

The available SSH parameters aren’t only these; if you need advanced features like tunneling I recommend reading the official SSH Config File guide.

Curiosities

These commands aren’t related to productivity, but they’re still noteworthy.

say '<TEXT>': did you know the Mac can talk and you can make it talk through the terminal? By combining this command with ssh we can make a Mac talk even remotely. So nobody forbids us from sending a love message to our girlfriend through the command ssh girlfriend@<IP_ADDRESS> say 'I love you!', try it and observe her reaction! (Spoiler: she won’t appreciate it, I can assure you ?).
yes: if you have a temporary chilblain and need a heat source, then I recommend starting yes from the terminal. This command does nothing but send the CPU to 100%!

Conclusions

I hope all these tricks of mine help you in your daily life to reach your work goals in the shortest time possible. There will surely be millions of other tips and tricks I’m not aware of: if I learn others I’ll try to add them here so I can share them with you.

See you soon, good work and see you next time!

How to download YouTube videos with the terminal

Sun, 17 Sep 2017 15:10:47 GMT

You’ve just found a beautiful video on YouTube and you absolutely want to download it, or you want to download only the audio part of the video to add it to your music playlist? The terminal doesn’t scare you? Good, then this guide is for you!

There are a lot of online sites that do these things: full of ads, sometimes they don’t even work and usually they give very little customization on what to download. I got tired of relying on these external programs and decided to do everything “at home”.

This guide is based on the Mac OS operating system. But with small adjustments it’s possible to follow it on Windows and Linux too, since the programs I’ll use are multi-platform.

Preparation

Basically to achieve the goal we need 3 programs:

Homebrew: the best application manager for Mac. With this software installing the other two will be a walk in the park.
Ffmpeg: when it comes to audio/video conversion it’s the reference point. Free, open source, cross platform, powerful… What more could you ask for?
Youtube-dl: the program that does the dirty work of downloading the video from YouTube.

Installation

Now that we know what we need, let’s proceed with installing these programs!

Let’s open the Mac terminal.

Let’s install Homebrew

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Let’s install Ffmpeg

brew install ffmpeg --with-fdk-aac --with-tools --with-freetype --with-libass --with-libvorbis --with-libvpx --with-x265

Let’s install Youtube-dl
```
brew install youtube-dl
```

Good! Now we have everything we need to download all the YouTube videos we want!

Let’s start!

As an example I’ll use the video by Gigi D’Agostino that he just uploaded online. I recommend watching and listening to it: the sobriety of his films is unmatched :D!

Automatic mode

Let’s start with the simplest thing: downloading the video without wasting too much time!

Let’s go to YouTube, open the video and copy the URL. In our case it’s https://www.youtube.com/watch?v=UyyxYha4j6o.
Let’s open the terminal and type youtube-dl and paste the URL from the previous step. So
```
youtube-dl https://www.youtube.com/watch\?v\=UyyxYha4j6o
```
you’ll notice that when pasting, some \ are added before ? and =. Don’t worry, they are escape characters that are necessary and that the terminal adds automatically.
Let’s wait and it’s done! In the folder we’re in we’ll have the video!

Choosing the video format

Ok Ok, maybe you don’t like the downloaded format or you want one with lower quality… To do this type:

Let’s add the -F option which shows us all the available formats
```
youtube-dl -F https://www.youtube.com/watch\?v\=UyyxYha4j6o
```
We’ll have a nice list of available formats. Let’s choose the one we like best and take note of the format code.
To download the chosen version we just need to add -f <format_code> where <format_code> is the code chosen in the previous step. So, for example, if we wanted to download the version
```
299          mp4        1920x1080  DASH video 5820k , avc1.64002a, 50fps, video only, 274.09MiB
```
we’d just type.
```
youtube-dl -f 299 https://www.youtube.com/watch\?v\=UyyxYha4j6o
```

Extracting the audio track

To get only the audio track we just need to choose the format code that has audio only in the resolution column. In our video, if we show the list of available formats we can notice that there are these audio tracks

format code  extension  resolution note
139          m4a        audio only DASH audio   48k , m4a_dash container, mp4a.40.5@ 48k (22050Hz), 2.26MiB
249          webm       audio only DASH audio   62k , opus @ 50k, 2.39MiB
250          webm       audio only DASH audio   79k , opus @ 70k, 3.16MiB
140          m4a        audio only DASH audio  128k , m4a_dash container, mp4a.40.2@128k (44100Hz), 6.03MiB
171          webm       audio only DASH audio  152k , vorbis@128k, 6.18MiB
251          webm       audio only DASH audio  153k , opus @160k, 6.24MiB

Very likely the audio format isn’t the one we want. So we need an audio conversion. Youtube-dl uses ffmpeg to perform the conversion. For example if we wanted an audio file in .m4a format with a bitrate of 160k we’d just type

youtube-dl -x --audio-format m4a --audio-quality 160k https://www.youtube.com/watch\?v\=UyyxYha4j6o

Where the options mean

-x: extract the audio track from the video.
—audio-format m4a: the format to download, or to convert if not present in the list, is m4a. The available formats are "best", "aac", "flac", "mp3", "m4a", "opus", "vorbis", "wav".
—audio-quality 160k: desired audio track bitrate.

Downloading multiple videos or a playlist

Now that we know how to download a video in the perfect format for us, why not apply the same settings to multiple videos at once? It’s very simple! Just add more URLs in our terminal separated by a space. For example we can type

youtube-dl -x --audio-format m4a --audio-quality 160k https://www.youtube.com/watch\?v\=UyyxYha4j6o https://www.youtube.com/watch\?v\=6L5a3Qt7cvg

to have both videos converted into audio tracks.

If instead we wanted to download a playlist instead of a single video we just need to insert the playlist URL!

Making our life easier with aliases

When I want to download songs from YouTube, I usually always use the same settings. And typing youtube-dl -x --audio-format m4a --audio-quality <URL> every time is a pain as well as hard to remember. It would be much better if you could just type yda <URL> (yda stands for Youtube Download Audio). The good news is that you can!

Let’s open the terminal and enter

echo "alias yda='youtube-dl -x --audio-format m4a --audio-quality 160k'" | tee -a ~/.bash_profile && source !$

What does this command do?

echo "alias yda='youtube-dl -x --audio-format m4a --audio-quality 160k'" | tee -a ~/.bash_profile: write the new alias at the bottom of the ~/.bash_profile file. This way, every time we open a new terminal we’ll have this alias ready to use.
&& source !$: after writing the new alias, reload the ~/.bash_profile file in the terminal, so that the new alias is available in the current terminal too.

Now, if we want to download any song from YouTube it will be very simple.

Let’s search for the video on YouTube, copy the URL.
Let’s open the terminal and type yda <URL>.
Done!

I want to know more!

Youtube-dl doesn’t only have these settings, these are just the ones I use. If you want to know more just type youtube-dl --help. This way you’ll have all, but really all, the settings of this program and I assure you there are a lot of them!

Conclusions

I hope this guide has given you one more tool to download the YouTube videos you care about most, without needing online sites full of ads that work once in a while. I also hope that now you’ve grasped the potential of the terminal, an application used daily by developers, but unjustly snubbed by many!

See you next time!

How to play videogames from two or three eras ago

Mon, 02 Jan 2017 07:03:47 GMT

During these holidays I got a moment of nostalgia for the old, very old videogames from the ’90s that I used to play on my first PC, an ancient Compaq Presario with a 133MHz CPU.

It seems impossible but it’s still possible to play these videogames without too many compatibility problems thanks to DOSBox. This program is nothing more than an emulator of the DOS operating system: in the ’90s many games supported this operating system and so we can take advantage of that.

In this guide I’ll explain how to play some videogames that marked my childhood.

Let’s install DOSBox

First of all you need to install DOSBox; I recommend using Windows as the operating system… On Mac I have some performance problems… It seems absurd but that’s how it is :D. The installation is nothing special: just go to the download section of the website and download the package and install it like any common software. Once this step is done we’re ready to dive into the past!

Grand Prix Circuit

Let’s start with a very simple game. Grand Prix Circuit!

Download

First we need to download the game. Download the package and extract the files from the .zip archive into a folder, for example C:\games\gp\.

Mounting the volume

Now let’s start DOSBox. A command prompt appears, which is nothing more than the classic DOS shell. The first thing to do is mount the volume where our game lives. In simpler words we need to give DOSBox a C:\ volume, a portion of memory, that has access to the game. Without this operation DOSBox can’t read any data from our hard disk precisely because the volumes (C:\, D:\ etc.) that Windows uses are completely different from the ones DOSBox uses. To do this operation just type in DOSBox

mount c c:\games\

Now that we’ve mounted our games folder as volume C, we can access it through DOS.

c:

You can see that now DOSBox shows C:\> instead of Z:\> precisely because it’s inside our newly created volume. Basically now in DOSBox we’re in the C:\games\ folder of our PC.

Starting the game

Now let’s use another DOS command:

dir

this command shows all the files and folders contained in the path we’re in. Our gp folder should appear. To access it we use the command

cd gp

Now we’re in the folder where the Grand Prix Circuit we just downloaded is. To start the game we just need to run the command

GP.EXE

Congratulations! Now you can have fun playing a great game that made history!

Summary

Good, so far we’ve learned to do the following things:

Download the game and start DOSBox.
Mount the games directory to give DOSBox a portion of memory from our PC.
Navigate between folders in DOS using the cd and dir commands. This guide doesn’t aim to teach you how to use this operating system so I’ve been quite brief. If you want to know more you can read up via this link.
Start the game.

These steps work for a good part of the old videogames, but there are others that require installation or inserting the CD-ROM to play. The next game falls exactly into this case.

Ignition

I really like this game a lot! It reminds me of the hours spent with my cousin many years ago!

The startup mode is similar to the previous one, except that in this case we need to install the game as if we had a CD-ROM.

Download

Let’s download the game and extract the archive into C:\games\ign\.

Mounting the volumes

Let’s mount the C: volume with the same command used before

mount c: c:\games\

Now let’s insert the Ignition CD-ROM! How? More or less like mounting a classic volume.

mount d: c:\games\ign\ -t cdrom

In this case we’re telling DOSBox to show our c:\games\ign\ folder, where the downloaded game is, as if it were a CD-ROM.

Installation

Now let’s enter the CD and start the installation

d:
DOS_INST.EXE

At this point the installation will start. Choose your language and keep pressing next.

Starting the game

Now we’re ready to play. Type these three commands and have fun!

c:
cd IGNITION
IGN_DOS.BAT

How to download songs from SoundCloud in a few simple clicks

Sun, 13 Nov 2016 07:03:47 GMT

SoundCloud, for those who don’t know it, is a great website that allows musicians to collaborate, promote and distribute their music.

Unfortunately it doesn’t allow songs to be downloaded, but by using Google Chrome’s developer tools it’s very easy to bypass this restriction.

Instructions

Go to SoundCloud using Google Chrome, and open the page of the song you want to download.
Open the developer tools by pressing F12 on Windows or cmd + opt + I on Mac. Click on the Network tab. In this section, click on the sixth icon from the left. This will open the timeline of the various downloads the browser is making.
Now you’re ready to intercept the .mp3 file to download! Start the song in the browser. As soon as the song starts you’ll see some downloads starting on the timeline, one line will surely catch your attention because it will be much longer than the others. This line represents the song in .mp3 format.
Select the song from the timeline. To do that, just drag the mouse over a small piece of the line; I recommend picking a part where this line is isolated so you don’t get the selection wrong.
Once you’ve selected the download in the timeline, the entry for your track will appear at the bottom. Right-click it and click on “Open link in new tab”.
That’s it! Now you have your track in .mp3! Click on File->Save as... and enjoy the song wherever and whenever you want!

Video tutorial

Do 6 steps seem too many? I assure you that once you get the hang of it you can download whatever you want in less than 2 seconds. Here are the 6 steps wrapped up in a short video.

See you next time!

How to improve your photos with Nik Collection

Thu, 11 Aug 2016 07:03:47 GMT

Digital has revolutionized quite a few industries, including photography. I entered this world right at the moment of transition… and I must say I really don’t miss the old film roll.

A great advantage of digital photography is certainly the fascinating world of post-production. Photography purists hate it, veterans of the old film roll think it’s an abomination. For me, on the other hand, it has become a phase almost as important as the shot itself: with post-production you can completely change a mediocre photo and turn it into a unique shot, add dynamism, give character to an insipid photo.

This little guide will teach you exactly that (or so we hope): the photo I’ll use was taken in France and it’s the very famous Notre Dame de Paris.

Before we start

To follow this guide you need professional software: I used Adobe Photoshop. You can also use Aperture or Lightroom. These programs cost a lot: the first thing to do is buy them or hope your company has a license for you, or that some friend of yours is particularly kind and passes you one of their license slots (Photoshop can be used on up to 3 PCs with a single license).

This guide doesn’t aim to teach you how to use these programs; let’s say having at least a basic knowledge of them is a prerequisite.

The star software of this guide, however, is this collection of plugins that rely on them: Nik Collection.

Until last year this suite cost about $500, and in my opinion it was worth every penny. Now instead Google has decided to make it free!

Choosing the photo

The first thing to do is choose which shot you intend to improve / modify. One thing must be clear: post-production can work miracles but there’s a limit to everything. If the initial photo is bad, it’s likely that even under heavy retouching it will still be bad. Post-production enhances the content, but if the initial value is zero it’s hard for something sensational to come out. So always try to take the best possible shot to have a good starting point. My photo is this one: I took it with my LG G3, not a DSLR, just a 3-year-old smartphone. During my stay in Paris I didn’t have anything better, but it still has great potential.

Once the photo is chosen you have to decide what to do: maybe if you don’t know what Nik Collection can do, this step will be a bit difficult for you. My goal in this photo is to enhance the cathedral, make the details stand out and give a dramatic tone, typical of cathedrals, to the whole scene.

Let’s get to know Nik Collection

Now that we’ve decided our starting point and our goal, it’s time to get to know the tools at our disposal.

If you haven’t already, download Nik Collection and install it. Photoshop must already be present on your PC.

Once installed, if you start Photoshop, you can see a small new box where you can choose the various plugins that make up this suite:

Dfine: it’s the first plugin to start. Its job is to remove noise from photos.
Analog Efex Pro: simulates the effect of old films and old cameras that made history with their particular effects. Do you remember Instagram filters? Well, you’ve grasped what this plugin can do.
Color Efex Pro: if you want to correct some tints, create effects that have to do with colors, this plugin is perfect.
Silver Efex Pro: very useful if you want to create a monochromatic image.
Viveza: great if you want to modify only specific parts of the photo, highlight subjects.
HDR Efex Pro: to create an HDR image. Without getting too technical, to create a real HDR image you need multiple shots of the same subject using different exposures. But this plugin manages to achieve the same result even from a single image.
Sharpener Pro: its job is to make details stand out and make the image less blurry.

If for some reason the Nik Collection box doesn’t appear, you can find all these programs in the top menu under Filters > Nik Collection.

Let’s remove the noise

Now it’s time to start! The first step I always take is removing noise from the photo. If the shot was taken with a camera that isn’t particularly powerful, like my smartphone, this step is practically mandatory. Noise, even if imperceptible, is amplified during the editing steps. So cleaning the image from these disturbances is very important.

This step is very simple. Load the image into Photoshop, start Dfine. As you open it you can notice that it does some magic and removes noise automatically. It might seem like nothing has changed; actually to appreciate Dfine’s work it’s useful to zoom the image to 100%. To do that double-click the image; finally, at the top left, where it says Views, you have three display modes:

Single image view: the default view. You only see the final output.
Split view: you can see the image before and after applying the filter at the same time. You can also move the red line to have a quick comparison between the two versions.
Side by side preview: basically you’ll have the two images, before and after applying the filter, side by side.

Once you’ve been able to appreciate the changes, press OK. So you can move on to the next step.

HDR Efex Pro, it’s your turn!

Now it’s time to add a touch of HDR to the image. HDR stands for High Dynamic Range and it’s a photographic technique that allows you to show more details than a normal photograph. Do you remember those situations where you take a photo of an environment that has both particularly bright and dark spots? It happens that the very bright part comes out as just a nice white patch and the dark part as a black patch. HDR manages to overcome these problems. This is just to give you an idea of what HDR is; you’d need to write a guide just on this… But it’s better to get back to our image!

Always from the Nik Collection box, in the HDR Efex Pro section choose Tone Mapping (Single Image).

On the left side you can find the Preset Library, in it there are already some great starting points. I chose Deep 1 which is in the Realistic library.

Once you’ve chosen the starting point you can focus on the part on the right of the window. Here you can find a sea of configurations to modify every single aspect of the filter. The macro areas are these:

Tone Compression: here are all the settings that only concern the HDR effect. Surely this is where you find the settings that radically change the appearance of your image.
Tonality: here you can find the classic image settings like exposure, shadows, highlights, contrast, etc. An option I particularly like is structure. By increasing it you can notice how the image becomes particularly defined. Be careful not to overdo it because in addition to enhancing details it also enhances noise. That’s why I strongly recommended removing it with Dfine… Now the chickens come home to roost!
Finishing: it’s the post-production of the post-production effect! I mean, it’s the final touches you can make before leaving this plugin. From this area I added a small vignetting effect. Near the Vignette text there’s a small selection menu. I chose Lens 2. The vignetting effect does nothing more than darken the edges of the image (or lighten them). It’s a great way to guide the observer’s attention. In this case my intent is to enhance the cathedral.

In this case too I haven’t gone into too much detail. My advice is to play and try all the various features yourself. Only this way can you master this suite… By using it!

Here’s the final result after this step.

Let’s correct the colors

Now the image has a much more dramatic tone than the original, it suits a cathedral very well. The details of the structure have been highlighted, we can be satisfied with our work! There’s just one small problem… The cathedral has become a canary! It’s too saturated and so we need to remove some saturation from the cathedral.

To do this let’s start Color Efex Pro. The window that opens should be familiar by now: only the starting filters on the left and the settings on the right change.

The filter we’ll use to desaturate the image is called Bleach Bypass. Let’s select it from the available filters and we immediately notice the changes on the image.

So far we’ve applied the filters to the whole image; in this case we want to apply it only to the cathedral without affecting the rest.

Say hello to control points!

To solve this problem we need to learn a very important feature that I’ve kept hidden from you until now. Control points.

These “control points” allow you to apply the filter only to specific areas. Let’s immediately try using one. On the right side look for the Control Points function; here you can notice two buttons: the first shows a circle with a ”+” next to it and the other a ”-“. The difference between the two is that the first adds a point with 100% opacity, the second with 0%. In other words the first applies the filter at the selected point, the second removes it.

We need to apply the filter at the point we select. So we need to press the button with the ”+“. Once pressed, move the cursor over the cathedral; you’ll notice that the cursor symbol has become a sort of crosshair; click on the cathedral. Nik Collection is smart enough to understand that you want to apply the filter only to the selected point, so it removes the filter from the rest of the image and leaves it only on the control point.

Once the control point is placed you can notice that there’s a small circle on the image with two sliders. The first determines the range of the control point, the second its opacity. Obviously the larger the range, the more our filter will affect a larger portion of the image, and the more we increase the opacity the more the difference is noticeable (in our case it’s already at 100%).

Since the control point has a circular range, I recommend adding many small points rather than having one big one to have more control over it.

But how do we understand the real range of a control point? In fact Nik Collection applies the filter intelligently: even though the range is circular, you don’t notice the point where it stops applying these corrections. Precisely because Nik’s strong point is its algorithms for creating the application masks. This sentence is probably gibberish for those who aren’t familiar with Photoshop, but it will be clear shortly. To understand what I wrote let’s go back to Color Efex Pro; in the Control Points section we can notice the list of our added points. Each entry in this list is made up of 4 points:

Checkbox to enable the control point.
Its name.
The percentage of the image affected.
Checkbox to show the application point (its mask).

Let’s check this last checkbox of any control point we’ve added. The image becomes monochromatic and from here we can actually see the real application point of the selected control point’s filter. Black indicates that the filter hasn’t been applied, white where it has been applied, the various intermediate shades indicate a different opacity of the application.

Add all the control points necessary to “make the cathedral white”; this way you’ll apply the filter only to the cathedral. Here’s my white cathedral, a sign that I’m applying the bleach bypass filter only to it.

In my opinion control points are the real strong point of Nik Collection. To create the same mask using only Photoshop, instead of taking 5 minutes, it would have taken us hours!

Once we’ve made sure the filter is applied only to the cathedral, we can uncheck the checkboxes of the control points’ masks and go back to viewing the normal image.

Finally here we are!

Conclusions

Here’s the final result next to our original image:

I hope this guide has given you a vague idea of the potential of this plugin suite. Obviously this is just the tip of the tip of the iceberg: my intent isn’t to explain the suite and all its commands, but to apply it in a real use case.

If you feel like learning more, I strongly recommend this introductory video tutorial… It’s in English and lasts an hour… But it’s really worth it!

See you next time!

How to remove viruses from your computer

Sun, 07 Aug 2016 07:03:47 GMT

“Hi! How are you? Listen, I think I have a virus on my PC, can you take a look at it?”

The variations of this sentence are endless, but more or less almost all the people who call me to get a hand with this problem start like that.

Today I’ll explain what you can do to solve this problem on your own, without spending a penny and without even needing any special skills.

There are viruses and viruses

Before starting, it’s better to know at least a little about the various types of viruses you might run into. In my opinion it’s useful to understand the error messages your antivirus shows you when it finds something unusual.

Malware: this is anything that can harm your PC. So it’s the macro category that contains all the entries below… In fact I should have written “There are malware and malware”, but whatever, you get the idea!
Virus: malware that is able to self-replicate and infect other users through USB sticks or removable media.
Worm: similar to a virus but it infects other PCs through the network.
Trojan: it shows up as useful software, but in reality it literally opens a door for an attacker to steal your sensitive data or take control of your PC. It really is a Trojan horse!
Spyware: spies on what you do on your PC and sells this information on the black market.
Adware: fills your PC with unwanted advertising.
Keylogger: records the keys you press hoping you type some credentials or, even better, your credit card number.
Backdoor: if the attacker manages to install this stuff, they can use your PC to do several things on the network: for example send emails or connect your PC to a website to overload it and cause an outage.
Rootkit: these are very hard to find; if you have one it’s as if the attacker had your PC in their house. They can simply do whatever they want remotely.
Scareware: these are malware that scare you, telling you that your PC has millions of problems and that you should pay them to fix everything. Too bad the real problem is the program itself pretending to help you. Most of the time people catch this type of malware.
Cryptolocker: it has been very popular recently. Basically, once started, it begins to encrypt all your documents and when it’s done it asks you for a ransom. If you want your documents back you have to pay!

Prevention above all!

Disclaimer: I don’t use antivirus, in my opinion it’s possible to live without it too. The important thing is to be careful and know what you’re starting… Always! If instead you don’t feel safe, then it’s better to install some antivirus that can cover your distractions in real time. Finally, if you already have malware on your PC then install it immediately. I usually install Bitdefender on the PCs I fix. There are really very few options and it does its job. Once installed, run a full scan of the PC and hope it manages to solve your problem.

Uninstalling the malware

The antivirus does two things well: keeping viruses away before they can do damage and removing them if they happen to already be inside your PC. The problem is that they are only very good at one category, viruses precisely. But as you read a moment ago, there are a myriad of malware that don’t fall into this category. If your malware is still there bothering you, you need something else… Maybe even just a little something else… Sometimes all it really takes is uninstalling the app that’s bothering you without needing any special software.

Doing this step is very simple:

Read the name of the app that opens and bothers you.
Go to Control Panel > Programs and Features > look for the app name and uninstall it.
If it happens to tell you the app is in use, close it. If you can’t close it, open the task manager by pressing ctrl+shift+esc. Go to the processes tab and terminate the app in question.

These steps can vary significantly depending on the version of Windows you have.

Broad-spectrum antibiotics

I imagine that if you’re reading here it means your dear malware loves you very much and is still there on your PC making your day hell. Good! Now it’s time to use two pieces of software: Malwarebytes and AdwCleaner. I call these two pieces of software broad-spectrum antibiotics precisely because that’s what they do: they try to remove any malware and therefore manage to remove more junk than a normal antivirus.

I recommend starting with Malwarebytes. Download it, install it (you don’t need the premium version), update its database and finally run a full scan. All these steps should be simple so I won’t dwell on them too much.

Once the scan is finished, it has probably found something; follow the steps and remove what it found. Finally it will ask you to restart the PC. Do it.

Now it’s time for the second piece of software: AdwCleaner. Here too the steps are simple. Download it, install it and finally start it. The screen is very simple: first press Scan to start the scan. Then, once it’s done, press Clean to start the cleaning. Finally it will ask you to restart the PC. In this case too, do as it says.

A little experience never hurts

At this point I hope your problems are over. If instead your malware is still there then it’s time to call a technician because the easy-to-use tools are finished and an expert hand is needed. All you can do is save what can be saved and pick up the phone to ask for help.

Last resort

Actually there is still one small thing you can do on your own, even though it’s quite dangerous: you can attempt a full reset of your PC. Over the years resetting a PC has become a very simple process and in my opinion it’s a step that has become within everyone’s reach.

The first thing you must do is save your data, and by save I mean to an external hard drive, a removable drive such as a USB stick. In short, save it outside the PC to be sure you don’t do any damage.

If you have Windows 10 the steps are simple: go to Settings > Update & Security > Reset this PC > Get started and you can choose between these options:

Keep my files: maybe try this option first. This way your data is safe, but Windows 10 will be reset and your applications will be deleted.
Restore factory settings: this option is the most effective because it makes your PC like new.
Remove everything: choose this option if you can’t find the previous entry.

Conclusions

I hope this guide helped you solve your little problem on your own… In the end there’s also a nice satisfaction in solving problems without anyone’s help, right :)? See you next time!

Welcome!

Wed, 03 Aug 2016 07:03:47 GMT

Here I am, on the Eurocity heading to Switzerland, obviously stopped because of the works at the Monza station.

For this very reason I decided to spend the time I pass on trains writing this little blog of mine (and creating it too).

I haven’t quite decided yet what or how much I’ll write. Probably many articles will be guides to solve the millions of problems I occasionally solve for those who “hire” me as a computer technician. Or something else, I really don’t know yet. For now I’ve had fun creating this little blog. Now only the content is missing! :)