Rui Alves Blog

The Return of the Jedi - Part 3

Mon, 26 Jul 2021 00:00:00 GMT

This is the final part of a multi-part post. If you haven't already, feel free to check out the other parts:

Part 1 - Boosting my chess club's website with Cloudflare and Heroku
Part 2 - Insights on my MSc Thesis
Part 3 - You're here!

I'm finally an engineer! 🎉

A couple of weeks ago, on the 12th of July, I finally defended my MSc thesis — The presentation went really well and I had a really nice and insightful discussion with the jury, managing to pull off a final grade of 20, which was pretty neat. With that, I've finally graduated as a Computer Engineer with a final GPA of 19/20 — the hard work paid off!

That being said, I'm now starting a new big journey. Earlier today I arrived in London where I will be staying in the future to work for Palantir Technologies as a Software Engineer! I'm really excited to start, and I'm sure I will be facing a lot of interesting challenges and that I will learn plenty of new things.

This wasn't much of a post, it was more of a status update than anything 😅 I'm probably going to be busy with settling in during the near future, so I will likely not be posting anything in the meantime. I will be back once I have something interesting to write about, though 😋

Cheers!

The Return of the Jedi - Part 2

Sun, 04 Jul 2021 00:00:00 GMT

Hey! This is the second part of a multi-part post. If you haven't already, feel free to check out the other parts:

Part 1 - Boosting my chess club's website with Cloudflare and Heroku
Part 2 - You're here!
Part 3 - Finally an Engineer and next steps

For this second part, I want to focus on the stuff I'm working on at the moment. Throughout the past months, I've been working on my master's degree thesis — Towards the Partitioning and Operation of Web Applications leveraging Platform and Function as a Service — I challenge and dare you to say the full name in one go, I double dare you! I know I certainly can't 😅

Jokes aside, it has been pretty insightful to be researching and learning so much about cloud computing. Although I didn't have the chance to explore as many services and "stuff" as I wanted to, I certainly did learn a lot! Unfortunately, I can't provide a lot of information about my thesis at the present moment since I will be defending it in my final presentation on the 12th of July — The presentation is going to be public and streamed on Twitch, so feel free to show up!

Anyway, I think it should be alright if I share just a few insights. My work consisted of facilitating the partitioning and deployment of web applications into multiple cloud service models (namely PaaS and FaaS) in order to allow developers to optimise their applications by using the best that each model has to offer. Although this may sound cute and idyllic, it is not quite trivial since it poses a great deal of challenges in infrastructure management, system observability and monitoring, traffic routing, overall robustness, and a bunch of other things.

That being said, my research in the past few months culminated in really-big-document-with-over-100-pages-dissertation and in the development of a plugin package for Python that works on Django web applications. A quick disclaimer: the package itself is simply a proof of concept of my dissertation's key findings and still needs a lot of work in order to be fully robust, work with a multitude of cloud providers (at the moment, it only operates with Azure and Heroku), and to be of real use to the cloud development community. Additionally, I also synthesised a part of the key findings (mostly regarding traffic routing among distinct cloud service models) into a small scientific paper that should be published in a cloud-related conference in the near future (more on that later)!

It is hard to work on other things while one is focused on something as "big" as an MSc thesis; However, in the meantime, I've finally picked up some motivation and started to add a couple of features to my personal chess website. Moreover, I've started to learn a bit about Kubernetes, something which has been in my backlog for ages now.

In the following and final part, I'm going to be writing about my future steps after finishing my degree in computer engineering — BIG things are coming...!

Cheers!

The Return of the Jedi - Part 1

Sat, 03 Jul 2021 00:00:00 GMT

I'm back! This is my first time writing something here in months since I haven't had much time (nor motivation, to be honest) recently since I've been focusing on my thesis and a couple other things. Nevertheless, I've got a bunch I want to write about, so I'll be breaking this post into different parts, so stay tuned!

Part 1 - You're here!
Part 2 - Insights on my MSc Thesis
Part 3 - Finally an Engineer and next steps

For the past few years, I've been developing and maintaining my chess club's (Grupo de Xadrez do Porto) website in my free time, as I've mentioned in a previous post. The website was developed using Meteor and features a bunch of information about the club's activities, players, and a few other things. I initially started by hosting it using AWS free tier, which offers a low-budget VPS. After the free tier terminated, the club decided to extend the AWS hosting for another year.

In terms of domain management, the club has been renewing the same domain for years and years now. Initially, the domain was being managed at dns.pt, which is terrible in terms of management flexibility. I "upgraded" the management to dominios.pt which, although it wasn't perfect, was surely a big improvement.

However, in the end of the last year, it was brought to my attention by a good friend that CloudFlare was not only pretty neat for domain management, but also the first company to offer SSL/TLS protection free of charge and without bothersome configuration. Since I had some free time on my hands, I decided to go all-in with CloudFlare, and I am very happy with the results. The process took less than a couple of hours, and I've then started using CloudFlare for my personal domain as well — It is a game-changer.

While I was at it, I also decided to make some changes to the website itself. Since it was hosted in an AWS IaaS, managing updates and adding new features was at times a bit cumbersome and not as automatic as I would like. After doing some digging around the internets, I found out that Heroku free tier features a Meteor buildpack which facilitates the deployment process. I then created a staging instance to check if it was "solid" enough to hold up in production, and did a bit of load testing using Locust to check if it held up with some traffic in its hands. Since we don't have (and won't have any time in the near future) a large number of people accessing the website, it proved to be more than enough. I have then deleted the AWS instance and moved everything to Heroku, and I now have a much easier time pushing updates and changes to production. Additionally, it was also quite easy to configure the custom domain in the Heroku PaaS instance — I simply added the domain and received a DNS target that allowed me to create a CNAME record in CloudFlare, easy peasy!

Finally, I also added a couple of new features to the website's management dashboard. Firstly, I added the capability of bulk editing players info, which is pretty useful, specially in the beginning and ending of tournament-season. Additionally, I enhanced the updating of photos (some of the pictures were hard-coded, which was far from ideal 😬) and did some minor tweaks in styling here and there. I'm finally moderately happy with the website 😅

In the following parts, I'm going to be writing a little bit about some side projects, my thesis, and my future work plans.

Cheers!

I miss Programming!

Wed, 24 Feb 2021 00:00:00 GMT

It's been a while since I last wrote something, mainly because I've been dedicating most of my time to my MSc Thesis (more on that later). However, I've been fiddling around with some side projects and background ideas.

Chess games page

Chess matches annotations follow a specific standard called PGN. Although my chess games pages already allowed exporting my games to this PGN format, it didn't follow the metadata standards and wasn't fully compatible with PGN readers. Thus, I took some time in adapting the "Export Game to PGN" functionality to ensure it followed the standard.

I had been using this functionality mostly to later import the match using Lichess so that I could analyse it. Since this takes a lot of clicks, I decided to also add an "Analyse with Lichess" functionality that uses the Lichess API to export the match and quickly analyse it (using Lichess' Stockfish).

Besides that, I fixed some small things here and there (mostly to improve UX a bit). I've also integrated Google Analytics in the website, which was actually fairly straightforward since Gatsby (I'm using Gatsby, incidentally) features a plugin that automates most of this purpose. Although it provides some pretty cool insights, most of the stuff looks to me like a bunch of codswallop, to be honest 😅

You can check this stuff out at a match's page on my chess website.

MSc Thesis

My thesis is named Annotation-Based Runtime Partitioning of Web Applications into FaaS. Basically, my premise is that if a cloud-based web application can automatically move parts of itself from IaaS/PaaS into FaaS and vice-versa at runtime, it should be able to improve stuff such as client latency and infrastructure costs — that is, Cloud-stuff goes beep boop.

I'm currently doing research on this domain and I'll be implementing a proof of concept plugin for the Django web framework, throughout the next few months, to validate my findings.

Competitive Programming

A few months ago I stopped doing LeetCode's daily challenges and competitions altogether and, meanwhile, I completed the Advent of Code 2020 challenge. Other than that, I've been completely out of action in this type of stuff. A few days ago I started to miss it, so my plan is to pick up Code Wars and some old AoC unfinished challenges that have been in my backlog for way too long.

Research work is weird and different from what I'm used to. However, this is the last step in my MSc and, after this, I'll be doing more practical stuff for a lot of time, so I guess I'll do my best to enjoy the ride.

Cheers!

Advent of Code 2020

Thu, 07 Jan 2021 00:00:00 GMT

Advent of Code is a Christmas-theme programming competition that takes place every year during the first 25 days of December. Usually, it's your job to save Christmas!

I've been participating every year for the past five years, and I always end up getting addicted and spending a lot of time on the challenges, so this year I decided that I wouldn't be participating. Naturally, this resistance lasted a single day, and, except for day 1, I managed to solve all exercises in their publish-day.

I wanted to share some insight into the challenges that I found interesting and enjoyed the most.

Day 10: Adapter Array

This challenge consisted of helping our hero to connect a set of joltage adapters. Part 1 was pretty straight-forward. However, the second part was a bit more challenging. This was the first challenge that wasn't solvable using a brute-force solution (as visible in the leaderboard). I ended up using a DFS + cache solution, although the challenge was screaming for dynamic programming:

def dfs(joltages: ArrayBuffer[Long], i: Int): Long = {
    if (cache.contains(i)) return cache(i)

    cache(i) = (1 to 3).map(j => {
        if ((i + j < joltages.length) && (joltages(i + j) <= joltages(i) + 3)) dfs(joltages, i + j)
        else 0
    }).sum

    cache(i)
}

Day 13: Shuttle Search

This one consisted of fixing a set of bus schedules. Similar to the previous one, part 1 was quite trivial. When reading part 2, I had some flashbacks of the Discrete Mathematics course I took about four years ago, specifically about the Chinese Remainder Theorem. In short, it consists of finding a number N, such that:

N = a (mod x)
N = b (mod y)
N = c (mod z)
...

With a few tweaks and tricks, this can be applied to our problem. Assuming that the input is 17,x,13,19 and that we want to find the earliest timestamp T:

T = 0 (mod 17)
T + 2 = 0 (mod 13)
T + 3 = 0 (mod 19)

...and by applying a bit of mod-space algebra ...

T = 0 (mod 17)
T = 13 - 2 (mod 13) = 11 (mod 13)
T = 19 - 3 (mod 19) = 16 (mod 19)

Although after browing through the AoC Reddit I found out that there were other ways of solving this, I'm absolutely sure this is the coolest one 😎

Day 17: Conway Cubes

Although not difficult, this challenge took Conway's Game of Life to the next level by creating a 3-dimensional and 4-dimensional version in parts 1 and 2, respectively. Even though I'm pretty happy about my solution, it doesn't need to be very efficient since the problem statement only asks for 6 iterations.

Day 18: Operation Order

This one consisted of evaluating a numeric expression. The twist is that operators have different levels of precedence, which was a bit counter-intuitive! I used a stack-based approach to solve it linearly, similarly to my solution for a LeetCode Calculator Problem I had solved about a year.

Day 23: Crab Cups

In this challenge, the goal was to play a version of the cups game with a crab (yeap, it's a Christmas-themed competition). Part 1 had a very small-sized input, so it was solvable without the need for a very elaborate solution. However, the input size was much larger in part 2. Since I was pretty much expecting the differences in input size to be substantial between both parts, I tried to come up with an efficient solution from the start.

This problem was, by nature, very similar to the LeetCode LRU Cache problem I had solved a few months before.

Day 24: Lobby Layout

I find hexagonal grids to be very interesting for some reason (as do bees, go figure!), so I really enjoyed this one. After reading the statement, I instantly remembered about a similar Advent of Code 2017 Challenge I had solved a few years back, so I ended up using the same 2D-matrix coordinate representation I used back then:

final val STEPS_DELTAS = Map(
    "w" -> new Cell(-2, 0),
    "e" -> new Cell(2, 0),
    "sw" -> new Cell(0, 1),
    "se" -> new Cell(1, 1),
    "nw" -> new Cell(-1, -1),
    "ne" -> new Cell(0, -1)
)

However, a friend of mine pointed me to an article that actually uses a lot let space with a slightly different representation:

final val STEPS_DELTAS = Map(
    "w" -> new Cell(-1, 0),
    "e" -> new Cell(1, 0),
    "sw" -> new Cell(0, 1),
    "se" -> new Cell(1, 1),
    "nw" -> new Cell(-1, -1),
    "ne" -> new Cell(0, -1)
)

I solved all the challenges using Scala, and I ended up improving my coding skills with this language quite a bit since I pretty much learned something new every other day. For me, this year's edition was the most enjoyable one by far, which motivated me to solve every challenge in their publish-day. I will be back next year with more Xmas-coding!

Cheers, I (sincerely) hope that 2021 is better than 2020 for everyone!

SINF 2020 Programming Contest

Fri, 13 Nov 2020 00:00:00 GMT

A few months ago I was invited by the SINF - Semana de Informática (Informatics Week) organization team to organize and develop the challenges for their annual Programming Contest.

A bit of context: SINF is an event organized by students for students, that takes place every year at FEUP, the Faculty of Engineering of the University of Porto, which aims to provide tech-related content and first-hand contact with tech companies to college students. The event has been organized by NIAEFEUP in the past 3 editions.

Although I've been a member of the organization team of this event in the past 2 editions, I was quite glad to take on a different role this time. This was my first time ever organizing a Programming Competition and creating new original problems, which was quite an interesting challenge! However, I must add that I got a lot of help from my friend Guilherme Silva, which not only was my beta tester but who also gave me a lot of feedback regarding how to structure and improve the problems. I would also like to thank Professor André Restivo for taking the time to review the problem statements.

The contest was held from the 8th to the 11th of November 2020 and featured a total of 6 original problems, which were released in pairs throughout the 3 days of competition. A total of 66 people from different parts of the country signed in and, in the end, a total of 5 people were able to solve all the problems!

The problems were organized in a sort-of-ascending-difficulty-manner, and their main theme was to help the organization team solve problems that required the help of programming! Each of them was assigned a score based on its difficulty:

Scheduling the First Meeting - 100 points (Easy), 88.89% success rate
Minimizing Costs - 275 points (Medium), 60.87% success rate
So many speakers - 150 points (Easy), 75% success rate
Grouping lecture themes - 250 points (Medium), 94.12% success rate
Organizing the workshop - 250 points (Medium), 73.33% success rate
Time to celebrate - 350 points (Hard), 41.67% success rate

More details about the competition can be found on the contest's Hackerrank page.

Since doing it by hand would be not only a nightmare but also not reusable in the future, I developed a set of scripts to generate all the inputs and outputs. All of this and my proposed solutions for the problems can be found in my GitHub repository.

I must say it was a lot of fun to get out of my comfort zone and accept this challenge and, for once, to be on the creating side of competitive programming. I hope I get the chance to do it again in the future!

Cheers!

Scala, Hacktoberfest and my MSc Thesis

Sat, 07 Nov 2020 00:00:00 GMT

Learning Scala and Competitive Programming

For a couple of months now, I've been wanting to learn Scala. I wanted to pick up something in the functional programming spectrum, and Scala seemed like a very good choice (lots of support and documentation, cute syntax, ...). I was quickly amazed at how clean and tidy scala code looks. Moreover, I was pretty happy with the fact I could start by coding more OOP-java-like code (which is what I'm used to) and slowly pivot towards the functional side.

Right at the same time, I found out about Code Wars (which is way too addictive incidentally) and start spitting out some cool one-liner solutions to the easier exercises while I got acquainted with the language. I also re-started Advent of Code 2019, which has a few nice problems that were cool to learn some new Scala concepts.

On another note, I decided to drop LeetCode's daily challenges, since most of them are repeated and/or a bit too easy. I guess I might come back to them during the next month's challenges if I start to miss them 😛

All in all, Scala seems to be pretty powerful and has a lot of ways of getting the same task done. As time goes by, I want to pivot towards the functional side of the scala spectrum as much as possible, since it was what got me into it in the first place.

Hacktoberfest

This year was my second time participating in Hacktoberfest, an open to everyone month-long celebration of open source software that takes place every year during the month of October.

Last year I decided to contribute to tldr and joined two local Porto hubs to hack alongside other people, which was pretty cool (I also won a pair of Sammy the Shark plush slippers that I am actually wearing right now).

However, in 2020's edition my participation was fully remote (due to the current pandemic situation) and was more focused in contributing to some projects from FEUP's Computer Engineering Student Branch (in which I've been a member of the board for 2 years and member for almost 5 years now). Earlier in October, I was aiming to contribute to Lichess, a free and open-source internet chess server. However, since most of their services run in Scala, I didn't feel quite ready to dive into their codebase (which was another motivation for wanting to learn Scala!).

MSc Thesis: Automated API Refactoring for Edge Computing

As part of my last year in the Computer Engineering MSc, I am now working on my MSc thesis and dissertation, under the mentorship and guidance of Professor Tiago Boldt Sousa. I will be working towards implementing a web framework capable of self-inspection and self-organization, with the capacity of moving parts of its own API from cloud to edge computing, and vice-versa.

I am quite happy with this topic, as it is a great opportunity to learn more about serverless computing models, as well as cloud and edge computing paradigms. Looking forward to investigating more about these subjects!

The SINF 2020 Programming Contest (the first programming contest I've developed by myself) is starting tomorrow and will be up and running for 3 days, with new originals challenges every day! I'm planning to write a bit about it (and maybe discuss some of the problems) in the near future.

Cheers!

So much to learn, so little time

Sat, 19 Sep 2020 00:00:00 GMT

This post will have a slightly different structure than usual since I want to share a bunch of things (without going into much detail into either of them). I can only start by saying this past month was a lot of fun, because I had a large amount of free time to work on a bunch of things that were on my backlog.

Organizing a Programming Competition

A few weeks ago I was asked by the SINF (a tech-related event for students) organization team to develop a set of programming challenges and host a competition for the event.

I was very happy to accept the challenge and I've come up with some pretty cool ideas that will, hopefully, make the competition challenging and fun, while at the same time being "friendly" for people who are just starting.

The competition will last for 3 days and problems will be handed as daily challenges, so there's a lot of room for imagination!

Ethical Hacking is pretty cool

Being a nerdy nerd as I am, I've always wanted to learn a bit about Ethical Hacking. Although I already had some concepts in mind that I have learned at college, I was mostly motivated to start it due to a friend's blog articles, in which he details his thought process when... "hacking stuff".

I decided to start an online course (so that I could learn about it in an "organized" way) and decided to pick The Cyber Mentor's course, since it had pretty good reviews. I'm now about 35% through the course and I feel like I have learned a bunch.

As suggested in the course's lectures, I'm trying to take notes regarding methodologies and tools in an organized way. For that, I've been using Zim Wiki. It is pretty user friendly and has a small learning curve (in terms of syntax). Although the wiki has no backup mechanisms, I've implemented a small workaround using a Git repository and anacron to periodically "save" changes.

I'm planning to dedicate some time to this and I'm aiming to finish the course within the next couple of months. Then it's time to test what I've learned in some CTFs! 😄

Improving my Chess Games app

As I've mentioned in a previous blog post, I've created and am currently maintaining a Chess Games app where I share all my past chess games, so that I can analyze them at any time, to better understand my mistakes and help me improve my game.

I had the idea of adding a chess engine live analysis to my board component to get an understanding of the computer's evaluation of given positions, while at the same time allowing the user to try on different possible moves on a given position. I'm currently working on this and I'm expecting to finish it within the next month!

Interview Preparation

Last but not least, I've been devoting most of my time in the past weeks into preparing for an important interview I'm having in the near future.

I'm taking it quite seriously, and the preparation has consisted mostly in:

Solving Programming Challenges, to brush up my problem-solving skills (I've completed about > 100 LeetCode problems in this process);
Going through the Cracking the Coding Interview book, to make sure I'm ready to answer possible System Design questions as well as possible;
Analyzing my notes of Past Interviews, to get a grasp of what I did well and what I need to work on;
Taking some Interview Simulations, which have been of great help by training me on how to answer "unexpected questions" (a special thanks to my friend Miguel for taking the time and patience to sit with me for a few hours).

I hope everything goes well and that I have a good "Return on Investment" regarding my preparation! 😄

On a last note, I'm starting my last year of my MSc in a couple of days. I'm working on defining my Thesis theme, which will hopefully be very exciting to work on!

Cheers!

A Meteoric regret

Thu, 13 Aug 2020 00:00:00 GMT

About 2 years ago I was asked to build a new website for my chess club. In this article, I want to share a bit of my experience in developing a web app with Meteor and why I've come to regret that choice.

Firstly, I'll share a few details about the project per se. Before I started working on the project, the existing website was very 90's looking (with some blinking buttons) and had plenty of broken and empty pages. From what I understood, it was an ASP.NET app that communicated with 4 Microsoft Access databases (3 of them being empty for some reason). All of this was hosted without backup / version control in someone's home server. Although I can't be sure, my understanding is that new articles and posts were added by manually editing HTML files via an ssh connection. It was time for a change!

The new website had two main requirements:

A back office for the chess club management team, where someone without a technology background could easily manage articles, players, club activities, etc
A clean fresh look to improve user experience, so that people would actually want to visit the website

When I started tackling the project, I had been studying Computer Science for about 2 or 3 years and didn't have much experience in web development. Since this was a "solo gig", I could choose whichever tech stack I wanted. After doing a bit of research (in hindsight, it was a very small bit), I thought that Meteor was a very good option. I was mostly attracted by the fact I would be developing the whole app using only one programming language and by Meteor using GraphQL (I had read about it in some article and really wanted to give it a try). Moreover, I had only worked with relational databases, so I wanted to give MongoDB a try. Although satisfying the "client's" needs was the priority, I saw this project as a big learning opportunity (and it was!).

As of August 2020, the new website has been up for a little less than a year. The full development process took about 10/11 months. Due to budget limitations, the app is hosted in a t2.micro AWS EC2 and the MongoDB database is hosted in MongoDB Atlas (using their free tier, which is actually pretty good). I'm using dominios.pt for DNS management (after a painful switching from dns.pt), but that's a story for another time.

In the middle of having 2 years of studies, 3 internships and a bunch of side-projects, I ended up leaving the website untouched after deploying it (I didn't even set up SSL certificates). I'm currently in the process of "dockerizing" the app and configuring SSL certificates to setup https.

Now, let's get to why I've come to regret the decision of choosing Meteor:

There is too much magic going on. It does a lot for you, which can be a downside. There are many things that were built in order to work in a specific way out of the box and without being tweaked. It can be difficult (or even impossible) to configure a certain thing in a specific way (e.g. Having custom authentication aspects or custom specific middlewares) and the community seems to value and prefer "hacks" that are not pretty nor maintainable to fix a known issue rather than adding support to fix it to the framework itself.
It is tightly-coupled with MongoDB. Even though there's (supposedly, I haven't tried it myself) support for other DBMSs, it was conceived in a way to be completely MongoDB dependant, both as persistence storage and cache - if you want to avoid a bunch of hacking, you're pretty much stuck with Mongo.
The lack of structure. This is probably my biggest pain-point. Unlike other frameworks such as Rails, Phoenix or Laravel, Meteor lacks a decent structure with a clear separation of data, logics and views (which is really well defined in the aforementioned alternatives). In the long run, this makes a project quite hard to maintain.
Very, very slow build times. Everything seems to take a lot of time, even if you're developing in a decent machine. Starting the development server, running tests, live-updating after applying changes, building for production - all these tasks take a lot more time than in other frameworks I've tried before.
Small community support. Although this is much more of a personal opinion than the former statements, there is a lot less community support than, for example, in the Rails community. Even though there are experts in all languages, it's just way harder to find the help you're looking for.

Even though there are other things I'm unhappy with (lack of basic core functionalities that can only be found in specific modules, unpredictability, etc), these are the biggest disadvantages to my eyes.

Note that there are still plenty of perks in using Meteor (such as being good for quick prototyping and relatively easy to learn). However, I don't feel they are enough to shift the "scale" to the positive side.

In conclusion, I chose Meteor for the wrong reasons, driven by lack of knowledge regarding alternatives. In the future, I should take more time choosing the ideal tech stack for the given scenario, which is a lot easier now since I have a bit more experience.

Cheers!

Chess and Tech

Tue, 04 Aug 2020 00:00:00 GMT

Besides programming, Chess has been the only "hobby" that I've kept 100% active at throughout the past few years. It's fun, competitive, strategical, and virtually impossible to master - everything I like in a hobby!

The past few weeks I've been having fun coding stuff that automates some of the daily chess activities I usually do. These tasks involve annotating and publishing my chess games, keeping track of my current ratings (both online and over the board) and creating and hosting a bunch of weekly online tournaments for my chess club.

Chess games page

I started by making some changes to my chess games page. I had to fix some package vulnerabilities that introduced breaking changes. After some minor tweaks here and there, I also added a small section displaying my online ratings using the Lichess API.

I already had a section featuring my over the board ratings, which was getting data from a scraper and API module I had developed to get ratings from FIDE's (the international chess federation) website since they don't expose a public API. However, some data appeared to be missing, so I decided to take a look at the scraper.

FIDE ratings scraper

As I've mentioned, FIDE does not feature a public API. Thus, to get ratings and other info from players in a programmatic way, I decided to develop this scraper about a year ago.

The missing data was due to a minor change FIDE made in a ratings HTML table section, which was quickly fixed by changing a query selector. However, the scraper's code was quite messy and needed a bit of a refactor, since the scraping logic was coupled with an express API I set up. Moreover, the test coverage was quite poor, so I also wanted to improve that.

After separating the scraping logic and the API into different modules, I decided that I might as well add CLI interaction (which although is not my main use-case, it might be useful to someone) and publish the package on npm.

Lichess tournament creator

Throughout the past few months, I have been manually creating online tournaments for my chess club in Lichess. The tournaments always take place at the same time (Wednesdays and Saturdays) and have exactly the same format. Although this is not a very demanding task, I wanted to automate this process, and finding that Lichess had a public API was the "kick" I needed to get me started. I decided to build a CLI tool to create tournaments, which had basically two requirements:

Creating a tournament (either in Arena or Swiss format) with a set of specifications (duration, match time, etc) in a single command
Creating tournaments in bulk from a given tournaments list input file (so that I could create a bunch of tournaments instantly, to save time)

They limit the number of tournaments a user can create daily (12 public tournaments), but creating tournaments for the following one and a half months was already pretty good for me. Although there are a bunch of settings a user can specify when creating a tournament, I didn't dive too much in adding all the features possible, since a simple version is enough to suffice my needs.

I also published this package on npm and listed some of the future work, so maybe someone interested will take a look and contribute with some more features!

Regarding future plans, I'd like to add engine analysis to my chess games page. I would also like to start contributing to the Lichess organization (although I'm already a patron, I want to contribute with something other than my money 😛).

In the meantime, feel free to take a look at my games page and/or at the packages I published:

Cheers!

Home Networking

Thu, 30 Jul 2020 00:00:00 GMT

Recently I decided to do a bit of a network setup improvement around my house. The main objective was to get a decent wi-fi coverage and "easy" cable accessing in a few key rooms. The only thing that makes this endeavor far from trivial is that my house is about 60 years old, with about 15cm thick walls, and without any cable-supporting infrastructure whatsoever. I wanted to share a bit of what went well and wrong with my planning and implementation, as well as some lessons learned.

Since I wanted to have the full hands-on experience, I decided I would assembly the cabling myself. I don't have gigabit ethernet at home (and I don't plan to have it anytime soon), so I decided to go with cat5e cabling. I got myself an RJ45 crimping tool, an FTP cat5e cable spool and a bunch of connectors. I picked up a wiring diagram from the internet (I used the Type B standard since it seemed to be the most common one) and tried to build a short 1m cable, just to get acquainted with the process - happily for me, it worked quite well. I found this video a bit helpful as well.

Now for the planning part: I live in a 2 story house and my ISP's router is placed on the bottom floor. I found that if I added an Access Point to my house's upper floor I would get decent wi-fi coverage, and then I could add a few cables to each of the bedrooms for wired connections. My ISP's router is placed in the garage and its location can't be changed due to logistics reasons, so I had to drill a small 1cm hole in the wall so that the wire could pass through. After doing a bunch of measures, I found that I needed to build a 20m cable to connect the ISP's router to the place I would later on place a router on the top floor.

I also wanted things to look as neat as possible, so I bought a bunch of wiring rails and installed them near to the ceiling. I didn't need a very good router since my ISP does not provide me a very good internet service anyway, so I ended up going with a budget option and got an Asus AC750 RT-AC51U router.

Now I only needed to add the cable and patch both ends to finish the process! - Not quite. After placing the cable across the rails and patching both ends, the data connection was failing. I double-checked the RJ45 connectors I had set up, and they looked perfectly fine (both in wiring order and cable length). After debugging the cable across the rails, it looked like the cable's bending radius had been exceeded, causing the inner copper wires to break. Doing a bit of research, I found out I had made a bad choice when choosing the cabling type:

FTP cables are shielded and commonly used in areas where the will likely be interference. However, the shielding causes cable flexibility loss.
UTP cables, being unshielded, offer more cable flexibility (at the cost of being more sensitive to interference).

However, interference was not a problem at all in my situation. Moreover, the lack of cable flexibility actually caused my setup to break, so I decided to switch to UTP cabling.

In fact, installing the new cable was much easier (due to the increased wire flexibility) and everything went smoothly! Even with a few bumps on the road, at the end of the day, the new network setup works pretty well and I've now got a fairly decent coverage around all the house. The main lesson here is that I should investigate more carefully before committing to a certain approach, especially when I'm not familiar with the domain.

Cheers!

A tour around Competitive Programming

Sat, 25 Jul 2020 00:00:00 GMT

During the past year, I've been exploring the "competitive programming problems" world. It all started around October 2019, when I began studying for coding interviews at Amazon, Bending Spoons and, most of all, Palantir Technologies.

I wanted to have a somewhat methodic and organized way of tracking my progress, while at the same time trying to keep focus on problem types that were common in large tech companies' interviews. After trying out LeetCode, Project Euler and Hackerrank, I particularly enjoyed the former, since most of the community seemed to have the same focus as I did. I started a GitHub repository featuring all my solution proposals, which were developed using Java.

As of the start of April, when I finished my interview process, I had solved 41 easy, 119 medium and 8 hard problems. I found that the "repetitive" process of solving similar problems (that is, problems that have analogous solutions) resulted in training my brain to quickly recognize how to solve a new problem, whether by identifying the most appropriate data structures for the given scenario or by applying the correct solving methodologies (dynamic programming, memoization techniques, etc).

In the last 4 months I've been less active, although I've been tackling problems from the LeetCode monthly challenges every day, which started back in April. During this time, I also participated in a couple of contests, although I didn't do particularly well in either of them.

Although there were many problems I enjoyed solving, I decided to pick two of them to share (not because they were particularly my favorites, but because they had something interesting by nature).

Problem 72 - Edit Distance [Hard]

Given two words word1 and word2, find the minimum number of operations required to convert word1 to word2. 
You have the following 3 operations permitted on a word:
- Insert a character
- Delete a character
- Replace a character

I had implemented some C++ code in the past using a "known" algorithm to compute two strings' edit distance. However, I wanted to try finding a solution by myself, without recurring to the literature (to simulate an interview-like situation). Based on the operations described in the formulation, this type of edit distance is also known as Levenshtein distance.

After scratching my head for a while, I realized that this could be solved using - that's right - the dreadful dynamic programming (for some reason, while studying algorithms in a curricular unit early on in college, I ended up getting the idea dynamic programming was very hard, although that's not the case at all).

By using some extra space to help the process, I ended up using a matrix to solve the problem. The matrix width represents word1 and its height represents word2. Each matrix cell (i, j) value is the edit distance from the first i characters of word1 to the first j characters of word2. Now I just needed to populate the matrix! After initializing all elements with 0 and filling the first row and first column with ascending values (which represent the edit distance of the word to an empty string), the edit distance is calculated using the following recurrence function:

if word1[i] == word2[j]:    // The char is the same, no distance
    M[i, j] = M[i-1, j-1]
else:
    M[i, j] = 1 + min(
        M[i-1, j],          // Delete char at word1[i]
        M[i, j-1],          // Insert char in word2[j]
        M[i-1, j-1]         // Replace char in word1[i] by word2[j]
    )

After iterating the full matrix, the edit distance is the value at M[word1.size-1, word2.size-1]! The full solution may be found here.

Problem 210 - Course Schedule II [Medium]

There are a total of n courses you have to take, labeled from 0 to n - 1.
Some courses may have prerequisites, for example to take course 0 you have to first take course 1, which is expressed as a pair: [0,1]
Given the total number of courses and a list of prerequisite pairs, return the ordering of courses you should take to finish all courses.
There may be multiple correct orders, you just need to return one of them. If it is impossible to finish all courses, return an empty array.

I found this problem particularly fun to solve. The way I came up with the solution shows that, sometimes, you only fully understand the problem and get good ideas when you "get your hands dirty" and start coding. The first thing that caught my eye is that prerequisites could be modeled as connections in a directed graph and that the input may not lead to a valid course schedule - the courses could have requirement loops (direct or not).

Finding if the input was valid or not was easily solved by applying a DFS starting at node 0 (the "root" of the graph) - If all nodes were visited without finding loops, the input could lead to a valid solution. That's when it came to my mind that applying the DFS would actually get me the solution itself if the input could lead to a valid course schedule!

The order in which nodes are visited using a depth-first search consists of a Topological Order (or topological sorting) - which in this scenario represents a valid course schedule. The full solution may be found here.

As of the of July 2020, I have solved 73 easy, 152 medium and 13 hard problems. I'll keep solving problems from these monthly challenges for as long as I'm having fun with them - which will likely be for a long time.

Cheers!

Interview Scheduler

Fri, 24 Jul 2020 00:00:00 GMT

For the past nearly 4 years, I've been a member of FEUP's Computer Engineering Student Branch. Every year we go through a "recruitment season", in which about 30 or so of our active members interview around 50 candidates. And every year, we face the challenge of matching the availability of our members with the candidates to schedule the interviews. Although we've been using Doodle polls to query people about their availability, we do the matching almost entirely manually, using some common sense heuristics.

Thus, I decided to develop a CLI tool to automate this process, calling it interview-matcher. Although we have been scheduling the interviews with 2 interviewers and 1 candidate during the past few years, I wanted to design the tool in a way it is flexible enough to accommodate changes in the process, if needed.

At first, the problem seemed to be a simple Maximum Cardinality Matching. However, after thinking deeper about the subject, I realized it could not be treated as such:

This isn't a classic 1:1 matching problem. Instead, matching is made in an N:1 ratio, so formulating the availabilities as a standard bipartite graph was out of the question;
The interviewers are not matched directly to candidates. They're matched to candidates via option slots (e.g. from 9:30 am to 10:00 am), which means there may be multiple ways to connect entities from the two parties;
Finally, an interview matching is only considered feasible if they feature the total number of required interviewers (e.g. if a 2:1 interviewers to candidate ratio is being used, an interview with only 1 interviewer will not be accepted).

Another non-functional requirement is that all interviewers should have a similar number of assigned interviews, in order to distribute work as much as possible.

Even though this wasn't a "classic" problem, I had the feeling this could be formulated as a Graph Maximum Flow Problem. The flow network needed to follow certain constraints:

Each candidate may only be assigned to a single slot, and vice-versa;
Each slot must have an entering flow capacity of N (where N is the number of interviewers required for each interview).

To follow these constraints, I formulated the following flow network schema:

Note that edges that do not display a value have a weight of 1. The infinity weighted edges from the source to the interviewers allow an interviewer to be assigned to more than one interview. The slot filters layer provides a way of having an entering capacity of N, while ensuring the slot is only assigned to one candidate (due to the edge of weight 1 connecting every node in the slot filter layer to its respective node in the slot layer).

Finally, this flow graph can be used to find a valid matching using an algorithm like Ford-Fulkerson. To both converge towards a solution faster and to improve the final solution's quality, I used some heuristics when searching for flow augmentation paths:

Prioritizing interviewers with less amount of work (to distribute interviews as evenly as possible);
Prioritizing slot filters with the most output flow, to avoid having "half-full" slots;
Prioritizing slots that flow towards candidates that don't have an assigned slot yet, to reach a feasible solution faster.

Although the output was near-optimal, it did not ensure optimal matching - a max flow algorithm focuses on (that's right) obtaining maximum flow from the source to the sink. Even with the prioritization heuristics, there's no certainty the matching will result in a similar number of interviews per interviewer!

To mitigate this, I ended up adding an iterative optimization layer, featuring an ad hoc algorithm that focuses on finding local optimizations to slowly improve the overall matching. To my surprise, applying this algorithm converged to the optimal solution quite fast! It focuses on trying to move slots from overworked interviewers to underworked ones.

I'm pretty happy with the final result. I have tested the tool with data inputs similar to the ones from previous years' recruitment seasons, obtaining great results. I tried to use a TDD approach while developing, which proved to be quite productive and allowed me to avoid mistakes. I also had some fun setting up a cool CI pipeline to allow maintaining the tool in the future. Hopefully, someone will take a look at the project and find room for improvement!

Finally, I decided to publish the tool as an npm package. I invite you to download it and try it out:

$ npm i -g interview-matcher

Cheers!