It’s not just about Sorting!

1. Merge Sort

Merge Sort is one of the most commonly taught sorting algorithm (beside Quick Sort) in any Computer Science and Information Technology courses. I will start by briefly reviewing the fundamental knowledge of Merge Sort. The basic idea behind Merge sort is a divide-and-conquer strategy, where you break the array into smaller halves, sort each half and then merge them back together. The recursive process continues until each half contains only 1 item (already sorted).

There are 2 variants of Merge sort implementation, using for and while loop. You can check my other post about Merge Sort here, which contains the for version from the algorithm course on Coursera. For the purpose of interview questions, I’ll demonstrate using the while version in this post.

As mentioned above, there are 2 essential steps in Merge Sort, the Divide step and the Merge step. Let’s begin with the Divide step

// WARNING: This is not working code and not the most efficient way to do
const mergeSort = (arr) => {
  // base case
  if (arr.length === 1)
    return arr;

  const mid = Math.floor(arr.length / 2);
  const firstHalf = mergeSort(arr.slice(0, mid));
  const secondHalf = mergeSort(arr.slice(mid, arr.length));
  return merge(firstHalf, secondHalf);
}

and here is how you merge 2 halves (2 sorted arrays)

// WARNING: This is not working code and not the most efficient way to do
const merge  = (arr1, arr2) => {
  const res = [];
  while (arr1.length && arr2.length) {
    const item1 = arr1[0];
    const item2 = arr2[0];
    if (item1 < item2) {
      res.push(item1);
      arr1.shift();
    } else {
      res.push(item2);
      arr2.shift();
    }
  }

  if (arr1.length) {
    res.push(...arr1);
  } else {
    res.push(...arr2);
  }

  return res;
}

It’s time for you to practice some easy Leetcode questions related to Merge Sort before reading the next part.

• Merge Sorted Array
• Merge Two Sorted Lists (solution)
• Squares of a Sorted Array (solution)

2. The Merge Function

We will continue by examining the above merge function in detail. Here is another way to visualize it. Think about it as Pick the first item from either array based on some criteria, then repeat until no items left

This time, ask yourself these questions

• What if the arrays are replaced with two linked lists?
• What if they are replaced with two queues?
• How about the arrays with unknown size?
• How do you handle datasets that are too large to fit in memory?

These scenarios often appear in interview problems. You can spot them when you see the input involves two datasets that can be ordered, such as sorted arrays, linked lists or queues. For more challenging questions, the interviewer might obscure these details, requiring you to transform the input into an ordered format (usually by sorting).

Let’s take a look at some scenarios

External Sorting

Merge Sort and its variants are useful for doing External Sorting, a technique to handle massive amounts of data that don’t fit in RAM. The most common question related to this type is to sort a large file (containing 1 million 8-bit integers) on a computer with only 1MB of RAM. The approach remains the same: divide the file into smaller parts that fit into memory, sort each of them individually, write the sorted parts to another file and finally merge the files back together.

// pseudo code only
let val1 = readLine(file1);
let val2 = readLine(file2);
while (val1 || val2) {
  if (!val1) {
    writeLine(val2, outFile);
    val2 = readLine(file2);
    continue;
  }

  if (!val2) {
    writeLine(val1, outFile);
    val1 = readLine(file1);
    continue;
  }

  if (val1 < val2) {
    writeLine(val1, outFile);
    val1 = readLine(file1);
  } else {
    writeLine(val2, outFile);
    val2 = readLine(file2);
  }
}

Read more Solution to the Sorting with 1MB RAM computer problem

Intersection and Union

Another interview problem is to find the intersection or union of two arrays. Union is an array containing elements present in either or both of the arrays, while intersection is an array containing only items present in both arrays. In both cases, all duplicates are removed and the order of the elements in the result may vary depending on the implementation.

For problems like these, sorting the input arrays beforehand can simplify it a lot. Once sorted, apply a modified version of the merge function to your 2 sorted arrays. For the union, maintain two pointers starting at the beginning of both arrays, append the smaller one to the result and advance the pointer for that array. If the two items are equal, append one of them and advance both pointers.

To help you imagine it better, here’s the visualization of each iteration inside the merge function

// pseudo code only
const res = [];
while (arr1.length || arr2.length) {
  // edge where only arr1 or arr2 has items
  if (!arr1.length) {...}
  if (!arr2.length) {...}

  // decide which array to pick item from and then move the point up
  if (arr1[0] < arr2[0]) {
    // pick from arr1
    res.push(arr1.shift());
  } else if (arr1[0] > arr2[0]) {
    // pick from arr2
    res.push(arr2.shift());
  } else {
    // pick from both
    res.push(arr1.shift());
    arr2.shift();
  }
}

A similar process can be adapted to compute the intersection, but only equal elements are added to the result

Here are some similar questions to practice

• What happens if the two input arrays contain duplicate elements?
• Longest Common Prefix

Concurrency Control

The merge function is not limited to solving mathematical problems, but also applied in concurrency control. It is widely used to combine and coordinate different data sources like processes, queues or linked lists.

Consider the following interview exercises

1. Two-Lane Bridge Problem: A bridge has two lanes, one for vehicles coming in and one for vehicles going out. The bridge can only support one vehicle per lane at a time. Vehicles arrive at different times, and each vehicle wants to either enter or exit the bridge.
   • If a vehicle wants to enter and another wants to exit at the same time, prioritize the vehicle that arrived first.
   • How would you determine the exact time each vehicle can cross the bridge, respecting the order of arrival and the last usage of the bridge?
2. Train Station Platform: A train station has a single platform and two queues: one for passengers wanting to board the train and one for passengers wanting to disembark.
   • The train can either be used for boarding or disembarking at any given time. Passengers arrive at different times and are either boarding or leaving.
   • The platform can be used for boarding or disembarking, but the direction depends on who arrived first when simultaneous arrivals happen.
   • How would you calculate the exact time each passenger can board or disembark the train?
3. Single Gate Airport Security: At an airport, there is a single security gate that allows people to either enter or exit. People arrive at the gate at different times and want to either board a flight or leave the airport.
   • If two people arrive at the same time with opposite directions (one entering, one exiting), the direction in which the gate was last used will determine who goes first.
   • How would you determine when each person passes through the gate?
4. Library Entry/Exit System: A library has a single turnstile and two queues: one for people who want to enter the library and another for those who want to exit.
   • Each person arrives at a specific time, and their direction (enter or exit) is known.
   • If two people arrive at the same time with different directions, the decision of who goes first depends on the turnstile’s last usage.
   • Calculate the time each person will be allowed to pass through the turnstile.
5. Parking Garage Gate: A parking garage has a single gate that vehicles can use either to enter or to exit. Each vehicle has a specific arrival time, and its desired direction (enter or exit) is also given.
   • If vehicles with opposite directions arrive at the same time, the gate gives priority based on its last usage.
   • Determine when each vehicle will pass through the gate, considering these rules.
6. Single Lane Toll Booth: A toll booth has a single lane, and vehicles either want to pay tolls or leave. Vehicles arrive at the booth at different times, and their direction (whether they want to pay or leave) is given.
   • If two vehicles arrive at the same time, one wishing to pay and the other wishing to leave, the decision of who goes first depends on the toll booth’s last usage.
   • Calculate the time each vehicle will cross the toll booth.
7. Hospital Entry System: A hospital has a single entrance, and people arrive either to enter the hospital or leave. Each person arrives at a given time with a desired direction (enter or exit).
   • If people arrive at the same time, the direction the entrance was last used will determine who gets to go first.
   • How would you compute when each person can pass through the entrance?
8. Amusement Park Entrance: An amusement park has a single gate, and guests either want to enter or exit. The entrance operates like a turnstile and can only process one guest at a time.
   • If two guests with different directions arrive at the same time, the direction the gate was last used (enter or exit) determines who goes first.
   • How would you determine the exact time each guest will pass through the gate?
9. Stadium Gate Entry/Exit: A stadium has a single turnstile and two queues: one for people wanting to enter the stadium and another for those wanting to exit.
   • If two people arrive at the same time, the one with the direction corresponding to the last operation of the turnstile will go first.
   • How would you calculate when each person passes through the gate?
10. Convention Center Turnstile: A convention center has a single turnstile that can be used for people to either enter or exit. People arrive at the turnstile at different times, and each person is either entering or leaving.
    • If two people arrive at the same time and their directions are different, the decision of who goes first depends on the previous usage of the turnstile.
    • Determine the time when each person passes through the turnstile.

Did you spot the pattern? The keyword of this type of challenge is to coordinate two different queues. The above questions are just the simplified version. In reality, they may require you to do some data transformation to form the two queues but the idea remain unchanged. Now, it’s your turn to solve the above problems yourself. 😉

Here is how you can prepare yourself for that type of interview question

1. Basic Recursion

Let’s start with this very basic recursion question

Given an array arr with N items, write a recursive function to calculate the sum of all items in the array

If you think about recursion in its simplest form, it’s like a divide and conquer strategy, where you break the problem into smaller ones, solve one of them first and then repeat with the other remaining ones (recursive case) until there is no item left (base case). Now, let’s take a look at the above problem, the solution can be described like this: The sum of an array is the sum of the first element (the head of the list) and all the remaining items (the tail of the array).

If you have worked in any lisp-like language (Emacs Lisp for example 😂), you will immediately see the pattern. They are the car and cdr function in Elisp.

From that, you can easily write a basic recursive function like this (in Javascript)

const sum = (arr) => {
  if (arr.length === 0) {
    return 0;
  }
  const [head, ...tail] = arr;
  return head + sum(tail);
};

An iterative solution

Of course, the easiest way is to convert the above one to a simple for (or while) loop to avoid adding to the call stack.

const sum = (arr) => {
  let sum = 0;
  for (let i=0; i<arr.length; i++) {
    sum += arr[i];
  }
  return sum;
};

Simple? Yeah that’s just what you learnt in university, but that will probably help you during the interview.

Tail-call optimization and Trampoline

Another way you can answer the interviewer is to use Tail-call optimization. In some programming languages, if you return the recursive call as the last expression in the function, they can optimize automatically by avoiding creating a new call stack. Using that concept, you can rewrite the above sum function like this

const sum = (arr, acc) => {
  if (arr.length === 0) {
    return acc;
  }
  const [head, ...tail] = arr;
  return sum(tail, head + acc);
};
sum(arr, 0);

Does it look like Array.prototype.reduce() in Javascript?

However, a new problem arises here. The above piece of code is written in Javascript, which doesn’t support tail-call optimization

Actually, it’s partially supported now. However, let’s assume we are working on a non-supported language.

There is another way you can rewrite the above one in a tail-call style without the support from the language. Instead of returning the recursive call, you can return a function calling the recursive function and use a wrapper function (trampoline) to execute and manage the stack.

const sum = (arr, acc) => {
  if (arr.length === 0) {
    return acc;
  }
  const [head, ...tail] = arr;
  return () => sum(tail, head + acc); // the difference is here
};

const sumTrampoline = (arr) => {
  let result = sum(arr, 0);
  while (typeof result === 'function') {
    result = result();
  }
  return result;
};

sumTrampoline(arr);

2. Dynamic Programming

Now, let’s come to a more complicated problem

You are climbing a staircase. It takes n steps to reach the top. Each time you can either climb 1 or 2 steps. In how many distinct ways can you climb to the top?

This is the most common DP problem that you will see every time you enter a coding interview (beside the Fibonacci and House Robber question). The most straight forward solution is to write a recursive function to count the ways to step through 1 or 2 steps.

For the above example, the solution can be explained like this

• Start with count(4)
• We can climb 1 or 2 steps each time, that means the total ways is the sum of climbing 1 or 2 steps, or count(4) = count(4-2) + count(4-1) = count(2) + count(3)
• Repeat the same problem with the smaller numbers until there is no more step to climb

From the above explanation, it’s not hard to come up with the simplest recursive implementation

const count = (n) => {
  // base case: only 1 way to climb if 0 or 1 step
  if (n === 0 || n === 1)
    return 1;

  return count(n - 1) + count(n - 2);
};

const n = 4;
console.log(count(n)); // print 5

Memoization

Let’s take a look at the above solution again, you may notice that both these recursive calls count(2) and count(3) contain an overlapping problem (the blue one in the below picture) because count(3) can be interpreted as count(1) + count(2).

With that in mind, we can store the overlapping calculation into an array and pass that array into each recursive call.

const count = (n, res) => {
  // same base case
  if (n === 0 || n === 1)
    return 1;

  // check if this problem has already been calculated
  if (res[n] !== undefined)
    return res[n];

  res[n] = count(n - 1, res) + count(n - 2, res);
  return res[n];
};

const n = 4;
const res = Array(n + 1);
console.log(count(n, res));

Tabulation

In the above memoization solution, after executing the count function, the res array will be built like this

The pattern here is similar to Fibonacci problem, where the value of each item in the array is the value of the previous 2 items. We can then build a solution to traverse and build the array from the beginning

const count = (n) => {
  const res = Array(n + 1);

  // base case
  res[0] = 1;
  res[1] = 1;

  for (let i=2; i<=n; i++) {
    res[i] = res[i-1] + res[i-2];
  }
  return res[n];
}

const n = 4;
console.log(count(n));

You can also optimize it even more using only 2 variables to store i-1 and i-2 values.

3. Depth First Search and Breadth First Search

Another type of recursion question that the interviewer usually ask is the question related to tree data structure, for example this one

Given a starting directory, build a JSON model representing the structure of the directory and all its nested items. Your function should output a json structure like this

{
  "name": "root",
  "contents": [
    "file.txt",
    {
      "name": "root/folder1",
      "contents": [
        "weavels.pdf"
      ]
    },
    {
      "name": "root/an_empty_folder",
      "contents": []
    },
    "test.txt"
  ]
}

• Each file is represented as a string
• Each folder is represented as an object with a contents array for all its child { "name": "xxx", "contents": []}

Depth First Search

I was quite surprised that many engineers couldn’t solve this problem using the simplest recursive implementation 🫠

At the very basic level, a software engineer should be able to write a simple recursive function like this to traverse the directory tree starting from the root node

const getContents = (path) => {
  // implement this
  return ['child1.js', 'child2.txt', 'childFolder1'];
}

const isFolder = (path) => {
  // implement this
  return true;
}

const buildNode = (parent) => {
  const children = getContents(parent.name);
  
  for (let i = 0; i < children.length; i++) {
    const name = children[i];
    const childNode = isFolder(name) ?
      buildNode({name, contents: []}) : name;
    parent.contents.push(childNode);
  }
  
  return parent;
}

const res = buildNode({name: entryPoint, contents: []});

Breadth First Search

After you have finished writing that, the interviewer will definitely ask a follow up question about your optimization when the input folder is a very big one with a lot of nested levels. There are many ways to solve this, but the most common way is to convert this to a Breadth First Search solution.

As long as you can mention that keyword, you are 50% to the success 😆

The idea behind BFS is to visit all nodes in the current level first (closest nodes first) before visiting the next level (further ones later), in constrast to DFS, that is to traverse all the nested child of the current node before visiting the next one. BFS can help you avoid the “Maximum call stack exceeded” error in DFS for very large trees. The implementation relies on a queue and sometimes an extra array to store the visited items.

Here is the quick and dirty BFS solution

const buildTree = (entryPoint) => {
  const root = {name: entryPoint, contents: []};
  const q = [root]; // use array for bfs queue
  
  while (q.length) {
    const current = q.pop();
    const children = getContents(current.name);
    children.forEach(child => {
      if (isFolder(child)) {
        const childNode = {name: child, contents: []};
        q.push(childNode);
        current.contents.push(childNode);
      } else {
        current.contents.push(child);
      }
    });
  }
  
  return root;
}

You can also use BFS for Binary tree, 2D maze and Unweighted graph problems

Time to practice

What I presented in this post so far are just the suggestions to the common recursive problems. They are not everything but can cover most of the interview questions related to recursion. I’ve been through several similar interviews and even the advanced problems given to Senior role are based on those 3 categories with a lot more edge cases to handle (FAANG and Big Tech interview could be different and harder).

You should also practice to get yourself familiar with those types of question before any interview. Here are some challenges that you could take

• Factorial
• Climbing Stairs
• House Robber
• Coin Change
• Fibonacci
• Binary Tree Level Order Traversal
• Flood Fill
• Max Area of Island
• 01 Matrix
• Populating Next Right Pointers in Each Node
• Rotting Oranges
• Binary Tree Right Side View
• Populating Next Right Pointers in Each Node II
• Symmetric Tree
• Find if Path Exists in Graph

Previous post: A Unified Development Environment Journey - Part 1

I won’t mention about setting up Codespaces or Devcontainer as it’s already on their documentation page. In this post, I’ll show you some of the problems and example workflows that I did with Codespaces/Devcontainer.

Dev credentials and Environment variables

Your organization may created some private packages that requires some layers of authentication in order to install (the npm packages hosted on GCP for example). One possible solution is to create an access token (or any other type of credential) and put set it in your development environment. For Codespaces, you can simply set it on your Github configuration page. However, for Devcontainer, it needs a bit more work. The idea is to mount the environment file from outside of the container and then load those variables to Devcontainer

Create an env file on your computer, for example /var/devcontainer-mount/workflow.env, which contains all the secrets that you want to inject

export GOOGLE_APPLICATION_CREDENTIALS="xxx"

Mount the folder to your Devcontainer by adding a mount volume to your devcontainer.json file. Read more here Add another local file mount

{
  //...other config
  "mounts": [
    {
      "source": "/var/devcontainer-mount",
      "target": "/workspaces/devcontainer-mount",
      "type": "bind"
    }
  ]
}

In your Devcontainer bootstrap file (usually defined under updateContentCommand section in your devcontainer.json file), load those variables before executing any script

if [[ -e "/workspaces/devcontainer-mount/workflow.env" ]]; then
  echo "Sourcing secret env file..."
  source "/workspaces/devcontainer-mount/workflow.env"
fi

You can also make these variables available by default in your VSCode Terminal or in SSH session by adding it to your shell’s startup files. For example, if you use zsh with a Node Devcontainer, add this to your bootstrap file

touch /home/node/.zshenv
cat /workspaces/devcontainer-mount/workflow.env >> /home/node/.zshenv

Git config not shared properly with Devcontainer

It’s suggested that you install git-credential-manager when working with Devcontainer. However, it only transfers the credential, not the configurations (git username and user email). Codespaces set these values automatically based your Github account. For Devcontainer, you need to do it manually.

Continue with the workflow.env file that I mentioned above, add your Git details

GIT_USER_NAME="Tony Tran"
GIT_USER_EMAIL="my.email@my-company.com"

and then in your bootstrap file (after sourcing the env file)

git config user.name > /dev/null || gitUserNameCheck=$?
if [[ -n "$gitUserNameCheck" && -n "$GIT_USER_NAME" ]]; then
  echo "Setting Git user.name"
  git config --global user.name "$GIT_USER_NAME"
fi

git config user.email > /dev/null || gitUserEmailCheck=$?
if [[ -n "$gitUserEmailCheck" && -n "$GIT_USER_EMAIL" ]]; then
  echo "Setting Git user.email"
  git config --global user.email "$GIT_USER_EMAIL"
fi

docker-in-docker and its bugs

Your Devcontainer/Codespaces instance is actually a Docker container (or can be treated as another computer). You can install any database engine into it and connect directly. However, for simplicity, we usually just pull a docker image with the database engine already bundled. You can also easily find other docker images for different backing services, for example Pubsub emulator.

Devcontainer supports running docker inside docker. Once you have enabled and recreated your Devcontainer, just use all the Docker commands normally inside your Devcontainer

This is an example of Docker running inside Docker Devcontainer

However, I usually get these below issues when using this feature, sometimes on first rebuild of the container or sometimes when the container is started

I figured out the issue is that Docker service inside Devcontainer sometimes fail to start. You can simply run this command inside Devcontainer/Codespaces to fix

/usr/local/share/docker-init.sh

Similar, if you use its sshd feature, sometimes it also fails to start. Run /usr/local/share/ssh-init.sh to manually start ssh service.

It may still fail after you run the above init command in case you run it inside your iTerm via an SSH session. In that case, simply run any Docker commands (docker compose up for example) inside VSCode Terminal and then come back to your SSH window. 99% it will work in both VSCode and SSH after that. I guess it’s because VSCode terminal has some environment variable set by default related to Docker RPC. 🫠

E2E and UI Tests

Devcontainer acts like another computer and we connect to it via our VSCode/Jetbrains terminal or SSH session. For UI testing with browser, you can only run in headless mode. Luckily, Devcontainer comes with a builtin feature, providing us a minimal VNC desktop UI. It can be connected using any VNC client or directly via WebUI. You can enable it following the instructions here.

After you have rebuilt your Devcontainer/Codespaces, simply install any browser or UI test framework that you want and start them in headful mode. You can easily access that VNC desktop via browser

Anything else?

My first attempt in university

Here is my first effort when I was a student, a classic solution.

In this method, you would run everything normally in your computer and use a Bash/Ansible script to install all the necessary tools as well as set up the development environment. Of course, this is the worst (but was the best with my knowledge at that time 😆).

• This setup won’t be reused for multiple engineers. Each engineer has a very different setup on that machine. The script can mess up the other people’s computer easily.
• Even if it’s just for me, that also won’t work. I install and set up new applications all the time. After a while, when I touch that project again, all the scripts were broken and I don’t even remember how to run my app again.

A better way

I began learning how to do it the professional way in my first job after graduated. The environment is configured using a Virtualbox VM (provisioned by Vagrant), which reflects the real setup on the production server. The server/VM runs everything, from the database server to the main application.

Later on, with the increase in Docker adoption, the setup had changed to this

This time, of course, it was a much better setup

• The VM/Containers could be created or destroyed easily.
• The VM/Containers reflect the real production server, which is a Linux server while my machine is a Macbook

However, there are still a lot of drawbacks with this approach. Each engineer in the team still has their own computer with different applications, different setup. We then need to clone the repo, install all the necessary tools (Nodejs, Homebrew, tmux,…), load the repo into our text editor, configure the plugins (ex eslint, prettier,…) to work.

• The only things that we can make sure that always behave the same is the VM or the containers running inside Docker. Everything else, No. It really depends on what is installed on that computer. An engineer could have multiple Nodejs versions (installed via nvm) and forget to switch to the correct version for the project. An engineer could install a different application on that computer, for example, the command like ls behave slighly different on Macos and Linux,…
• We have to configure all the IDE/Editor plugins to follow the correct convention. What will happen if we accidentally install a different version of prettier/eslint or install a different version of the plugin? What will happen if we misconfigure it?
• As a result, still the classic problem: It works on my machine.
• What will happen if you mess up everything and want to do a fresh setup of the project?
• What will happen if you want to run 2 instances of the same project, to test some configuration?
• How to share a same setup across all team while still maintain the customizability of each engineer’s habit?

Some people also mount the git repo into Docker to run the app directly, but the performance is not very good so I won’t mention that solution here.

Adopting Github Codespaces

At Skutopia, we then adopted Codespaces as a solution to the above problems. The idea behind is to transform Your Computer into Somebody else Computer. Codespaces is a Cloud technology, where you can quickly spin up a new VM running your whole dev environment. Your computer acts as a UI only.

Here are the benefits of running Codespaces

• A single unified development environment for the whole team, where you define a configuration file describing what you want to set up for that machine, even the VSCode plugins.
• You can configure the exact software and the version that should be used for the whole team
• You can configure the exact plugins to make sure everybody in the company follow the same convention
• Once created, the application will just work. You can start working immediately, using the debugger or running a test case. Everything will be configured already.
• No more It works on my machine!
• You can still customize it! Simply set a dotfiles repository to install your favorite software. It won’t mess up your local computer. You can also override specific VSCode settings if needed, for example your favorite key bindings, favorite editor theme, favorite shell setup, etc.
• And of course, fix all the problems mentioned above with previous methods

However, I have never been satisfied enough with Codespaces

• It’s someone else computer (I mean cloud computer), it’s laggy, even if you have a good internet. It has been a really annoying feeling for everybody working on Codespaces
• It has limited disk space. The instance we can afford has just 32gb of ram and there is no way to increase just the disk space. We have to pay for the higher machine specs. Some of our applications consume quite a lot of disk space.
• Its performance is terrible. An M1 macbook with 16gb ram really outperforms a Codespaces instance with 4 cores and 16gb ram. We all have M1+ Macbook but cannot utilize that power.
• It’s pricey!

To Devcontainer

Let’s go into more details about Codespaces

• Codespaces is a Linux VM
• It runs Devcontainer, which is actually a Docker container, hence you build it using a Dockerfile
  • Devcontainer is just a wrapper around normal Docker container, providing you some other utilities to make that Docker container a dev environment.
  • Codespaces uses Devcontainer, that’s why you see we configure it using devcontainer.json file
• Your VSCode UI doesn’t connect to that Codespaces VM. It actually interacts with that Devcontainer. The gh command abstracts everything for you, from starting the VM+container to connect to the correct one.
• Devcontainer is a separate technology. Codespaces just build the wrapper around it to provide the infrastructure. VSCode actually supports us running Devcontainer separately, directly on our computer using Docker.
  • You have already installed this VSCode extension when working with Codespaces. If not, simply install it from VSCode marketplace

To get the best of both worlds, fix all the existing problems with Codespaces while still maintaining the benefit of a unified development environment, we should run Devcontainer directly on our machine, not via Codespaces.

My blog is also configured with Devcontainer now! 😆

Aren’t we going back to our initial solution?

No. Here is the comparison table

Next

• Problems/Solutions when setting up Codespaces/Devcontainer
• Sample Dev workflow with Codespaces/Devcontainer

My last words: Why is Jetbrains (my favorite IDE and I paid for it) losing to VSCode in this race? I have been waiting for so long but its support still far behind compare to VSCode 🫠

const promise1 = new Promise((resolve) => {
  for (let i = 0; i < 10000; i++) {} // longer loop
  resolve('promise1');
});

const promise2 = new Promise((resolve) => {
  for (let i = 0; i < 100; i++) {} // shorter loop
  resolve('promise2');
});

Promise.race([promise1, promise2]).then((value) => console.log(value));

Surprisingly, most of their answers are the same, with promise2 to be printed at the end!

If you have been working with Nodejs long enough and understand its event loop, you can immediately see the problem in the above code. Even though the code is inside a Promise, which should be run concurrently (in normal mindset). However, the for loops contain nothing inside. They are called blocking code in js. Since Nodejs is single-threaded, the above functions will be executed sequentially, in the order of declaration.

It will behave differently only when you add some non-blocking (async) operation inside the Promise, for example setInterval.

Turns out a lot of them also use ChatGPT to get the answer 🥸

In conclusion, use these AIs to speed up and improve your workflow, but don’t completely trust them. Always verify the answers yourself.

It’s the classic logging issue again! In the Warehouse Management system that I’m working on, the team usually needs to add this logging pattern

const sendToteToPackingStation = (warehouseId: string, toteId: string): Promise<Result> => {
  logger.info('Sending tote to packing station', { warehouseId, toteId });

  const result = await someLogic(...);

  logger.info('Send tote result', { result });
  return result;
};

The purpose is simple. It’s what you have to do for production debugging in every system. You should write out some unique ids so you have a place to start querying your logging system. From that point, you will then trace the related entries using a correlation id that your system provides.

From time to time, when the system scaled up, there were new areas that could slow down the system. We then added more logging logic to the system, for example, execution time logging to help build some visualization dashboards to identify the root cause.

const sendToteToPackingStation = (warehouseId: string, toteId: string): Promise<Result> => {
  const startTime = performance.now();

  logger.info('Sending tote to packing station', { warehouseId, toteId });
  const result = await someLogic(...);
  logger.info('Send tote result', { result });

  logger.info('Execution time', { durationMs: elapsedTimeMs(startTime) });
  return result;
};

Of course, when this was repeated multiple times, we started thinking about making a higher order function (HOF) to reuse everywhere is the system

First implementation…

Let’s begin with the type definition. Here are the generic types how a HOF looks like. It’s a function that receives a function and return another function with the same signature with the input function

export type WrappedFunction<
  FunctionArgs extends ReadonlyArray<unknown>,
  FunctionReturn
> = (...args: FunctionArgs) => FunctionReturn;

export type HigherOrderFunction = <
  FunctionArgs extends ReadonlyArray<unknown>,
  FunctionReturn
>(
  func: WrappedFunction<FunctionArgs, FunctionReturn>
) => WrappedFunction<FunctionArgs, FunctionReturn>;

And here is the first basic implementation

export const withLogging: HigherOrderFunction =
  (wrappedFunction) =>
  (...args) => {
    const functionName = wrappedFunction.name || wrappedFunction.constructor.name;
    logger.info(`Running function ${functionName}`, { args });
    const startTime = performance.now();

    const result = wrappedFunction(...args);
    logger.info(`Finished ${functionName}`, {
      result,
      durationMs: elapsedTimeMs(startTime)      
    });

    return result;
  };

const sendToteToPackingStationFn = (warehouseId: string, toteId: string): Promise<Result> => {...}
export const sendToteToPackingStation = withLogging(sendToteToPackingStationFn);

…but it introduced other problems

• There are some cases that we want to log only the input and the execution time because the output object is too large and can slow down the system. We had to introduced another HOF withParamsLogging just for that purpose. Finally, we ended up with a lot of overlapping HOFs like that in our system.
• For function output logging, depending on the case, we have different ways of returning data. Sometimes, it’s a normal object with a status/outcome prop. In some other cases, we have to return an object from neverthrow library. It’s hard to maintain an HOF for multiple purposes. It usually becomes too complex and contains a lot of if/else to make sure we log the right way.
• We cannot use just part of the logic of the HOF, for example, only log execution time of the input function.

Composable Small HOFs

Inspired by Redux compose function, it doesn’t take much time to resolve all the above problems with a composable approach. Instead of making big and generic HOFs, I decided to split them into smaller HOFs, each HOF is responsible for 1 and only 1 single purpose, for example

• withInputLogging and withOutputLogging to log the input and output of a function
• withNeverthrowOutputLogging for logic specific to logging neverthrow output object
• withExecutionTimeLogging
• withPubsubLogging automatically sends the data to pubsub
• withUnhandledErrorLogging beautifies the unhandled error and rethrows it
• …

To combine them, implement a compose function like this

export const compose =
  (...hofs: Array<HigherOrderFunction>): HigherOrderFunction =>
  (wrappedFunction) =>
  (...args) => {
    if (!hofs.length) {
      return wrappedFunction(...args);
    }

    const [head, ...tail] = hofs;
    const finalFunc = tail.reduce(
      (res, hof) => hof(res),
      head(wrappedFunction)
    );

    return finalFunc(...args);
  };

and then you can decide which HOFs you want to use in each individual case

export const sendToteToPackingStation = compose(
  withInputLogging,
  withOutputLogging,
  withExecutionTimeLogging,
)(sendToteToPackingStationFn);

Make it work with Promise

You may notice that the above logging utils don’t really play nicely with Promise. To fix it, just add a finally block if the return data is a Promise.

//...
const result = wrappedFunction(...args);

if (isPromise(result)) {
  result.finally(() => logFinishTime());
} else {
  logFinishTime();
}
//...

Include extra information

You may find yourself need to include some extra information to the HOFs, for example, to log a key that you can use to correlation the business logic. In that case, simply turn that HOF to a function that return the actual HOF (yes, it’s a function returning a function returning a function 😅).

export const withInputLogging =
  (opts: { action?: string } = {}): HigherOrderFunction =>
  (wrappedFunction) =>
  (...args) => {
    logger.info('Start hof', { action, args })
    //...
  };

// or to specify the pubsub topic to publish to?
export const withPubsubLogging =
  (opts: { topic: string }): HigherOrderFunction =>
  (wrappedFunction) =>
  (...args) => {
    client.publish(topic, { args });
    //...
  };

In the previous post, I have shown you a real example about how to solve the problem as a Software Engineer. This time, we gonna do it by the Product Engineer approach.

Make Agile great again!

If you refer to the first post that I wrote, Building Product is about delivering user values, collecting feedbacks and constantly adapting to the change of business. Does it sound like Agile? Yes, in my opinion, Agile seems to be the best fit out there for Product company at a small and medium size.

Don’t do this

Let’s start with the non-Agile way by this picture-by-picture story

When you put this into a Software perspective, it’s pretty much the same with the first example that I showed in my previous post. The way most people would choose is to implement the whole feature, from backend to frontend before delivering to the customer. Again, do NOT do this.

but use this Incremental Delivery model

How about taking 1 parcel at a time? Begin with the most valuable one and repeat until you’re done?

Does it feel repetitive? Yes, but it’s less risky because it’s much simpler for you to implement. You can ship the values continuously, collecting the feedbacks from your customers. When there are any problems, you can quickly change the route (the approach). Sometimes, you can even throw away the unnecessary features based on the feedbacks. By starting with the most valuable ones, you also focus on resolving 80% of the problem using 20% of the effort, gain the most revenue for the company.

Start from User Values

Ok. If all the above sound so theoretical, how about taking the Integration project that I mentioned in part 2 of this series and do it the Agile way?

Instead of defining the tickets/milestones based on the code implementation, try re-writing the Acceptance Criteria following this format

- As a user (a staff, a system admin,...), I want...
- (Optional) so that I can solve (don't have to, be able to,...) ...

• As a user of XXX, I want to be able to log in to Shopify from XXX so I can verify that the credentials are correct.
• As a user of XXX, I want my order to be synced to XXX when it is placed on Shopify.
• As a user of XXX, I want my order to be updated in XXX when it is changed on Shopify.
• As a user of XXX, I want my fulfillment information to be reflected in XXX when I mark the order as fulfilled on Shopify.

then come up with the implementation

For instance, you have this AC from the above

As a user of XXX, I want my order to be synced to XXX when it is placed on Shopify.

you will then define the steps to execute it

• Set up the webhook endpoint to handle OrderCreated event from Shopify.
• Configure Shopify to send updated data to the new webhook endpoint.
• Implement the mapping logic to map from Shopify sales order to XXX sales order schema.
• Add a new API to get list of orders (very basic information).
• Design a simple UI so the user can verify the information.

We should try to make this as simple and as achievable as possible. This ticket should not target on building a beautiful UI. It should follows exactly the pre-defined AC and provide a very simple UI to verify the data only. After that, there should be another ticket to enhance the UI.

Defining User Value rules

Here are some rules for you to follow to ensure that the project will be on the right track

• Usually, you still need a scaffolding phase in the beginning.
  • Again, be agile, try to make it as small as possible. Don’t try to be perfect!
• A User value isn’t necessarily a big and complete feature.
  • If it’s still too big, split it until it’s small enough for an S, M or maximum L estimate. Never create XL+ ticket
• However, a User value/ticket should be a complete flow, frontend to backend
  • That means the Users can use it
• Be clear about the target users and decide whether this feature is necessary for that target users.
• A clear definition of Done
  • Done is when the feature flows all the way to Production and is shipped to the user. You don’t have to come back anymore
• Do NOT include future work.
• Prioritization: be 80-20, highest value with smallest effort first.

To be continued

More real world examples…

Previous post: Let’s build Product, not Software - Part 1

In the first part of this series, I have walked you quickly through some differences between Building Product and Building Software and why Building Product is important. In this post, I’ll show you a simple example, analyze its problem and come up with a solution following the Product Engineer mindset.

A Software Engineer approach

Let’s take a look at this simple project

You are working for an Automation marketing platform for E-commerce merchants. The product needs the data about the sales orders of the merchant and your task is to build a 1 way Sales Order integration feature to sync data from Shopify to your system

After going through several discovery steps with your customers and the PO, you decide that it’s time to make the implementation plan. You now come up with this plan

Sounds good? Yes, this is a perfect plan from a Software Engineer perspective, but…

Assume that it’s 01/10/2023 and you have finished the first Backend phase. Everything works as expected, you bring this to the weekly project update meeting and present to all the stakeholders. However, oposite to what you expected, they feel so disappointed. What has happened?

• They have been waiting for 1 month and they still have nothing.
• They have to be patient for another month, for 4 other project update meeting to start using the feature.
• The PO and QA have also waited for that long, sitting there without doing anything beside preparing all the test cases

Another month has passes, you now finish the beautiful frontend so everybody can start experimenting it. The PO now realizes that the workflow is so confusing, could lead to a lot of complaints from customers. They ask you to go back and change the main workflow. You say that the business needs to either extend the deadline or accept that flow because it would take a lot of time to adapt to this change. The whole team start debating because nobody is satisfied.

After several discussions, you have convinced the team to follow the original solution. It’s time to release to some beta customers. What the PO worried about has happened now. The customers say that the feature is too hard to use and they don’t really like this. It’s when the project really go off-track

How about making smaller milestones?

After reading the above part of this post, you decide to do it differently. You now go back and update the milestones like this

Sounds better, huh? Yeah this is closer to the target. Smaller milestones make it easier to change. However, there are still some questions you need to ask here

• Do we look into this from the User perspective? What are the problems that you are trying to solve for your customers?
• What does it mean when you say I have finished "Handle Sales order requests" phase? Does it mean that you finished setting up the API to handle that type of webhook requests? Or do you mean that you have completed the function to map from Shopify schema to your system schema? Does it include the frontend? Can the user start using it?
• Or more precisely, the question is What is your definition of Done?. Do you mean it’s done when you say it’s done? Do you need to go back and change it if the frontend doesn’t fit the data?
• When the project goes off-track and you are going to miss the deadline, what features can you drop but not affecting the workflow?

Are you building a Software or a Product?

To be continued

A Product-oriented solution will be presented in Part 3

This may not always be true. It really depends on the type of company that I will go through in this post. This also sometimes sounds strange from a Software Engineering perspective, but as we get used to, it really did help FMG (my old company) grow to the leading player that niche market

Software engineers love technology, for sure. We love building things, love applying the latest technology into our product. However, we usually forget one important thing: The technology that doesn’t fit in the product, cannot make a profitable business, is a useless one. That sounds obvious, right? Surprisingly, a lot of Software Engineers that I’ve met made this mistake, especially the talented ones.

Let me walk you through the 2 approaches, compare the differences between them and analyze some real examples. In latter posts, you can also find some techniques that I’ve applied in order to help build a better Product Engineer mindset.

This is converted from a presentation that I made at work

What is Building Product?

In short, Building Product could be described as

Delivering USER VALUES

Building Product usually includes these 3 main steps

• Identify the user problems
• Provide the solutions, using technology, and optionally, do it better than your opponents
• Constantly update and adapt to the changes of the business. If you don’t keep up with the market, you are losing already. A good product you build today could be a crap tomorrow.

In Building Product, you should NOT prioritize the technology. Yes, tech still plays an important role but it is not the most important factor. You should always be focusing on the User Values. Building Product is also not about Building a Perfect solution. Nothing is perfect. The business changes every day. Spending time making one thing perfect means getting nothing done. A perfect feature that you write today could be a shitty one tomorrow.

Perfect is the enemy of good

Why Building Product instead of Software?

Let’s ask these questions

• Do the users care if you are using latest technology or not?
• Do they care if you have spent a lot of effort to develop a feature on your own or do they care if you use a third party solution?
• Do they care if you have some tech debt to resolve?
• Do they care if you use PHP to develop? 🤣

Sadly, the answer to all those questions is NO. They don’t (fucking) care. All they care are their problems. Building Product is to deliver User Values, and that’s what matters to your customers.

Also, a good technology doesn’t always mean a good product if it doesn’t solve user problems. Even a shitty technology sometimes is a suitable choice because it does the job. And then, it simply becomes a high quality one.

A better product, of course, helps generating better revenue. By providing what the customers need, you can attract new clients, retain the existing ones longer and reduce the churn rate. Once you got the money, you can even throw away some crappy modules that you made and buy another solution. That is really important to keep in mind and is cruelly how the business works.

“Start with the customer experience and work backwards to the technology. You can’t start with the technology and try to figure out where you’re gonna try to sell it” - Steve Jobs

When to apply?

However, you don’t always need to apply that mindset. If you are working in an outsourcing company or in a large organization, where all the requirements are clearly defined, sometimes it’s better to just build the software.

Building Product mindset is more suitable for startup or small/medium product company, where you have to move fast, where nothing is clear and the requirements change everyday.

To be continued

In the next post, I’ll show some real examples and examine the problems as well as come up with the solution that we can use.

Read it here Let’s build Product, not Software - Part 2

Just a collection of tips to make working with Postgres (and other SQL-like databases) easier

Integration data

Usually when you build a system that integrates with other 3rd party service, you will need to store integration information related to the entity, for example the id of the entity on the 3rd party system or some of its configuration on that system. Imagine that you are building an e-commerce related product, you may want to sync the Sales order information from Shopify to do the analytics on customer behavior. The first solution you can think of is to add a column like shopify_entity_id on that table.

• What will happens if you introduce another integration later? Does the name shopify_entity_id still make sense? You may consider renaming it to external_entity_id. How do you know where it comes from? Adding another source column? How do you store extra 3rd party information about the sales order? Keep adding columns like external_something? Do those columns actually belong to the sales_order table itself?
• What will happen if an single entity exists on multiple 3rd party system? For instance, the sales order may be presented on both Shopify and on another Shipping service. How would you deal with it? Keep adding more columns? What if we introduct another integration?
  • A Json (Jsonb) column could solve the above issue but also creates a whole new problem. How about schema enforcement and constraint? How do we make sure that nobody will accidentally update it the an incorrect schema? How about null and undefined values (in case you are working with Javascript)? How about indexing the values for quick access? You can index inside the json but it just makes things more complicated due to those schema problems mentioned above.

The solution, of course, is a SQL approach: make an entity integration table (sales_order_integration in this case). It’s a 1-N relationship, 1 sales order could have 0 or multiple integrations

sales_order table

sales_order_integration table

You may add another integration table (and sub-tables if needed) as a generalization to store other information like the access/refresh token, for example

and then the sales_order_integration table would become

Here are some query scenarios

• Find sales_order by id: of course, you always have index on primary key
• Find external ids by sales_order_id: simply add an index on sales_order_id column of the sales_order_integration table and you’are good to go.
  • You could use this when you want to refecth new data of the entity from a specific 3rd party system
  • You could also use this when you want to write updated data to the other systems
• Find sales_order_id by external_entity_id: you can add an index on external_entity_id column. Another option is a unique constraint on integration_id, external_entity_id (which makes more sense since each entity in your system should correspond to only one entity in the other system). A unique constraint also acts as an index. A simple JOIN like this will utilize both the unique index and the primary key index on 2 tables

  SELECT *
  FROM sales_order s INNER JOIN sales_order_integration i
  ON s.id = i.sales_order_id
  WHERE i.integration_id = $1 AND i.external_entity_id = $2
  

  • This is usually seen in webhook handler, where you receive the update about each individual sales order. You can use this method to quickly look up the sales order using its external id.