jiahao.blog

The Math Behind Burn Rate Detectors

Fri, 04 Sep 2026 00:00:00 GMT

This article is my understanding of the Google SRE Workbook chapter on “Alerting on SLOs”.

I highly recommend reading it to get more examples!

SLOs and error budgets

An SLO, or Service Level Objective, is an internal target for a service to meet over a period of time. Common SLOs are used for service availability and latency SLOs, such as having a 5 nines uptime (99.999% availability).

When defining an SLO, you also (inherently) define an error budget: the number of errors that can occur within the period of time that stays within the SLO.

For instance, at an SLO of 99.9% (3 nines), the error budget is 0.1%. So if the service has 1,000,000 requests a month (which we can standardize as 30 days for simplicity), the error budget is 1,000 errors in the entire month. This means that every day, you can have about errors. Remember this number because we will keep coming back to it as reference!

If we get 30 errors every day, we would have only gotten 900 errors, and we would have still met our SLO of a 99.9% availability.

Why error rates don’t scale well

Typically, most operational health metrics are based on the rate of errors that occur at a period of time, compared to the SLO. We call these error rate alerts.

These error rate alerts are very one-dimensional. They primarily focus on detecting regressions of your system at a point in time, and do not adapt well to the total traffic over time.

For instance, the error rate threshold for the availability of a service with an expected uptime of 99.9% might look like:

This alert could be configured with a lookback window of 15 minutes.

Then, take the following scenario where the total requests at a point in time is equivalent to the number of 500s:

In this scenario, if we used an error rate alert, it might detect the increase in 500s over the lookback window, and it would compute the error rate at the point in time:

It would calculate something like:

Number of 500s: 1
Number of requests: 1
Error rate: 100%

Given that the error rate is 100% and the error rate threshold is 0.1%, the alert would fire. But this is likely a false alarm, as there were no other recurrences of the errors across the rest of the service.

This is a problem because now the alert has a high false positive rate, creating alert fatigue for responders.

Why not increase the lookback window?

A very natural first instinct would be to say, “Why not increase the lookback window so it takes more errors to trigger the alert?”

And this is absolutely spot on! In this scenario, suppose that the lookback window is increased to 1 hour instead (4x greater than the previous window):

Now, the calculations could look like (albeit rather exaggerated but point notwithstanding):

Number of 500s: 1
Number of requests: 500,000
Error rate: 0.0002%

This time, the error rate is much less than the error rate threshold, so an alert doesn’t occur.

However, suppose that we took a step back and zoomed out, and realized that every hour, we were getting 1-3 errors (averaging 2 an hour).

(Excuse my awful mouse drawing skills)

Individually, at each lookback window, none of these errors would raise an alert, but if this problem persisted throughout the day, it would have generated an average of 48 alerts in the day, above the error budget of 33.3 errors a day.

Now, we have a problem of false negatives where a clear issue is going unnoticed because our error rate alert is too loose.

What about enforcing that errors need to be firing for an extended duration?

Another great instinct!

And this is not the same as increasing the lookback window!

Increasing the lookback window is like saying, “I want to have a wider sampling window so that my error rate is less prone to one-off spikes”, which gives the error rate alert more “breathing room” to ingest more events - error and success - to avoid false positives.

Increasing the alerting duration is like saying, “I want the problem to last for a sustained duration”. For instance, the service might get bursts of 100% error rates that last for a minute. But a proper issue would be when these 100% error rates last for 10 minutes. In this case, the alert duration would be 10 minutes.

However, once again, a burst of 100% error rates that last for 5 minutes can then go un-alerted, creating another problem of false negatives.

So if we can’t really improve the false positive and negatives by using error rate alerts, what can we do?

It’s a bird, it’s a plane, it’s a burn rate detector!

Rather than modelling and focusing on the error rate, we should design alerts around how quickly they burn through the error budget.

If our error budget is 33.3 a day, a serious issue would chew through the budget in less time, which could mean that the problem is a slow-burning issue (where it is chewing through the budget consistently throughout half the day) OR a sudden spike (where it immediately chews through the budget).

To model the rate the error budget is burned, we can use the following:

Burn rate	Error rate at 99.9%	Time to exhaust error budget
1	0.1%	30 days
2	0.2%	15 days
10	1%	3 days
1000	100%	43 minutes

We start shifting our focus from “what are the errors happening at this moment?” to “are the error occurring at this moment going to burn through my error budget more quickly than expected?”

Then, we define our consumption threshold, which is the threshold that the burn rate exceeds that we deem necessary for alerting. Maybe we want to be alerted when 2% of our error budget is consumed, so the consumption threshold would be 2% of the error budget within the available lookback window.

With these, we can now define the actual burn rate detector and the alert threshold:

The SLO period is the entire month in this instance.

So, if we have a consumption % of 2% for a lookback window of 1 hour, we would have a consumption threshold of:

So if an issue is burning through our error budget at a rate that is 14.4 (or 1.44% error rate at 99.9% SLO) or greater, we deem it alert-worthy.

To illustrate this, we can reverse the question, asking ourselves, “How many errors need to occur in the 1 hour lookback window to have a burn rate of 14.4?”

Let’s constraint this problem by defining that the number of requests is 500,000 and SLO is still 99.9%:

This means that we would have to have 7,200 errors to trigger this alert. If this number seems too high, then lowering the consumption % would help to increase the sensitivity of the alert as well. For instance, we can lower the consumption % to 0.5% (so if 0.5% of our error budget is burned, we want to be alerted). This yields a total error count of 1,800 within the lookback window. You could also tighten the SLO so that the error budget decreases, for instance, keeping the consumption % at 2%, but tightening the SLO to 99.999% (5 nines), then the total error count would be 72 instead.

I highly recommend taking a pen and paper and trying these computations several times to build up an intuition for how each component influences the others.

Tiered burn rate detectors

We can also combine multiple burn rate detectors to create tiers of alerts, where each tier has a different alerting mechanisms (like creating Slack messages, opening tickets, paging, etc) associated with the rate of error consumption. These can be paired with different lookback windows as well:

Consumption %	Lookback window	Alerting	Burn rate
2%	1 hour	Page	14.4
5%	6 hours	Page	6
10%	3 days	Ticket	1

(These are the recommended thresholds from Google)

These create higher granularity alerts: the first alert detects sharp spikes in errors while the last alert detects slow-burn issues that don’t immediately impact the availability but still burn through the error budget.

Reducing false positives

Even with burn rate detectors, false positives can be present: an issue could no longer be ongoing but its tail-end could still exist at the very start of the lookback window, triggering the burn rate detector:

To reduce the rate of false positives, we can introduce a second shorter window that answers the question, “Is this problem ongoing?”

According to the Google SRE Workbook, this shorter window should be 1/12th the length of the longer window:

Lookback window	Shorter window
1 hour	5 minutes
6 hours	30 minutes
3 days	6 hours

The consumption threshold is tied to the lookback window, and the shorter window inherits this same window, but its calculation uses the much shorter lookback.

The shorter window will be much spikier than the longer window. We only alert when both the long and short windows exceed the consumption threshold at the same time (see the colored area). In the longer lookback window’s slice (in blue), the short window burn rate does not exceed the consumption threshold, so we do not alert.

In doing so, we only alert when we confirm that:

There is a problem chewing through the SLO (long window)
The problem is still ongoing (as opposed to having occurred in the past and is currently recovering) (short window)

Let’s run through a simple exercise to illustrate the point:

Short window burn rate
- Errors in 5 minutes: 2,000
- Requests in 5 minutes: 10,000
- Burn rate: 200
Long window burn rate
- Errors in 1 hour: 8,000
- Requests in 1 hour: 500,000
- Burn rate: 16

Both of these exceed the threshold of 14.4, so an alert fires. Now, when we shift the lookback window further:

Short window burn rate
- Errors in 5 minutes: 100
- Requests in 5 minutes: 10,000
- Burn rate: 10
Long window burn rate
- Errors in 1 hour: 7,500
- Requests in 1 hour: 500,000
- Burn rate: 15

In this scenario, even though the long window still contains the remnants of the errors, the short window does not capture it, so the alert does not fire.

Conclusion

I hope that this article was able to motivate the math behind burn rate detectors and how it solves problems that error rate alerts do not.

I have become a big proponent of favoring burn rate detectors over error rate alerts to develop golden signals for system health that error rate alerts simply could not provide.

Unified Data Modelling with Elixir

Fri, 12 Jun 2026 00:00:00 GMT

The LTA public bus schedule API is an amazing public resource for anyone looking to integrate Singapore’s public transport data into their applications. However, a core limitation with the API is its relatively aggressive pagination (500 rows per response). It is understandable why it has to exist given that the API is free to use and has virtually no rate limits, but it was not feasible for the project I am working on (I’ll publish a short about it soon!) as I wanted to fetch all of the data at once.

To fetch all of the data at once while avoiding spamming the API, I built an intermediate API that would bulk-load the data from the endpoints I was interested in (bus routes, bus stops, and bus arrivals) and cache the results for a day to avoid repeated API requests.

Of course, being an Elixir fanboy, I wanted to write this intermediate API in Phoenix. While writing this API, I encountered a fun application of metaprogramming: unified data modelling.

Traditional API response data modelling

Traditionally, if you were building a service that ingests an API, you would introduce a data transfer object (DTO) to parse the API response, along with a DTO for your own API responses:

The intermediate API service will receive the LTA API response, transform it into an API DTO to match the fields, and then convert that DTO to a different format that can be returned to the user (called the internal DTO).

Common conversion steps include:

Removing fields
Renaming fields (like changing the case from PascalCase to snake_case)
Parsing data

Suppose that the API responds with the following format:

// GET /api/people?$skip=0
[
  {
    "name": "John",
    "occupation": "Student",
    "homeAddress": "18 Baker Street"
  },
  {
    "name": "Mary",
    "occupation": "Teacher",
    "homeAddress": null
  }
]

A DTO could look like:

defmodule API.Person do
  defstruct [:name, :occupation, :homeAddress]
end

And have the serialization library convert the API responses to the DTO.

Then, the output DTO could look like:

defmodule Model.Person do
  defstruct [:name, :occupation, :home_address]
  
  @derive {Jason.Encoder, []}
  
  @spec from_api(API.Person) :: %__MODULE__{}
  def from_api(api) do
    %__MODULE__{
      name: api.name,
      occupation: api.occupation,
      home_address: api.homeAddress
    }
  end
end

This is discounting providing type safety to both models by declaring @type t, providing custom parsing for fields, and the cost to add new fields or modify fields as both DTOs would have to change, and many more features.

Introducing… unified data modelling

Rather than creating two separate data models, Elixir allows us to create a single model that handles both parsing and output using some clever metaprogramming constructs.

We would also like to ideally address the following:

Automatically generating consistent @type t definitions
Support custom parsing for fields like date times and special formats
Make it easy to add/delete/modify/hide fields
Generate OpenAPI specifications
Generate defstruct for all fields

However, for the sake of simplicity, let’s keep it extremely simple and design a unified data model that only takes in string fields.

The full unified data model for BusWhere can be found in this GitHub repository, where it covers more complex features like custom parsing and enum handling.

Working backwards, let’s come up with what a unified data model might look like:

defmodule Model.Person do
  use Model.ModelBase
  
  field :name, :string, "name"
  field :occupation, :string, "occupation"
  field :home_address, :string, "homeAddress"
end

We want to declare fields with the field macro, passing in the internal DTO’s field name, the field type, and the API DTO’s field name. These should then generate

defstruct [:name, :occupation, :home_address]
@type t
def from_body()

So the final Model.Person looks like:

defmodule Model.Person do
  defstruct [:name, :occupation, :home_address]
  
  @derive {Jason.Encoder, []}
  
  @type t :: %__MODULE__{
    name: String.t(),
    occupation: String.t(),
    home_address: String.t()
  }
  
  def from_body(body) do
    %__MODULE__{
      name: body["name"],
      occupation: body["occupation"],
      home_address: body["homeAddress"]
    }
  end
end

And handle any other work that is necessary to then allow the intermediate API to return the model directly as a JSON response (i.e. serialization).

Macros recap

If you wish to have a deep dive into metaprogramming in Elixir, we have a series on it you can read here.

We will cover most concepts in this article again, but one core idea to learn before diving in is how macros are treated in Elixir, particularly how the compiler handles them.

Elixir is a compiled language, so the compiler first parses files, converts the syntax into an Abstract Syntax Tree (AST) and then compiles the AST into BEAM VM bytecode which is then converted to machine code.

While parsing files, the compiler will encounter macros, which is then executed, expanding the macro in place of the macro call, using the body of a quote:

defmacro foo do
  quote do
    IO.puts("Hi")
  end
end

defmodule Bar do
  foo
end
# ===> results in
defmodule Bar do
  IO.puts("Hi")
end

quote converts its body into an AST representation so all values are preserved as-is till compilation. To evaluate values before they are converted to an AST, use unquote within a quote :

defmacro foo(number) do
  quote do
    x + number # this translates to `x + number` which is not valid
    x + unquote(number) # this translates to `x + 11` for e.g.
  end
end

More precisely, unquote inserts the AST directly, so numbers are valid ASTs, but when dealing with more complex data structures, make sure that they are an AST first, before using unquote on them.

After the compiler expands all the macros and evaluates any behavior, the final AST is then compiled into the module.

Step 0: Creating the model base

We first need to declare a ModelBase that all unified data models will use :

defmodule Model.ModelBase do
  @callback from_body(map()) :: struct()
  
  defmacro __using__(_opts) do
    quote do
      @derive {Jason.Encoder, []}
      
      import Model.ModelBase
      
      Module.register_attribute(__MODULE__, :fields, accumulate: true)
      
      @before_compile Model.ModelBase
    end
  end
end

To use this ModelBase:

defmodule Model.Person do
  use Model.ModelBase
end

The use macro in Elixir triggers the __using__ macro, resulting in the following module upon compilation:

# after macro expansion
defmodule Model.Person do
  @derive {Jason.Encoder, []}
  
  import Model.ModelBase
  
  Module.register_attribute(__MODULE__, :fields, accumulate: true)
  
  @before_compile Model.ModelBase
end

It behaves very similarly (but not exactly) to inheritance in OOP, allowing you to embed common behavior across modules.

@derive {Jason.Encoder, []} — support serialization
import Model.ModelBase — use functions and macros in Model.ModelBase without using Model.ModelBase.xxx
Module.register_attribute — registers the @field module attribute that can accumulate values every time you call it
@before_compile Model.ModelBase — calls the before_compile macro before the Model.Person module is actually compiled from AST into BEAM VM code

This sets up the core of how the unified data model will work, we will now need to add macros to Model.ModelBase to make it a reality.

Step 1: Declaring fields

We want to declare fields using a single macro — providing both the internal and API DTO field names, along with the data type. We use the @fields accumulating module attribute that we declared using __using__ :

# Model.ModelBase
@spec field(atom(), atom(), String.t()) :: Macro.t()
defmacro field(internal_name, field_type, api_name) do
  quote do
    @fields {
      unquote(internal_name),
      unquote(field_type),
      unquote(api_name)
    }
  end
end

When we use the field macro inside Model.Person, the macros will get expanded to:

# Model.ModelPerson
# field :name, :string, "name"
@fields {
  :name,
  :string,
  "name"
}

# field :occupation, :string, "occupation"
@fields {
  :occupation,
  :string,
  "occupation"
}

# field :home_address, :string, "homeAddress"
@fields {
  :home_address,
  :string,
  "homeAddress"
}

And because we set the field to accumulate: true, the @fields module attribute actually looks like this once all of the field macros are expanded:

@fields [
  {
    :name,
    :string,
    "name"
  },
  {
    :occupation,
    :string,
    "occupation"
  },
  {
    :home_address,
    :string,
    "homeAddress"
  }
]

Step 2: Declaring the struct

We want to now use the values of @fields to provide a defstruct call in Model.Person , like this:

defmodule Model.Person do
  # values from __using__
  defstruct [:name, :occupation, :home_address]
end

However, as we are expanding the macro calls, we do not have access to the full value of @fields because we don’t know when field calls end.

To combat this, we use the __before_compile__ macro, which slots itself after the macro expansion but right before the compilation of the module, so it has access to the latest AST of the module after all other macros are expanded, allowing you to make one final update to the AST of the module with the knowledge of things like the final value of module attributes like @fields .

So the compiler steps look like this:

Parse raw module code — AST 1
Identify and expand all macro calls — AST 2
Call __before_compile__ to modify AST 2 — AST 3
Compile AST 3 into final module

This is what we need __before_compile__ to do:

Load the value of @fields
Extract the name component of each field
Use these values in defstruct

# Model.ModelBase
defmacro __before_compile__(env) do
  fields = Module.get_attribute(env.module, :fields) || []
  field_names = Enum.map(
    fields, 
    fn {field, _type, _key, _opts) ->
      {field, nil}
    end
  )
  
  quote do
    defstruct unquote(field_names)
  end
end

Module.get_attribute — retrieves the @fields value
Enum.map — converts the @fields list of tuples to an array of atoms
defstruct unquote(field_names) — unquotes the array of atoms from the previous step into the macro

This produces the intended compiled Model.Person module as seen above.

Step 3: Converting API to internal fields

With the @fields loaded, we can then parse each of the values from the API response into the internal fields. We can centralize this behavior in the function from_body (previously from_api):

# Model.Person
@spec from_body(map()) :: %__MODULE__{}
def from_body(api) do
  %__MODULE__{
    name: api.name,
    occupation: api.occupation,
    home_address: api.homeAddress
  }
end

Given that we may want to parse the API fields like converting strings to integers or booleans, handling enums, parsing special formats, we can build a general parsing layer by declaring parse functions that are responsible for taking the type of each field and applying the appropriate parsing rules to each:

# Model.ModelBase
def parse(:string, nil, _), do: nil
def parse(:string, api_value, _), do: api_value

def parse(:integer, nil, _), do: nil
def parse(:integer, api_value, _) 
  when is_integer(api_value) or is_float(api_value), do: api_value
def parse(:integer, api_value, _)
  when is_binary(api_value), do: String.to_integer(api_value)

To setup the from_body definition in every model, we add to the __before_compile__ macro:

# Model.ModelBase
defmacro __before_compile__(env) do
  # @fields preprocessing...
  
  escaped_fields = Macro.escape(fields)
  
  quote do
    # defstruct...
    
    def from_body(body) when is_map(body) do
      Enum.reduce(
        unquote(escaped_fields), 
        fn {field, type, key, opts}, acc ->
          raw_value = Map.get(body, key)
          value = parse(type, raw_value, opts)
          Map.put(acc, field, value)
        end
      )
    end
    
    def from_body(_), do: %__MODULE__{}
  end
  
  # parse functions
end

Note that we need to escape fields and then unquote it because unquote inserts the AST in-place, so we need to insert the AST of the fields array, not the fields array directly (which is invalid).

Step 4: Declaring `@type t`

The final piece we want is to automatically generate the @type t definition so that we can guarantee type safety and type usage through Model.Person.t(). Normally, you would need to declare @type t manually:

# Model.Person
@type t :: %__MODULE__{
  name: String.t(),
  occupation: String.t(),
  home_address: String.t()
}

But with macros, we can dynamically generate them based on the field values:

# Model.ModelBase
defmacro __before_compile__(env) do
  # @fields preprocessing...
  
  field_types = Enum.map(
    fields,
    fn {field, type, _, _} ->
      {field, type_spec(type)}
    end
  )
  
  struct_type = 
    {:%, [],
      [
        {:__MODULE__, [], nil},
        {:%{}, [], field_types}
      ]}
  
  quote do
    # defstruct...
    
    # from_body
    
    @type t :: unquote(struct_type)
  end
  
  # parse functions
end

def type_spec(:string), do: quote(do: String.t())
def type_spec(:integer), do: quote(do: integer())

Similar to parse, we declare helper functions type_spec to convert the declared type values to the appropriate type signatures, like String.t() and integer().

We also need to explicitly build the @type t struct’s AST and unquote it. The AST results in the exact @type t definition you see above. This needs to all be done before quote since we want to evaluate these values before the macro is expanded.

And…

We now have a fully unified data model Model.Person that works as both the API and internal DTO, with all of the type generation and parsing handled automatically. It scales with any number of fields and adding new support for field types or parsing is as simple as supporting new parse and type_spec helper functions. You can also use the Model.ModelBase in any number of data models, plugging in just use Model.ModelBase and using all the field macros.

If you want to see how enum values from the API is handled, or how union types can be generated, or how custom parsing for fields is implemented, you can refer to the ModelBase written for BusWhere.

This is the final Model.ModelBase:

defmodule Model.ModelBase do
  @callback from_body(map()) :: struct()
  
  defmacro __using__(_opts) do
    quote do
      @derive {Jason.Encoder, []}
      
      import Model.ModelBase
      
      Module.register_attribute(__MODULE__, :fields, accumulate: true)
      
      @before_compile Model.ModelBase
    end
  end
  
  @spec field(atom(), atom(), String.t()) :: Macro.t()
  defmacro field(internal_name, field_type, api_name) do
    quote do
      @fields {
        unquote(internal_name),
        unquote(field_type),
        unquote(api_name)
      }
    end
  end
  
  defmacro __before_compile__(env) do
    fields = Module.get_attribute(env.module, :fields) || []
    field_names = Enum.map(
      fields, 
      fn {field, _type, _key, _opts) ->
        field
      end
    )
    escaped_fields = Macro.escape(fields)
    
    field_types = Enum.map(
      fields,
      fn {field, type, _, _} ->
        {field, type_spec(type)}
      end
    )
    
    struct_type = 
      {:%, [],
        [
          {:__MODULE__, [], nil},
          {:%{}, [], field_types}
        ]}
      
    quote do
      defstruct unquote(field_names)
      
      def from_body(body) when is_map(body) do
        Enum.reduce(
          unquote(escaped_fields), 
          fn {field, type, key, opts}, acc ->
            raw_value = Map.get(body, key)
            value = parse(type, raw_value, opts)
            Map.put(acc, field, value)
          end
        )
      end
      
      def from_body(_), do: %__MODULE__{}
      
      @type t :: unquote(struct_type)
    end
  end
  
  def parse(:string, nil, _), do: nil
  def parse(:string, api_value, _), do: api_value
  
  def parse(:integer, nil, _), do: nil
  def parse(:integer, api_value, _) 
    when is_integer(api_value) or is_float(api_value), do: api_value
  def parse(:integer, api_value, _)
    when is_binary(api_value), do: String.to_integer(api_value)
    
  def type_spec(:string), do: quote(do: String.t())
  def type_spec(:integer), do: quote(do: integer())
end

Running Docker in GitHub Action steps

Tue, 10 Feb 2026 00:00:00 GMT

Working on the packaging and publishing pipeline for the Git-Mastery app revealed many interesting problems in software packaging, particularly for Linux distributions like Debian and Arch Linux.

In my last post, I shared about the packaging and publishing process for Debian, and how it is pretty non-trivial to get going, especially if you want to support multiple architectures and automate the pipeline using GitHub Actions.

A silver lining?

One benefit of packaging and publishing for Debian through GitHub Actions is that there is a GitHub-hosted runner for Ubuntu, a Debian-based distribution. This means we can easily translate the steps we would take regularly to package and publish a .deb manually to the steps a GitHub Action job would use.

But packaging for Arch Linux does not share the same benefit. There is no GitHub-hosted runner for Arch Linux, so when performing certain actions during the packaging process (like using makepkg), we do not have access to these commands in the Ubuntu hosted runner, and this is a major limitation in the packaging process.

So, what do we do?

Luckily, we can run jobs in a container, where you can:

Define a Dockerfile to set up the container
Run all your steps within this container

In doing so, we can define a job that runs an Arch Linux base image as a Docker container

That’s it right? Simple as that.

Well, not exactly…

Wait, what’s wrong?

Depending on the container we select, we might need to perform additional set up to prepare the container to include the typical dependencies of the GitHub-hosted runners.

Common tools like python and git may not exist, and it is up to us - the designers of these pipelines - to set up these dependencies and ensure that the container is fit to use.

It might seem simple enough to fix, but some actions that you rely on may also make assumptions about their dependencies, which creates a pretty painful experience all the way down.

Instead, it may be easier (given your specific context), to use Docker containers in certain steps, so you get the full benefit of the GitHub-hosted runners while giving you the option to access a different underlying operating system for a step or two.

What the title promised

To achieve this, an implicit requirement exists: the Docker container and the runner must be able to share files with each other.

This is very easily achievable by mounting a volume in Docker to a folder in the runner.

By doing this, we can now communicate between the Docker container and the runner.

The rest is relatively straight forward:

docker run --rm \
  -v $PWD:/pkg \
  archlinux:base-devel \
  bash -c "
    pacman -Sy --noconfirm sudo git base-devel && \
    useradd -m builder && \
    echo 'builder ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers && \
    chown -R builder:builder /pkg && \
    su builder -c 'cd /pkg && makepkg --printsrcinfo > .SRCINFO'
  "

sudo chown -R $(id -u):$(id -g) .

That seems like a whole lot of nonsense, but in essence, we are:

--rm: telling Docker to remove the container when it exits
-v $PWD:/pkg: mounting the current working directory of the runner with a directory called /pkg in the Docker container
archlinux:base-devl: specifying the image that Docker will use
bash -c: runs the following string as a bash command

You don’t need to concern yourself with the specific bash commands running, these are all specific to publishing on Arch Linux (I will be making a post on this soon!).

The final step involves fixing the permissions of the files generated by the Docker container so that the runner is able to properly access and interact with them.

When the files are generated from within the Docker container, even if they are stored in a mounted volume, they will still “belong” to the user within the container. This user will differ from the user in the runner, and this can be problematic for permissions on the runner.

So sudo chown -R $(id -u):$(id -g) . sets the permissions of all files (recursively) within the current directory (aka the mounted volume) back to the runner’s user’s details.

You can extend this by treating the mounted volume as a communication protocol between the Docker container and the runner: using it to store values in shared files that can be read by the runner and used to set environment variables afterwards:

docker run --rm \
  -v $PWD:/pkg \
  archlinux:base-devel \
  bash -c "
    echo 'FOO=hi' >> /pkg/test.txt
  "

sudo chown -R $(id -u):$(id -g) .

cat test.txt >> "$GITHUB_ENV"

Packaging and Publishing on Debian

Thu, 01 Jan 2026 00:00:00 GMT

As part of developing Git-Mastery, I had to publish the app across as many operating systems as possible for students to download it.

Debian was a target operating system as students may be using Debian/Ubuntu/some Debian-based Linux distro as their daily driver or they may be using Ubuntu for Windows Subsystem for Linux (WSL) on Windows.

Surprisingly, the process of packaging the app for Debian was way more involved than I had originally anticipated, taking over a day of experimentation to integrate with GitHub Actions.

While there are several nice guides covering individual steps of this process, I found myself having to piece them together as I was working on this that, and I had eventually compiled a set of pretty comprehensive notes on my findings.

I wanted to use this opportunity to consolidate my process and hopefully remove the awkwardness on figuring out how to package and publish applications on Debian!

I am not claiming to be subject matter expert, and I may have some inefficient steps below!

If you spot any, feel free to drop me an email and I will improve it!

What will be covered?

This guide hopes to cover the following core topics:

Packaging your application on Debian (creating a .deb file)
Creating a Debian repository using reprepo and GitHub Pages
Publishing multi-architecture packages
Automating packaging and publishing using GitHub Actions

Prerequisites

Before packaging your application, you first need to create an ELF binary of your program (which I will refer to as a “binary” or “executable” from here on out). I will use the Git-Mastery app binary for reference.

If you wish to follow along, you can get v7.1.2 of the binary on ARM64 here.

The Git-Mastery app uses PyInstaller to bundle the CLI into a single executable, and it uses the Ubuntu GitHub Actions image to generate this executable across architectures.

If you are not using GitHub Actions, the next easiest option is to use a Virtual Machine (VM) to build it. You can also use the VM to run through the individual steps of packaging, so I highly recommend setting one up now. I have included my set-up for a Debian ARM64 VM on MacOS Apple Silicon in the appendix below.

Finally, you will need to ensure that you have the devscripts and debhelper-compat packages installed in your environment.

In Debian, you can install it using apt:

sudo apt-get install devscripts build-essential debhelper-compat

Core concepts

The official introduction to packaging on Debian provides some high-level details about the packaging process that I will (shamelessly) copy over to explain some fundamental concepts.

Upstream tarball: a tarball (.tar.gz or .tgz) that contains the software that has been written (i.e. it contains the executable)
Source package: built from the upstream source; “contains the upstream source distribution, configuration for the package build system, list of runtime dependencies and conflicting packages, …”
Binary package: built from the source package which is then distributed and installed; contains (among other things) the executables and resources required for the executable to run

In our context, the upstream tarball simply contains the executable that you intend to publish and the binary package is the resulting .deb file that is consumed by users, so these should be relatively straightforward to understand.

Instead, the focus of this guide is the construction of the source package, allowing tools like dpkg-buildpackage (man) to build and create the binary package.

The simplest source package consists of:

The upstream tarball
A debian/ directory containing the changes made to the upstream source and all files required for the creation of the binary package
A description file (.dsc) which contains metadata about the above files

This guide will focus on these three as the primary packaging workflow.

Creating the upstream tarball

Given that you are packaging your own application, you will first create the upstream tarball.

Start by creating a folder for your executable. I chose to name mine using the following convention <name>-<version>-<architecture>.

mkdir gitmastery-7.1.2-arm64/

Then, copy the executable into this folder.

cp gitmastery-7.1.2-linux-arm64 gitmastery-7.1.2-arm64/

Finally, generate the tarball following naming convention: <name>_<version>.orig.tar.gz.

tar -czf gitmastery_7.1.2.orig.tar.gz gitmastery-7.1.2-arm64/gitmastery-7.1.2-linux-arm64

Creating the `debian/` directory

Navigate into the folder containing your executable.

cd gitmastery-7.1.2-arm64/

Then, create a sub-folder named debian.

mkdir debian

This is where all the configuration about packaging your application will be held.

There are several files you will need within debian/ to tell the packaging tool how to create your package:

debian/changelog

The Debian Changelog is where you (as the maintainer) log the changes to a package. If you are packaging the application using GitHub Actions, you could lift it from the latest commit message on the repository.
```
git show v7.1.2 --no-patch --pretty=format:%s
```
Otherwise, you can specify it directly when generating the file using dch (man).
```
dch --create -v 7.1.2-1 -u low --package gitmastery "Changed things"
```
- --create : generates a new debian/changelog file (note that it expects the debian/ sub-folder so run this command outside of the debian/ sub-folder)
- -v 7.1.2-1: specifying the version of the application listed in the changelog; -1 indicates the release number of the package
- -u low: indicates the urgency of the changelog entry; tied to the Debian “testing” distribution; low implies that the package may transition to “testing” in 10 days
- --package: name of the package
debian/control

The control file describes the source and binary package, providing information about the name, maintainers, and build dependencies.

You can generate this file using the following template:
```
Source: gitmastery
Maintainer: Jiahao, Woo <woojiahao1234@gmail.com>
Section: misc
Priority: optional
Standards-Version: 4.7.0
Build-Depends: debhelper-compat (= 13)

Package: gitmastery
Architecture: arm64
Depends: ${shlibs:Depends}, ${misc:Depends}, libc6 (>= 2.35), python3
Description: execute Git-Mastery
  gitmastery is a Git learning tool built by the National University of Singapore School of Computing
```
It is broken up into two parts: source and binary.

In the source section:
- Source: source package name
- Maintainer: name and email of the person responsible for the package; follows the $NAME <$EMAIL> format
- Priority: priority of the package to indicate whether the package is important to the standard functioning system; one of required, important, standard, optional
- Build-Depends: list of packages that need to be installed to build the package (note this does not imply that the package can run with these dependencies, just that it can be built)
In the binary section:
- Package: name of the binary package which might differ from the source package name (in our case it doesn’t)
- Architecture: specifies the computer architecture where the binary package is expected to work, including arm64 and amd64 (the two primary architectures I chose to focus on)
- Depends: list of packages that must be installed for the program in the binary package to work; ${shlibs:Depends} are the packages that contain built shared libraries and executables and ${misc:Depends} contain the packages like debhelper (here we also list libc and python3 as extra dependencies)
- Description: full description of the binary package; first line is a summary and the rest of the lines are a longer description
debian/copyright

The name is pretty self-explanatory - this contains the copyright information about the package.

You are able to leave this blank, but when packaging the Git-Mastery app, I had opted to instead copy over the LICENSE of the application, which happens to be the MIT license.
debian/rules

These provide the rules for installation, including how/where the application should be installed.

The file format is that of a Makefile.
```
#!/usr/bin/make -f
%:
        dh $@

override_dh_auto_install:
        install -D -m 0755 gitmastery-7.1.2-linux-arm64 debian/gitmastery/usr/bin/gitmastery
```
While the official installation guide starts with a rather sparse file (that they build upon):
```
#!/usr/bin/make -f
%:
        dh $@
```
I have found - through experimentation - that the binary package will not work unless you provide the override_dh_auto_install step (more information) where you need to specify how and where the application is getting installed using the install (man) command.
- -D: create all leading components of the destination except the last (i.e. create all sub-folders) and then copy the source to the destination
- -m 0755: set the permission mode
- debian/gitmastery/usr/bin/gitmastery: the leading debian/gitmastery is necessary because it indicates the fakeroot before packaging; the ultimate path upon installation is /usr/bin/gitmastery (this can be verified upon installation).
debian/$NAME.dirs

This serves as a directory declaration file used by the dh_installdirs (man).

For the Git-Mastery, I have left it blank, but if you create your file as debian/test.dirs, and specify a path usr/bin, it would then create debian/test/usr/bin.
debian/source/format

This specifies the version number for the format of the source package.
```
3.0 (quilt)
```
quilt (docs) is used to manage patches to Debian packages.
debian/source/include-binaries

Given that we’re directly installing the binary, we need to provide its path in this file.
```
gitmastery-7.1.2-linux-arm64
```

Creating the `.deb` file

That is all the files we really need to package the application.

With this, we can finally package the application:

dpkg-buildpackage -us -uc -a arm64

You want to run this command in the root of your directory, not within debian/!

-us: do not sign the source package
-uc: do not sign the .buildinfo and .changes files
-a arm74: specify the architecture

This produces the .deb file with the name <NAME>_<VERSION>-1_<ARCHITECTURE>.deb.

You can verify the package using sudo dpkg -i <package>. Once installed, you can also check to confirm where the binary was installed using which gitmastery.

Now, if all you wanted to do was to create a one-off .deb and share it with someone directly, you’re done! Congratulations 🎉!

But if you want to publish this .deb to a Debian repository for the apt package manager or automate the entire process, read on!

Setting up the Debian repository using `reprepro`

reprepro (repository) is used to create a Debian repository, but it also works well to host your package so that it is discoverable by through the apt package manager via third-party repositories.

As a Debian repository is just a set of files organized in a special directory tree with different infrastructure files, it can be hosted through a static site platform like GitHub Pages. But before we get ahead of ourselves with hosting, let’s figure out how to first generate the necessary files.

I had initially followed this wonderful guide on the DigitalOcean community tutorial blog on “How to Use Reprepro for a Secure Package Repository on Ubuntu 14.04” but I have adapted it for my (and hopefully your) use case!

Install reprepro on your machine.

apt-get update
apt-get install reprepro

Then, create a folder to store the files:

mkdir repo && cd repo

With this, we can start the process of setting up the Debian repository.

Create a GPG key. If you already have a GPG key, you might find this discussion useful in deciding if you should generate another one just for publishing your package, or you can consider using a subkey instead.

gpg --gen-key

List the key.

gpg --list-secret-key

The returned value will include your key which we will refer to from here on out as <key> .

[keyboxd]
---------
sec   ed25519 2025-07-04 [SC] [expires: 2028-07-03]
      <key>
uid           [ultimate] Jiahao, Woo <woojiahao1234@gmail.com>
ssb   cv25519 2025-07-04 [E] [expires: 2028-07-03]

Then, within repo/, create a conf/ folder.

mkdir conf

Create a repo/conf/distributions file:

Origin: GitHub
Label: GitHub Git-Mastery
Suite: stable
Codename: any
Components: main
Architectures: arm64
SignWith: <key>

Origin / Label / Description: free-form text displayed to the user or used for pinning
Suite: stable or testing
Architectures: space-separated list of Debian architecture names; for now, we’re only publishing to the arm64 architecture (we will explore how to support multi-architecture packages later)
SignWith: signing fingerprint from the PGP certificate above using <key>

Then, you can start to add packages to your repository. For the Git-Mastery app, because we store the generated .deb packages as release artifacts, we need an additional step to download the files onto the local machine, but if you have the .deb package on hand, you can just use it directly.

curl -L https://github.com/git-mastery/app/releases/download/v7.1.2/gitmastery_7.1.2-1_arm64.deb -o gitmastery_7.1.2-1_arm64.deb

With this package, use debsigs (man) to crytographically sign it using the GPG key created earlier:

debsigs -v \
  --gpgopts="--batch --no-tty --pinentry-mode=loopback" \
  --sign=origin \
  --default-key="<key>" \
  gitmastery_7.1.2-1_arm64.deb

--gpgopts
- --batch: non-interactive mode
- --no-tty: don’t use the terminal
- --pinentry-mode=loopback: allow passphrase input programmatically
--sign=origin: creates an origin signature that indicates that it is the official signature of the organization that distributes the package (although you might use maint instead)
--default-key="<key>": uses the GPG key created

Then, add the signed .deb to the Debian repository using reprepro:

reprepro -Vb repo includedeb any \
  "gitmastery_7.1.2-1_arm64.deb"

-Vb repo: verbose and use the repo folder as the base directory
includedeb: command to insert the .deb into the repository and update the metadata
any: target distribution

This creates the necessary files for the Debian repository to host and publish the .deb package you had just included.

Then, copy your GPG public key over into repo/ along with creating an empty index.html file.

You can copy your GPG public key to a file named pubkey.gpg via

gpg --output pubkey.gpg --armor --export <email>

Publishing the Debian repository to GitHub Pages

As mentioned before, a Debian repository is just a set of files with some set of special infrastructure files. So you can host it on a static site hosting platform like GitHub Pages.

To do so, initialize repo as a Git repository.

git init

Then, create a GitHub repository and add it as a remote to the local Git repository.

git remote add origin https://github.com/woojiahao/demo-apt-repo.git

Add all of the generated files in a commit.

git add . && git commit -m "Setup Debian repository"

Push the changes to the repository.

git push -u origin main

Finally, enable GitHub Pages on the repository and GitHub should handle the rest.

You should then be able to install the package from the repository.

# Set-up to recognize the Debian repository as a trusted source
echo "deb [trusted=yes] https://git-mastery.github.io/gitmastery-apt-repo any main" | \
  sudo tee /etc/apt/sources.list.d/gitmastery.list > /dev/null
sudo apt install software-properties-common
sudo add-apt-repository "deb https://git-mastery.github.io/gitmastery-apt-repo any main"

# Installing the package
sudo apt update
sudo apt-get install gitmastery

Remember to substitute your username and repository name accordingly!

Amazing, we are able to now install the package from the new Debian repository, so users can avoid directly downloading your .deb package just to try your package.

This just leaves two big questions:

How do we support multiple architectures?
How do we automate this process?

Multi-architecture support

Supporting multiple architectures is actually a lot simpler once you understand the packaging and publishing steps.

Multi-architecture packaging

To create package your application for multiple architectures, you will need to:

Create multiple executables for each architecture
Create an upstream tarball for each architecture
Update the Architecture field in the debian/control file
Update the executable path in the debian/rules file
Update the executable name in the debian/source/include-binaries file
When running the dpkg-buildpackage command, pass the architecture to the -a flag

These changes should generate a new .deb file for the new architecture.

Multi-architecture publishing

With this new .deb file and you can sign it and add it to the Debian repository.

Add the new architecture to the Architectures field in repo/conf/distributions field
Sign the new .deb file with debsigs
Run the same reprepro command again (with no changes!)

This should generate the necessary files for your Debian repository. Remember to commit these changes and push them, and let GitHub Pages do the rest.

Now, depending on your user’s architecture, the Debian repository should be able to intelligently download the appropriate signed .deb package for that architecture.

Let’s now wrap everything up and see how we can automate this entire process using GitHub Actions so that it triggers when a new tag is pushed.

Automating the packaging and publish process

For this, you will need two GitHub repositories:

The source repository that contains your application code
The Debian repository

You should already be working with the first, but you may need to create the second (note that if you were following along, you might need a new repository for this portion).

If you are new to GitHub Actions and need an introduction, I have given a talk on CI/CD with GitHub Actions along with a guide on the contents covered!

Setting up the source repository

Create a new workflow file in your source repository under .github/workflows and you can name it whatever you want. I chose to name the one for Git-Mastery publish.yml (workflow file).

To automatically package your application on tag, you can use the workflow trigger action of push:

name: Build and release Git-Mastery CLI

on:
  workflow_dispatch:
  push:
    tags:
      - "v*.*.*"

Then, you will need three jobs to run in sequence:

Building the Linux executables for both AMD64 and ARM64
Building the .deb packages per architecture
Triggering the publish job

Building the Linux executable for both AMD64 and ARM64

To build the Linux executables for both AMD64 and ARM64, we can use a matrix in GitHub Actions to run the same job on both an amd64 and arm64 runner.

Then, the steps are relatively straightforward:

Checkout the current source repository
Setup the necessary dependencies to build the application (Python 3.13 for Git-Mastery and using the libraries in requirements.txt)
Extracting the ARCHITECTURE and FILENAME from the matrix information
Build the actual binary (using pyinstaller for Git-Mastery)
Create a GitHub release with the generated binary
Publish the generated binary as a workflow artifact

We use a workflow artifact because it can be shared across jobs within the same workflow, reducing the need for us to directly access the release files.

jobs:
  linux-build:
    strategy:
      matrix:
        os: [ubuntu-latest, ubuntu-24.04-arm]
    runs-on: ${{ matrix.os }}

    steps:
      - name: Checkout source
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.13"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Get binary name
        id: binary-name
        env:
          OS_VERSION: ${{ matrix.os }}
        run: |
          if [ $OS_VERSION = "ubuntu-latest" ]; then
            ARCHITECTURE=amd64
          else
            ARCHITECTURE=arm64
          fi
          FILENAME=gitmastery-${GITHUB_REF_NAME#v}-linux-$ARCHITECTURE
          echo "binary=$FILENAME" >> $GITHUB_OUTPUT

      - name: Build binary
        env:
          BINARY_NAME: ${{ steps.binary-name.outputs.binary }}
        run: |
          echo "__version__ = \"${GITHUB_REF_NAME}\"" > app/version.py
          pyinstaller --onefile main.py --name $BINARY_NAME

      - name: Create GitHub Release
        uses: softprops/action-gh-release@v2
        with:
          files: dist/${{ steps.binary-name.outputs.binary }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Publish package as artifact
        uses: actions/upload-artifact@v4
        with:
          name: ${{ steps.binary-name.outputs.binary }}
          path: dist/${{ steps.binary-name.outputs.binary }}

Building the .deb packages per architecture

Then, we declare the next job that requires the linux-build job from above to start. We use a matrix again to build the .deb on each architecture.

These steps follow how we regularly package the .deb , except this time, we care a little more about where we store the files for publishing:

Extracting the ARCHITECTURE and FILENAME from the matrix information
Install the necessary dependencies on the runner
Create the root folder as discussed above
Create the upstream tarball
Create the files for debian/
Build the .deb using dpkg-buildpackage
Add the .deb to the GitHub release

Note that when we generate the debian/rules file this way, we need to preserve all the tabs and newlines accordingly, so we use echo -e.

We also do not need to publish any workflow artifacts as there are no more downstream jobs that will use these files.

jobs:
  linux-build:
    # ...

  debian-build:
    needs: linux-build
    strategy:
      matrix:
        os: [ubuntu-latest, ubuntu-24.04-arm]
    runs-on: ${{ matrix.os }}

    steps:
      - name: Checkout source
        uses: actions/checkout@v3
        with:
          path: "app"

      - name: Extract variables
        env:
          ARCHITECTURE: ${{ matrix.architecture }}
          OS_VERSION: ${{ matrix.os }}
        run: |
          if [ $OS_VERSION = "ubuntu-latest" ]; then
            ARCHITECTURE=amd64
          else
            ARCHITECTURE=arm64
          fi
          echo "VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
          echo "ARCHITECTURE=${ARCHITECTURE}" >> $GITHUB_ENV

          # Get the tag's commit message
          cd app/
          CHANGELOG_MESSAGE=$(git show ${GITHUB_REF_NAME} --no-patch --pretty=format:%s)
          echo "CHANGELOG_MESSAGE=${CHANGELOG_MESSAGE}" >> $GITHUB_ENV

      - name: Install Debian packaging tools
        run: |
          sudo apt-get install devscripts build-essential debhelper-compat

      - name: Create folder structure for ${{ env.ARCHITECTURE }} distribution
        run: |
          mkdir gitmastery-${VERSION}-${ARCHITECTURE}

      - name: Download ${{ env.ARCHITECTURE }} binaries from artifacts
        uses: actions/download-artifact@v4
        with:
          name: gitmastery-${{ env.VERSION }}-linux-${{ env.ARCHITECTURE }}
          path: gitmastery-${{ env.VERSION }}-${{ env.ARCHITECTURE }}/

      - name: Create upstream tarball .orig.tar.gz
        run: |
          # Create .orig.tar.gz file
          tar -czf gitmastery_${VERSION}.orig.tar.gz gitmastery-${VERSION}-${ARCHITECTURE}/gitmastery-${VERSION}-linux-${ARCHITECTURE}

      - name: Generate Debian packaging files
        working-directory: gitmastery-${{ env.VERSION }}-${{ env.ARCHITECTURE }}
        env:
          EMAIL: woojiahao1234@gmail.com
          NAME: Jiahao, Woo
        run: |
          file gitmastery-${VERSION}-linux-${ARCHITECTURE}
          # Create the debian folder
          mkdir debian

          # Generate the changelog
          dch --create -v ${VERSION}-1 -u low --package gitmastery "$CHANGELOG_MESSAGE"

          # Create the control file
          echo """Source: gitmastery
          Maintainer: $NAME <$EMAIL>
          Section: misc
          Priority: optional
          Standards-Version: 4.7.0
          Build-Depends: debhelper-compat (= 13)

          Package: gitmastery
          Architecture: ${ARCHITECTURE}
          Depends: ${shlibs:Depends}, ${misc:Depends}, libc6 (>= 2.35), python3
          Description: execute Git-Mastery
            gitmastery is a Git learning tool built by the National University of Singapore School of Computing
          """ > debian/control

          # Copy over the MIT license from the main app to this release
          cat ../app/LICENSE > debian/copyright

          mkdir debian/source
          echo "3.0 (quilt)" > debian/source/format

          # Provide the rules for installation, using -e to preserve the tab character as per:
          # https://wiki.debian.org/Packaging/Intro
          echo -e $"""#!/usr/bin/make -f
          %:
          \tdh \$@
          \n
          override_dh_auto_install:
          \tinstall -D -m 0755 gitmastery-${VERSION}-linux-${ARCHITECTURE} debian/gitmastery/usr/bin/gitmastery
          """ > debian/rules

          echo """usr/bin
          """ > debian/gitmastery.dirs

          mkdir -p debian/source
          echo """gitmastery-${VERSION}-linux-${ARCHITECTURE}
          """ > debian/source/include-binaries

          # Build the package
          dpkg-buildpackage -us -uc -a ${ARCHITECTURE}

      - name: Create GitHub Release
        uses: softprops/action-gh-release@v2
        with:
          files: gitmastery_${{ env.VERSION }}-1_${{ env.ARCHITECTURE }}.deb
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Triggering the publish job

The final step is to trigger the publish job, taking the generated .deb packages (stored in the source repository’s GitHub releases) and uploading them to GitHub Pages on the Debian repository.

For this step, we will simply call the workflow file from the other GitHub repository. We pass the github.ref_name through as a variable since we will need it for versioning the published files.

jobs:
  linux-build:
    # ...

  debian-build:
    # ...

  debian-publish-apt:
    needs: debian-build
    permissions: write-all
    uses: git-mastery/gitmastery-apt-repo/.github/workflows/debian-apt-repo.yml@main
    with:
      version: ${{ github.ref_name }}
    secrets: inherit

For the entire workflow file, refer to the appendix below.

Setting up the Debian repository

Now, in the Debian repository, create a new workflow file under .github/workflows. For Git-Mastery, I opted to call it debian-apt-repo.yml (workflow file).

There are some additional set-up steps required to work with your generated GPG key:

Add the GPG private key to the GitHub repository’s secrets, you can retrieve your private key using gpg --armor --export-secret-key

For simplicity, the GPG should not require a passphrase, otherwise it will fail when this workflow runs.
Add the GPG public key (<key>) to the GitHub repository’s variables
Add a blank index.html to the repository
Add the GPG public key file to the repository
Set your GitHub Pages to publish from the gh-pages branch instead

Then, the rest of the steps are similar to what we have described before.

The main difference here is that we iterate over both architecture types to support multi-architecture publishing.

We also use a GitHub workflow to automatically publish the new files under a separate branch gh-pages instead, treating the main branch as just a store for the necessary files.

name: Build & Publish Debian Package

on:
  workflow_call:
    inputs:
      version:
        required: true
        type: string

jobs:
  build-and-publish:
    permissions:
      pages: write
      contents: write
    environment: Main
    runs-on: ubuntu-latest
    env:
      DISTRIBUTION: any
      COMPONENT: main
      RELEASE: 1
      GPG_PRIVATE_KEY: ${{ secrets.GPG_PRIVATE_KEY }}
      KEY_ID: ${{ vars.KEY_ID }}
      VERSION: ${{ inputs.version }}

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
        with:
          repository: git-mastery/gitmastery-apt-repo
          token: ${{ secrets.GITHUB_TOKEN }}
          path: aptrepo

      - name: Install dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y reprepro dpkg-dev curl jq gnupg debsigs

      - name: Import GPG key
        run: |
          echo "$GPG_PRIVATE_KEY" | gpg --batch --import
          echo "$KEY_ID:6:" |  gpg --batch --import-ownertrust --pinentry-mode=loopback

      - name: Setup repository structure
        working-directory: aptrepo
        run: |
          mkdir -p repo/conf
          echo "Origin: GitHub" >repo/conf/distributions
          echo "Label: GitHub Git-Mastery" >>repo/conf/distributions
          echo "Suite: stable" >>repo/conf/distributions
          echo "Codename: ${DISTRIBUTION}" >>repo/conf/distributions
          echo "Components: ${COMPONENT}" >>repo/conf/distributions
          echo "Architectures: arm64 amd64" >>repo/conf/distributions
          echo "SignWith: $KEY_ID" >>repo/conf/distributions

          cp pubkey.gpg repo/pubkey.gpg
          cp index.html repo/

      - name: Add .deb
        working-directory: aptrepo
        run: |
          for ARCHITECTURE in "arm64" "amd64"; do
            echo "Publishing for $ARCHITECTURE"

            DOWNLOAD_URL=$(curl -s https://api.github.com/repos/git-mastery/app/releases/tags/${VERSION} | jq -r --arg suffix "${ARCHITECTURE}.deb" '.assets[] | select(.name | endswith($suffix)) | .browser_download_url')
            TRIMMED_VERSION=${VERSION#v}
            FILENAME="gitmastery_${TRIMMED_VERSION}-1_${ARCHITECTURE}.deb"
            TRIMMED_VERSION=${VERSION#v}

            curl -L "$DOWNLOAD_URL" -o $FILENAME

            debsigs -v --gpgopts="--batch --no-tty --pinentry-mode=loopback" --sign=origin --default-key="$KEY_ID" $FILENAME

            reprepro -Vb repo includedeb ${DISTRIBUTION} "gitmastery_${TRIMMED_VERSION}-${RELEASE}_${ARCHITECTURE}.deb"
          done

      - name: Deploy APT repository to GitHub Pages
        uses: peaceiris/actions-gh-pages@v4
        with:
          personal_token: ${{ secrets.ORG_PAT }}
          publish_dir: ./aptrepo/repo
          external_repository: git-mastery/gitmastery-apt-repo

Voilà! You have now created a fully automated package and publishing pipeline!

Visualizing the pipeline will look like this:

Conclusion

Great job getting this far! Packaging and publishing on Debian can seem very confusing because of the little amount of comprehensive documentation there is, but I hope that this guide can help to coalesce all of the available documentation into something practical and usable!

I could have used off-the-shelf solutions for this, but I liked the process of understanding how to package and publish from scratch!

These were some of the resources I had relied on when I was first starting out:

Appendix: Setting up a Debian VM on MacOS Apple Silicon

Thankfully, Debian has an ARM64 image for you to virtualize on MacOS Apple Silicon (which is what I used to experiment with packaging).

The ISO isn’t immediately obvious because it’s not (to my knowledge) listed on the same page where you regularly download ISOs. Instead, you can find this image here.

I used VirtualBox because it’s very simple to get started. Create a VM and mount the ISO. I picked XFCE as the desktop environment because I wanted to interactively download the releases from GitHub.

You can then use this VM to test the commands and try to load the packaged application!

Appendix: Packaging workflow

name: Build and release Git-Mastery CLI

on:
  workflow_dispatch:
  push:
    tags:
      - "v*.*.*"

permissions:
  contents: write
  pull-requests: write
  packages: read
  issues: read

jobs:
  linux-build:
    strategy:
      matrix:
        os: [ubuntu-latest, ubuntu-24.04-arm]
    runs-on: ${{ matrix.os }}

    steps:
      - name: Checkout source
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.13"

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      - name: Get binary name
        id: binary-name
        env:
          OS_VERSION: ${{ matrix.os }}
        run: |
          if [ $OS_VERSION = "ubuntu-latest" ]; then
            ARCHITECTURE=amd64
          else
            ARCHITECTURE=arm64
          fi
          FILENAME=gitmastery-${GITHUB_REF_NAME#v}-linux-$ARCHITECTURE
          echo "binary=$FILENAME" >> $GITHUB_OUTPUT

      - name: Build binary
        env:
          BINARY_NAME: ${{ steps.binary-name.outputs.binary }}
        run: |
          echo "__version__ = \"${GITHUB_REF_NAME}\"" > app/version.py
          pyinstaller --onefile main.py --name $BINARY_NAME

      - name: Create GitHub Release
        uses: softprops/action-gh-release@v2
        with:
          files: dist/${{ steps.binary-name.outputs.binary }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

      - name: Publish package as artifact
        uses: actions/upload-artifact@v4
        with:
          name: ${{ steps.binary-name.outputs.binary }}
          path: dist/${{ steps.binary-name.outputs.binary }}

  debian-build:
    # We support both ARM64 and AMD64 since Debian comes with support for
    # these two out of the box
    needs: linux-build
    strategy:
      matrix:
        os: [ubuntu-latest, ubuntu-24.04-arm]
    runs-on: ${{ matrix.os }}

    steps:
      - name: Checkout source
        uses: actions/checkout@v3
        with:
          path: "app"

      - name: Extract variables
        env:
          ARCHITECTURE: ${{ matrix.architecture }}
          OS_VERSION: ${{ matrix.os }}
        run: |
          if [ $OS_VERSION = "ubuntu-latest" ]; then
            ARCHITECTURE=amd64
          else
            ARCHITECTURE=arm64
          fi
          echo "VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
          echo "ARCHITECTURE=${ARCHITECTURE}" >> $GITHUB_ENV

          # Get the tag's commit message
          cd app/
          CHANGELOG_MESSAGE=$(git show ${GITHUB_REF_NAME} --no-patch --pretty=format:%s)
          echo "CHANGELOG_MESSAGE=${CHANGELOG_MESSAGE}" >> $GITHUB_ENV

      - name: Install Debian packaging tools
        run: |
          sudo apt-get install devscripts build-essential debhelper-compat

      - name: Create folder structure for ${{ env.ARCHITECTURE }} distribution
        run: |
          mkdir gitmastery-${VERSION}-${ARCHITECTURE}

      - name: Download ${{ env.ARCHITECTURE }} binaries from artifacts
        uses: actions/download-artifact@v4
        with:
          name: gitmastery-${{ env.VERSION }}-linux-${{ env.ARCHITECTURE }}
          path: gitmastery-${{ env.VERSION }}-${{ env.ARCHITECTURE }}/

      - name: Create upstream tarball .orig.tar.gz
        run: |
          # Create .orig.tar.gz file
          tar -czf gitmastery_${VERSION}.orig.tar.gz gitmastery-${VERSION}-${ARCHITECTURE}/gitmastery-${VERSION}-linux-${ARCHITECTURE}

      - name: Generate Debian packaging files
        working-directory: gitmastery-${{ env.VERSION }}-${{ env.ARCHITECTURE }}
        # TODO: Update to something agnostic
        env:
          EMAIL: woojiahao1234@gmail.com
          NAME: Jiahao, Woo
        run: |
          file gitmastery-${VERSION}-linux-${ARCHITECTURE}
          # Create the debian folder
          mkdir debian

          # Generate the changelog
          # TODO: Maybe detect if major version change, then make it urgent
          dch --create -v ${VERSION}-1 -u low --package gitmastery "$CHANGELOG_MESSAGE"

          # Create the control file
          # TODO: Maybe detect if major version change, then make it mandatory
          echo """Source: gitmastery
          Maintainer: $NAME <$EMAIL>
          Section: misc
          Priority: optional
          Standards-Version: 4.7.0
          Build-Depends: debhelper-compat (= 13)

          Package: gitmastery
          Architecture: ${ARCHITECTURE}
          Depends: ${shlibs:Depends}, ${misc:Depends}, libc6 (>= 2.35), python3
          Description: execute Git-Mastery
            gitmastery is a Git learning tool built by the National University of Singapore School of Computing
          """ > debian/control

          # Copy over the MIT license from the main app to this release
          cat ../app/LICENSE > debian/copyright

          mkdir debian/source
          echo "3.0 (quilt)" > debian/source/format

          # Provide the rules for installation, using -e to preserve the tab character as per:
          # https://wiki.debian.org/Packaging/Intro
          # $(DESTDIR) resolves to debian/binarypackage/ as seen in
          # https://www.debian.org/doc/manuals/debmake-doc/ch06.en.html#ftn.idp1797
          echo -e $"""#!/usr/bin/make -f
          %:
          \tdh \$@
          \n
          override_dh_auto_install:
          \tinstall -D -m 0755 gitmastery-${VERSION}-linux-${ARCHITECTURE} debian/gitmastery/usr/bin/gitmastery
          """ > debian/rules

          echo """usr/bin
          """ > debian/gitmastery.dirs

          mkdir -p debian/source
          echo """gitmastery-${VERSION}-linux-${ARCHITECTURE}
          """ > debian/source/include-binaries

          cat debian/rules

          # Build the package
          dpkg-buildpackage -us -uc -a ${ARCHITECTURE}

      - name: Create GitHub Release
        uses: softprops/action-gh-release@v2
        with:
          files: gitmastery_${{ env.VERSION }}-1_${{ env.ARCHITECTURE }}.deb
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  debian-publish-apt:
    needs: debian-build
    permissions: write-all
    uses: git-mastery/gitmastery-apt-repo/.github/workflows/debian-apt-repo.yml@main
    with:
      version: ${{ github.ref_name }}
    secrets: inherit

Unpacking Options in Python

Sat, 27 Dec 2025 00:00:00 GMT

I have been working on pretty interesting problems as part of Git-Mastery (we are looking for open-source contributors!) and one of the most interesting problems I have encountered recently is having to design a wrapper around the Git CLI.

A Git command (or any CLI in general) can comprise of arguments, options, and flags.

git commit [-a | --interactive | --patch] [-s] [-v] [-u[<mode>]] [--amend]
	   [--dry-run] <commit>_ | --fixup [(amend|reword):"><commit>]
	   [-F <file> | -m <msg>] [--reset-author] [--allow-empty]
	   [--allow-empty-message] [--no-verify] [-e] [--author=<author>]
	   [--date=<date>] [--cleanup=<mode>] [--[no-]status]
	   [-i | -o] [--pathspec-from-file=<file> [--pathspec-file-nul]]
	   [(--trailer <token>[(=|:)<value>])…] [-S[<keyid>]]
	   [--] [<pathspec>…]

How do we exactly model so much information within a single function?

git.commit()

Arguments will become parameters:

def commit(
    self,
    pathspec: Optional[str] = None,
) -> None:

But because the order of the flags and options can vary, it does not make sense to expand every single one of them into a separate parameter (because it’s not very sustainable to maintain it in my opinion).

Instead, we represent the flags and options as a TypedDict (docs) instead:

class CommitOptions(TypedDict, total=False):
    all: bool
    reuse_message: str
    message: str
    allow_empty: bool
    no_edit: bool

Then, we can use Unpack (docs) to expand these parameters while providing type hints for the fields of the TypedDict:

def commit(
    self,
    pathspec: Optional[str] = None,
    **options: Unpack[CommitOptions],
) -> None:

It works in tandem with ** to expand these into the respective fields.

When we call this function, we will see that the type hints include every field along with its associated type:

Then, to access the fields, you can use options.get() (since we have specified total=False).

The result is a function that we can call with any order of parameters without exploding our function signature at once:

git.commit(
  "dest", 
  message="Add dest/", 
  no_edit=True, 
  allow_empty=True
)

This works especially well when you have a much larger number of flags and options:

class RestoreOptions(TypedDict, total=False):
    source: str

    worktree: bool
    staged: bool

    ours: bool
    theirs: bool

    merge: bool
    conflict: Union[Literal["merge"], Literal["diff3"], Literal["zdiff3"]]

    ignore_unmerged: bool
    ignore_skip_worktree_bits: bool

    recurse_submodules: bool
    no_recurse_submodules: bool

    overlay: bool
    no_overlay: bool

If you are interested in how we have used this pattern in Git-Mastery, feel free to refer to the repo-smith repository: https://github.com/git-mastery/repo-smith

Building Components for Markdown in Astro

Tue, 29 Apr 2025 00:00:00 GMT

MDX allows you to use JSX in Markdown, embedding these components within the content, adding more flexibility in the type of content Markdown can generate.

import {Chart} from './snowfall.js'
export const year = 2023

# Last year’s snowfall

In {year}, the snowfall was above average.
It was followed by a warm spring which caused
flood conditions in many of the nearby rivers.

<Chart color="#fcb32c" year={year} />

However, I use Neovim, and Neovim has a shocking lack of good support for MDX files.

After trying to replicate Hugo’s shortcodes in Markdown and spending a bunch of time trying to get MDX to work nicely with Neovim for @git-mastery, I decided to look into how I could do it purely in Markdown.

Shortcodes in Hugo are custom templates that you could embed in your markdown files:

{{< gist user 23932424365401ffa5e9d9810102a477 >}}

Problem

As part of @git-mastery’s setup guide, I wanted to collapse information into tabs, such as having different OS instructions. I wanted users’ preferences to be persisted into localStorage, and shared across all other tabs with the same “key”. However, without MDX, this was not easy to create.

Additionally, the syntax to create these tabs should be as minimal as possible, so that authors of these files do not need to write extra convoluted Markdown.

::::tabs{key="bash_installation"}
:::tab{value="windows" header="Windows"}
If you are using Windows, you should try the following:
:::

:::tab{value="unix" header="Linux/MacOS"}
If you are using Linux or MacOS, you should try the following:
:::
::::

::::tabs{key="bash_installation"}
:::tab{value="windows" header="Another windows"}
If you are using Windows, you should try the following:
:::

:::tab{value="unix" header="Another Linux/MacOS"}
If you are using Linux or MacOS, you should try the following:
:::
::::

Markdown processing

Before diving into the implementation details, let me briefly cover the high-level of how Astro processes Markdown files and generates the resulting HTML pages.

Astro uses remark, remark-rehype, and rehype to process Markdown files, converting them into HTML files that are rendered (like this page!).

remark: processes Markdown files as mdast or Markdown Abstract Syntax Tree (AST)
rehype: processes HTML files as hast or HTML AST
remark-rehype: transforms mdast into hast

To build the tabs system, we essentially have to “modify” the final HTML that we generate, so the syntax used above is correctly transformed into the necessary components.

Using directives

To achieve shortcode-like syntax, we can use remark-directive, an extension on top of remark that allows :::container to be parsed as a custom component in remark.

So,

:::tabs
:::

Would be recognized as creating a “container directive” with name tabs which we can identify in the mdast generated.

remark-directive also supports parent-child nesting:

::::tabs
:::tab
<!-- tab is a child component to tabs -->
:::
::::

Directives are declared via :::name[label]{attributes}.

For this use case, we only care about the attributes, which have the following syntax:

{a=b} or {a="b"}: creates an attribute a with value "b"
{a="b" c="d"}: creates two attributes a and c
{.big .red}: creates the class attribute where the class is "big red"
{#big}: creates the id attribute where the id is "big"

We use these attributes to provide the following properties for our tabs:

tabs{key=""}: global key of tabs
tab{value=""}: value associated with the parent tabs’s key; persisted into localStorage
tab{header=""}: header to render at the top of the tabs

Setting up `remark-directive`

To setup remark-directive in Astro, download the package:

yarn add remark-directive

Then, update astro.config.mjs to include it:

import remarkDirective from "remark-directive";

export default defineConfig({
  vite: { 
    //... 
  },
  markdown: {
    remarkPlugins: [
      remarkDirective,
    ],
  },
  // ...
});

Now, Astro knows to run this remarkDirective remark plugin.

Plugins

Notice that remark and rehype aren’t just single one-off extensions, but rather, a suite of plugins that incrementally and sequentially modify the mdast and hast respectively:

As such, we can build our own plugins to implement custom transformations on top of the original mdast and hast.

Creating a `remark` plugin

Now that we have access to directives, we can build a custom plugin that will transform the tabs directives into the necessary elements.

`remark` plugin skeleton

First, install the necessary dependency to write a plugin:

yarn add mdast unified unist-util-visit

Then, create a new plugin src/plugins/remark-tabs.ts:

import type { Root } from "mdast";
import type { Plugin } from "unified";
import { visit } from "unist-util-visit";

const remarkTabs: Plugin<[], Root> = () => {
  return (tree) => {
    visit(tree, (node: any, index: number, parent: any) => {
    });
  };
};

export default remarkTabs;

A remark plugin, denoted by the type Plugin<[], Root>, is a function that visits the mdast, modifying the node or the node’s parent during traversal (using the visitor pattern).

Our plugin’s mdast transformation will be written within the visit function callback.

Detecting directives

There are several types of directives, but we are only interested in the containerDirective.

if (node.type === "containerDirective" 
  && node.name === "tabs") {
  const tabs = node.children.filter(
    (child: { type: string; name: string }) =>
      child.type === "containerDirective" 
        && child.name === "tab",
  );
  // ...
}

The plugin only executes the transformation when the current node is a container directive and when the node’s name is tabs.

Creating wrapper elements

Normally, directives are not used in remark, so they will not be converted to a meaningful HTML element. But, we can change that by modifying the hName of a node:

node.data = {
  hName: "div",
  hProperties: {
    className: "tabs"
  }
}

We can even modify the hProperties to include a className for the div.

But, what we want is not just a container for tabs. We want a header for the tabs and a container for the content of each tab.

So instead, we need to generate a wrapper element, which “replaces” the current mdast of the tabs container directive with a new AST element that contains all of the necessary components:

const wrapper = {
  type: "element",
  data: {
    hName: "div",
    hProperties: {
      className: "tabs",
      "data-tabs-key": node.attributes?.["key"] ?? "",
    },
  },
  children: [
    {
      type: "element",
      data: {
        hName: "div",
        hProperties: {
          className: "selector",
        },
      },
      children: tabs.map((tab) => ({
        type: "element",
        data: {
          hName: "div",
          hProperties: {
            "data-tab-selector-value": tab.attributes.value,
          },
        },
        children: [
          {
            type: "text",
            value: tab.attributes.header,
          },
        ],
      })),
    },
    ...tabs.map((tab) => {
      tab.data = {
        hProperties = { 
          "data-tab-value": tab.attributes.value 
        }
      }
    }),
  ],
};

The specifics of mdast syntax can be found in the Github repository.

There is a ton going on, but essentially, we are converting the initially naive tabs container directive into one with the header, selectors, and tab content.

Along with these transformations, we are attaching additional properties (through hProperties) to each component, such as data-tabs-key to be the key attribute of the tabs container directive.

The final step is to set the current node’s parent to this new wrapper element, effectively replacing the current node with this new wrapper:

parent.children[index] = wrapper

Using the plugin

To use the plugin we just created, update astro.config.mjs again, adding the remarkTabs plugin AFTER the remarkDirective plugin.

Recall that plugins run sequentially, so if we used remarkTabs before remarkDirective, we would not have access to the containerDirective.

import remarkTabs from './src/plugins/remark-tabs';

export default defineConfig({
  markdown: {
    remarkPlugins: [
      remarkDirective,
      remarkTabs,
    ]
  }
})

Then, you can verify that the modified mdast changes are reflected by rendering a tabs component and verifying the HTML:

<div class="tabs" data-tabs-key="bash_installation">
  <div class="selector">
    <div data-tab-selector-value="windows">Windows</div>
    <div data-tab-selector-value="unix">Linux/MacOS</div>
  </div>
  <div data-tab-value="windows">
    <p>If you are using Windows, you should try the following:</p>
  </div>
  <div data-tab-value="unix">
    <p>If you are using Linux or MacOS, you should try the following:</p>
  </div>
</div>

Great! But we are missing the critical interactivity of selecting tabs and persisting the choices.

Creating a `rehype` plugin

Not everything can be embedded within the remark plugin. So, we have to modify the hast generated as well, adding the necessary interactivity for the tabs.

`rehype` plugin skeleton

Just like the remark plugin, rehype plugins are defined as functions with the visitor pattern.

First, install the necessary dependency (the others were already installed previously):

yarn add hast

Then, create the plugin src/plugins/rehype-tabs.ts:

import { visit } from "unist-util-visit";
import type { Plugin } from "unified";
import type { Root } from "hast";

const rehypeTabs: Plugin<[], Root> = () => {
  return (tree) => {
    visit(tree, (node: any) => {
    });
  };
};

export default rehypeTabs;

This time, we are visiting the hast, so we can think of each node as a HTML element.

Detecting elements

Again, like the remark plugin, we will need to detect when we are visiting the various tabs elements created earlier.

if (node.type === "element" && node.tagName === "div") {
  const classNames = node.properties?.className?.split(" ") ?? [];
  if (classNames.includes("tabs")) {
    // ...
  } else if ("data-tab-selector-value" in node.properties) {
    // ...
  }
}

Each node has a type and tagName which we use as the preliminary filters. Then, we can check if the class HTML attribute contains tabs to know that we are visiting the tabs container, or if the node’s properties contains the data-tab-selector-value property, which we added to the selectors.

Storing tab keys on load

When we first load the tabs element, we want to ensure that our localStorage contains the global key.

To do so, we can append a <script> element to the end of the hast:

const tabsKey = node.properties["data-tabs-key"];
const availableKeys = JSON.parse(node.properties["data-tabs"]);
const firstKey = availableKeys[0];
const localStorageTabsKey = `tabs-${tabsKey}`;
tree.children.push({
  // @ts-ignore
  type: "element",
  tagName: "script",
  properties: {},
  children: [
    {
      type: "text",
      value: `
      (() => {
        if (localStorage.getItem("${localStorageTabsKey}") == null) {
          localStorage.setItem("${localStorageTabsKey}", ${JSON.stringify(firstKey)})
        }
      })()
      `,
    },
  ],
});

The specifics of hast syntax can be found in the Github repository.

This is the key to embedding interactivity in Markdown components. It’s not very fancy, but it certainly gets the job done.

The script above will execute when the page loads and store the key tabs-X into localStorage. The key is specified by the author of the Markdown file.

Loading and updating selection

Then, we need to incorporate the user’s selection of tabs, ensuring that their selection:

Propagates to all other tabs that share the same tabs key
Updates the localStorage value associated with tabs-X
Switches the visible tab to their selection

const tabValue = node.properties["data-tab-selector-value"];
const tabsKey = node.properties["data-tabs-key"];
const localStorageTabsKey = `tabs-${tabsKey}`;
tree.children.push({
  // @ts-ignore
  type: "element",
  tagName: "script",
  properties: {},
  children: [
    {
      type: "text",
      value: `
      (() => {
          const selectorTabsAcross = document.querySelectorAll("div.tabs[data-tabs-key='${tabsKey}'] > div.selector > div[data-tab-selector-value='${tabValue}']");
          const tabs = document.querySelectorAll("div.tabs[data-tabs-key='${tabsKey}'] > div.tabs-content > div.tab");
          const selectorTabs = document.querySelectorAll("div.tabs[data-tabs-key='${tabsKey}'] > div.selector > div[data-tab-selector-value]");

          const value = localStorage.getItem("${localStorageTabsKey}")
          select(value)

          function select(value) {
              for (const tab of tabs) {
                  if (tab.attributes["data-tab-value"].nodeValue !== value) {
                      tab.classList.remove("active")
                  } else {
                      tab.classList.add("active")
                  }
              }

              for (const selectorTab of selectorTabs) {
                  if (selectorTab.attributes["data-tab-selector-value"].nodeValue !== value) {
                      selectorTab.classList.remove("active-selector")
                  } else {
                      selectorTab.classList.add("active-selector")
                  }
              }
          }

          for (const t of selectorTabsAcross) {
              t.addEventListener("click", function() {
                  select("${tabValue}")
                  localStorage.setItem("${localStorageTabsKey}", ${JSON.stringify(tabValue)})
              })
          }
      })()
      `,
    },
  ],
});

Essentially, we are selecting the necessary elements, loading the initial state, and specifying the click event listeners. When a user selects a tab, it will add the active class to the tab element and the active-selector class to the selector tab, along with updating the localStorage.

Loading the plugin

Like the remark plugin, we need to update astro.config.mjs to use it by adding the rehypeTabs plugin.

import remarkTabs from './src/plugins/remark-tabs';
import rehypeTabs from './src/plugins/rehype-tabs';

export default defineConfig({
  markdown: {
    remarkPlugins: [
      remarkDirective,
      remarkTabs,
    ],
    rehypePlugins: [
      rehypeTabs
    ]
  }
})

Then, you can verify that scripts were added to the end of the hast of the parent container:

Styling the component

We are almost done with creating our component. We just need to make sure that the component is styled correctly. The most important part is ensuring that only the active tab is shown.

div.tabs {
  @apply mb-4;
}

div.tabs .selector {
  @apply flex flex-row;
}

div.tabs .selector div[data-tab-selector-key] {
  @apply inline-block px-4 py-2 border-1 border-neutral-300 border-b-0 hover:cursor-pointer;
}

div.tabs .selector div[data-tab-selector-key]:first-child {
  @apply rounded-tl-md;
}

div.tabs .selector div[data-tab-selector-key]:last-child {
  @apply rounded-tr-md;
}

div.tabs .selector div[data-tab-selector-key]:not(:first-child) {
  @apply border-l-0;
}

div.tabs .tabs-content {
  @apply w-full p-4 border-1 border-neutral-300 rounded-b-md;
}

div.tabs div.selector div[data-tab-selector-key]:not(.active-selector) {
  @apply bg-neutral-50 text-gray-500;
}

div.tabs div.tab:not(.active) {
  @apply hidden;
}

div.tabs div.tab.active {
  @apply block;
}

I use Tailwind to style the tabs, but you can use whatever you want. Make sure you import this style into whichever component renders your Markdown files.

Voila! You now have a fully functioning tab system with persistent storage!

Closing

Yes, I built this out of spite due to the lack of good MDX support in Neovim. Yes, it might not be the most intuitive thing. Yes, MDX could have done this with much less effort.

But this was a fun dive into Markdown and HTML ASTs with remark and rehype.

If you are interested, I have written other Markdown components for @git-mastery here.

Understanding Systems with Sequence Diagrams

Fri, 25 Apr 2025 00:00:00 GMT

Uncle Bob (Robert C. Martin) wrote in Clean Code that “the ratio of time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code…”

As software engineers, reading code is our bread and butter. Maintaining a legacy system? You have to read the legacy code. Want to build a website? You’re reading framework and library documentations. Contributing to a project? You are combing through existing code to figure out how everything works before contributing meaningfully. Writing a Request For Comments (RFC)? Guess what? You have to read the old code to propose changes to it.

It is an immutable part of software engineering, yet after talking to various engineers, I have come to realize that there is no well-defined system to understand and navigate complex codebases.

So today, I will try to introduce a system I have used throughout the years to understand large systems and start contributing to them within hours: sequence diagrams.

Practical sequence diagrams

You might remember sequence diagrams from a software engineering class you took where your takeaway was that UML diagrams like sequence diagrams are fraught with rigid rules and terminology. [1]

However, my approach to sequence diagrams focuses on its practical applications, and adapting it to work within a more “realistic” software engineering environment.

I believe that sequence diagrams should be used to understand how software components interact with each other. So, it is less important that you remember what a lifeline is or ensure that class names are underlined, but rather, focus on using sequence diagrams as a visual tool on top of note-taking.

In fact, my flow for tracing codebases using sequence diagrams is very simple:

Open Notion or VS Code
Create a new page or Markdown file
Create a codeblock for Mermaid
Find an entry point for the code you are interested in
Trace the code by jumping through definitions and following each line
Add the important details into the sequence diagram
In the rest of the document, add any additional details to supplement the diagram

Before I explain what is considered “important details” for the sequence diagram or how to construct the sequence diagrams, I will first cover the basics of Mermaid, the tool I use to generate sequence diagrams using code.

Mermaid crash course

I will only cover the very bare minimum you need to create sequence diagrams. These are all the notations I use 99% of the time.

If you are interested in learning more, feel free to refer to the official documentation.

To start with creating a sequence diagram in Mermaid, you need to declare the diagram type:

sequenceDiagram

Then, you can declare actors via actor <name>:

sequenceDiagram
  actor User

This generates a stick figure you can use to represent users:

I use the actor to represent users of the system. Very often, they are the people who trigger some codeflow that I am interested in.

You can also generate lifelines via participant <name>:

sequenceDiagram
  actor User
  participant UI

These participant objects are used to represent the various files/classes/components that the sequence diagram aims to capture.

Then, to illustrate messages between entities, you can use ->> for solid lines and -->> for dashed lines (often associated with return values). After each line, you have to specify the message through : <message>

sequenceDiagram
  actor User
  participant UI

  User ->> UI : click big red button 
  UI -->> User : "Don't click the button!"

The messages can either be function call signatures, or descriptions of the function calls and return types.

Loops can be expressed using the loop <name> <condition> ... end syntax:

sequenceDiagram
  actor User
  participant UI
  participant API

  User ->> UI : subscribe to data
  loop polling
    UI ->> API : fetch data
    API -->> UI : return data
    UI -->> User : display data
  end

You can also specify “alternative” code paths (or branches) using alt <condition> ... else <else> ... end:

sequenceDiagram
  actor User
  participant UI
  participant API

  User ->> UI : subscribe to data
  loop polling
    UI ->> API : fetch data
    alt has data
      API -->> UI : return data
      UI -->> User : display data
    else no data
      API -->> UI : no data
      UI -->> User : "no data found"
    end
  end

You can also enable sequence numbers by adding autonumber to the top of the diagram:

sequenceDiagram
  autonumber

  actor User
  participant UI
  participant API

  User ->> UI : subscribe to data
  loop polling
    UI ->> API : fetch data
    API -->> UI : return data
    UI -->> User : display data
  end

That’s all you really need to know. There are plenty of other notations like par and note, but I generally do not use them.

Now that you have a general understanding of Mermaid, I can now introduce how I approach using sequence diagrams through the concept of “fidelity”.

Sequence diagram fidelities

I borrow the term “fidelity” from prototyping, which refers to the level of detail, accuracy, and realism of a prototype compared to the final product. [2]

Similar to prototyping, there are the three levels of fidelity I work with:

Low-fidelity
Medium-fidelity
High-fidelity

The choice of fidelity is clear when you ask the question, “How much details do I need to capture in the diagram?”

Low-fidelity

Low-fidelity sequence diagrams focus on component-component or user-component interactions. A lot of low-level details are omitted, such as individual function calls.

Often, I use these as a starting point to understand the “big picture” of a system.

For instance, this was a low-fidelity sequence diagram I created to understand how Supabase authentication works.

Rather than spending time trying to map out the individual underlying function calls, such as how user signup submission is handled, I focus on representing the system through the actor User and participants Dashboard, Supabase, and API (representing the high-level components of the system). Every message describes the high-level interactions between actors and participants.

Low-fidelity sequence diagrams are very useful to give you a bird’s eye view of how a system works, and how user interactions look like. However, you may be interested in understanding how certain components work while maintaining a bird’s eye view of the entire system. This is where medium-fidelity sequence diagrams come in.

Medium-fidelity

Medium-fidelity sequence diagrams often involve both high-level component-component or user-component interaction, and some low-level details within these interactions.

For instance, this was a medium-fidelity sequence diagram I used to understand how the Google OAuth2.0 authorization flow works (which might not be 100% accurate).

This diagram contains both high-level component-component and user-component interactions (e.g. between Client and Google) and some lower-level information (e.g. between Server and Database) where details like how the information is retrieved and what is returned is captured as individual messages.

However, I prefer to avoid using medium-fidelity diagrams. As you can see, these diagrams can grow unwieldy very quickly as more high-level components are introduced, which becomes hard to understand and maintain.

Instead, what I often do is create a low-fidelity sequence diagram with message numbers (using autonumber), and attach additional notes (through text or as high-fidelity sequence diagrams) to each of the relevant message numbers.

But for cases where the number of components are still reasonable, medium-fidelity diagrams are still a good middle ground to provide a bird’s eye view of the system while having some details fleshed out.

High-fidelity

Rather than focusing on the “big picture” like low-fidelity or medium-fidelity sequence diagrams, high-fidelity sequence diagrams are designed to drill deep into the interaction between a very small subset of components.

For instance, this was a high-fidelity sequence diagram I created to explain a change I was proposing for a pull request:

Here, I focus solely on the interactions between the IssueService and GitHubService, with Caller acting as an “anchor” for what initiates this interaction. In high-fidelity sequence diagrams, focus on mapping out the individual function calls like createTeamReponse(Issue) or providing high-level descriptions of each step, explaining the “how” behind the interaction.

Some functions may include many individual function calls that are not relevant to understanding the current interaction. For these cases, I typically summarize them as a high-level message. High-fidelity diagrams try to capture the relevant low-level details of the interaction, so any irrelevant details can be omitted or summarized.

General guidelines

Now that I have established my system of creating sequence diagrams, these are some general guidelines I try to follow to keep myself sane:

Keep it to under 50 messages

If you realize that the diagram is growing extremely large, then it is probably time to break it out into separate diagrams.
Avoid adding notes within the diagram itself

Instead, add message numbers using autonumber and add notes outside of the diagram to reduce clutter.
Don’t get sucked into representing every little interaction

Depending on your goal, you should try to aim for clarity over comprehensiveness in your messages. Save the little interactions for other sequence diagrams or notes.
Treat sequence diagrams as living documentation

Upload these diagrams to your repository, revisit them, and update them as changes are made. Do what you can to keep these diagrams and notes up-to-date as it benefits both you and future maintainers.
Keep asking questions

As you comb through the codebase and add the interactions to your sequence diagrams, leave questions (or TODO comments) about components that you find interesting or are unsure about. If these impede your understanding of the codebase, you have found a blocker, else, these act as good things to return to down the road.

Closing

That’s it! It’s a relatively simple system that I started using a few years ago and found it to be quite effective in both personal projects or during my internships, allowing me to create pull requests within hours of being introduced to a codebase.

“Oh, but this feels like a lot of extra effort…” you might lament.

And to that, I answer, “You are absolutely right!”

This is a system that takes a bit of time to get used to, and it does add a layer of overhead to your approach to codebases. However, I think that benefit of having a living learning document/page/repository is crucial in becoming more productive as a developer, understanding systems, and realizing where your contributions fit into the “bigger picture”.

I also find that the act of creating documentation and writing about the systems I interact with has made me a better communicator of my ideas and understanding. These diagrams have also helped my teammates understand the system and some have even become staples within the team’s repositories!

So, give this system a go and let me know what you think! Enjoy!

[1] IBM’s introduction to sequence diagrams

[2] Low-Fidelity vs. High-Fidelity Prototyping: Key Differences Explained

Evolution of Resumes

Mon, 21 Apr 2025 00:00:00 GMT

I was reading Kenneth Lu’s page on his evolution of resumes and felt it would be interesting to look back at my past resumes and reflect on the lessons learnt at each junction, and explain my thoughts on crafting a good resume and building a strong portfolio. [1]

Before starting, I would like to make a disclaimer that this is by no means a definitive guide to creating a good resume, nor is it your golden ticket to landing internships. I believe a lot of my success in finding internships thus far has been a mixture of incredulous luck and technical interview preparation. Having a good resume is just one step in the process, so you should be focusing on improving all facets of the technical interview preparation process, not just your resume. [2]

By focusing on both resume building and developing a portfolio over the years (and with quite a bit of luck), I was able to secure internship offers from companies like Citadel, Palantir, Google, Paypal, and Stripe. I hope that you find my experiences useful and insightful in your own journey.

General advice

Your resume should follow these general principles:

Keep it to one page
Highlight your most relevant experiences first
Use action words (verbs) for each bullet point
Use a legible font like Arial
Avoid cramming content, whitespace is good

You can find more on the r/EngineeringResumes wiki, so I will not go into repeating everything.

Polytechnic year 3

I went to DBS Bank during my polytechnic internship placement.

I will admit that I was pretty “privileged” that I did not start out from absolute zero. During my first two years in polytechnic, I was building various side projects such as Taiga and working on several school projects like BookIt!.

These led me to land a short-term unofficial stint as an Android developer for MightyAim, a startup that I was introduced to by a friend, Li Haoquan, where I was able to build some experience in working on real-world engineering problems.

These experiences opened the door for me to give a workshop, teaching students the fundamentals of Android development using Kotlin, a language I was obsessed with back then.

So, going into my internship job hunt in my final year of polytechnic, I had the following going for me already:

A good GPA that would stand out amongst other candidates
Some relevant work experience through my time with MightyAim
Substantial projects with technical depth that I could talk about
An interesting extracurricular activity that was relevant to tech

“These are great Jiahao, but what can I do if I do not have such a starting position?” you might ask.

It is definitely not a lost cause, I would focus on the following:

Include any projects you might have worked on previously. Something is better than nothing, even if it is a school project that other students may have attempted as well.
If your GPA is good, include it, it is something to be proud of.
Include all past work experiences, even if it is not immediately relevant. A data entry job may not be software engineering related, but it can demonstrate that you possess some of the skills that employers might care about.
Think about what employers want to see and frame your experiences and projects that way. This might differ based on the role.
Pursue your interests and develop your portfolio that way, focusing on going deep rather than trying everything at a beginner level.
Meet new people and network to unlock more opportunities. I am admittedly not the best at this still, but it is always worth attending events and talking to people.
Make your relevant skills and experiences clear and apparent with dedicated sections and emphasis.

Post polytechnic

After graduating from polytechnic, and while awaiting for my enlistment into the Singapore Armed Forces, I decided to pursue an internship in research, a field I was unfamiliar with. This led me to do an internship as a Natural Language Processing (NLP) research intern at DSO National Laboratories.

I kept using the same LaTeX template, but this time, I had included my experience from DBS Bank, along with a different set of projects, all self-initiated. I also started including my academic achievements, like graduating with a Diploma with Merit and receiving the IMDA Gold Medal. I also moved my skills down to the end, highlighting my work experience first.

These are some comments on this resume:

Focus on including the impact of your work on your resume using concrete numbers. I don’t think I was the best at representing it then, but looking back, this was something I would have done differently.
Prefer side projects over school projects. Once you have established some set of personal projects, they are often more impressive than mandatory school projects.
Bubble all your experiences to the top, prioritizing experiences and projects that demonstrate your skills over simple lists of technologies you know.
Use action words (verbs) for each bullet point. One thing I don’t like about this resume is the fact that I placed my “role” at the start, but the bullet points should allow readers to infer the role.

University year 1

I have shared about my struggles in finding an internship in the first year of university. Despite having relevant experiences, I did not have the right technical interview skills to overcome the interviews. [2] However, I was very fortunate to have still received three offers: one from the Computing for Voluntary Welfare Organisations (CVWO), an internship that NUS students have access to, another from Nodeflair, a tech career platform, and the final one from Betafi, a startup building a unified user research platform. I eventually took up the internship at Betafi.

I did not receive these offers because I was a good technical interviewee. Instead, I was able to leverage my experience from building past projects and technical writing.

The CVWO application process involved building a project over the winter break, and I had spent an obscene amount of time on it, refining the product. [3] The online interview was a discussion on my engineering decisions. The process was not simple, but having used the technologies in other projects reduced the barrier to entry.
The Nodeflair interview process was the most similar to the traditional technical interview process. However, the questions revolved around the application of programming to solve common software engineering problems (like reversing words in a string). This process allowed me to tap into my general problem solving skills that I have developed, as opposed to knowing specific data structures and algorithms.
Ethan Sherbondy, the CTO of Betafi, reached out to me via Linkedin about interning at Betafi. However, this was not because I was a “stacked” engineer, but rather because I was very deeply involved in writing about Elixir, and I had written several articles on my personal blog (here!) and for a company called AppSignal. He noticed my portfolio and reached out because Betafi was building their platform in Elixir.

The key points I would hope you take away from this is:

Play to your strengths. Apply to companies that have processes that best align with your strengths and interest. This may not always be possible, but there are some that are like that.
Building your portfolio and doing things you are interested in matters. I do not think I would have been able to secure as many opportunities if I had not spent my time working on projects or writing technical articles about things I enjoyed.
Highlight the things that make you stand out. For me, it was my teaching and writing experience, so I included those as a dedicated section.
Pursue all possible opportunities. Rather than optimizing for glamorous internships, try applying for all opportunities, you never know what could happen.

University year 2

Learning from my struggles from the previous summer, I started preparing for technical interviews religiously [2] and eventually was very fortunate to receive offers from Citadel, Palantir (in London), Google, and Paypal. I eventually accepted the offers from Citadel for the summer, and Palantir for the fall.

I also added some tweaks to my resume:

Moving to Google Docs. [4]
Squeezing as much content by shrinking the margins. While this felt like a good idea then, looking back, I think it would have been good if I kept some whitespace to allow readers to breathe when reading my resume.
De-prioritizing old experiences. Rather than trying to fit every experience I’ve had, I decided to trim my experiences to only include my most recent three internships.
Removing my polytechnic education. In Singapore, polytechnic is similar to high school, so there was no reason to include it now that I have been in university for two years.

University year 3

I did not apply to as many companies as I did the year before because I had other commitments and was too busy having fun in London. Naturally, I received fewer Online Assessments (OAs) and interviews, but I still managed to get interviews with companies like Hudson River Trading (HRT), Jane Street, and Stripe. I was fortunate to have friends who provided referrals to both HRT and Jane Street. Ultimately, I received an offer from Stripe.

Regardless, I would still like to point out the following changes to my resume:

Moving to Typst. [5] I wanted to consistently generate sections without tweaking Google Docs.
Highlighting my work authorization. I was also applying to firms in the US and UK, so having my work authorization at the very top helped to cut through the uncertainty these firms might have about my ability to work abroad. As a Singapore Citizen, you have access to the H-1B1 US visa, and as a graduate of NUS or NTU, you have access to the HPI UK visa.
Including objective metrics of impact. I started quantifying my impact in numbers, focusing on demonstrating the improvements made through my contributions.
Using ChatGPT to paraphrase bullet points. I would first write a couple of sentences for each bullet point, and then use ChatGPT to summarize and paraphrase the bullet point, tweaking the result. A simple prompt could be, “Can you help to generate a resume bullet point of this contribution? Emphasize the impact of my contribution.”
Ask for referrals from people you know. It might seem embarrassing, but asking for a referral from people who you have worked with, and can vouch for you, is super helpful in getting a foot in the door.

Conclusion

I have been so fortunate to be where I am today. As much as I would love to claim that all you need is passion, having a support network of friends and seniors, being in the “right place, right time”, and having the privilege and time to work on things I enjoy, have all significantly contributed to my success thus far. If you would like to see a more exhaustive list of my working history, you can check out my Linkedin profile.

All the best!

[1] My reflection on my time in tech: Ten Years in Tech

[2] If you are starting the technical interview preparation process or are looking to polish your skills, I have prepared a technical interview study guide with my notes that got me into firms like Citadel, Palantir, and Google: Technical Interview Study Guide

[3] My CVWO web forum project: hermes

[4] I have moved over to using Typst to generate my resume, but my Google Docs resume template is still available: Jiahao Woo Blank Resume Template

[5] My Typst template can be found in this Github repository. Use the repository as a template and get started: woojiahao/resume-template

Reframing Failure

Sat, 15 Mar 2025 00:00:00 GMT

I saw this message on an online forum and it reminded me of the uncertainty I felt before starting every one of my internships.

After sharing some of my sentiments in the forum, it got me thinking about my past experiences and how I’ve started reframing failure in tech and I wanted to consolidate those thoughts here.

What is failure?

Before I dive into how I reframe failure, I think it’s good to set some precedence about what exactly constitutes a failure in tech — specifically during a tech internship.

To some, failure might look like this…

And perhaps, incidents like these are the reason why we are so afraid of failures in our own internships.

To me, another mode of failure I often worry about is not being able to complete a feature on time or even finishing the internship project on time.

Broadly speaking, I think of failure as not being able to live up to the expectations that others might have of us — be it not destroying production or not being able to deliver something.

It’s okay to be scared

The most important step in all of this is acknowledging that this fear of screwing up is valid, normal, and very human. Everyone is susceptible to worrying about the unknown.

It is in our nature to be worriers — after all, if we worry and imagine every possible scenario where we screw up, maybe we could would be ready for when it happens and combat it right?

Absolutely not. Although it may be in our nature to worry about the what-ifs, it is unproductive to be paralyzed by the fear of the unknowns and what-ifs.

This is why acknowledging this fear is step 0 — just being aware of it does not fix the problem, but it allows you to see that a problem exists. Only then can you start working towards addressing it.

If not now, then when?

Internships are the PERFECT environment for you to learn and grow and fail. You are supposed to be inexperienced, and you are supposed to stumble around. It is all a part of the process, so embrace it.

While there are many reasons why you are doing the internship, I want to highlight what I believe are the three key purposes of an internship:

Learning opportunity
Exploring a field
Meeting and learning from professionals

Whenever you feel lost and hopeless, remind yourself that you are in the best position to be feeling this way, and turn to the three purposes to figure out what you should be prioritizing. Use them as your North Star throughout the experience.

Self-imposed expectations are not always reality

It is easy to feel like you have to produce the best code, write the best documentation, generate the most value, and deliver the most impact. However, take a step back and ask yourself, “Are these actually what is expected of me? Or are these just what I believe I should be doing?”

As the intern, you are NOT expected to be perfect. Heck, even as a junior/senior engineer, no one expects perfection out of you. No sensible person would place such unrealistic expectations on others, which begs the question — why do you place such unrealistic expectations on yourself?

Be conscious about your self-imposed expectations and realize that the world does not demand perfection from you, you do. So be kind to yourself and allow yourself to make mistakes.

Perfection requires practice, practice entails failure

Let’s suppose that you do want to be perfect and become amazing at what you do. The only way you could even achieve perfection and mastery is through repeated practice which naturally invites failure.

Some of the very best in their craft achieved their mastery through hundreds and thousands of hours of practice and failures.

So, rather than shrinking in fear of failure, embrace it and treat them as roadbumps on your road to mastery. Grow to fill your potential, and you will learn to enjoy the process of failing.

Failure == learning

The best learning opportunities often arise from the greatest failures. I know that the times where I had the greatest delta in learning were born from hours — if not days — of head scratching and banging my head against a wall trying to get something working but failing repeatedly.

The process might be brutal and it might suck in the moment, but more often than not, these are the points where you get to learn and grow the most. In fact, I would argue that if you are not experiencing failure, you are simply not moving out of your comfort zone. So, use these moments as motivations to push through the suffering, knowing that you will grow so much at the end of it.

Adopting a “so what can I do” attitude

If, and when, you do encounter a roadblock, ask yourself this very simple phrase, “So what can I do?”. Rather than worrying about the consequences of a failure that may or may not materialize, focus your efforts on figuring out what’s the next steps you can do to improve the situation.

During one of my production incidents, I felt a flurry of emotions that went from:

Oh no
What did I do?
*Panics*
Okay, what can I do?

Going to my mentor, I wanted to fix the problem I had started and it became an opportunity for me to understand how our production systems worked, and how such issues were recalled and fixed. It proved to be a positive learning experience.

So as difficult as it may feel to face a problem you caused, there can be much to learn from such an experience.

Trust in the safety nets

While I have been talking a lot about how you can reframe failure in YOUR eyes, a big part of reframing and embracing failure is realizing that there exists some set of safety nets for you to fail safely.

Most companies with decent engineering cultures should have some form of guardrails for you to fail. Having things like proper sandboxed or staging environments, robust CI/CD, comprehensive tests, frequent backups, and detailed rollback plans should give you confidence in being able to fail as much as you want, knowing that these failures do not result in immediate and permanent damage.

Beyond the engineering capacity of the company, you should trust in your coworkers/mentors/managers/seniors to “have your back” and rally around you to fix your mistakes.

These safety nets (or support structures) are quintessential. Knowing that your failures will not cause systems to end up in flames reduces the stress you feel from every mistake and allows you to focus on trying and learning as much.

Conclusion

Failing is scary. Screwing up is horrifying. But doing nothing because of these fears should scare you even more. You need to embrace failure and reframe how you think about failure to be the best version of yourself.

All the best!

Ten Years in Tech

Wed, 08 Jan 2025 00:00:00 GMT

2025 marks my tenth year in tech. 15 year old me would not have imagined that 25 year old me would be having so much fun with a random hobby he picked up but here I am, pursuing tech as a career!

To mark ten years in tech, I thought it would be fun to document my journey in tech so far. I also wanted to point out the amount of “failures” I received throughout my ten year journey, to serve as a reminder that everyone starts somewhere and getting started it always the hardest step.

Secondary School: 2014 - 2016

Got a Samsung Galaxy Ace 2
Got into Android ROMs, installing ROMs like Cyanogenmod and MIUI
Started lurking in XDA forums
Decided to (try) learn Android development from a YouTube series by xda-developers (I couldn’t find the series anymore 😔)
Did not really understand Android development and gave up (Failure #1)
Printed out tutorials on Java and trying to learn Java (yes, I printed them out and bound it)
Did not enjoy learning Java too much and gave up again (Failure #2)
Wanted to find the “perfect language” to start programming
Visited the public library many times to borrow books on programming like C# and web development
Tried setting up the family computer (Toshiba Satellite) to dual boot Ubuntu and Windows
Accidentally wiped the Windows partition instead 🤡 (Failure #3)
Discovered a video about why programming is important

Ignited a deeper interest in me to learn more about it and treat it as a fun hobby and find a wider community to enjoy it with
Dad brought me to Bras Basah to browse second-hand bookstores
Got books about C++ and iOS development
Installed Visual Studio to learn C++
Did not have an Apple machine to learn iOS development (yes, the book became useless) (Failure #4)
Started picking up C++ and understanding the fundamentals of programming
But really struggled with understanding pointers (who doesn’t on their first attempt) (Failure #5)
Took the O-Levels and did really well (L1R4 of 4 and L1R5 of 6)
Faced with making a tricky decision of choosing Junior College or Polytechnic
Unsure if I should study chemical engineering or information technology

Took a leap of faith and decided to pursue my interest in tech by studying IT at Singapore Polytechnic

Polytechnic: 2017 - 2020

Started my diploma in Information Technology!
Started learning Java as my first language in school from Derek Banas

Built my very first proper project for school (a chat bot!)
Joined The Programmers Hangout
Met an amazing community of people who were equally passionate about tech
Became a moderator on the server and continued making friends from around the world
Started side projects to explore different technologies like Kotlin like a Discord moderation bot, Spotify API wrapper, and extensible Markdown to PDF conversion library
Introduced to MightyAim by a friend
Working at MightyAim for a while as an Android developer, building an SDK to allow existing applications to adopt a gesture-based workflow
Received my first paycheck (of SGD 300!)
Went back to Bras Basah to buy books on algorithms and operating systems with my newfound wealth 🤑
Attended PHP Conference Asia 2018 even though I didn’t know PHP (and still don’t)
Invited to teach a workshop for students over the semester break
Put together “Introduction to Android Development with Kotlin” and taught over 15 students live!
Discovered my love for teaching and writing through this experience
Started my blog
Got placed to DBS Bank for my polytechnic internship under the SOI Re-platform team
Worked across teams to get 90% UAT completion (across over 90 teams) and did a bunch of root cause analysis
Attached to XCLR8 for my Final Year Project
Migrated the backend from Java to Kotlin Spring Boot and the database from MongoDB to PostgreSQL
Became valedictorian of my cohort! Graduating with a 4.0 out of 4.0 GPA
Still felt like I wanted to find my tribe in person
Was rejected from the PSC Scholarship (Failure #6)
Got accepted into NUS Computer Science
Started a research internship with DSO National Laboratories
Integrated social media information for unsupervised stance detection and was exposed to NLP research!
Enlisted in the Singapore military

National Service: 2020 - 2022

Focused on technical writing instead
Got really into Elixir through 2020 Advent of Code
Started writing about Elixir on my blog
Invited to write for company blogs
Got articles featured on Elixir newsletters
Studied calculus in my free time to prepare for university
Applied to US universities like MIT and Stanford for Computer Science
Got rejected by every US university (Failure #7)
Built chris, a Pratt parser written in Go
chris was featured on the Golang Weekly newsletter

University: 2022 - Now

Enrolled in the National University of Singapore’s School of Computing, studying Computer Science
Joined Residential College 4
Met many close friends in RC4
Joined NUS Google Developer Student Club as a backend software engineer on Project Sight Word
Met more amazing friends from GDSC
Realized I was not cut out to be an academic weapon after struggling in my first semester (Failure #8)
Realized that getting a job in tech was very hard after flunking a bunch of technical interviews (Failure #9)
Discovered the existence of Leetcode
Joined NUS Hackers and finally discovered my tribe IRL
Articles were noticed by Ethan
Ethan offered for me to do an internship at Betafi using Elixir + Phoenix!
Joined Betafi as a software engineer intern and worked on improving the video processing pipeline and adding project templates
Started the Technical Interview Study Guide to compile my notes from interview preparations
Taught a workshop on Git and another on databases for Orbital
Became the top student for CS2103/T (Software Engineering)
Returned to CS2103/T as a teaching assistant (and got 4.8/5.0 for teaching feedback!)
Taught a web development with Elixir workshop for Hackerschool

Taught another workshop on Git for Hackerschool
Got internship offers from Google, Citadel, Palantir, and PayPal!
Organized Hack&Roll 2024, supporting the web app used by over 800 live participants and 40 organizers
Realized on-call isn’t fun and that the web app had to be rewritten to be more intuitive (Failure #10)
Taught yet another workshop on Git for Orbital (again)
Joined Citadel as a site reliability engineer intern over the summer on the Central SRE team
Met an amazing team and mentor, Nelson
Worked on preparing the Commodities Observability Platform for a disaster recovery exercise and implementing an automated incident post-mortem tool
Flew off to London immediately after to join Palantir as a software engineer intern on the Developer Console team
Met awesome people both within the Dev Console team (shoutout to Zeyad, Amish, Nadira, Xinyi, and Tim!) and outside of the team, creating amazing memories in London
Integrated Platform API support into Ontology SDK clients and made many memes
Working intensely towards getting the rewritten web app out (and I did!)
Returning to school!

Wow! That took a while to recap! It has been such a wonderful journey and I am so happy that 15 year old me decided to get into tech. I got to meet so many amazing people and work on so many interesting problems.

Although I have had many successes as well, I have faced many failures as well (10 in total!) and these failures have taught me so much. The past ten years has been so fun and I can’t wait to see what the next 10 years has in store for me! 🎉

Maximizing learning at internships

Tue, 17 Dec 2024 00:00:00 GMT

“Congratulations! XYZ company would love to extend an offer to you for the summer! Please find the offer letter attached!”

The dopamine rush from finally seeing your efforts pay off is unbeatable! After going through countless rounds of interviews with recruiters, engineers, and hiring managers, you finally have a foot into XYZ!

However, as you start the internship, you quickly realize that getting in was step 0. The real battle starts here and now you are wondering how you are going to maximize your three-months as an intern. I will attempt to share the insights that I have gained throughout my internship experiences and focus on answering four fundamental questions:

How do I ask questions?
How do I navigate the complex codebases?
How do I understand the overall product easily?
How do I contribute to the projects?

P.S. If you just want a TL;DR of the advice as bullet points, find them here.

Asking questions

Just like raising your hand in class to ask a question, going to someone in a new environment to ask seemingly silly questions can feel like a daunting task. But as with anything new, the first step is always the hardest. These are some tips to help you figure out the best way to phrase and ask questions to maximize your takeaway.

Compile questions

As you are reading through the codebase or using the product, keep a document/notebook/napkin of questions you have in mind and try connecting the dots as you go through the system(s). Cross out questions you figured out the answers for and highlight those that you need help with. This helps collect your thoughts and tracks important questions to ask, ensuring that you fully grasp the system.

Research beforehand

Perform preliminary research as you compile your questions. Here is a hypothetical: would you rather have to explain a concept to someone who has no idea what is going on? Or would you rather have a discussion about the concept with this person who is actively clarifying their understanding with you? I am sure everyone would much rather do the second so do to others what you would have them do to you.

Pay attention and ask follow-up questions

As someone is answering your questions, pay attention and think of any follow-up questions you might have. Try connecting the explanation to your understanding of the other systems and think about whether this new information supports or contradicts what you know already. Ask follow-up questions based on this and ensure that you leave the session with a better understanding than before.

Just ask

Do not wait for the perfect opportunity to ask a question, it does not exist. It is not 3 hours later, it is not a day later. It is as soon as you realize that you are going nowhere with your own research and efforts. Once you have hit the brick wall, you should ask the people around you for help. If you do not know when you have hit the brick wall, a rule of thumb I have is that if I have made no concrete progress in 30 minutes, it is time to ask someone for help. Keeping your questions to yourself will only harm your learning.

Navigating complex codebases

No matter where you go, there is a 90% chance you end up having to comb through a relatively complex codebase. As opposed to working on side projects or school work, you will rarely be creating new systems from scratch. So learning to navigate a complex codebase is a crucial skill to learn.

Use search tools

Search tools like SourceGraph are amazing ways of searching across repositories. If your organization does not use SourceGraph, GitHub’s built-in search functionality works well too. Even a simple grep command can be leveraged to navigate through the codebase quickly! Understanding these search tools can be key in getting the relevant code in front of you as soon as possible.

Search and trace smartly

Knowing WHAT to search for is as important as having good search tool. For instance, if you are looking at a front-end project and want to make changes to a page, the fastest way to figure out what file the page belongs to is by looking up words on the page. For back-end projects, you may want to use tools like “Jump to implementation/references” to figure out what are the parent classes and definitions. There are many ways to search smartly and navigate a complex codebase, so the best way is to figure out what a good starting point is and go from there.

Read other codebases

While it may seem like self-inflicted torture, getting good at navigating complex codebases is a learned skill and the best way to train that skill is to read other codebases. Whether it is other services within your organization or open-source projects, you can only get good at understanding code by reading code. One thing I did a few years ago was to analyse open-source Elixir projects, take detailed notes, and write blog posts about them.

Understand underlying libraries and tools

It is very easy to treat underlying libraries and tools as blackboxes. However, this mental model of systems is usually unsustainable in the long term. So it is usually better to treat the underlying libraries as though they are part of your projects and delve deep when and where you can. This way, when you are faced with a random error thrown from a dependency, you will not be stuck staring at the stack trace. The same goes for the tools you are using, whether it’s your IDE or logging tools like Grafana Loki or CI/CD tools. Knowing the tools you use well ensures that you do not get stuck the moment something in the background stops working.

Organize teach back sessions

You have amassed all of this new knowledge, that’s awesome! But are you sure that you have fully understood the systems? This is why teach back sessions are amazing, they give you the platform to share your knowledge with others. Yes, it is going to be scary presenting your potentially flawed ideas to the “experts” but remember: everyone is there to see you succeed, not to see you fail. Having these “experts” listening to your interpretation of the system can boost your confidence and if you are mistaken, you have access to immediate feedback and correction. It is a win-win! So put some time on your team’s calendar and present your findings to them!

Start with low-hanging issues

If your team labels their issues with good first issue (or something equivalent), focus on knocking those out first. Try to solve at least one a day (this is not a hard rule, just something that I try to achieve to ensure that I have a healthy pace of learning) and pick the ones that allow you to work across all the core platforms that your team manages. They might feel like menial tasks, but they are essential in wetting your feet and giving you confidence! If you are lost on where to start or if you are going in the right direction, ASK!

Contributing to codebases

Now that you have understood how to navigate a complex codebase, the tricky part is actually contributing code to it! Try out these tips to maximize your contributions.

Start small

Building on top of “Start with low-hanging issues”, you should start with small issues and work your way up. Starting small also applies to the types of changes you are making. Focus on making small changes to one affected area at a time and moving to the next. So start by breaking down the problem into smaller parts and focus on solving them individually until you have managed to fix the problem or add the feature!

Use existing PRs as reference

No, it is not cheating! You do not need to come up with the solutions all on your own and bang your head against the wall trying to work with a foreign system! A big benefit of working in a team and working on a developed codebase is having fallbacks and references in the form of other people’s code! So leverage this advantage and use existing PRs as reference when writing your code! Not only will it help you get over the hurdle of “This task feels too big”, it can also save you a ton of time re-inventing what others have thought of and done. Additionally, existing PRs set precedence on the best practices and structure of codebases and it is always best to follow (and sometimes question) these practices.

Git blame excessively

While Git blame can be used to, well, blame people for making changes that might have broken production, it can also be an invaluable tool when contributing to a codebase because you now can (nicely) approach the latest editor to figure out their rationale of things. There have been many times where the decisions for code felt confusing and I was itching to fix it, but through Git blame and speaking to the last editor, I was able to understand the rationale and (a) identify new issues, (b) learn something new, or (c) work with them to streamline the code.

Review your own code

When you create your PRs, do not just create a giant essay for the PR description. Invest time into using the “Files” panel (or equivalent) to review your changes. Use GitHub’s “Viewed” to methodically check off files you have reviewed yourself. Add comments about questions you had about the code, rationales behind the code you have written, follow-ups, descriptions, and more. Think about the code you have written and try to identify code smell that you may have introduced. Test your changes and try to break your feature. These help to criticalensure that you are producing the best possible code, giving clarity to actual reviewers, and ultimately, training you to critically analyze your own code effectively.

Get frequent feedback

It is easy to feel like your changes should be perfect on the first go to avoid having back and forth with reviewers. But that is a very flawed way of viewing code reviews. Code reviews are built to review your code (seems silly to explain it but it is important to remind yourself from time to time). It gives you the opportunity to get other people’s eyes on your changes to ensure that they are sensible. So, rather than spending time trying to create the perfect PR, create the PR, review your own code, and open the floodgates for comments. Iterate on the feedback quickly and ask questions about the comments. This allows you to work closer with your team and helps accelerate your pace of work!

Understanding the overall product

Being a software engineer is not all about producing the most amount of code or reviewing the most number of PRs. It is also about understanding the product that your company is building and how your team and your work fits into the bigger picture. Knowing all of this helps you to think about the “What’s next?”.

Treat the product like an end user

The easiest way to understand a product is to be a consumer. Approach it with skepticism and play around with the features that the product has. Come up with use cases and use the product to solve them. Use these opportunities to pinpoint flaws in the product and identify core pain points. This process is commonly known as “Dogfooding” and it is crucial in giving you insight into how your end users approach your software. It is not easy pretending to be the end user if you know all of the quirks of your own system, but perhaps those quirks are signals into making use case paths clearer or improving some feature.

Look for corner cases

While dogfooding or running your changes, always think about “What if I tried this?” and act on it. Look for corner cases in your feature and see if it has been built to handle these cases. Similar to dogfooding, this explicit chasing of corner cases will help in highlighting critical paths that may not have been accounted for and this gives you critical “What’s next?” signals.

Talk to teams who own features or services

The world is much bigger than your team and your impact within the team. Speak to teams of other features and services to understand how their features work. Ask about the decisions that have gone into shaping the software and think about how their work fits into the overall product that the company builds. These conversations are great ways to meet more people and understand more systems. They may also uncover potential collaboration between teams, where you discover that your work can be intersected with theirs to create even better user experiences.

Conclusion

Securing internships are hard so do not let all of that effort go to waste by not learning as much as you can. You should be a sponge and use every opportunity to learn and grow as an engineer.

All the best with your internships!

Tips at a glance

These are the tips at a glance:

Asking questions
1. Compile questions
2. Research beforehand
3. Pay attention and ask follow-up questions
4. Just ask
Navigating complex codebases
1. Use search tools like SourceGraph
2. Search and trace smartly
3. Read other codebases
4. Understand underlying libraries and tools
5. Organize teach back sessions
6. Start with low-hanging issues
Contributing to codebases
1. Start small
2. Use existing PRs as reference
3. Git blame excessively
4. Review your own code
5. Get frequent feedback
Understanding the overall product
1. Treat the product like an end user
2. Look for corner cases
3. Talk to teams who own features or services

Appreciating 'Largest Rectangle in Histogram'

Sat, 13 Apr 2024 00:00:00 GMT

Problem statement

The ‘Largest Rectangle in Histogram’ problem is as follows:

Given an array of integers heights representing the histogram’s bar height where the width of each bar is 1, return the area of the largest rectangle in the histogram.

I love this problem as it provides an amazing introduction to problem solving, more specifically, formalizing proofs of why algorithms work. Let’s start with an example:

Let the array of heights be [1, 4, 7, 5, 7]:

The answer is 16:

If you have solved this problem before, or just looked at the answer, you may have found it hard to convince yourself why the algorithm works. Fret not, I am here to try formalizing the proof of the algorithm so that you can understand how exactly it works!

The general solution is to keep a monotonically increasing stack of indices

Invariants

We can think of the monotonically increasing property as an invariant that the stack must maintain.

As a refresher, an invariant is a property that is always true across iterations (of a loop) or for all elements in a data structure.

A monotonically increasing stack of indices maintains the following invariants:

Notations

For each index where is the array of heights and . Let be the monotonically increasing stack.

Operations

Many monotonic stack problems involve thinking about two key operations:

When to push?
When to pop?

Push

When or , then we push to the stack as it maintains the invariant.

Pop

When and , then we pop from the stack, receiving index , and try computing the largest rectangle using as the shortest rectangle.

To calculate the , let . The height of , .

To derive the width of , , we need to find the leftmost and rightmost rectangles that are still . We can label the indices of these rectangles as and .

Otherwise, will no longer be the shortest rectangle. After computing , we can push onto the stack as well.

Solving

Finding the rightmost index

Notice that if and is monotonically increasing, then must be the rightmost tallest rectangle. Formally, the claim can be written as

The above claim implies that can go from to without encountering a shorter rectangle in between.

Suppose that , then it must have caused to be popped off of prior to . However, this causes a contradiction as . Therefore, the claim holds and as a result, .

Finding the leftmost index

Case 1:

If , then by monotonic stack. Thus, the as that implies that the leftmost rectangle that is is itself.

Case 2:

If , then what we want to show is that

To prove this, notice that given that . Also notice that , otherwise, would not be in . This proves the first part of the claim where as can be interchanged with .

Given that as well (otherwise ), then , otherwise, which is a contradiction. That means would have caused the final to be popped from the stack.

Therefore, the claim is proven and we know that as the leftmost rectangle that is is given that by monotonic stack.

Case 3:

If . When this happens there is no leftmost rectangle, so we can set .

Calculating

Therefore, we have found both the rightmost index where , and the leftmost index where , or . The width of is therefore, .

If , then .

Implementation

When we implement the algorithm, we see that we naturally derive the ideal stack solution:

def largestRectangleArea(self, heights: List[int]) -> int:
    stack = []
    heights.append(0)
    n = len(heights)
    ans = 0
    for i in range(n):
        while stack and heights[i] < heights[stack[-1]]:
            j = stack.pop()
            h = heights[j]
            w = i if not stack else (i - stack[-1] - 1)
            ans = max(ans, w * h)
        stack.append(i)
    return ans

Conclusion

There you have it! The largest rectangle in histogram algorithm has now been formally proven and the magic numbers have been answered! Of course, when working on these problems during an interview, it is not expected of you to derive these proofs from scratch, but having such an intuition about monotonic stacks help a lot in reasoning with the magic values that are used!

Preparing for Technical Interviews

Sat, 30 Mar 2024 00:00:00 GMT

My friend, Devansh, recently wrote an article about “Debunking Myths Regarding Tech Internship Applications” , and I agree with many of his points. Proofreading his article gave me the opportunity to reflect on my own personal journey through internship hunting and technical interviews in summer 2024 and I would love to share more about the systems and techniques I had used to prepare for my technical interviews.

I understand that the tech industry is very volatile at the moment and a lot of my “success” this internship cycle boils down to timing and luck along with a lot of preparations. I am incredibly thankful for the offers I have received and I will never know what combination of factors led to my ultimate success.

If you are reading this and you have not had as much success so far, please do not stop trying. Keep pushing and you will definitely get an amazing offer :)

Who am I?

While you are reading this, you may be thinking to yourself, “Who is this person telling me about technical interviews?” No worries, an introduction is in order. Of course, if you are just interested in the systems and techniques, you can jump to them here.

My background

I am Jiahao, a Computer Science undergraduate from the National University of Singapore. Before entering university, I was studying Information Technology in Singapore Polytechnic.

I was also a complete “noob” at data structures and algorithms and technical interviews (my course in polytechnic never taught us these subjects). As such, while I had built several projects in my own spare time and worked with various teams, I was never well-versed in solving these data structure/algorithms problems and would struggle with even simple LeetCode easy problems (no exaggeration, I would spend an eternity trying to solve Two Sum).

First failures…

My first true exposure to technical interviews was when I had started applying for internships when I first entered university. I remember my very first online assessment (OA) was for the Grab engineering internship. Looking back, the questions really were super straightforward, yet I spent the entire one and a half hours to solve them, and I had still answered them ALL wrongly 😱.

The rest of the application cycle played out similarly, having failed many of the OAs and even when I had passed to the subsequent rounds, I would choke and fail at the live codings.

Thankfully, I still managed to secure a software engineering internship offer at Betafi where I worked with Elixir + Phoenix to improve the platform (I am very grateful to my supervisor, Ethan, for reaching out about this opportunity).

Formal exposure to data structures and algorithms

The following semester, I took the mandatory data structures and algorithms course that all CS undergraduates had to go through. It was my first real brush with proper data structures and algorithms and it instilled in me the very fundamentals of algorithmic thinking (thank you Prof Seth!). While I did not score amazingly (I got an A-), it gave me the basic tools to start exploring algorithms on my own during the summer.

Turning point

During the summer, I devised a study plan and techniques to help with my preparation for the next internship cycle, committing myself to becoming good at technical interviews (on top of juggling my internship, summer projects, and commitments with NUS Hackers).

As a result of the consistent effort, I was able to subsequently secure 4 offers (out of over 250+ applications), receiving offers from companies like Citadel, Palantir, Google, and PayPal (all of which I am extremely thankful for). While the following diagram is not 100% accurate, it hopefully gives you a rough idea on my overall success rate:

Why am I writing this?

While I am not claiming to be an absolute authority of technical interviews, I believe that my experiences, systems, and techniques had proven to be successful and as such, I would love to share more about them in this article so that you too can benefit and experience success in your own internship hunt! This article focuses on the “meta-discussion” of studying for technical interviews and the state of technical interviews in general.

If you are interested in my condensed notes for the various topics involved in technical interviews (data structures, algorithms, system design, behavioral, and resume writing), I have open-sourced my notes as a “Technical Interview Study Guide” that you can find here: https://interviews.woojiahao.com.

Technical interviewing as a skill

I broadly classify every component of the SWE/SRE interviewing process as a technical interview, so it covers the OA, initial phone screen, technical onsites, behaviorals, etc.

To me, technical interviewing is a skill. Much like how you would invest time and energy into getting good at games or your studies, I believe technical interviewing deserves similar treatment. Similarly, much like gaming or studying, there are fixed systems and techniques that you can use to streamline this process of improving.

I will talk about four key aspects of technical interview preparation that I briefly mentioned in my Technical Interview Study Guide:

Creating a good study plan
Compiling useful notes
Starting early
Interviewing with a passion

Creating a good study plan

TL;DR

Understand your weaknesses, create/use a study plan, and follow a reasonable timeline

Prior to creating a study plan, you should first identify your strengths and weaknesses. Before even thinking about the questions I wanted to finish, I first sat down for about 30 minutes and thought hard about the topics that I found particularly challenging from my data structure and algorithm course and noted them down. My list looked a little something like…

Arrays
Linked lists
Dynamic programming
Greedy algorithms
Etc…

As you can see, there is no shame in admitting that you may not be good at what some may even consider as “basic” topics (after all, you’re doing this preparation for yourself).

Once you have established what you know and what you don’t know, you can start thinking about a plan of attack (study plan). A study plan comprises of two key ingredients: a fixed timeline and an exhaustive list of subjects to cover.

Luckily, Yangshun had published a study plan on his Tech Interview Handbook, where he provides a detailed list of questions per major topic of technical interviews and a sample timeline to follow. So the first thing I did when I started was to first compile his list of questions into a Google Sheets document with a slightly adjusted timeline. I used these questions as a guideline on what to complete. I have since moved this study plan to my Technical Interview Study Guide, but this was what planning my study plan looked like in Google Sheets:

The final secret sauce is to create and follow a reasonable study plan. For me, I used the number of questions and familiarity with each topic as a gauge for how long I should spend on each topic. For instance, I felt more comfortable with a topic like recursion so I allocated just a day for eight questions, but for dynamic programming, I dedicated over five days for it (including the time to internalize and understand each solution).

You should also include the time it would take to revisit and revise topics. I usually try to revise topics about a week after having finished their questions to ensure that I’m not forgetting the key concepts.

Compiling useful notes

Once you have a study plan, you can start attempting the questions. Of course, having a systematic way of tackling problems is quintessential for learning.

TL;DR

Focus on systematically capturing the intuition of problems and jotting down recurring/unique patterns in problems

I use Notion to store my notes for every LeetCode question I have attempted since starting serious preparation. LeetCode has a built-in notetaking functionality and I know others who use tools like Google Sheets to make such notes. The medium is secondary to the actual process of notetaking and internalizing the intuition of each problem.

I recommend capturing the following “metadata” about each question:

Concepts used
Difficulty level (I follow LeetCode’s grading system)
Should you redo it? (I use this to gauge if I should redo a set of questions)
Status of completion (for instance, did you give up? Did you solve it the first try?)
Where the question came from

These metadata speed up the process of revision and let you revisit questions/topics that you found difficult in the past.

After noting these metadata, I use a template for jotting down my notes per question. This template includes the following:

Intuition (base intuition for solving the problem written in your own words)
Optimizations/notes (any notable optimizations/notes to make of this problem; if I gave up, I would often include the intuition for my failed attempt here)
Alternatives (from the LeetCode “Solutions” tab where I try internalizing multiple ways of solving a problem)
Remarks (any remarks about the problem)

For instance, this is what my note for Range Sum in BST:

When approaching a question, try to allocate a fixed time to solve it by before looking at the solution. This varies per person. This allows you to stay on track with your timeline. However, do not just space out for the alloted time because you know you will see the answers eventually. Try really hard to come up with the solution and implementing it within the time. After seeing a solution, start by running through a few examples with the new algorithm, building a strong intuition for it. Then, start jotting down your notes for the question. Take some time to also read any alternative solutions as it allows you to learn more about different ways of approaching problems.

In addition to notes about questions, I also highly recommend notes about the topics. Much like how my Technical Interview Study Guide includes information about the common patterns in each topic, I highly recommend making your own notes for these topics (the content of the study guide comes from my own personal notes!). While you can refer to online sources for these information, having a personalized set of notes is helpful when it comes to linking concepts and revising! This was what my personal notes looked like:

Starting early

All these preparations would be in vain if you only start in the middle of the internship cycle or if you start applying late.

TL;DR

Start practicing as early as possible so that you are ready and able to apply to more roles

My personal timeline for summer 2024 was as such (taken from my Technical Interview Study Guide):

Start of May 2023: sourcing for problems and figuring out my weaknesses when it comes to interviewing
May 2023 to mid June 2023: working on my fundamentals by completing questions by topic
Mid Jun 2023 to Jul 2023: working on other problems to expand my exposure and speed
Jul 2023 to Oct 2023: start applying for roles in Singapore, US, and UK
Aug 2023 to Mar 2024: started interviewing and received my offers

As you can see, as “kiasu” (Singaporean slang for “fear of losing”) as it is to start the moment summer starts, it is necessary if you’re attempting to cover as much breadth and achieve as much depth as possible before the internship applications open. Many large firms in the US opened while it was still the summer in Singapore and many large firms in Singapore opened soon after the summer ended.

Interviewing with a passion

It does not matter how much you prepare for the interview if you cannot convince your interviewers to want you on their team.

TL;DR

Research every company in depth and understand their business, projects, and culture and find an alignment with them

Every interview with an engineer/manager/HR with the company is a two-way conversation. It is both a way for them to get to know you as a candidate, but also for you to get to know about the company. However, in order for this conversation to be effective and meaningful for both parties, you MUST do your due diligence in understanding more about the company.

A great example is learning about a company’s open-source efforts and understanding these projects and talking to the engineers about them! I find that finding these “talking points” about the company helped me to not only appear interested in the company (because I was), but also helped me to ask meaningful questions about the company as my curiosity would drive me to want to learn more.

However, this does not just apply to knowing more about the company. Being passionate about your past experiences like past projects and internships also helps to show that you are someone who is invested. Be proactive in learning about the industry you were in, the problems you were solving, the technical and non-technical challenges you faced, and how you contributed to the team. These are all key things that help inform your perspective of these experiences and gives the interviewer the sense that you CARE about the project beyond just being a “code monkey”.

Conclusion

I have rambled for long enough.

If you were to take away just one thing from this, I hope it is this: technical interviews is a skill that takes time and dedication to improve on. You cannot expect to be amazing at it without any practice, just as you would not expect to be Magnus Midtbø when starting out climbing.

I hope that this article gives you both hope and guidance on how to navigate your technical interview preparation and I hope that my Technical Interview Study Guide can become your study buddy throughout your journey.

I am wishing you all the best and I am rooting for you! 🤩

P.S. If you have used my guide or just want to talk to someone about technical interview preparation, feel free to email me at woojiahao1234@gmail.com!

The Story of Elixir

Mon, 15 Jan 2024 00:00:00 GMT

This post was published on https://osshistory.org/p/elixir

A Comprehensive Guide to Using JSON in Go

Sat, 23 Dec 2023 00:00:00 GMT

This post was published on https://betterstack.com/community/guides/scaling-go/json-in-go/

Redis Caching in Go: A Beginner's Guide

Sat, 23 Dec 2023 00:00:00 GMT

This post was published on https://betterstack.com/community/guides/scaling-go/redis-caching-golang/

How to Work with SQL Databases in Go

Tue, 19 Dec 2023 00:00:00 GMT

This post was published on https://betterstack.com/community/guides/scaling-go/sql-databases-in-go/

How to Get Started with Logging in Django

Wed, 06 Dec 2023 00:00:00 GMT

This post was published on https://betterstack.com/community/guides/logging/how-to-start-logging-with-django/

Installing UxPlay on Arch Linux

Tue, 01 Feb 2022 00:00:00 GMT

I got my iPad Air 4 a few months ago and have been wanting to test the AirPlay functionality on it. However, as I am on Manjaro Linux - a Linux distribution based on Arch Linux, I was unable to get the native support that MacOS has.

I stumbled upon UxPlay as a potential solution for this problem.

UxPlay is an AirPlay Unix mirroring server that acts like an AppleTV for screen-mirroring on the machine that is running the server. It only works on UNIX systems.

The README is quite verbose and only contains the package names for Debian, Red Hat, Fedora, CentOS, OpenSUSE, FreeBSD distros.

So, this guide aims to help install UxPlay on Arch/Manjaro.

Installation

Install the necessary dependencies.

yay -S cmake pkgconf
yay -S openssl libplist
yay -S avahi gstreamer gst-plugins-base gst-libav gst-plugins-bad
yay -S gstreamer-vaapi
yay -S libx11
# If you are using Manjaro, you can should also install the manjaro-gstreamer
yay -S manjaro-gstreamer

Clone the UxPlay repository.

git clone https://github.com/antimof/UxPlay.git
cd UxPlay

Build and install the server.

cmake .
make
sudo make install

UxPlay will be installed to /usr/local/bin/uxplay.

Run the server.

sudo /usr/local/bin/uxplay

That’s all there is to this!

Optimizing for GoodNotes 5

One of the main uses I have for AirPlay is to annotate and write notes in GoodNotes 5 without having the worry about the size constraint. Typically, I would use a 50:50 layout, where one side hosts my annotated notes (like a textbook) and the other is a notebook for my notes.

However, with AirPlay, we can configure GoodNotes 5 to mirror only the notes side (displaying it on screen). The other side will remain the same. This way, the layout can be 25:75 without sacrificing the readability of the notes as the notes are displayed on a bigger screen while still being controlled via the 25% on the iPad.

The Javascript Ecosystem Can Be Frustrating to Work With

Tue, 21 Dec 2021 00:00:00 GMT

I have been working with Electron for the past two months or so — building Apollo (open-source RSS aggregator). And working with Electron meant I had to work with the Javascript ecosystem.

And boy is the Javascript ecosystem a confusing beast. From trying to bundle Electron via Webpack, to installing React, to getting hot reloading to work, everything has been a hassle.

To give even more context, lately, I had finalized the majority of the core functionality of Apollo — which included things like adding a new feed, and bookmarking feeds. I wanted to polish the application and so I started to modifying the styles of the React components. However, as I hadn’t setup Electron to allow for hot reloading of the renderer, it was taking minutes to build and render the changes. This was not very economical as I could be changing a single text color and have to wait minutes to view the changes.

So, I did what any sensible developer would do and look for ways to hot reload an Electron app. I stumbled upon electron-reloader and tried it out. No dice. Apparently, it did not work really well with Webpack + React.

So, I tried the next best thing, using a pre-packed Webpack plugin for electron-reloader. Again, no dice. The changes simply were not rendering.

Exasperated, I decided to re-write my Webpack to use electron-webpack as it had promised hot reloading out-of-the-box. But this time, I could not get my existing Webpack configurations to work. electron-webpack was simply too abstract for my liking.

The same happened with electron-forge.

Frustrated with the lack of flexibility and documentation for both electron-webpack and electron-forge, I decided to roll my own “minimal” Electron setup.

I decided to hardcode the tech stack to be Typescript, React, Tailwind.CSS, and TypeORM. This was the stack that was already working for me with Apollo and I did not want to rewrite Apollo.

So, I started reading the documentation and looking at various posts about setting up these tools. I cannot express just how many times I would follow a guide to the tee and have some random error message appear because the tutorial was like three months old.

THREE MONTHS

Only in Javascript will you get breaking changes by following a three-month old article…

I was pulling my hair at the multitude of errors with just setting up Webpack to work with every single technology. The implementation of Apollo was never a trouble, but the setup was the one giving me the most amount of problems…

Finally, after nearly eight hours of repeated trial and error and constant Google-fu, I finally got the minimal setup working.

I think this experience really speaks to the almost tragic state of Javascript. Everything is moving so quickly that three-month old articles can go obsolete. Tools are so abstracted that it’s almost impossible to make flexible decisions without needing to pull everything apart. There are so many tools that perform the same operation but differ ever so slightly that using one over the other would cause red errors to fill your screen. And yet, we are recommending web development as a “beginner” domain. Sure, having opinionated tooling can make it easy to start out with — but staying within the confines and diction of each opinion can become stifling and frustrating.

Sure this whole process has been eye-opening and has taught me a lot about Webpack and Babel. But maybe it’s just me, but I think that development should be less about confusing setup and debating standards and more about actually developing a product that matters. I just feel that time can be better invested elsewhere if the ecosystem and tooling is a lot more cohesive and complete.

I know my new “minimal” Electron setup is contributing to the problem, but for me, working with the tools at the most fundamental level — away from the abstracted nonsense — is the easiest way for me to get something working.

Of course, Electron development is not exactly the traditional starting point for web development and my past experiences with these libraries have always been less problematic. I think I am just frustrated at the fact that just by adding a new framework or feature to the mix, all of these tools suddenly stop working properly and the debugging process is doubly difficult because of the lack of proper documentation.

So what can we do about this? Well, I believe that this problem stems from a lack of proper documentation. Rather than adopting the traditional mindset of “This works with a single click”, we should be striving to document how a layer of abstraction works behind the scenes. What dependencies does it rely on? How do we properly add new dependencies that don’t strictly play into the tight integration? How do we use this library, add new features, and maintain it as our project grows? By taking this first step to making libraries easily understandable and modifiable, we minimize the need to double down on previous standards and can instead, just stick to a single modular library that just works.

In case you are interested, my “minimal” Electron setup can be found here.

Under the Hood of Metaprogramming: Pitfalls of Metaprogramming in Elixir

Tue, 16 Nov 2021 00:00:00 GMT

This post was published on https://blog.appsignal.com/2021/11/16/pitfalls-of-metaprogramming-in-elixir.html

Under the Hood of Metaprogramming: How to Use Macros

Tue, 26 Oct 2021 00:00:00 GMT

This post was published on https://blog.appsignal.com/2021/10/26/how-to-use-macros-in-elixir.html

Under the Hood of Metaprogramming: Under the Hood of Macros in Elixir

Tue, 05 Oct 2021 00:00:00 GMT

This post was published on https://blog.appsignal.com/2021/10/05/under-the-hood-of-macros-in-elixir.html

Under the Hood of Metaprogramming: An Introduction to Metaprogramming in Elixir

Tue, 07 Sep 2021 00:00:00 GMT

This post was published on https://blog.appsignal.com/2021/09/07/an-introduction-to-metaprogramming-in-elixir.html

Open-source Deep Dive: Broadway (Part 2) - Inner workings of Broadway

Tue, 13 Apr 2021 00:00:00 GMT

This open-source deep dive has been split into two parts! The first part covers the prerequisite knowledge that would be good to know when trying to understand the inner workings of Broadway. The second part is an in-depth analysis of the implementation of various features of Broadway.

This is the second part of the deep dive and the following topics will be covered:

Rate limiting
Batching messages
Telemetry
Creating a built-in testing support for pipelines
Achieving graceful shutdowns
Other interesting bits of code

If you want a refresher on the concepts behind Broadway (like message queues and concurrency in Elixir) or to better understand Broadway’s pipeline architecture from a bird’s eye view, you can find the first part here!

What’s the scoop?

Now that we have explored the overall architecture of a Broadway pipeline, we can look at how certain features in Broadway are implemented.

Rate limiting

Rate limiting refers to the act of limiting the amount of data that can be requested or processed in a given period of time

Rate limiting is applied across producers within a single pipeline to control the number of events emitted within a given period of time.

This is especially useful when the hardware of the machine running the pipeline is not able to keep up with processing large numbers of events demanded at a time — possibly due to a poorly configured pipeline.

Some producers do not leverage the rate limiting feature of Broadway. For instance, the RabbitMQ producer creates an active listener, which means that event emission is not inhibited by the rate limiter. Instead, events are emitted the moment messages are published to the message queue ( unless otherwise configured) .

But for the producers that do leverage the rate limiting — such as the Amazon SQS producer — rate limiting is applied in two instances:

When consumers make demands to the producer-consumer or producer

If the producer can still emit events, any demand made by the consumer will be handled by the producer. We take into account the rate limit threshold. If there are too many events to emit, the excess messages are stored in a message buffer that will have to be cleared later on.

Each message that can be emitted will be transformed into the standard event structure that Broadway uses.

If the producer can no longer emit messages, any demand made is stored in a demand buffer that is cleared later on.
When the rate limit is being reset after the given interval

After the given interval, the rate limit threshold can be reset. However, we may have accumulated demands and messages in their respective buffers. We may find that the threshold has not been met before we reset it. Thus, we can use this remaining threshold to clear any lingering demands and messages stored in their respective buffers.

Once we have cleared as many messages as our remaining threshold allows, we will reset the threshold and schedule for another reset. These resets are scheduled at fixed intervals.

The rate limiting threshold is maintained as an atomic (discussed later on). This atomic array is generated by the RateLimiter process. This module handles all behavior surrounding working with the rate limit threshold. ProducerStage handles the actual logic of managing the demands of consumers.

When the producer cannot emit any more events, i.e. the threshold has been reached, an internal state is set to :closed to avoid future demands from being handled.

Batching

Batching groups events based on given properties and sends them to designated “sub-pipelines” or batch processors to be handled. For instance, we might design a pipeline that stores events with even numbers in an S3 bucket and ones with odd numbers on Google Drive.

The Batcher process is assigned unique names for identification and events that are emitted from the producer must be tagged to a batcher. Failure to do so will result in a runtime error. This only applies if batching is enabled.

In order for the producer to send the appropriate events to the respective batcher, a PartitionDispatcher is used. Essentially, it defines the behavior of how events are emitted to consumers. A PartitionDispatcher dispatches events to certain consumers based on a given criteria (defined as a hash function). In this case, the criterion is the name of the batcher from the given event. This means that when we assign a batcher to the event, it will be dispatched to only that batcher. More information about dispatchers in GenStage can be found in the official documentation.

Even within the batcher, further grouping can be made based on a batch key assigned to the event. This may be used to ensure that certain events are processed together. Internally, the batcher will accumulate events before emitting them. However, as it cannot sit around accumulating events forever, a batch is emitted at regular intervals regardless of how many events are stored in it.

The BatchProcessor process handles a single batch at a time. It is similar to a regular processor, except it works on a batch of events. The handle_batch callback is used here.

Telemetry

Telemetry is used in Broadway to benchmark certain operations that occur such as the duration that a handle_message callback takes.

Broadway relies on the telemetry library. Within the code, events are emitted when these operations occur and key measurements such as duration are tracked. Handlers/listeners of these events can be setup to respond to these events.

Telemetry is not an Elixir-only feature. It is commonly used to perform application monitoring. OpenTelemetry is a really interesting framework that offers powerful application monitoring through telemetry.

Built-in testing

To test the pipeline, we should focus on ensuring that the data processing aspect of the pipeline works as intended. However, as we rely on external services for input, it would be hard to coordinate a test suite to work with a live data source as we may not be able to replicate the data source or publish data to the data source at will due to access limitations. Thus, Broadway has designed a testing utility that allows us to test the pipeline’s data processing capacity without relying on the data source.

Broadway provides a placeholder producer module. This producer does not rely on any data sources. Instead, messages are emitted directly into the pipeline.

The producer module should be tested separately if there is core behavior that cannot be tested along with the pipeline.

This form of unit testing ensures that we reduce potential points of failure in our test suite if any of the aforementioned problems with using the original data source should surface.

Graceful shutdowns

Broadway boasts about having graceful shutdowns. This is a rather interesting concept to explore as it relies heavily on the concurrency system of Elixir.

Essentially, the pipeline can only exist in two states — when all components are online and when all components are shutting down. There is no point in time where a single component will shutdown on its own without being restarted. This is because of the way that the supervisor of each component declares restart strategies ensuring that should a child process encounters any errors, it will be restarted without a hitch. This way, the only time where our components can shut down is when we shut down our main process or pipeline supervisor process. When either process is terminated, we want to properly handle all remaining events in the pipeline before shutting off every component.

This is achieved through a mix of concurrency features. But before we can explain how it works, a simple introduction of exit signals and process termination is due.

Exit signals and process termination

Processes can be linked to one another. When either process receives an exit signal — which can occur when the process is terminated forcibly or when it receives an exit signal propagated from its parent — it will propagate the exit signal to the linked process and that process will terminate as well.

However, these exit signals can be trapped instead. When this occurs, rather than terminating the process that receives the propagated exit signal, the exit signal is sent as a message, allowing the receiving process to handle the exit as though it was just another message.

def handle_info({:EXIT, from, reason}, state) do
	# ...
end

When a process is terminated, an optional terminate/2 callback can be declared to perform any cleanup before the process is actually terminated. This is useful if we have any lingering operations that should be completed before we terminate the process.

Supervisors can start a list of child processes and is responsible for managing the restart strategy of each child. The interaction between a supervisor and terminate is rather interesting. When a child is terminated, it is restarted accordingly. When a supervisor terminates, all of its children will also be terminated. If a child process traps exits, the terminate callback is called. If not, it will simply terminate immediately without calling the callback. More information about how supervisor interact with shutdowns can be found in the official documentation.

Back to our regularly scheduled deep dive…

With a basic understanding of exit trapping and process termination, we can actually understand how graceful shutdowns in Broadway works.

When the main process or the pipeline supervisor process is terminated, the main process — which traps exit signals — will invoke its terminate callback which will inform the Terminator process to begin trapping exits and terminate our pipeline supervisor. As this Terminator process is a child of the pipeline supervisor, it will invoke its implementation of terminate.

@impl true
def terminate(reason, %{name: name, supervisor_pid: supervisor_pid, terminator: terminator}) do
  Broadway.Topology.Terminator.trap_exit(terminator)
  ref = Process.monitor(supervisor_pid)
  Process.exit(supervisor_pid, reason_to_signal(reason))

  receive do
    {:DOWN, ^ref, _, _, _} -> :persistent_term.erase(name)
  end

  :ok
end

The exit signal propagates to the other components through their supervisors terminating and they will also invoke their terminate callback if they trap exits such as producers disconnecting from the data source.

The Terminator process is responsible for ensuring that all events still within the pipeline are processed before terminating the pipeline entirely.

It does so in three phases:

Notify that the processors do not resubscribe to producers through a state flag
Drain the producers of any events remaining by emitting the events through the pipeline
Wait for the batch processors (which will be the very last component in the pipeline) to terminate before terminating the supervisor

@impl true
def terminate(_, state) do
  for name <- state.first, pid = Process.whereis(name) do
    send(pid, :will_terminate)
  end

  for name <- state.producers, pid = Process.whereis(name) do
    Broadway.Topology.ProducerStage.drain(pid)
  end

  for name <- state.last, pid = Process.whereis(name) do
    ref = Process.monitor(pid)

    receive do
      {:done, ^pid} -> :ok
      {:DOWN, ^ref, _, _, _} -> :ok
    end
  end

  :ok
end

Interestingly, as the producer may be waiting to drain events, we may not want to cancel all of its consumers immediately. Thus, we rely on GenStage#async_info to queue the message to cancel all consumers at the end of the GenStage message queue — effectively waiting for all other events to be processed before cancelling all consumers. If batching is enabled, the processors will also wait for the batches to be processed before cancelling all consumers.

@spec drain(GenServer.server()) :: :ok
def drain(producer) do
  GenStage.demand(producer, :accumulate)
  GenStage.cast(producer, {__MODULE__, :prepare_for_draining})
	# The :cancel_consumers message is added to the end of the message queue
  GenStage.async_info(producer, {__MODULE__, :cancel_consumers})
end

These mechanisms ensure that all events left in the pipeline is properly processed before the pipeline terminates, thus achieving graceful shutdowns.

Fascinating discovery!

These are some interesting bits of code that Broadway has.

`using` configurations

Like other libraries in Elixir, use Broadway is where it all begins. As discussed in the previous open-source deep dive, the behavior of use can be altered by defining the __using__ macro.

defmacro __using__(opts) do
  quote location: :keep, bind_quoted: [opts: opts, module: __CALLER__.module] do
    @behaviour Broadway

    @doc false
    def child_spec(arg) do
      default = %{
        id: unquote(module),
        start: {__MODULE__, :start_link, [arg]},
        shutdown: :infinity
      }

      Supervisor.child_spec(default, unquote(Macro.escape(opts)))
    end

    defoverridable child_spec: 1
  end
end

There are three interesting bits of code in the __using__ macro:

location: keep

Used to report runtime errors from inside the quote. Without this, errors are reported where the defined function ( in quote) is invoked. This is to ensure that we are aware of where the errors are occurring. More information about this configuration can be found here.
bind_quoted

Used to create bindings within the quote. When a binding is created, the value is automatically unquoted (which includes evaluation) and the value cannot be unquoted again. This is especially used when we do not want to re-evaluate the value multiple times.

More information quoting and unquoting in Elixir can be found in the official tutorial and a simplified explanation and example of binding can be found here.

@behaviour

Used to define interface-like behavior where modules that adopt these behaviors can implement callbacks defined. In this case, when a module use Broadway, it will have to implement certain callbacks like handle_message while other callbacks like handle_batch remain optional.

@callback prepare_messages(messages :: [Message.t()], context :: term) :: [Message.t()]
@callback handle_message(processor :: atom, message :: Message.t(), context :: term) ::
            Message.t()
@callback handle_batch(
            batcher :: atom,
            messages :: [Message.t()],
            batch_info :: BatchInfo.t(),
            context :: term
          ) :: [Message.t()]
@callback handle_failed(messages :: [Message.t()], context :: term) :: [Message.t()]

@optional_callbacks prepare_messages: 2, handle_batch: 4, handle_failed: 2

More information about typespecs can be found in the official documentation .

Module metadata processing

While on the topic of meta-programming, module metadata can also be processed.

ensure_loaded? ensures that a given module is loaded. In Broadway, this is used to ensure that the :persistent_term module from Erlang is available for Elixir — the only time it will not be available is when the version of Elixir is too old. Documentation here.

unless Code.ensure_loaded?(:persistent_term) do
  require Logger
  Logger.error("Broadway requires Erlang/OTP 21.3+")
  raise "Broadway requires Erlang/OTP 21.3+"
end

function_exported? returns whether a module contains a definition for a public function with a given arity. Used to execute functions from modules if they are defined. Documentation here.

if Code.ensure_loaded?(producer_mod) and
     function_exported?(producer_mod, :prepare_for_start, 2) do
  case producer_mod.prepare_for_start(module, opts) do
    {child_specs, opts} when is_list(child_specs) ->
      {child_specs, NimbleOptions.validate!(opts, Broadway.Options.definition())}

    other ->
			# ...

Dynamic process naming

As the pipeline can comprise of any number of components, Broadway supports dynamically generated processes. These dynamically generated processes are assigned names that follow a fixed convention — comprising of the name of the pipeline, the process type, and the index of the component among the other components of the same type.

defp process_name(prefix, type, index) do
  :"#{name_prefix(prefix)}.#{type}_#{index}"
end

defp process_names(prefix, type, config) do
  for index <- 0..(config[:concurrency] - 1) do
    process_name(prefix, type, index)
  end
end

The names are returned as quoted atoms where the atom has a space in it so it has to be declared via :"" .

Storage options in Elixir

Besides the basic data structures like lists and dictionaries, Elixir and Erlang offer other ways of storing data within processes.

Atomics

:atomics are a way of performing atomic operations on a set of mutable atomic variables.

Used to maintain the rate limiting threshold.

Previously, the rate limiter used ETS instead but atomic operations are much better for concurrent systems as they avoid race conditions when multiple producer processes are attempting to modify the rate limit.
```
counter = :atomics.new(@atomics_index, [])
:atomics.put(counter, @atomics_index, allowed)
```

Persistent term

Storage for Erlang terms that is optimised for reading terms at the expense of writing and updating terms.

Used to store pipeline metadata like producer names etc.

:persistent_term.put(config.name, %{
  context: config.context,
  producer_names: process_names(config.name, "Producer", config.producer_config),
  batchers_names:
    Enum.map(config.batchers_config, &process_name(config.name, "Batcher", elem(&1, 0))),
  rate_limiter_name:
    config.producer_config[:rate_limiting] && RateLimiter.rate_limiter_name(opts[:name])
})

Queue

Manage first-in, first-out queues.

Used to manage message and demand buffers in the producer.

# A queue of "batches" of messages that we buffered.
message_buffer: :queue.new(),
# A queue of demands (integers) that we buffered.
demand_buffer: :queue.new()

Process dictionaries

Store state within a process directly although its usage is generally frowned upon .

Used to store batches in the batcher.
```
defp init_or_get_batch(batch_key, state) do
  if batch = Process.get(batch_key) do
    batch
  else
		# ...
  end
end

defp put_batch(batch_key, {_, _, _} = batch) do
  Process.put(batch_key, batch)
end
```
A better alternative might have been to use an Agent or ETS instead.

Edit! I clarified with the team about their decision to use process dictionaries over ETS, this was their response:

The correct solution here would be to simply use a map. But because this is very intensive code, we need a mutable option, and the process dictionary is the most efficient one. ETS would be slow as data has to be copied in and out of ETS. This is one of the very cases where using the pdict for performance is justified. :)

So, the reason why they decided to use a process dictionary over ETS is due to the performance requirement of batching! Very interesting!

Options validation

Dashbit — the team behind Broadway — developed an options validation library called NimbleOptions that aims to be a small library for validating and documenting high-level options.

A set of definitions for the available options are created first and these can be used to validate a keyword list — aka the options.

If the options are invalid, an error is returned, otherwise an :ok status along with the options are returned. The returned options have default values filled in.

Default values in dictionaries

Broadway has an interesting way of fanning out default values for the options keyword list. In the options keyword list, a “parent” value for :partition_by, :hibernate_after, and :spawn_opt is provided.

options = [
	partition_by: ..., # these are parent values
	hibernate_after: ...,
	producer: [
		hibernate_after: ... # this is a child value
	]
]

The parent value will be used for producers, processors, and batchers if no explicit child value is provided. Alternatively, we might want to fan out a parent value to only two of the three unset child values while maintaining the original child value of the third.

This is done by merging the child options into the parent options. Thus, if the child does not define a value for the option, the parent value is inherited.

opts =
  opts
  |> carry_over_one(:producer, [:hibernate_after, :spawn_opt])
  |> carry_over_many(:processors, [:partition_by, :hibernate_after, :spawn_opt])
  |> carry_over_many(:batchers, [:partition_by, :hibernate_after, :spawn_opt])

defp carry_over_one(opts, key, keys) do
  update_in(opts[key], fn value -> Keyword.merge(Keyword.take(opts, keys), value) end)
end

defp carry_over_many(opts, key, keys) do
  update_in(opts[key], fn list ->
    defaults = Keyword.take(opts, keys)
    for {k, v} <- list, do: {k, Keyword.merge(defaults, v)}
  end)
end

Closing the curtains

To conclude, Broadway is a powerful library for building data processing pipelines. These pipelines are built on top of the robust concurrency system that Elixir boasts.

Broadway is a very versatile library and the documentation contains detailed guides about using it with various data sources. Check out the Github repository and documentation!

If you want to get a basic understanding of the underlying concepts of Broadway or better visualise the architecture of a pipeline in Broadway, check out the first part here!

Open-source Deep Dive is a series where I pick apart open-source projects to explain the underlying concepts that power these projects and share my findings about the project!

Open-source Deep Dive: Broadway (Part 1) - Message queues, concurrency in Elixir, and Broadway architecture

Mon, 12 Apr 2021 00:00:00 GMT

This is the first part of the deep dive and the following topics will be covered:

A brief introduction to what Broadway is
Message queues
Concurrency in Elixir
Producer/consumer model & GenStage
Architecture of a Broadway pipeline
Construction of producer & processor components

If you wish to jump right into the meat of Broadway, you can find the second part here!.

Act 1, Scene 1

You have just received your latest feature to work on and it is to build a system that receives transaction information from a message queue, maps the customer code in this transaction information to the customer’s information, and stores this collective information in a separate database to be queried for customer transaction analysis. Your boss has developed an obsession with Elixir recently and is now pushing for every project to use it. Gasp.

You start researching for libraries that can do exactly that and stumble upon Broadway.

…build [concurrent] and [multi-stage] [data ingestion] and [data processing] [pipelines]…

Oh boy… that — that is a mouthful… Let’s break it down, shall we?

concurrent - having two or more computations in progress at the same time; in progress meaning that they do not have to be executed at the same time ( definition here)
multi-stage - successive operating stages ( definition here)
data ingestion - process of moving data from one source to a destination for further storage and analysis ( definition here)
data processing - conversion of data into a usable and desirable form ( definition here)
pipelines - series of data processing elements ( definition here)

In essence, Broadway builds systems that behave like factory assembly lines. Raw materials (data) is fed into the assembly line (Broadway pipeline) which is then pieced together to create the end product or other components used in the final product. The factory has multiple identical assembly lines running so raw material can be fed into any of these lines to be worked on.

For your use case, the flow of data will look something like this:

So how does Broadway achieve all of this?

Lights! Camera! Action!

Before understanding the internals of Broadway, we should establish some basic knowledge of the technologies we will be using so that we won’t be headless chickens running into this.

Broadway revolve around the following concepts:

message queues
concurrency in Elixir

What are message queues?

Note! While Broadway can integrate with many types of data sources, the core examples given in the project focus on message queues as the primary data source.

Message queues are like containers that hold sequences of work objects — called messages — that are to be consumed and processed. It aids with building asynchronous modular and concurrent systems.

Messages are created and delivered to these queues by producers and taken from these queues for processing by ** consumers.** These messages can vary from something as simple as plain information to more complex structures like requests or — in our case — transaction information.

Message queues are useful for decentralising the communication mechanism of large systems by acting as a medium for exchanging events between systems which allows for systems to be easily scaled and distributed.

This is a reduced explanation of what a message queue is and what it is capable of. For more information about message queues, the Amazon documentation and this blog post by CloudAMQP are good places to start.

Concurrency in Elixir

Broadway relies heavily on concurrency in Elixir. The topology (architecture) of a pipeline is built on top of processes and many of the features are achieved using the robust concurrency model of Elixir. So what exactly is the concurrency model in Elixir?

Elixir employs the actor concurrency model. In this model, actors are defined as self-isolated units of processing. In Elixir, these actors are called processes and they are managed by the Erlang VM . Elixir code is run in each process and a default/main process is akin to that of the main thread in other concurrency models.

Each process communicates via asynchronous message passing. Think of a process as a mailbox of sorts; it has a “bin” to receive incoming messages and it possess an “address” for other processes to identify it by.

The unique aspect of this model is the lack of shared mutable state that other concurrency models rely on. Rather, state is exclusive to each process.

In order for the state of a process to be altered, the owner process must make the alteration either on request or internally due to certain changes.

The topic of concurrency in Elixir is vast and Elixir provides many other features surrounding its concurrency model such as GenServer. This section is a short preview of what the actor concurrency model and concurrency in Elixir is all about. For more information, you can refer to this thesis paper and the Wikipedia article talking about the actor concurrency model and the official documentation and this tutorial on OTP in Elixir for more examples of concurrency in Elixir.

Cue the producer/consumer model

Using the actor concurrency model as a foundation, another concurrency pattern can be modelled in Elixir — the producer/consumer model.

This model aims to allow for decoupled data production and consumption by setting up two separate processes to handle each task — effectively creating a logical separation of concerns.

However, the producer/consumer model faces a critical issue — what happens if the producer generates excessive messages for the consumer? The consumer will be overwhelmed and will eventually fail trying to keep up with processing that many messages. This is where back pressure comes into play.

Back pressure is a control mechanism for how much a producer should emit based on consumer demand, consumer message buffering, or limited sampling

Back pressure avoids the problem of overloading the consumer with messages by applying one of or a combination of the three methods mentioned above (more information in the link here).

The next frontier (of concurrency): GenStage

Seeing the value of having a standard implementation for the producer/consumer model, the Elixir team decided to develop exactly that.

GenStage is a specification for exchanging events between producers and consumers with back pressure between Elixir processes

Producers emit events to consumers for processing. The events can be of any structure.

The control mechanism used is a demand system. Consumers inform producers of how many events they can handle (demand) and producers emits no more than the demanded amount. This ensures that the consumers are capable of handling the events emitted.

Producer-consumers behave like both producers and consumers. They are used to perform transformations on events emitted by the producer before they are emitted to the consumer.

Similar to GenServer, stages in GenStage exchange events through callbacks.

When a demand is handled — i.e. producer emits events and demanding consumer handles these events — another demand is made, creating a cycle where both stages are always working - ideally.

GenStage is a powerful tool in an Elixir developer’s arsenal. More information can be found in the official announcement where a little bit of history of how GenStage came to be was discussed and in a talk by José Valim — creator of Elixir.

With a better grasp of the overarching concepts used in Broadway, we can finally discuss what Broadway is all about and how it does what it does!

Pipeline architecture

It is at this juncture where it would be important to clarify the term “producer”. In both message queues and GenStage, a producer is a creator of messages or events. However, in Broadway, a producer is both a consumer of messages and an emitter of events.

For the rest of the article, the following definitions for the following terminology will be used:

producer — producer of events in Broadway
message — message in a message queue or any other data source
event — GenStage events

When messages are consumed by the producer, they will be transformed into events with a fixed structure defined by Broadway.

Each component is a separate process and they are dynamically generated as different topologies (architectures) can be designed. The order of initialisation for a typical pipeline looks something like this:

The producers and processors are both created using interesting conventions that is will be explored now. Other components will be discussed later on as they tie into other features Broadway has.

How it’s made: Producers

Producers are built using a pattern similar to the strategy pattern but modified to integrate with the concurrency system in Elixir.

Different data sources require different methods of establishing connections and receiving messages. Thus, we break up the producer process into two modules — ProducerStage defines the behavior for enforcing the rate limit while a dynamically loaded module defines the behavior for establishing a connection to the data source and receiving messages.

ProducerStage assumes that the dynamically loaded module contains the typical GenStage callbacks like handle_call and handle_demand and uses them for things like rate limiting.

The ProducerStage behaves as the context while the dynamic module behaves as the strategy. The dynamic module adopts the Producer module — which defines two callbacks for managing the overall producer life-cycle.

To load the module dynamically, the module name is passed to ProducerStage as an argument. To keep the producer as a single process, we call the init function of the module directly when initialising the ProducerStage. This way, the module will initialise under the newly spawned process for ProducerStage rather than spawning an entirely new process.

@impl true
def init({args, index}) do
  {module, arg} = args[:module]
	# ...
  state = %{
    module: module,
    module_state: nil,
    transformer: transformer,
    consumers: [],
    rate_limiting: rate_limiting_state
  }

	# Calling the init function of the dynamically loaded module
  case module.init(arg) do
		# ...
  end
end

When start_link is called, a new process is spawned first before the init function is called under the new process.

This is done as certain message queue providers like RabbitMQ attach active listeners to the calling process so spawning a separate process for this would mean having to manage two separate processes for a producer.

How it’s made: Processors

Processors are created using a concept similar to inheritance in object-oriented programming. This idea comes from the need to standardise the subscription logic of producer-consumers and consumers.

When a processor is started using start_link, a process of the Subscriber module is started with the current processor module passed as a argument.

def start_link(args, stage_options) do
  Broadway.Topology.Subscriber.start_link(
    __MODULE__,
    args[:producers],
    args,
    Keyword.take(args[:processor_config], [:min_demand, :max_demand]),
    stage_options
  )
end

The current module is initialised in the Subscriber process through init.

@impl true
def init({module, names, options, subscription_options}) do
  {type, state, init_options} = module.init(options)

	# ...
end

Other producer-consumers and consumers like batcher and batch processors also use this pattern to create their respective GenStage stages.

A separation of concern is achieved using this pattern. The processor is responsible for event handling while the subscriber handles the subscription logic.

That’s a basic rundown of the concepts underpinning Broadway. While it may not be a complete and intensive explanation of everything, hopefully it is able to provide some clarity. In the next part, we will be exploring how features in Broadway have been implemented!

Hop on over to the second part here!

Open-source Deep Dive is a series where I pick apart open-source projects to explain the underlying concepts that power these projects and share my findings about the project!

Open-source Deep Dive: Hound

Sun, 17 Jan 2021 00:00:00 GMT

I do not condone the use of browser automation or torrenting for illegal purposes. Any links or discussions about the mentioned subjects are purely for educational purposes and should remain as that.

What is Hound?

For browser automation and writing integration tests in Elixir

Let’s inspect this definition a little closer…

What is browser automation?

Browser automation is effectively the process of using a proxy (like Selenium or Hound) to perform browser actions on behalf of the user (like the test case). Essentially, we are automating the usage of the browser.

It is often associated with illegal applications like sneaker-botting but much like torrenting, there are positive applications and we will be exploring one of them in this post - integration testing.

What is integration testing?

When building software, we first build individual components to support given functional requirements. These individual components can be tested using unit tests - which ensure that given a set of inputs, the component returns a predictable set of outputs (predictable means that the functions tested are pure).

However, while components may work well on their own, when combined with other components (to form larger components/whole systems), unexpected behavior may be exhibited. For instance, the input from component A is transformed before it is used as input to component B, thus, the combined components returns an unexpected result.

Hence, integration tests serve to bridge the gap between individual components testing and full system testing.

When combined with browser automation, we can ensure that a website works end-to-end. We can ensure that the data validation on the front-end works as intended and that the forms submitted by users are properly sent to the back-end and saved in the database.

Approaching browser automation integration testing…

We can take two approaches to browser automation integration testing. We could either

build our own interfacing system to communicate with the browser, or
rely on existing interfacing systems

The former is time-consuming and requires a lot of care during development as we have to account for varying browser APIs and quirks. Thus, it is wiser to chose the latter when approaching browser automation integration testing. Doing so minimizes the number of components we have to manage.

Introducing Hound!

This is where Hound comes into the picture. Hound provides a clean API to build browser automation tests. It relies on Selenium, PhantomJS, and ChromeDriver as the interfacing systems to perform the “dirty” work of coordinating requests/responses to/from the browser.

This introduces a larger question, what exactly is Selenium, PhantomJS, and ChromeDriver? More importantly, in fact,

“How is browser automation performed?”

Understanding how browser automation is performed provides us with a better foundation to grasp how these technologies work and how Hound works under the hood.

The world of web drivers…

The key driving (pun intended) of browser automation is web drivers. But before we can understand what they are, we should establish some basic understanding of what a driver is in general computing terms.

What are drivers?

Drivers are pieces of software that behave as a proxy between a caller and a target. Callers can be something like the print prompt in Google Chrome or a computer peripheral. Targets can be something like the printer or computer.

In general, drivers are responsible for translating the caller’s request into a given format that the target can understand.

There may be variations of a caller to the same target so each driver must be able to translate their respective caller’s request into a common request format for the target. For instance, there are multiple types of keyboards that can be connected to a single computer but the computer can only understand a single request format. So the respective keyboard drivers are responsible for converting the unique keyboard’s requests into the format that the computer accepts.

Back to web drivers

Similar to general drivers, web drivers behave as proxies for the caller (Hound) to communicate with the target ( browser). It allows the caller to send instructions for the browser to perform. In the web development world, a browser is also referred to as a user agent.

The Selenium project proposed a W3C specification to guide the development of web drivers. For the rest of this discussion, we will be relying on this specification. The specification can be found here.

According to the specification, there must exist a separation of concern when designing a web driver. More specifically, there are two components to a web driver:

Local end - API for developers to send requests to the browser (libraries like Selenium and Hound)
Remote end - responsible for communicating with the browser, i.e. a browser driver (can you infer what this means?)

In essence, a web driver is comprised of an API and a browser driver. Ideally, the API should be able to work with different browser drivers for different browsers.

The remote end must also provide an HTTP compliant wire protocol where each endpoint maps to a command for the browser.

This means that the remote end relies on HTTP to communicate requests with the browser. The remote end is a HTTP server that the local end writes HTTP requests to. The remote end translates each HTTP request (based on endpoint and method) to a command for the browser. Note that a wire protocol is a method of getting data from one point to another. It dictates that requests should follow a given format.

The specification also provides an outline for the endpoints that the remote end must make available for the local end. This ensures standardization and ease of adoption for future browser drivers.

One advantage to using a HTTP server for the remote end is that it is possible to host the remote end on a remote machine. This means that we can delegate the job of integration testing to another machine, a process commonly known as distributed testing. By enabling distributed testing, the local machine is not burdened with the responsibility of testing potentially extensive and rigorous integration tests which the machine may not support.

So, how does Selenium work?

Selenium implements the web driver specification (they did author it). The remote end uses the JSON Wire Protocol as the HTTP compliant wire protocol to communicate with the browser driver. Note that the documentation provided by Selenium (for the JSON Wire Protocol) has been obsoleted in favor of the one defined in the specification.

How does Selenium differ from PhantomJS?

PhantomJS is a headless browser library. Headless browsers are essentially web browsers without the graphical interfaces. Selenium, on the other hand, is a web driver. The key difference between the two is the way requests are routed and managed (PhantomJS is a rather interesting project so Open-source Deep Dive: PhantomJS edition maybe?).

However, Selenium supports headless browsers as well and more importantly, Selenium is still in active development while PhantomJS has been archived due to a lack of active contributions.

What is ChromeDriver then?

ChromeDriver is a browser driver developed as part of the Chromium project. It is used by Selenium as one of the supported browser drivers. However, Hound supports raw requests to ChromeDriver as the underlying HTTP server works the same with or without the use of Selenium. It is an interesting project so I may explore it in another installment of Open-source Deep Dive!

Hound: Under the hood

Leveraging browser drivers

With a better understanding of how browser automation and web drivers work, we can in fact see that Hound doesn’t rely on the entirety of Selenium (including the local end APIs). Instead, it relies on the remote end of the Selenium web driver (along with PhantomJS and ChromeDriver) to minimize the number of “moving components” that need to be managed while reaping the benefits of the existing technologies. Thus, it can focus on delivering a seamless API for developing browser automation integration tests.

Exploring a basic use case

We will inspect a basic use case of Hound before diving into how Hound works.

defmodule HoundTest do
	use ExUnit.Case
	use Hound.Helpers

	hound_session()

	test "the truth", meta do
		navigate_to("https://google.com")
		element = find_element(:class, "search")
		fill_field(element, "Apples")
		submit_element(element)

		assert page_title() == "Apples"
	end
end

There is quite a bit to unpack here. Let’s first understand the core of how a test suite with Hound is setup.

First, we use ExUnit.Case as Hound works hand in hand with ExUnit, a built-in Elixir library for developing unit tests. It relies on two components of ExUnit: setup and on_exit. This allows Hound to work as expected.

Then, we use Hound.Helpers which, with the power of macros, imports all helper functions that are required to access the browser session.

Finally, we call hound_session() which creates a new session (an instance of the browser) and initializes the setup and tear down functionality of a Hound browser automation test.

Once the core of the browser automation test is built, we can write test cases as per normal, leveraging on functions like navigate_to() and fill_field() to perform browser actions. The intended behavior of these functions are easy to understand and the documentation for them can be found here.

Breaking it down

With a basic understanding of how a test suite can be setup in Hound, we can start to decompose Hound to better understand what makes it tick.

First, we need to inspect the following file: lib/hound/helpers.ex which houses the Hound.Helpers module, the same one that we use in the example above.

By overriding the __using__ macro, Hound is able to import all of the helper functions into a given file with a single use statement. This helps to minimize the boilerplate for users to get started. Macros are meta programming constructs that inject code during compile-time. More on macros here.

defmacro __using__([]) do
  quote do
    import Hound
    import Hound.Helpers.Cookie
    import Hound.Helpers.Dialog
    import Hound.Helpers.Element
    import Hound.Helpers.Navigation
    import Hound.Helpers.Orientation
    import Hound.Helpers.Page
    import Hound.Helpers.Screenshot
    import Hound.Helpers.SavePage
    import Hound.Helpers.ScriptExecution
    import Hound.Helpers.Session
    import Hound.Helpers.Window
    import Hound.Helpers.Log
    import Hound.Helpers.Mouse
    import Hound.Matchers
    import unquote(__MODULE__)
  end
end

Hound.Helpers also defines the hound_session() function which relies on setup and on_exit() of ExUnit to setup and tear down a session between every test case.

defmacro hound_session(opts \\ []) do
  quote do
    setup do
      Hound.start_session(unquote(opts))
      parent = self()
      on_exit(fn -> Hound.end_session(parent) end)

      :ok
    end
  end
end

Each helper function constructs a HTTP request to the browser driver server using Hackney . For instance, navigate_to - which opens a given URL in the session - creates the following HTTP request:

def navigate_to(url, retries \\ 0) do
  final_url = generate_final_url(url)
  session_id = Hound.current_session_id
  make_req(:post, "session/#{session_id}/url", %{url: final_url}, %{}, retries)
end

Hound rolls its own HTTP request/response management system that supports multiple retries. This can be found in the lib/hound/request_utils.ex file.

We have managed to break down the core functionality of Hound. There are additional interesting components to Hound that I would like to explore as well.

Processes

Applications are started according to standard OTP specification (here). lib/hound.ex starts a link to Hound.Supervisor which initializes two workers: Hound.ConnectionServer and Hound.SessionServer. These are child processes (Hound isn’t fully up-to-date with Application convention) that the supervisor manages.

def init([options]) do
  children = [
    worker(Hound.ConnectionServer, [options]),
    worker(Hound.SessionServer, [])
  ]

  supervise(children, strategy: :one_for_one)
end

Let’s explore what the connection server and session server are all about next.

More information on processes in Elixir here.

Connection server

This process is responsible for managing the details of the browser driver and providing information to construct the HTTP server endpoints. It stores the driver information using Agent to allow the information to be accessed across processes.

More information on Agent here.

Session management

Sessions, as mentioned earlier, refer to instances of the browser that we want to run our tests on. As Hound supports multiple sessions across different processes, it has rolled a session management system.

Session management in Hound relies on ETS, a built-in storage option provided by Erlang and available in Elixir. When the session server first starts, it creates a new ETS table to hold the session information. This server is setup as a GenServer which allows it to support asynchronous and synchronous callbacks from other processes.

def init(state) do
  :ets.new(@name, [:set, :named_table, :protected, read_concurrency: true])
  {:ok, state}
end

When a session is first created by hound_session(), the current process’s ID (by default, it’s the main process) is related to the session. The process is monitored and the new session is created. Under the process ID, multiple sessions can be created, thus allowing Hound to support multi-session testing. Each session is identified by an ID. Each session is also assigned a name. By default, we use the session name of :default. The ETS table holds the following information (mapped to JSON for illustration purposes. In reality, ETS tables store tuples of data so the actual data stored does not include any keys, just the values in the given order):

[
   {
      // Process ID
      "pid": ...,
      // Process monitoring ref
      "ref": ...,
      "session_id": ...,
      // Map containing all sessions running
      "session": {
         "<session_name>": "<session_id>",
         ...
      }
   }
]

As you can see, a session name can be assigned to the same session ID, but not the other way around (I am not too sure why this is setup as such, more investigation would be required).

With the ETS table setup to manage session information, we can avoid a major problem: passing around the session ID to various functions. If we had done so, we would have increased the overhead required when using the session ID as functions would have to be designed to accept the session ID and we would have to devise a method of passing the session ID around.

Instead, the session ID is retrieved from the server on demand using the current_session_id() function in lib/hound.ex.

As the current session ID is related to the calling process ID, multiple processes can have different sessions, thus, providing multi-session testing support. This also means that if the calling process changes, the associated session will be retrieved or a new session will be created dynamically.

If a process dies - i.e. Process.monitor() sends a DOWN message - the session server will destroy all associated sessions with that process asynchronously.

def handle_info({:DOWN, ref, _, _, _}, state) do
  if pid = state[ref] do
    destroy_sessions(pid)
  end
  {:noreply, state}
end

Configurations

Configurations are managed using Elixir’s Config API which uses keyword parameter lists to manage configurations.

import Config

config :hound,
  browser: "firefox"

The configurations are stored as application environment variables which are retrieved by the connection server.

driver = options[:driver] || Application.get_env(:hound, :driver, "selenium")

More information on the Config API here.

Coding conventions

Other rather interesting bits of Elixir convention that Hound employs are:

^ (pin) operator

The pin operator ensures that a variable, when matched during assignment, is the same as the existing variable of the given name

In Hound, this is used to ensure that the retrieved process ID of session (from the session server) is the same as the given process ID (from argument).

If the retrieved pid does not match the pid argument, an error is raised.
```
def all_sessions_for_pid(pid) do
  case :ets.lookup(@name, pid) do
    [{^pid, _ref, _session_id, all_sessions}] -> all_sessions
    [] -> %{}
  end
end
```
More information on the pin operator here.
defdelegate

defdelegate dictates that a function’s underlying behavior is deferred to that of another function in another module.

This allows a module to house the functionality of different modules without breaking the modularity afforded by the module system.

Interestingly, the __using__ override in Hound.Helpers can be replaced with a multitude of defdelegate to the helper functions but it would, understandably, create a lot of confusion.

More information on defdelegate here.
= (match operator) in function parameters

As = is the match operator in Elixir, it can be used to perform pattern matching while assigning the matched pattern to a variable name.

This is very useful when working with structures as you may not want to deconstruct the entire structure while ensuring that arguments follow the given structure.
```
def foo(%User{} = user) do
	IO.puts user[:name]
end
```
More information on the match operator here.

Pattern matching as enums

Pattern matching with atoms can be used as substitutes for typical enum behavior.

An enum in Kotlin may look like:

enum class MatchClause(val name: String) {
	CLASS("class"),
	CSS("css selector"),
	NAME("name"),
	ID("id"),
	ELEM("element")
}

In Elixir, it can be written as such:

def match(:class), do: "class"
def match(:css), do: "css selector"
def match(:name), do: "name"
def match(:id), do: "id"
def match(:elem), do: "element"

Conclusion

To conclude, Hound is a browser automation and integration testing library built on top of web driver - more specifically, browser driver - technologies as it leverages Selenium, PhantomJS, and ChromeDriver to build a highly abstracted and simple to use API for building integration tests.

Under the hood, Hound is an intriguing project that uses fundamental constructs to build powerful internal libraries that support complex operations.

If you are interested in the topics discussed in this post, here are some additional readings:

I do not condone the use of browser automation or torrenting for illegal purposes. Any links or discussions about the mentioned subjects are purely for educational purposes and should remain as that.

Open-source Deep Dive is a series where I pick apart open-source projects to explain the underlying concepts that power these projects and share my findings about the project!

Elixir has opened my eyes...

Tue, 29 Dec 2020 00:00:00 GMT

Functional programming (FP) is a mythical beast to the fledging developer. It can seem daunting to approach the idea of designing systems with the idea of pure functions in mind and rightfully so. I was the same a couple of years back - right when I was starting to get into computer science. Coming from object-oriented programming (OOP), FP was like trying to fit a square peg into a round hole for me. My first exposure to FP was through streams in Java 8. Then, I moved onto Kotlin and Javascript which had better “built-in” support for FP-like syntax by focusing on developing a language that has first-class functions. Finally, I approached the beast: Elixir!

Learning Elixir has been a joy, honestly. It filled the gaps that Kotlin and Javascript left as they had to accommodate for OOP. In order to practice Elixir, I decided to work on Advent of Code (AoC) 2020. Working on AoC highlighted the strengths of Elixir/FP and my weaknesses in recursion and problem solving without OOP. It has also trained me to write far cleaner code without wasting crucial man-hours writing convoluted code.

Recently, I was reading some Kotlin code and the most unusual thing happened: I had no idea what I was reading! I certainly understood what the code was meant to do, but I didn’t at the same time… The once elegant and easy to reason FP-esque code had become a challenge to interpret and that made me appreciate Elixir and “proper” FP more - specifically, function chaining, pattern matching, and guard clauses.

Function chaining

Function chaining is by far the easiest of the three to explain so let’s knock it out in one fell swoop.

In Kotlin or Javascript, functions are chained as such:

lst.filter{ it % 2 == 0 }.map{ it + 3 }

This syntax is made possible because lst is a List<T> object and these objects have methods that provide such functionality (so really, we’re performing method calling).

In Elixir, however, we do not trifle with the mundane like objects or methods. Instead, we have modules. And in these modules, we have our functions. So, in order to chain function calls, we do not rely on object method calls, instead we do something like this:

lst
|> Enum.filter(&is_even(&1))
|> Enum.map(&(&1 + 3))

We are quite literally chaining function calls as opposed to chaining method calls.

lst is passed directly as the first parameter of the first function, then the result of Enum.filter is passed to the second function and so on.

I like this syntax as opposed to method call chaining as it provides a clearer way of reading the code - where we do not worry about whether an object supports a given operation. We just focus on the data type of the input and pick the methods that suit this information.

Pattern matching

Traditionally, return values or arguments are often passed in a given format (list or map, etc.) and it is the function’s responsibility to decode this structure in the function body. However, pattern matching eliminates this additional overhead by providing the necessary syntax for performing all of this decoding for us. Let’s look at several examples and contrast them against “traditional” solutions written in more OOP-oriented languages.

Let’s say we declare a function that returns a tuple. This tuple represents a simple HTTP connection string: the URL and port. Let’s see how we can write this in Kotlin:

fun getConnString() {
  return "193.67.13.1" to "2203"
}

fun main() {
  val connString = getConnString()
  val url = connString.first
  val port = connString.second
}

Let’s see how we would approach this in Elixir:

def get_conn_string() do 
  {"193.67.13.1", "2203"}
end

def main() do  
  {url, port} = get_conn_string
end

A keen Kotlin developer might argue that, “Hey, since Kotlin already supports Pair unpacking, this problem is trivial right?”

Wrong.

While Kotlin provides basic, single-level destructuring, it does not support far more complex ones, such as:

def big_bag() do 
  {"192.67.13.1", ["2203", "2204"]}
end

def main() do
  {address, [primary_port, secondary_port]} = big_bag
end

Thread lightly for brave souls have tried but failed in achieving this level of destructuring in “traditional” OOP languages.

But beyond a simple application of destructuring the return values of a function, we can also use pattern matching in our function parameters.

def read_conn({url, [primary_port, secondary_port]}) do 
  "#{url}:#{primary_port}"
end

Not only can we use pattern matching for destructuring parameters of a single function, we can use method overloading in conjunction with pattern matching to cater to corner cases in our parameters (more can be done with guard clauses but that is a topic I shall reserve till later on):

def ex([instruction | rest], result), do: ex(rest, result ++ instruction)
def ex([], result), do: result

In this particular example, the function ex extracts the immediate instruction from the list of instructions and adds it to the result. It does so recursively until the list of instructions is empty. At that point, the function will simply return the list of instructions once again (recall that result simply holds the instructions in order).

This recursive function is able to halt (or encounter a stop case) because pattern matching dictates that any non-empty list will call the function again while an empty list returns the result.

You can also use pattern matching to detect when a function parameter (destructured) does not match a given format:

def foo([name, age]), do: "#{name}, #{age}"
def foo(_), do: :error

Pretty cool innit? The official Elixir documentation talks about pattern matching in much greater detail but I wanted to highlight the applications of pattern matching that I found interesting.

Pattern matching has allowed me to write far more concise code and that has been such a wonderful feature that I sorely miss now when I work on Kotlin codebases.

Guard clauses

Imagine we’re writing an implementation of the Fibonacci sequence in Kotlin. In order to inform the program that we have hit a base case, we would often design the function to include the base case, as such:

fun fib(n) {
  if (n == 0 or n == 1) return 1
  return fib(n - 1) + fib(n - 2)
}

I don’t know about you, but the if-statement just makes the code look untidy. Instead, we can use guard clauses to spruce this up a little:

def fib(n) when n in 0..1, do: 1
def fib(n), do: fib(n - 1) + fib(n - 2)

Rather than embedding an if-statement inside of the function, we overload the method with a guard clause - denoted by the when keyword. When the program executes, it checks every overload of the method and tries to match the patterns AND guard clauses to ensure the right function implementation is called.

This is very useful as we can expand our code to account for many base cases without polluting the core behavior of the function. This allows us to design code that scales vertically (more functions), not horizontally (more if-statements).

Conclusion

~~Elixir is cool and you should use it!~~

In all honesty, Elixir has taught me to value OOP more, ironically. I have realised that I am very used to the OOP way of solving problems that my fundamental problem solving skills without this hammer has become extremely dull and I am very happy that Elixir has allowed me to express my code in an elegant manner. I cannot wait to try using Elixir to build sites using Phoenix!

The truth behind software projects

Tue, 26 May 2020 00:00:00 GMT

It’s 3am. You are one commit away from launching your first release of your latest idea. It’s an AI system to help people recommend names for their pets through pictures (clearly, Spot isn’t a great name). You take a swig of coffee. You fix the bug that has been sitting on the top of your Trello board for days now. git add . && git commit -am "Fix issue where AI takes over the world" and you are off. Time to tag this release. This AI will be a hit and pet owners all around the globe can rejoice knowing that the arduous task of naming their pets will finally be over (now only if you had an AI to recommend you variable names). You push the tag and head off to bed. Weeks go by and you continue working on your revolutionary project. You add a feature that allows you to name pet rocks too. School starts. You get swept up with a sea of assignments and presentations. Soon, your baby starts to collect dust as you scramble to finish your assignments. Eventually, school finishes and you heave a sigh of relief. “Finally, I can add object naming too!” you think to yourself. But when you open up the project, you realise that the spark, that allure of the project is gone. You stare blankly at the Trello board of “To-Do” features and begin to see this project as a chore. Those new features and promised rewrites now seem like a hassle. Those surges of dopamine you got every commit, now reduced to a mere flicker. But this can’t happen, right? After all, this was your baby! You had spent an entire semester break building it, raising it. It can’t all be for nothing! What will happen to the poor pet owners? You try convincing yourself that you can power through this lack of motivation. You think to yourself, “Maybe if I just started here, I would be able to find joy in working on this again”. But alas, no matter how hard you will yourself, you cannot find an ounce of motivation to fix that bug, to merge that pull request, to add that feature. You sigh and start questioning yourself. If you cannot finish a mere side project, are you even that worthy of being a developer? But eventually, you find another side project to occupy your time. One that makes your heart race every commit. And the cycle continues.

What I had just shared is - what I believe - a common narrative many developers experience. In programming, we are often bombarded with the idea that side projects are a necessary staple of a developer - that one must have a Github page filled to the brim with open-source contributions and personal projects to be considered a good developer. But is it really necessary? Or are we - as a collective - creating unecessary barriers for ourselves. Is this mindset really good for us? Or is it slowly destroying our motivation and sucking our weekends dry?

In this post, I would like to explore this aspect of the industry and talk about my experiences working on side projects and share some of my personal opinions on it. Now note, I do not claim to be a programming genius nor do I assume that my word is gospel. I am merely sharing my observations and experiences as a developer who was mostly self-taught. More importantly, this post is my way of performing introspection on my past five years as a developer and how much I have grown since then.

Separating the truth from fiction

As easy as it is to denounce the notion of side projects as a parasite to one’s sanity. There are many truths and value that can be derived from making side projets. So, I would first like to explain why I believe side projects are a useful tool for developers.

It is hard to deny that side projects can be incredibly fun to work on. When you find a project idea that you believe in, its siren’s call is hard to resist. You’ll find yourself spending hours on end on it. Your weekends can be engulfed by the pile of features you have planned for.

When someone stars our repository. When Google Analytics reports a new user. When someone makes a pull request. When we get a new issue. It is an undeniable fact that seeing people use what we have built is incredibly rewarding and often times, that drives us to want to work on it more; we now have an audience of users that we cannot disappoint.

Of course, many of us embark on side projects for the intrinsic rewards it presents to us. This fun is usually derived from the prospect of learning something new. Learning how a network protocol works. Learning how a language is used. Learning a new framework. These are all carrots-on-sticks for developers. Naturally curious, we all look for mental stimulation and seek the next idea we can get our fingers on. Side projects serve as a great way to explore different technologies and collaborate with others.

Side projects also offer a way for us, as developers, to view how much progress we have made. Remember that simple chat application you made when you were 16? Now look at you creating web applications from scratch or learning a whole network protocol on your own. These side projects document your journey as a developer and it is refreshing to look back and see how much you have grown through these projects.

A case against side projects

However, while the benefits of side projects are bountiful, it is hard to deny the “downsides” of side projects. Now, I use the word downsides with heavy quotations as I do not believe that these are direct downsides of the idea of side projects itself. Rather, I believe that the downsides to side projects are derived from the value we have placed on them as an industry and how we - as driven individuals - have warped this perception of a side project.

I believe that side projects are unnecessarily prized and emphasised. The naturally competitive environment in tech has caused this idea that side projects are a quintessential component of any developer’s arsenal to become overly emphasised.

It is due to this over-emphasis that many who embark on side projects often develop feelings of guilts when they are unable to see a project through. It has caused the idea of side projects to transform from one of learning and exploration to one of a “do-or-die” nature. Worse still, those who do not devote time to side projects may feel as though they are losing out against others and in turn develop an obsession with trying to “catch up” to them. This also devalues the efforts of those who try at the beginning. I believe that this boils down to the fundamental behavior of humans: competition. We will use any - and every - metric we can get our hands on to compare ourselves against one another. This is also why I do not agree with those who try to micro-optimise their daily lives - trying to squeeze every second that passes to achieve optimal productivity. I was one of those who were also obsessed with ensuring that my every waking moment was meticulously planned to ensure that my productivity remained at its peak. This was born out of fear that the rapid advancements of this field would eventually overwhelm me and leave me in the lurch. However, the reality is, as much as it is important to be driven, we must recognise that there is more to life than an eternal grind to be productive. It is important that we look at the time we are spending on side projects objectively and questioning whether we are neglecting other aspects of our life in pursuit of this “grind”.

This can lead to people developing an unhealthy obsession with creating side projects and working on that “hustle”. While many feel incredibly motivated at the beginning, those who fail to properly pace themselves will end up burnt out. This can lead to a vicious cycle where they push themselves too hard only to feel exhausted, give up, and feel guilty about giving up - spurring them on to try even harder. Burn out - in tech - is one of the worst things that could happen to a developer in my honest opinion. When one’s livelihood depends on their ability to produce good code under a time crunch, a general sense of demotivation can lead one to become careless; introducing bugs into their system. It also diminishes one’s interest in the field and in the worst case, can drive them away from the field entirely.

There are those who also pursue the idea of side projects with the incorrect mindset. Often when I speak to beginners, they often state that they feel this obligation to start a side project because they have noticed that others are doing so. Some start this side project with the goal to become the number one starred repository on Github. Some dive into one because they think that the only way to improve as a developer is to work on side projects. Some also get swept up with the common myth that “side projects == necessity to work in a company” and end up pursuing this belief.

However, what many of these individuals fail to realise is that side projects are not the only measure of a developer’s worth. As alluring of an idea it is that a company will hire you if you have a portfolio the length of a PhD graduate, companies are often looking to hire those who demonstrate problem-solving abilities. So what if you have made 20 APIs before? If you fail to demonstrate the most fundamental skill a developer should possess, you won’t be able to think critically to solve new problems. While side projects can build problem-solving abilities, some may only venture within the range of their comfort zone to try new side projects. In some cases, this could mean never trying any “hard problems” as they do not believe that they possess the capacity to solve them on their own.

It is this over-emphasis on side projects that has placed so much unnecessary pressure on new developers. I have met many developers who have been wrapped in this worry that if they don’t start building a spectacular portfolio right when they start, they will end up becoming a terrible developer. It is sad to see that they place this burden on themsevles when learning to program should be a fun and enjoyable thing.

So what now?

In summary, I believe that side projects are not inherently evil. It is the warped perception of them that our industry has created that has de-valued the true essence of them.

I have been on both ends of the “side project cycle” before. I have been the naive developer who thought having hundreds of followers on Github or thousands of stars on my projects mattered and I have also been burnt out, swamped with school work, and dissatisfied with my perceived notion of productivity.

As with anything in life, the key is balance. Understanding that side projects are not - in any way - sole markers of a developer’s abilities is important. Having a large following on Github does not mean you are a terrific programmer and vice versa. Learning to appreciate the learning process, rather than working on something for the potential rewards, is an important aspect of life and moreso in this culture where developers often try to one-up one another with false measurements of success. In the end, what you work on should matter to you. You can build something to benefit others, I am not discouraging that, but you must bear in mind that at the end of the day, the one who benefits most from it should be you.

I think we have to look at each project objectively. “Have I learnt something about myself, this technology, or programming in general?” If yes, then I believe that you have already benefited greatly from it. Whether or not the project is “failed” or “incomplete”, as long as you have learnt something along the way, I do not believe that you should belittle that effort.

So go out and build something that inspires you! Work on the project that gets you excited. Explore technologies that you are interested in. Don’t worry so much about the perceived value of the project you are embarking on. This is YOUR journey, so enjoy it to the fullest!

Applications of Kotlin's delegate properties

Sat, 13 Jul 2019 00:00:00 GMT

Announcement time!

After a long hiatus from kMD2PDF, I revved up my engines and began to work on my latest planned feature for the project…

YAML support!

Yes, now YAML is in the works and you can now control basic attributes of your exported document using only front matter YAML.

This is an incredibly big milestone as this allows anyone to quickly customise their document without having to write a single line of Kotlin, and with another planned release to create a GUI exporter for markdown documents, this would greatly streamline users’ experience.

Okay, so what are delegate properties?

kMD2PDF introduces a theming engine where you are able to change the entire color scheme of an exported document just by specifying an attribute in code as such

val converter = markdownConverter {
  // ...

  settings {
    theme = Settings.Theme.DARK
  }
}

Design considerations

Before building this system, we have to make several consideration

How do we specify that certain attributes have different configurations depending on the theme?
How do we allow changes to the theme propagate to each element such that the changes are reflected during the document creation?

Using singletons

The second inquiry is rather easy to answer, so we will tackle that issue first - we can store the Settings as singleton, since the state of a singleton is unique and static, as long as the settings are configured before the document is created, then the settings will be chosen during runtime and have the exported document reflect the changes.

object Settings {
  var font = FontFamily("sans-serif")
    get() = field.clone()
  // ...
}

inline fun settings(configuration: Settings.() -> Unit) = Settings.apply(configuration)

Kotlin comes with a language construct to create singletons easily - object keyword (read more here).

This creates everything we need in a singleton but reduces all the boiler plate that would be involved with creating the singleton, unlike other languages like Java.

When we update the Settings singleton, the changes will reflect with the exported document since the document style is lazily generated only up till the latest minute before it gets exported.

Multi-value properties

Now to tackle the hard question, how do we store multiple values in Kotlin? Ideally, what we would want with this system is to have a single variable and have that variable store both the dark theme and light theme setting. That is, if we had a single variable textColor, we would want to be able to store both the light theme and dark theme settings inside this variable. Depending on the current Settings.theme configuration at the time where the stylesheet has to be generated, the textColor variable would return either the configuration for DARK theme or LIGHT theme.

Traditionally, we would approach this issue using a class to store the information and have that be the end of things - and this was indeed the approach I ended up employing due to certain limitations in that design which I’ll discuss.

However, in Kotlin, there exists a language construct called delegated properties where you are able to call an object constructor to initialise a variable and provide it with a base of data. Subsequent times accessing this variable masks the object constructor and will only allow you to access the data type specified by the delegated property. You can think of a delegated object as an object that defaults all variable references to the given getValue and setValue attributes. This effectively means that once you’ve delegated a property, you are no longer entitled to modifying the object that created the delegate.

import kotlin.reflect.*

class DelegateExample(private var internalValue = "") {
  operator fun getValue(thisRef: Any?, property: KProperty<*>)
    = "Internal value is ${internalValue}"

  operator fun setValue(thisRef: Any?, property: KProperty<*>, value: Int) {
    internalValue += value.toString()
  }
}

In the example above, we have created a delegated property that when referenced as a variable (get), will return a string. But when called with an assignment operator (set) will only accept Int.

var value by DelegateExample()  // internalValue = ""

println(value)                  // Internal value is
value = 10                      // internalValue = "10"
println(value)                  // Internal value is 10
value = 5                       // internalValue = "105"
println(value)                  // Internal value is 105

As you can see from, in order to instantiate a delegated property and use the features of getValue and setValue, we have to set the variable with the by keyword. This way, when you have variable references, it will always call the respective setValue/getValue methods.

Now that you’ve had a crash course with delegated properties in Kotlin, can you think of how we can apply this concept to our current problem? To recap, we already have established a singleton to manage the configuration across all elements - and the configuration we are most interested in is the theme attribute which indicates which theme the document will create. In that case, since the state is singular and shared across the entire program, we can create a delegate property to hold the configuration of all the themes available, and when the time comes to generate the CSS for the document, the delegated property is called and it will return the corresponding attribute for the specific theme configured during the time of generation (since the document HTML/CSS are generated per call of MarkdownConverter#convert as opposed to generating them all at once).

In that case, we can create the delegated property as such:

class CssProperty <T>(
  private var light: T?,
  private var dark: T? = light,
  private var fallback: T? = null
) {
  operator fun getValue(thisRef: Any?, property: KProperty<*>) =
    when(Settings.theme) {
      LIGHT -> light
      DARK -> dark
    } ?: fallback

  operator fun setValue(thisRef: Any?, property: KProperty<*>, value: T?) =
    when(Settings.theme) {
      LIGHT -> {
        light = value
      }
      DARK -> {
        dark = value
      }
    }
}

This way, we can create a CssProperty to hold the configurations required for each theme (light/dark for now) and then when the variables are accessed, they will return the corresponding value stored depending on the current theme configured in the settings. The fallback property is a “default” for each CssProperty in the event where the light or dark property are both set to null.

To use the CssProperty class, we will use the following syntax:

var textColor by CssProperty<FontFamily?>(c("00"), c("fa"))
print(textColor) // Light theme text color -> #000000
Settings.theme = Settings.Theme.DARK
print(textColor) // Dark theme text color  -> #fafafa

And this provides us with such a convenient interface to modify the values of the configured properties and to quickly change the theme settings for each element on a whim.

Roadblocks 😢

However, when attempting to create the YAML feature, I had noticed a huge flaw in the delegate property system used for configuring the style components. Since the YAML formatting had to be rendered during runtime and as such, the YAML had to modify the existing style which while possible using the Singleton pattern, certain components like relying on a fallback would not be able to register the changes made. Therefore, I had to revert to using a simple class to house the CSS properties of an element and convert the Settings singleton to be a regular object that has to be passed to each element for the configuration to take place.

class CssProperty<T>(
  val theme: Settings.Theme,
  private var light: T? = null,
  private var dark: T? = light,
  private var fallback: T? = null
) {
  var value: T?
    get() = when(theme) {
      LIGHT -> light
      DARK -> dark
    } ?: fallback

    set(input) {
      light = input
      dark = input
    }
}

Then in order for the converter to apply the changes of the YAML to the document being generated, we iterate over every element within the style and apply the changes accordingly.

fun <T> List<Element>.updateStyles(input: T?, update: Element.(T) -> Unit) {
  input?.let { forEach { e -> e.update(it)  } }
}

The lesson learnt here is that while a structure might look good on paper and work for a specific use case, this might not always be the case and this can result in rewriting of the codebase. I was lucky that the codebase was one that I was very familiar with and I could afford the rewrite to get this new feature up and running. However, I would not always be so lucky and might encounter a codebase that might take too long to rewrite.

Porting to FlexMark

Mon, 08 Apr 2019 00:00:00 GMT

kMD2PDF now finally uses FlexMark as the back end for the markdown to html conversion. This is a big moment as now a lot more flexibility has been introduced for the library and that means more features.

Following my promise of making smaller and more frequent release updates, this port has ushered in version 0.2.1 of kMD2PDF and I’m really excited!

Changelog

Backend library using flexmark
Unit testing framework set up for testing node rendering
Task list items now supported

Original design

Originally, the library used commonmark to handle .md to .html conversions, but it was severely limited as the number of useful extensions was severely lacking and it resulting in a terrible codebase. However, with FlexMark, this problem is alleviated as it has all the features I need.

Hiccups

There was a slight hiccup with the port as the .html to .pdf library, flyingsaucer would not accept the HTML output produced by FlexMark. So I had to make use of Jsoup to parse the output of FlexMark to become valid XML that the flyingsaucer library would accept.

Another issue faced is the node renderers and visitors. In commonmark, these node renderers were a single class that would be added the parser (seen here) and visitors would be accepted after the document is parsed (seen here). This made it really easy to create custom node renderers and visitors which were used for figure generation and table of contents processing. However, with FlexMark, due to the increase in flexibility, the overhead for creating both increased as well and this resulted in requiring a parsing extension to be created, which would create a custom NodeRenderingFactory which in turn be responsible for creating custom NodeRenderers to render the needed node, which in this case was the figure elements. Whilst this may seem all complicated, it was actually outlined in their sample repository where I was able to successfully create the figure renderer here.

The table of content processor was similar in nature. Due to the increased flexibility offered by FlexMark, additional steps had to be taken to create a visitor to properly create the table of contents. This highlighted the idea that developing flexible software would often entail having to increase the overhead of the software because it just takes that many extra steps to provide that flexibility.

Unit testing the DOM

I also worked on attempting to create a system to unit test the node rendering aspect of the library since there can be a lot of edge cases involved with markdown and it might be useful to have an automated system to be able to ensure that the code is reliable and performs within my range of expectation.

I tried to design this API to be as seamless as possible, reducing the moving parts exposed to the user so that they would not have to fiddle with too many configurations to get it working. What I came up with was rather interesting. To ensure that the rendering was correct, I had to first find a way to test that the converted markdown file would produce a certain result. To do so, I exposed the HTML conversion process of MarkdownConverter to be able to hook into this using the API.

The essence of the API is to compare the processed HTML and an expected HTML input using Jsoup to ensure that the they are the same. This required some basic recursion to assert that every single node matched.

fun assertMarkdown(folder: String, file: String) {
  require(file.indexOf(".") == -1) { "File should not include extensions as they are added within the method" }

  val markdownFileName = "$file.md"
  val htmlFileName = "$file.html"

  val markdownFile = getResource(folder, markdownFileName)
  val htmlFile = getResource(folder, htmlFileName)

  val converter = setupConverter(markdownFile)

  val expectedDocument = parseDocument(htmlFile.readText()).body()
  val actualDocument =
    parseDocument(converter.generateBody())
      .getElementsByClass("content")
      .first()

  // Ensure that they both have the same number of children
  assertEquals(expectedDocument.childCount, actualDocument.childCount)

  // Ensure that they both have the same set of elements
  val expectedDocumentBody = expectedDocument.children()
  val actualDocumentBody = actualDocument.children()

  expectedDocumentBody.zip(actualDocumentBody).forEach { compare(it.first, it.second) }
}

As you can see, the markdown file will be the actual output produced by the library whilst the html file represents the expected output.

This was the bulk of the API, with the recursive function looking like:

private fun compare(ex: Element, ac: Element) {
  assertElementEquals(ex, ac)

  if (ex.childCount != 0) {
    (0 until ex.childCount).forEach {
      compare(ex.child(it), ac.child(it))
    }
  }
}

This allows the users to simply execute the assertMarkdown() function, providing the resource folder and resource name of the markdown file and html file. This would set in motion an automated set of testing to ensure that every aspect of the generated markdown file would produce the appropriate html.

One limitation that is present with the API is that the file name of the markdown file and html file would have to be the same, otherwise the assert function would fail. That said, it also encourages for developers trying to use the API to always stick to the same name for their markdown and html file, which reduces confusion.

Lessons on software release

Wed, 03 Apr 2019 00:00:00 GMT

Oh boy, I screwed up bad, like really badly. Ok ok, let’s go back and see where it all began. I’ve learnt countless lessons from this and I hope you (reader) will too if you’re developing your own software.

Story time!

2 months ago, I embarked on a journey to write a markdown to PDF converter, called kMD2PDF. The initial architecture was developed such that:

.md file -> .html file + .css styling -> .pdf

In order to facilitate this system, I used 2 libraries:

commonmark-java - this facilitated with the .md -> .html conversion
flyingsaucer - this facilitated with the .html -> .pdf conversion

In between, I used my own code to create a style DSL to generate the CSS styling. This system worked nicely as I was able to create an easy to use API for developers using this library.

fun main() {
  val document = MarkdownDocument("~/Desktop/README.md")
  val convert = markdownConverter {
    document(document)
  }
  converter.convert()
}

Failure to research

However, I made several big oversights. I failed to plan that the libraries I used were severely limited in what they could offer. For instance, commonmark-java does not support task list items, even as an extension and in order to implement this feature, I had to create a custom NodeRenderer that would convert bullet lists to task list items if necessary. Then I realised that because I overrode the default rendering behavior of the bullet lists, I no longer could create nested lists, so I had to remedy that myself and soon, the project became a bunch of band-aids stuck on top of the library and it caused the project to steer into a direction of just bug fix upon bug fix, as I attempted to introduce features into a tightly-coupled system. This made for an incredibly hard time working on the library as I was de-motivated to implement features since they would result in a mess.

Worst still, as I researched more libraries for handling markdown parsing, I noticed that there were libraries like Flexmark that did provide the support for the features I wanted such as task lists without requiring a lot of hacky work to be performed to the existing library.

Similarly, a big hurdle I had to cross with flyingsaucer was getting HTML5 xml code to render as the library required only XML or XHTML documents. This was a huge bottleneck because now some elements render incorrectly and others require even longer HTML that isn’t always necessary. That’s where I discovered OpenHTMLToPDF, which had a similar API to flyingsaucer but it allowed for HTML5 code and it doesn’t constrict users by forcing them to use XHTML (for the uninitiated, XHTML is a stricter form of HTML, where single-enclosed tags are not permitted, rather, every tag must be closed off with an ending tag).

This failure to plan ahead and research properly made my life hell as I spent most of my time fixing issues I created for myself.

Over committing

Another issue I had stupidly created for myself is over-committing to creating a huge feature push. Initially, after releasing version 0.1.2, I started work on version 0.2.0 - where many new features would come, along with a set of changes to the existing API to improve the lives of developers. But… that’s where I failed to realise that I had bitten off more than I could chew.

For an entire month, I focussed my efforts on trying to make version 0.2.0 feature complete. And within this period of time, the stable version of the library never once got updated. This spelt bad news for those who are using 0.1.2 as they are waiting for over a month for a newer library, whilst they’re stuck with a bug-filled library. It also meant that I would burn out quickly working on the library as I tried pushing to complete the features by an arbitrary deadline.

This caused me to be extremely stressed when users of the library would hound me for changes. However, during this month, I did learn a lot about Kotlin and software development so it wasn’t a complete waste of time. I just know it could have been so much better and smoother.

This is where I also begin to see the value of Agile development, where we should break up the development into deliverables across weeks to reduce the workload and to improve the end user experience.

No testing

An area lacking in the library is unit testing. As the markdown can have many corner cases, it’s hard to cater to each of these. The lack of unit tests meant that the bugs I encounter are always by accident, rather than the result of methodical checking and testing to ensure that nothing slips the radar.

Backtracking to unit test everything is also a pain as by this point, I would have forgotten about some of the classes.

Learning points

So what exactly did I take away from this experience and how will I be improving my approach. I’ve learnt to be more methodical with my research before starting a project, I should carefully scope for the best libraries and options available for the job. I’ve also come to realise just how key portioning work and creating frequent deliverables is to developing good software. Finally, I’ve learnt to use TDD or at least, unit test my code as I write them, rather than leaving it till the last minute.

So what exactly does this mean for kMD2PDF? Well, I’ll begin by announcing several changes to my workflow. First and foremost, I’ll be porting my codebase to use Flexmark and OpenHTMLToPDF over the next few releases, so that’s exciting! Secondly, I’ll shorten the release cycle to have new versions of the library be released every 2 weeks where the versioning will go from 0.2.0 to 0.2.1 etc. Each release will contain a few features and fixes and the goal is to make the library stable and ready to use as frequently as possible. Lastly, I’ll begin to rolling out a suite of unit tests for the codebase. I’ll adopt TDD in my workflow and hopefully, I can reach > 80% code coverage before version 1.0.0.

Whilst I’m incredibly frustrated about my set backs, I will not allow them to hinder the development and progress of the library and with a new found resolve to achieve better things, I’ll be taking on each new challenge with pride. I have also greatly enjoyed the past month learning various skills and practicing Kotlin.

If you’re curious about the library, it can be found here.

An investigation into generics in Java

Mon, 26 Nov 2018 00:00:00 GMT

Generics was a feature introduced in Java 5 and it has changed how Java developers write code.

This article will be discussing some of the things I’ve picked up as I dug into generics in Java.

Uses of generics

Pre-generics era

Imagine you wish to create a container to store an object that can be of any type, pre-generics, you would need to do something like:

class Container {
  private Object obj;

  public Object getObj() {
    return obj;
  }

  public void setObj(Object obj) {
    this.obj = obj;
  }
}

Container container = new Container();
container.setObj("Hello");
System.out.println((String) container.getObj()); // Prints "Hello"
container.setObj(1);
System.out.println((Integer) container.getObj()); // Prints 1

As you can see, this is not type safe as you can put anything into the Container and have it spit out something because the Container doesn’t care for the type of the inputs. You can get around this lack of type safety by giving the class type of the intended inputs into the Container constructor and check that against the inputs, but that will be extra code for something that seems so logical.

Generics era

With generics, you can now create the Container with type safety built in and no more casting.

class Container<T> {
  private T obj;

  public T getObj() {
    return obj;
  }

  public void setObj(T obj) {
    this.obj = obj;
  }
}

Container<String> container = new Container<>();
container.setObj("Hello");
System.out.println(container.getObj()); // Prints "Hello"
container.setObj(1); // Does not compile because container can only receive Strings

What’s going on?

This was a question I had asked a lot prior to reading up, because it was such a mysterious process that I could somehow enforce type safety just with this construct.

The type safety generics offer is limited to just compile time, this is because of a process called Type Erasure where the generic type is removed during runtime. This is done in an effort to ensure backward compatibility to pre-generic code.

This not only means that the type safety isn’t extended to runtime, it also means that any operations requiring the use of this generic type at runtime is not permitted because the type will be erased by that point. For example:

<T> void foo() {
  System.out.println(T.class); // This does not work!
}

Java isn’t able to recognize the data type of T during runtime thanks to erasure and thus this line throws and exception. This is especially troubling as you might want to be able to cast the data types from one generic type to another, but this is not possible.

This state can also be referred to as generics being non-reified, meaning, it is not present during runtime.

Kotlin to the rescue!

Since Kotlin is based off the JVM, it also inherits this type erasure behavior of generic types, which means behavior with generics you see in Java will be the same in Kotlin. However, Kotlin offers a means of enabling a generic type to be preserved during runtime, thus removing the “burden” imposed on it.

Reified generic types + inline functions

In Kotlin, the inline keyword, makes it so the function is not expanded into a separate object but rather “copy-pasted” into the call site of the function, thus allowing some interesting behavior to occur.

One such intriguing behavior is allow generic types to be reified, or retained at runtime. Because the function is now inline, the generic type given to the function can be preserved, thus, allowing you to access the generic type information during runtime.

fun <T> foo() {
  println(T::class.java)
}

Conclusion

Generics are an interesting construct with a lot of thought put into designing them.

Some articles that might peak your interest:

Familiarising yourself with lambdas in Kotlin

Sun, 28 Oct 2018 00:00:00 GMT

Kotlin is a language that was built on top of the JVM and what really sets it apart from Java is the emphasis on functions being a first-class construct, meaning many fancy things like lambdas are directly available to Kotlin developers without having to include bulky constructs like the concept of functional interfaces.

What are lambdas?

In my article on Diving Into Streams, I explained how lambdas were this miraculous constructs in Java that removed a lot of the boilerplate required when passing behavior from one method to another.

However, this definition was really under the assumption of pure Java. Lambdas, as a whole, are a mathematical concept. In programming, a lambda can be seen as a “portable” function or a piece of behavior that can be passed around and used accordingly.

Then, what are functions?

Functions are simply pieces of behavior that take in some input (or none at all) and return some output (or none at all) , this is very much the same as the mathematical concept it is based off of.

A case for lambdas

Scenario:

Say you were working in a performance-critical situation and you just discovered that there is a major memory issue with a function and it is causing some major performance drawbacks and you want to see how long it takes to benchmark and fix this issue.

“No biggie, I can take the time before and after the execution and check how long that took to benchmark the function” would be something you are thinking to yourself and that would be true in a simple case like ours.

You begin to write some code to test this function:

fun foo(upper: Int): Int {
  var sum = 0
  for(i in 1..upper) sum += i
  return sum
}

fun main(main: Array<String>) {
  val before = System.currentTimeMillis()
  println("foo returns: ${foo(10000)}")
  val after = System.currentTimeMillis()
  println("foo took ${after - before}ms to run")
}

Awesome! It worked, you can now go on your merry way and fix the bug. Then you manager comes around and tells you yet another performance issue was found and you think “Eh, this is kind of tedious to redo so I guess I could copy-paste the code I made earlier”.

Now, would you be able to continually repeat this if you suddenly had a whole class of functions that needed to be tested?

Lambdas to the rescue!

Going back to how I mentioned how a lambda is a "portable" function or a piece of behavior that can be passed around and used accordingly.

In this scenario, we ideally, would want to have a function that can time any function with a method signature like foo() and this timing function would not need to know anything about the function it receives, all it needs to know it that it should calculate the time before and after this function took to run and print that out, nicely formatted.

Some basic pseudo-code would look like this:

	Details
Input	Name of the function Function to be timed
Output	None, this function will make use of a side effect
Steps	Take the time before running the function Run the function Take the time after running the function Print out the difference between the before and after, this will be total time it took to for the function to run

Defining structure

In Kotlin, in order to declare a lambda, you have to declare a function type, or some kind of contract for the functions to be passed to follow. To simplify things, all the memory leaks that have surfaced all take in exactly 1 Int input and return an Int, so the function type for this case would look something like:

func: (Int) -> Int

Note how the function type is literally just the method signature of a potential function without the parameter names. As mentioned previously, these function types are merely just contracts, it defines what a function to be passed in should look like, it does not dictate what the inputs will be used for (generally what the parameter names represent) nor does it dictate the specific behaviors of this function.

Creating the timing function

Now that we have the general structure of the function we want to be able to time, we can then proceed with declaring the function body of this timer.

fun functionTimer(name: String, upper: Int, func: (Int) -> Int) {
  val before = System.currentTimeMillis();
  println("$name returned ${func(upper)}")
  val after = System.currentTimeMillis();
  println("$name took ${after - before}ms to run")
}

As you can see, this function’s body is very much the same as the original method of timing the function in the main method. However, whilst there were some similarities, the benefits of creating a specialized function for timing is as follows:

Modularity - your code is now modular and can be reused if needed
Flexible - because this is a function, you can apply consistent changes to the timing and the changes made will be reflected in any function timed by this function made
Abstraction - a new user of your codebase does not need to think about how a function is timed, all they need to know is how to call this timing function and they know that this function will help to time and do everything for them
Consistency - we can remove the upper argument and apply a consistent argument to be passed to func so that the timing is not skewed by changes in the upper, thus ensuring a degree of consistency

Timing things

With the newly made timing function, we can then begin to time how long each function took in our class. Let’s begin with the first function we tested, which simply added a bunch of numbers from 1 up till an upper value.

fun main(args: Array<String>) {
  functionTimer("add", 10000) {
    var sum = 0
    for (i in 1..it) sum += i
    sum
  }
}

What we have used is the lambda syntax Kotlin provides. We declare the arguments of the functionTimer function as per usual, but we then use {} which houses the body of the function. We also use the it keyword, to refer to the one and only input this lambda receives, which we can assume would be the upper limit (in this case, 10000).

Another thing to note is we omit the return keyword from the lambda body even though the function type specifies that each function has to return an Int. We simply put the value to be returned on a line of its own and Kotlin is clever enough to know to return this value.

Now, we can extend this function to work with any other function that takes in a single Int input and return an Int…

functionTimer("multiply", 100) {
  var result = 1
  for (i in 2..it) result *= i
  result
}

functionTimer("wacky", 1000) { (1..it).map { num -> num * 3 }.sum() }

Sample output:

add returned 50005000
add took 1ms to run
multiply returned 0
multiply took 0ms to run
wacky returned 1501500
wacky took 58ms to run

You can see how flexible we can make this timer be, taking in all sorts of functions and timing them.

Timing existing functions

Sometimes, you might not want to rewrite a long function just to time it, and that is perfectly fine, you can pass a reference to an existing method for it to be run as such:

functionTimer("add", 10000, ::add)

Conclusion

Lambdas are far more than just useful for timing stuff, it is extremely versatile and Kotlin uses them extensively throughout their language, in Collections to APIs that leverage off the design of lambdas.

Some resources for further reading:

Diving into (Java) streams

Sun, 08 Apr 2018 00:00:00 GMT

What are streams?

Streams was introduced in Java 1.8 and it had completely changed how we write code. The majority of what I will be discussing will be what I have learnt from watching this talk by Venkat Subramaniam. His talk was what had originally got me into using streams and the concept of lambdas.

Let’s revise

How do we implement a lambda in Java?

Lambdas are simple constructs with very powerful use cases in Java and many other languages. Most commonly, lambdas enable the everyday programmer to reduce their clunky anonymous inner classes into simple one-liners. It can also be used to pass methods around between methods without having to redeclare these method over and over again.

The common components that make up a lambda in Java are:

A functional interface
A method that matches the signature of the method in the functional interface

What is a functional interface?

A functional interface is an interface that contains a single method.

public interface StringOp {
  String perform (String in);
}

Using a lambda:

Code:

public interface StringOp {
  String perform (String in);
}

public class LambdaDemo {
  public static void main (String[] args) {
    StringOp operation = in -> new StringBuilder(in).reverse().toString();

    System.out.println(operation.perform("Hello World"));
  }
}

Output:

dlroW olleH

In this example, I created a functional interface (StringOp), declared an instance of that functional interface (operation) and gave it’s definition all in one line using a lambda.

The core syntax of the lambda is as such:

(parameters) -> {
  actions
}

In cases where there is only 1 parameter, the parantheses can be omitted, as seen in the example, and if the method body a single line, you can also omit the curly braces.

If lambdas did not exist, I would have to declare the method like this:

StringOp operation = new StringOp () {
  @Override
  public String perform (String in) {
    return new StringBuilder(in).reverse().toString();
  }
};

operation.perform("Hello World");

These are just simple examples of what lambdas are capable of, there a many more uses for them and you can check out a more comprehensive guide here: https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html

Starting streams

I will first begin by showing an example of a typical program and then showing the power of streams and how they can be used to simplify your work.

Problem: Write a program to print out all numbers that are multiples of a given number within a given range.

Traditional Solution:

public class Solution {
  static void printMultiples (int multiple, int upper) {
    for (int i = 1; i <= upper; i++) {
      if (i % multiple == 0) {
        System.out.println(i);
        }
      }
  }

  public static void main (String[] args) {
    printMultiples(2, 10);
  }
}

Streams Solution:

public class Solution {
  static void printMultiples (int multiple, int upper) {
    IntStream
      .rangeClosed(1, upper)
      .filter(i -> i % multiple == 0)
      .forEach(System.out::println);
    }

  public static void main (String[] args) {
    printMultiples(2, 10);
  }
}

Output:

As you can see, both methods produce the same output, however, the latter is a lot neater than the former. Not only is the solution simpler to understand than the solution that introduces loops and if statements, it is a lot easier to read and understand.

Explanation:

The reason why methods like .filter(i -> i % multiple == 0) works is due to the use of functional interface as mentioned previously. According to the Java documentation on streams, .filter() receives a Predicate interface as a parameter.

Represents a predicate (boolean-valued function) of one argument.

This means in order to create a lambda that receives one argument and returns a boolean condition.

Method references

Another unusual syntax you might have noticed is this forEach(System.out::println), you might be scratching your head and wondering that this :: symbol is doing. Well, it is known as a method reference. The core idea with method references would be as Mr. Venkat put it

Since the value is a simple pass over, you can use a method reference.

To illustrate this, let’s see how you would use the .forEach() method normally:

String[] menu = { "Pizza", "Cola", "Salad" };

Arrays.asList(menu)
  .stream()
  .forEach(menuItem -> System.out.println(menuItem));

Output:

Pizza
Cola
Salad

As you can see, for forEach(), the menuItem argument you receive is simply being passed onto the System.out.println method call, and since no other modification is being made to this menuItem value, you can use a method reference to System.out.println to shorten to code.

In this particular instance, since println is a static method of the System.out object, the method reference will be a reference to a static method, which means the syntax would simply be having the object name followed by the :: symbol and then the target method name.

jiahao.blog

The Math Behind Burn Rate Detectors

SLOs and error budgets

Why error rates don’t scale well

Why not increase the lookback window?

What about enforcing that errors need to be firing for an extended duration?

It’s a bird, it’s a plane, it’s a burn rate detector!

Tiered burn rate detectors

Reducing false positives

Conclusion

Further reading

Unified Data Modelling with Elixir

Traditional API response data modelling

Introducing… unified data modelling

Macros recap

Step 0: Creating the model base

Step 1: Declaring fields

Step 2: Declaring the struct

Step 3: Converting API to internal fields

Step 4: Declaring @type t

And…

Running Docker in GitHub Action steps

A silver lining?

So, what do we do?

Wait, what’s wrong?

What the title promised

Packaging and Publishing on Debian

What will be covered?

Prerequisites

Core concepts

Creating the upstream tarball

Creating the debian/ directory

Creating the .deb file

Setting up the Debian repository using reprepro

Publishing the Debian repository to GitHub Pages

Multi-architecture support

Multi-architecture packaging

Multi-architecture publishing

Automating the packaging and publish process

Setting up the source repository

Setting up the Debian repository

Conclusion

Appendix: Setting up a Debian VM on MacOS Apple Silicon

Appendix: Packaging workflow

Unpacking Options in Python

Building Components for Markdown in Astro

Problem

Markdown processing

Using directives

Setting up remark-directive

Plugins

Creating a remark plugin

remark plugin skeleton

Detecting directives

Creating wrapper elements

Using the plugin

Creating a rehype plugin

rehype plugin skeleton

Detecting elements

Storing tab keys on load

Loading and updating selection

Loading the plugin

Styling the component

Closing

Understanding Systems with Sequence Diagrams

Practical sequence diagrams

Mermaid crash course

Sequence diagram fidelities

Low-fidelity

Medium-fidelity

High-fidelity

General guidelines

Closing

Evolution of Resumes

General advice

Polytechnic year 3

Post polytechnic

University year 1

University year 2

University year 3

Step 4: Declaring `@type t`

Creating the `debian/` directory

Creating the `.deb` file

Setting up the Debian repository using `reprepro`

Setting up `remark-directive`

Creating a `remark` plugin

`remark` plugin skeleton

Creating a `rehype` plugin

`rehype` plugin skeleton