Mike Lowen

Books for getting involved in Software Architecture

2025-05-10T00:00:00+12:00

Earlier this week I had the opportunity to speak at the Auckland JuniorDev meetup where I presented 10 Things I wished I knew about architecture earlier in my career (slides). I’m a bit rubbish with speaker notes (e.g. I don’t do them) so the slide deck is lacking a bunch of context, I’m planning to remedy that in a follow up post.

I had a lot of awesome discussion afterwards with the attendees regarding career growth and getting more involved in architecture with lots of asks for book recommendations so here are my recommendations on books to start you down that path:

The Phoenix Project & The Unicorn Project - Must reads for any technologist.
Staff Engineer - A great overview/guide to leadership outside of the management path.
Domain Driven Design Distilled - A really useful overview of DDD which has underpinned a lot of my thinking.
Fundamentals of Software Architecture - A good general purpose guide to software architecture.

I appreciate all of these books enough that I own physical copies of all of them. There are plenty of other books that I also think are great or I have on my reading list but these would be my starting point.

Note: this was adapted from a post I orginally put on linkedIn, I believe in also owning my own content on my own domain so I’ve also posted it here.

Test data factories in javascript

2022-05-14T00:00:00+12:00

Friday the 13th was a quiet evening and when the rest of the family were in bed or watching TV I chose to play around with a bit of the thought experiment. This time unlike others I wrote up what I was doing as I went along, what follows is a tidied up and expanded version of the notes that I took. While I was doing this I was primarily using two tools - RunJS for my javascript REPL playground and Typora for my notes.

In the ruby world I’m a fan of factory_bot for generating objects that will serve as test data in my unit tests. I had a bit of time to kill this evening and thought it would be fun to implement the subset of the functionality that I use in factory_bot in javascript to see how I would approach it.

For those unfamiliar with factory_bot is an implementation of the factory pattern to generate objects as alternative to using fixtures in tests. This approach allows you to generate your test data in a known and expected way will allow you to overwrite any properties of the object as appropriate for the test. This approach allows you to remove a bunch of duplicate code from your tests. As an example of how I would use this in my current project I have a front end driven by APIs below is a sample payload from one of the APIs

	{
  "_links": [
    {
      "href": "/clients/898f3b58-6c16-40f7-bf29-c9dac868bb2c",
      "rel": "self"
    },
    {
      "href": "/clients/",
      "rel": "up"
    },
    {
      "href": "/organisations/1f7c161a-8de1-44ff-9ac3-e36ecc01dfdc",
      "rel": "organisation"
    }
  ],
  "_type": "urn:emboss:client",
  "name": "Testing",
  "oauth": {
    "id": "738542d0-e1fb-40c3-8786-8b47119d9151",
    "secret": "************************f8c644399992"
  }
}

Rather than have multiple copies of this object in the codebase each just slightly different to the last, what I’d like to do is have something like the following:

const payload = build(payloadFactory, { name: "Development" });

Where the second argument contains any overrides you want to make to the base object. I’ve got a bit of a quiet evening so lets play around with this idea, first lets define what we want a factory to look like. Considering that I’m restricting the functionality to only building objects what’s wrong with what was defined in the example payload? It’s simple and it works, we can assign it to variable call it our factory and pass it into a build function with some overrides to modify some of the properties to create a resulting object. Following that line of thinking the first version of our build function is as simple as:

function build(factory, overrides = {}) {
  return { ...factory, ...overrides };
}

As a generalisation this works great until we want to have more than one instance of our object, the simple solution to that is you overwrite more data to make each unique but you end up finding yourself having to do that a lot, wouldn’t it be neat if we could build in some automated uniquness into our factories? To acheive this what I’ll do is extend our definition of a factory such that if any property of the factory is an function then it will be called to generate the value of the field in the resulting object. A simple example would be this factory:

const factory = { foo: 1, bar: () => true };

would result in

{ foo: 1, bar: true }

This would mean that should we actually want a property of the resulting object to be a function we’d have to do something like:

const factory = {
  foo: 1,
  bar: () => { 
    return () => console.log("I'm a method on the object!");
  }
}

What would that mean for our build method though? That gets a little more complicated now, first we want to generate the appropriate value for any field, which would take the key of the field, the factory which defines it and finally the overrides the method to do that would look like:

function populate(key, factory, overrides) {
  if (key in overrides) { return overrides[key]; }
  if (factory[key] instanceof Function) { return factory[key](); }

  return factory[key];
}

Next we want to construct the object from the factory based on the properties in the factory like so:

function build(factory, overrides = {}) {
  return Object.keys(factory).reduce((obj, key) => {
    return {
      ...obj,
      [key]: populate(key, factory, overrides)
    };
  }, {});
}

This will do what we want, it will allow you to use a library like Faker to generatedata to populate your object, it may not be the fastest way to do it but I think it’s pretty legible. A nice side effect of this approach is this allows our factories to use other factories to define some of their properties, like so:

const factory1 = {
  foo: 1,
  bar: () => true
};

const factory2 = {
  tar: "string",
  baz: () => build(factory1)
}

The next step in complexity for our factories is to introduce dependencies, there are times when you want to populate data in your factory in a specific order, a simple example of this could be a factory like:

const factory = {
  firstName: () => faker.name.firstName(),
  lastName: () => faker.name.lastName(),
  fullName: () => '??'
}

We don’t have a way to set it up so that fullName will be populated with firstName + ' ' + lastName , my thoughts around this would be to declare the dependencies on the method like so and pass the in progress object in like so:

const factory = {
  firstName: () => faker.name.firstName(),
  lastName: () => faker.name.lastName(),
  fullName: (obj) => `${obj.firstName} ${obj.lastName}`
}

factory.fullName.dependencies = [ 'firstName', 'lastName' ];

It would also be nice if that was wrapped in a helper function so the factory looks (IMO) cleaner:

const factory = {
  firstName: () => faker.name.firstName(),
  lastName: () => faker.name.lastName(),
  fullName: dependantProperty((obj) => `${obj.firstName} ${obj.lastName}`, [ 'firstName', 'lastName' ])
}

This however breaks our current build function, first we don’t pass the object into the populate method and secondly we have no gaurantee on the order in which the properties will be populated. To solve this I’m going to order the keys to populate them in the appropriate order (though I’m not handling circular dependencies). Then to make sure that the population methods have the appropriate context to populate the object so it will now pass the object into the method when populating.

function populate(key, obj, factory, overrides) {
  if (key in overrides) { return overrides[key]; }
  if (factory[key] instanceof Function) { return factory[key](obj); }

  return factory[key];
}

function build(factory, overrides = {}) {
  const order = (prev, current) => {
    if (prev.includes(current)) { return prev; }
    if (factory[current] instanceof Function && factory[current].dependencies) {
      return [ ...factory[current].dependencies.reduce(order, prev), current ];
    }

    return [ ...prev, current];
  }

  return Object.keys(factory).reduce(order, []).reduce((obj, key) => {
    return {
      ...obj,
      [key]: populate(key, obj, factory, overrides)
    };
  }, {});
}

Again I’ve prioritised readability over performance, this was actually my second attempt my first attempt combined everything into a single function and felt very messy. Lastly I with the build method only populating the resulting object with the fields present in the factory I thought it gives us the oppotunity to pass additional context data into the factory via the overrides parameter and then pass that along to the property population method, allowing you to write population methods like so:

const factory = {
  name: dependantProperty(
    (obj, context) => context.nickName ?? `${obj.firstName} ${obj.lastName}`, 
    [ 'firstName', 'lastName' ]
  ),
  firstName: () => faker.name.firstName(),
  lastName: () => faker.name.lastName(),
}

build(factory, { nickName: 'Bob' });

Another neat aspect of the approach taken to defining factories is that you get extending factories for free by using the spread operator. If we take the above factory and wanted a new factory that extends it to includ an email address we could achieve it like so:

const subfactory = {
  ...factory,
  email: () => faker.internet.email()
};

This was a fun thought experiment about how I could bring some functionality from a library I like in a different language into javascript. It isn’t however a 1-1 port of the functionality of factory_bot just the pieces that I found interesting to play around with in an evening. I’ve setup the code in a git repository if you want to have a look at it in its entirety, given time I may come back and further tune it and publish it to NPM.

Skills to focus on for staff+ roles

2022-04-10T00:00:00+12:00

Recently a senior developer at work who is on the individual contributor (IC) path asked for advice as what they should be focussing on as they progress towards a staff+ role. Reflecting on the feedback I realised that with a bit of generalisation it would be useful for anyone looking to step up into a staff+ style of role. With the permission of the developer who requested the guidance here is an edited and slightly expanded version of the feedback I gave.

A bit of context around the advice given the company I work for is a SaaS product company currently going on a journey to transform our monolith into a service based architecture, this does influence some of the advice and recommendations that have been given.

I’m the sort of person whose learning style suits asorbing information from books and then using little projects (when I can) to apply that knowledge. As such I make recommendations of a few books throughout this post, these are all books that I have myself.

As you progress your technical skills while important become less relevant in many respects. In my mind what differentiates an intermediate from a senior from a staff+ engineer is increasing less about technical skill as you go along and much more about what are traditionally called “soft skills”, though I’ve never been a big fan of the term as I think these are the skills tend to be more difficult to master. As a generalisation communication is always a good thing to keep working on as it will become probably your core tool in your toolbox. The IC path is one that is centered around influence without authority and most of what you end up doing is trying to convince people to do things in the way you suggest.

In terms of specific skill sets that become more prominent when working in staff+ roles it’s my opinion that you would want to focus on the following.

Strategic thinking

As you progress down the IC path you will have more and more influence on company and engineering strategy so it is important to get your head in that space. This can range from how do you ensure you are supporting the broader strategy and how do you foster and create the appropriate strategies and vision for engineers informed by what the company is doing. Part of the role of a staff+ engineer is to provide the guiding light to rest of the engineering capability. A good write up around formulating strategies can be found at StaffEng, a book mentioned in there that I have also had recommended to me by others is Good Strategy / Bad Strategy, I must admit this is one on my to read list but I’ve had enough people I respect talk well about it that I got a physical copy.

Product thinking

You will find yourself making more and more product decisions or at the very least in assisting in them. While a product manager is ultimately accountable for a product roadmap or strategy as a staff+ engineer I believe it is a shared responsibility between the two roles to develop those artefacts. You’ll also find yourself being asked more and more product orientated questions by engineers as well so it is important that you build up that product skillset especially working if you continue working for product companies. A few books I might recommend are:

Tradeoff analysis

This again is another key skill set that will come into play, to quote Neal Ford

There are no right or wrong answers in architecture—only trade-offs.

Understanding the tradeoffs is also about how do you influence where you system is going in the future. Many times this may be unintentionally setting the direction for the architecture for the system. Where possible you should avoid concreting over a door and making a decision reversible if possible. When making faced with deciding between tradeoffs and making decisions a good question to ask yourself is “do I need to make this choice now?”. Sometimes it’s better to wait to gather more information and make an informed decision and wait for the last responsible moment.

One of the core responsibilities of a staff+ engineer in my opinion is balancing emergent vs intentional architecture. Intentional architecture is what you set out to build initially and emergent is what presents itself as you begin to actually build the system. In my experience most of the time the emergent architecture is usually the preferred option, it usually presents itself due to the discovery of new information. At the heart of this balance is making sure you understand the tradeoffs of following either path.

During delivery this quite often presents itself as when do we take on technical debt, which is always a tough decision and one where the manta “Perfection is the enemy of good enough” comes into play. At the end of the day the most important thing is deliverying business value, the code and architecture can be perfect but if it’s not out in the world being used it doesn’t really matter. You can always come back and fix up incurred debt (it’s a different problem if not given that opportunity). In my experience and opinion the systems that usually have the higher levels of tech debt are new systems which brings to mind a quote by Reid Hoffman:

If you are not embarrassed by the first version of your product, you’ve launched too late.

Now all this is not to say you should always take the option to incur technical debt but it’s important to understand the tradeoffs when making any decision. I think it was Neal Ford or Sam Newman who also said that (and I’m paraphrasing):

Every decision has trade offs, if you think you’ve found one that doesn’t it just means you haven’t found it yet.

Systems Thinking

Perhaps of all of the areas of focus here this is the one that comes most naturally to a developer, the key here is lifting up the level of abstraction which you are working at. Where you may be used to working at the class/function/story level you are having to hoist yourself up to understanding the impact on the entire system. I’d go so far that you need to make sure that you are operating at a level of abstraction where you understand the impacts on the people and process of either the company or the customer.

Stakeholder Management

Thinking on all of the above another of the key responsibilities of a staff+ engineer is stakeholder management. You’ll be wanting to understand who they are balancing their various needs and wants and how that will impact the system that is being built. This also includes making sure they understand the impacts that technical strategies and decisions will have on them. One of the things I see people new to the role overlook is that the development teams are also stakeholders that you need to account for.

General Advice

From here I would like to offer more generalised advice from my own journey that I hope you find valuable. Starting with some advice that I got when I first moved into an IC leadership role regarding the developers in my team (I had just stepped into a technical lead role):

Just because it’s not the way you’d do it, doesn’t mean that it’s wrong.

This is something that I still struggle with at times (I think it comes from being overly opinionated when the mood takes me), it is something that I’ve needed to loosen up on. Though loosening up can also be detrimental at times, where teams may go too far off the reservation, which is likely more to do with a lack of guard rails for the team to operate within. I’ve also in my time come to value a hunger to learn and adapt over straight skill sets when it comes to working with developers. Skills can be taught but that type of mindset it a lot harder to embed in someone who doesn’t have it.

An architects or staff+ engineer (I consider the roles to be largely analogous) is a very amorphous role and will generally change to fill the “gaps” in an organisation this has also been referred to as being glue. While your focus will be on the more technical side of things, the people side should not be ignored. Much like DevOps can be described as the intersection of people, process, and technology the same can very much be said about architecture - after all that’s why Conway’s law is a thing. You will find yourself involved and helping to guide and define organisational change it can be helpful to have a view on how these sort of things could work and the impact it will have on how you build your systems, I’ve found team topologies to be one that works well for my mental model.

Now I know I said that I don’t think that technical knowledge isn’t as important it is not something that should be neglected, here are the list of technical books that I’ve found to be useful in my role/career:

And because I do love me some dead trees on my shelf here are a set of books I have on my backlog that I’m working my way through slowly but surely that I think could be applicable/interesting.

Staff Engineer (Currently reading)
EDGE: Value Driven Digital Transformation
Project to Product
Building Event Driven Microservices
Coders
An Elegant Puzzle
Lovability
The Mom Test

Introducing Emboss

2022-01-13T00:00:00+13:00

I would like to introduce a side project that I’ve been working on Emboss. Emboss provides a RESTful API to transform HTML & CSS into a PDF and a management UI on top of it. This is a tool that is intended to be integrated into a broader system that needs to generate dynamic PDFs.

Now the first thing your asking is why HTML? There are already libraries available for generating PDFs. That there is but those libraries generaly have their own DSL that involves precisely specifying exactly where to draw things on a document that feel anything but user friendly, whereas HTML is a common technology that many people already know and have to work with. It also allows us to open up who can build the underlying PDF from developers to anyone who knows how to write HTML.

The next question you’re probably asking is couldn’t we just use wkhtmltopdf and be done with it? Sure if you really want to, it’s a good and popular tool, but do you really want to build more software that you have to maintain? Further below in the post I’m going to go through my current roadmap for Emboss that goes beyond the capabilities that go beyond what you get from a tool like wkhtmltopdf

I’ll do a longer post at some point but at a high level Emboss is comprised of an RESTful API using Python & Flask, an application with a React + Redux client and a Node back end managing sessions and acting as a backend for frontend. It also utilises Keycloak as the identity provider.

Now without further ado, I’d like to present my first demo of Emboss. In this demo I walk through the first time experience of a user signing into Emboss:

Creating an organisation.
Creating a client.
Transforming some HTML into a PDF.

While this demo walks through the application itself I’ve built out more of the API layer which you can find the spec (I wouldn’t call it documentation yet) on stoplight.

The Why

In my day job I don’t spend much time hands on keyboard writing code, I still get to do some but not a lot. Most of my time is spent in conversation with folks or writing about patterns, practices, architecture, and strategy. In my opinion the goal of a role like mine (I’m currently a principal engineer) is to enable others to help and allow them to build great things.

I also believe that for me to be useful and relevant in my role I also need to keep cutting code to try new things and keep my skillset reasonably current in an effort to not end up holed up in an ivory tower disconnected from the changes that happen in our industry.

Usually with my side projects I generally solve what I think is the interesting problem and then lose interest and put it on the metaphorical shelf to gather dust. As such my hard drive is full of half finished projects, in September Sara Chipps tweeted the following:

What is the difference between a Staff level engineer and a Manager?
.
.
.
.
.
.

One of them thinks they code for a living.
— sarajo.eth (@SaraJChipps) September 3, 2021

I’d be lying to say this tweet didn’t hit a little too close to home and while it wasn’t the trigger for this project it did light a fire in me to prove to myself that I want to ship something, if for nothing else to remind myself that I can do it. In this case I’m being generous with myself and counting this blog post and the demo video above as shipping something I do plan to deploy Emboss for others to use in the future this is just the first stepping stone in that direction.

So far I’ve written very much about the personal motivation to build something, but not Emboss in particular. The reason for that is a lot shorter, during my career I have seen the problem of generating PDFs arise multiple times and each time whoever I’m working for ends up rolling their own solution to this problem. Now there are solutions that already exist in this space but for a variety of reasons they don’t quite provide what I believe is required in this space, so I thought I’d take a crack at it.

What’s Next?

What I’ve presented in this demo is very bare bones and not yet MVP, there’s still plenty of functionality that I want to build. In rough priority order the road map contains the following items:

UI for managing user access and permissions (the API is already built).
Accept a JSON payload for generating a document, this will allow for consumors of the API to supply both HTML and CSS as seperate attributes along with meta attributes to resolve relative links and set meta data on the PDF.
Add templates, this would allow users to manage the templates within Emboss and then when making the request to generate the PDF the request only needs to contain the data to populate the template.
Support “meta” formats for the content in templates and when generating PDFs, this would include supporting things like CommonMark and Sass.
Storing generated PDFs.
Supporting batch jobs, this would allow a user to specify a template and supply a list of datasets used to generate multiple documents in a single submission.
Data regarding the number of documents that have been generated.

One of the bigger questions I am still to answer in regards to this project is how do I want to make this available? Is this something that I want to run as a side hustle, offering it as a SaaS with paid subscriptions? Or do I want to package it up and make it available as an open source solution? At this point if I’m honest I’m not sure.

I plan to blog at least monthly with a demo video to provide an update on my progress, I hope you follow along on the journey. If you have any feedback or feature suggestions please feel free to reach out.

The Other API Version

2021-12-29T00:00:00+13:00

Versioning RESTful API endpoints kind of sucks, there are a bunch of ways to do it all with different tradeoffs which are largely semantic and no real consensus on the “right” way to do it. Thinking about versioning endpoints brings to mind that Winston Churchill quote about democracy:

democracy is the worst form of government – except for all the others that have been tried.

Where you can replace democracy & government with your preferred style & versioning respectfully. I think this largely comes from the fact that there isn’t an ordained way to go about making breaking changes to your API outside of either don’t or version it.

Endpoints though are not the only thing that should be versioned in an API, you should also be thinking about how you want to version your data. If you aren’t versioning the data in your API and taking the default naive approach you are left in a place where the last write wins where whoever sends their request last will potentially overwrite any previous changes. You can visualise the problem like so:

In our hypothetical example both Client A and Client B have fetched and modified the same resource from the API but because Client A submitted their change last they have overwritten the changes made by Client B and none are the wiser Client B doesn’t know their changes have been overwritten and Client A didn’t know they weren’t working on the latest version of the resource. In the worst case scenario from there those two clients may continue working working on the increasingly divergent resource each time overwriting the others changes.

This issue is most prevalent when updating an entity using the PUT method where you are replacing the entire resource (as opposed to PATCH which is only a partial modification). You could find yourself changing fields back to a previous state that someone else had modified.

Unlike endpoint versioning HTTP does provide us a couple of “right” ways to version the data in our APIs utilising standard headers. The first of these is If-Unmodified-Since this header accepts a datetime as a value that should correspond to the the datetime which the client retrieved the resource, when updating the resource if the last changed time on the server is more recent than that supplied in the header the change is rejected.

While this is likely this is the easier of the two options to implment (IMO) it doesn’t completely mitigate the last write wins problem. With this approach we are reliant on clients to track when they retrieved the resource, if the client so chooses they could just specify the current datetime when making the update and the API would be none the wiser.

The second approach utilises two HTTP headers ETag and If-Match. The ETag header is supplied by an API when returning a resource and the value is intended to represent the version of that resource. What you use to represent that version is up to you as an API implementor, I often use a UUID that I store with the resource in the database but more commonly you will see APIs use a hash of the content or a revision number. The If-Match header is the other half of the solution, the client uses this header when sending the update to the API and it should contain the version of the resource (received via the ETag) that the client has modified. If the version supplied in the request does not match what the API is expecting then it will reject the update.

My preference is to use the ETag / If-Match combination as while it is a little more effort to implement it provides better protection against overwritting other clients changes to resources. Regardless of how you approach versioning of data in your API I think that it’s important that you do version the data. It provides clients integrating against your API a consistent and expected experience where they can rely on any data they submit not being accidentally overwritten.

Autoloading Handlers in Tornado

2020-02-02T00:00:00+13:00

In one of my current side projects I’m building out a service written in Python which exposes a RESTful API. In situations like this the library that I reach for is Tornado as I find it to be lightweight and doesn’t do more than I need it to (much like my ruby preference Sinatra).

When structuring my Tornado projects I create a module per handler and restrict a single handler to a single resource (I would count /url-path-a/, /url-path-b, & /url-path-b/id to all be separate resources). The basic file structure will look something like the following:

/-
 | - app.py
 | - handlers/-
              | - __init__.py
              | - handler_one.py
              | - handler_two.py

In the handlers/__init__.py I import the handler classes from the respective modules so I can write handlers.HandlerOne rather than handlers.handler_one.HandlerOne. Tying it all together is the app.py which defines the routes (the mappings between URL and handler), in it’s simplest form looks it like the following:

import asyncio
import tornado.ioloop
import tornado.web

import handlers

if __name__ == '__main__':
    app = tornado.web.Application([
        (r'/one', handlers.HandlerOne),
        (r'/two', handlers.HandlerTwo)
    ])

    app.listen(3000)

    tornado.ioloop.IOLoop.current().start()

You could imagine with an app of a reasonable complexity that this could become a sizable list. There are a couple of things that I’m not a fan of with this approach. The first is that adding a new handler requires touching three files, two of which are minor tweaks and something that could be easily forgotten. I’m a fan of convention over configuration and would prefer a situation where I add a file to the right directory and it is all wired up automatically. The second is that I like to have the definition of the route to be as close to the handler as possible so that when working on the handler you can have as much context as possible without needing to jump around the code base.

While neither of these are big annoyances (they aren’t deal breakers by any means) I thought this would be an interesting opportunity to play around with solving these issues with the reflection capabilities built into python. To begin I moved the definition of mapping of the route to the handler to be closer to the handler, for those unfamiliar with Tornado a route is defined as a tuple. After defining the handler I added the route, assigning it to the ROUTE constant like so:

from tornado.web import RequestHandler

class HandlerOne(RequestHandler):
    def get(self):
        self.write('Hello from one')

ROUTE = (r'/one', HandlerOne)

Each module that contains handler should also define the ROUTE constant and later we will use that convention to determine if we can load a route from the module.

Now that we have our routes defined we need to load them. To do this we will add some code to the __init__.py of the Handlers module which will be responsible for loading the routes. All of the routes that are loaded from the modules will be added to a ROUTES constant that will be exposed from the Handlers module. The first part to do is to iterate through all of the files in the directory and skip any files which are not python files:

FILES = listdir(dirname(realpath(__file__)))

for name, extension in [splitext(f) for f in FILES if f != __file__]:
    if extension != ".py":
        continue

Once we are only dealing with the python files we are able to load them in to the running application:

mod = import_module("{0}.{1}".format(__name__, name))

I’m using the the import_module method from the importlib package in the standard library to load the files into the application. This method takes the name of the module to load and returns a reference to the module. Because in python a module maps to a directory or a file we can use the file name (sans extension) for the last part of the module and as we are running this from __init__.py we know the name of the parent module can be found in the __name__ variable so we can create the full name of the module to be loaded without hard coding any module names.

Once the module has been loaded getattr is used to load the ROUTE constant we defined earlier:

route = getattr(mod, "ROUTE", None)

I am using getattr rather than using mod.ROUTE because the final argument to getattr allows us to define what will be returned if ROUTE has not been defined in the module. In this case I’ve specified None as the default value. If we just accessed mod.ROUTE and it did not exist on the module an AttributeError would be thrown that would need to be handled.

We’ve loaded something from the module that we believe is a route, we need to verify this before we expose it to the wider application. We do this by first checking that the module actually does expose a route (e.g. the value of the route variable is not None) and that it is of the type we are expecting. Earlier I mentioned that a route is defined as a tuple, we can use isintance to test this is the case.

if route is not None and isinstance(route, tuple):
    ROUTES.append(route)

When we bring all of the individual parts of loading our routes together the contents of the __init__.py look like the following:

from os import listdir
from os.path import dirname, realpath, splitext
from importlib import import_module

ROUTES = []

if not __name__.endswith("__init__"):
    FILES = listdir(dirname(realpath(__file__)))

    for name, extension in [splitext(f) for f in FILES if f != __file__]:
        if extension != ".py":
            continue

        mod = import_module(".{0}".format(name), __name__)
        route = getattr(mod, "ROUTE", None)

        if route is not None and isinstance(route, tuple):
            ROUTES.append(route)

You’ll note in the full example that we are not running this if the __name__ variable ends in __init__ this is being done so that we don’t try and run this code multiple times which will cause it to throw a run time exception.

This then has an impact on app.py where it no longer is has to be responsible for defining all of the routes for the application, leading to something which looks much cleaner in my opinion:

import tornado.ioloop
import tornado.web

import handlers

if __name__ == '__main__':
    app = tornado.web.Application(handlers.ROUTES)
    app.listen(3000)

    tornado.ioloop.IOLoop.current().start()

The trade off with using this approach is there is no single place in code that you can go to see all of the routes that are exposed by the application. I believe that having the fuller context with each route is worth the trade off. For those who still want to view all of the routes for an application they could do so using the python shell:

For those so inclined you could also build this functionality into the app itself. A working version of the example code for this article can be found over at Github.

Autonomy and Coupling in Microservices

2018-04-23T00:00:00+12:00

When discussing microservices and communication there seems to be a sentiment that I keep seeing crop up, which paraphrasing goes something like the following:

Microservices shouldn’t communicate with one another otherwise they lose their autonomy and that defeats the purpose of using microservices.

I can’t help but think that those who say this sort of thing are somewhat missing the point, while autonomy is important (more on that in a moment) when talking about communication the focus should be on loose coupling between services. Now don’t get me wrong, if you find that there is a lot of communication between two services this can be a smell that your modelling may not be correct, but in any sufficiently complicated system there will be some level of coupling incurred. If we go back to the sentiment that I mentioned earlier it’s my opinion that those making statements like this are often conflating autonomy and coupling, so I want to define my interpretation of those two terms in this context.

Autonomy - If a service has the ability to run independently of any other service and if it or any other service fails it does not cause the entire system to fail.
Coupling - Is the dependence on the implementation of a service by another. When services are tightly coupled a change in one necessitates a change in another, when loosely coupled a change in one service can be made independently of any other service.

When we consider those definitions and revisit the original sentiment I think that there are two things at play here, the first is that autonomy and coupling are being conflated and the second is the assumption that any coupling between services is tight coupling. In order for the latter not to be true there does need to be some guidelines around communication using implementation agnostic approaches (e.g. HTTP, RabbitMQ, etc). One argument that I’d like to address that is often brought up when discussing communication between services is that when one service needs to communicate you incur deployment coupling and the services will no longer be able to be deployed independently. I find this notion to be something of a red herring, there is some truth to it in that there will be deployment coupling however this is a once off scenario; Once the upstream dependency has been deployed both services can once again resume being deployed independently, in the situation where you are unable to wait for the upstream dependency there are techniques that can be used to mitigate that situation.

In the remainder of this post I want to go through some of the approaches you can take when your services need to communicate with one another.

Database

I thought we’d get this one out of the way first, there is always the option to access the database of the service that you need to interact with. I’d strongly advise not to do this as it would be tightly coupling your services together where a change to your underlying data structures could necessitate changes in multiple services and quite likely also the reimplementation of the same business logic in multiple places making changes to that logic more costly. Once you start down this path you are effectively creating a distributed monolith where you suffer from the drawbacks of both architectural styles.

Synchronous APIs

This approach sees your service interacting with the upstream dependencies synchronous APIs (e.g. RESTful API) on an as needed basis and is best used when you require data from your upstream service (which could be creating/updating/deleting its own concerns) to complete the transaction, in this situation this approach tends to be easier to implement than the equivalent asynchronous approach.

The cost of it is that you incur runtime coupling between both of you services as such they both need to be running for your service to complete its transaction, the level of complexity within a service does increase when your service requires multiple upstream dependencies to modify their concerns, careful thought needs to be given to how to handle the situation where one of the dependencies fails.

Graceful degradation

When interacting with another service via synchronous APIs it’s my opinion that you need to implement graceful degradation to be a good citizen. For those of you who are unfamiliar with the term of graceful degradation it is when you limit the functionality provided by your service when its dependencies are not available rather than taking the entire service offline. An example of this would be service A depends on service B when it is creating entity C, however it does not require service B when exposing entity C for retrieval in the case where service B is not responsive service A will continue to allow clients to perform a read however it will not allow for the creation of new entities until service B is responding once more.

Graceful degradation is a technique which you can use to get around deployment coupling, in the example above if service A is deployed before service B you may not be able to create entity C but the service can still perform any other commands which service A provides.

Asynchronous Messages

In this approach your service listens to messages published by the service that it is dependant on and upon receiving them performs the appropriate business logic, this allows a service to react to changes within another service without the burden of the runtime coupling of using asynchronous communication. In the situation where the service needs to create/update/delete an entity based on this command this approach is best suited when the result of the command is not required by the service that published the message to complete its own transaction.

In the situation where you need data from another service to complete a transaction but do not require the dependency to perform a command and latency dictates that retrieval from the upstream dependency is too slow you can use asynchronous messages to create a cache of data within your service which can be quickly accessed. The drawbacks of this approach that needs to be considered is that it does increase the complexity of your service now that it has to maintain a cache of data, this cache could contain stale data as your service may not have processed all of the asynchronous messages to bring the cache up to date.

Moving Concerns in a Microservices Architecture

2018-04-05T00:00:00+12:00

For better or worse you’ve found yourself in the situation where the concern you’re dealing with no longer fits in the service that it currently lives in. Whether it’s because something has changed with the concern or the initial modelling wasn’t correct is neither here nor there, these things happen. What happens next is what is important.

In this article I want to cover some of the techniques we can use when we find ourselves in this situation. For the purposes of this article I’m going to use the following definitions:

Commands - The synchronous communication for the service used when you want your service to do something, these are commonly mapped to the CRUD actions offered out on a RESTful interface.
Events - The asynchronous communication for the service for when you want to notify the ecosystem that something has happened in the service (usually as a result of a command or another event). These are generally offered out over a messaging technology such as RabbitMQ or Kafka.

The main scenarios that we will be covering is when the is a 1-to-1 move of the concern (lifted and shifted) from one service to another and a 1-to-many split of the concern between potentially multiple services.

Synchronisation

While this approach could fall under techniques that can be used to cater for events I wanted to address this option separately because all of the other approaches deal with moving the ownership of the concern to another service. In this approach you add the concern or concerns to their destination without removing it from the original service, then the data store for each service is populated and kept in sync by consuming the events emitted by the other services.

Personally I’m not a fan of this approach as it is effectively introduces dual mastery into your system and all of the complexity and issues of keeping two data repositories synchronised (e.g. what happens when one gets out of sync with the other or if one event breaks the other service?) and this only gets worse if you are splitting your concern in a 1-to-many fashion. In cases where I’ve seen something like this done I’ve found that it also leads to confusion amongst the developers in regards to which is the go forward option. The advantage of this approach is that compared to some of the others listed below the old service does not incur any run time coupling between the services as they do not need to be up at the same time.

Commands

Redirection

This approach is best utilised when commands are exposed by a RESTful API, when the service receives a request for the concern (in the example below a GET) the service will respond with the new location for concern for the client to request.

This approach is best used when the concern is either a straight lift and shift from one service to another with no changes to the contract or the new contract is a superset of the previous version as in the case with the semantics of a 301 redirect the client will be expecting the redirected request to behave the same as it previously did.

API Composition

This approach is best suited for the situation where when removing a concern from a service you split it amongst multiple services. The original endpoint for the concern becomes a facade across the new APIs that are now responsible for the sub-components of the concern, responsible for taking the results from the new APIs and transforming them into the shape that is expected from the original endpoint.

This approach is also applicable in the situation where a concern has moved between services but the redirect approach is not applicable e.g. the contract has a breaking change. In this case the original endpoint serves as an adaptor over the new endpoint transforming the request and response into the expected formats.

Events

Multi-Publish

In this approach the new service publishes both the event from the existing service and if needed a new event for the concern now that it has shifted.

This approach is best suited when the concern has been directly lifted and shifted or when the data can be transformed into the shape of the old event when a breaking change has been made. If the case where a breaking change has been made and additional data is needed to augment the services data in order to transform it into the original shape some care needs to be taken to ensure that you don’t start leaking concerns between services.

Republish

In the case where the concern has been split amongst multiple services and you want to avoid the risk of leaking concerns between services you would be better off modifying the original service to listen to the events being published for the new concerns and then retrieving the data required from the other services needed to publish the original event.

What next?

Now that the concern has been moved what are the next steps? No matter what approach you take there will be some level of technical debt incurred and at some point you need to retire the original commands and events. The simplest approach is to announce the deprecation of the concern and a date at which point it will be removed from the original service, putting the onus on the consumers of the concern. While at times this approach is appropriate it is not one that I particularly care for as this can lead to friction between yourself and the consumers due to potentially conflicting and competing priorities.

Alternatively you can maintain the technical debt until usage of the original concern dissipates. In the case of commands (especially when using HTTP APIs) which are externally exposed this can be tricky as you may not be in control of all of the clients interacting with your services. What you should have hopefully is monitoring of the usage of your APIs which you can use, when the usage of the original endpoints drop below a certain threshold the endpoints can be removed. In the case of events you can hopefully raise a technical dept with each of the consumers of the event and once those pieces of work has been completed then remove the publication of the original event.

Introducing Knockout-SinglePage

2015-06-01T00:00:00+12:00

I’ve found that when working on a side project I have the tendency to write a component which I think will stand really nicely on its own so I start extracting it out, I effectively had a side project from my side project. It last happened when I was playing around with commonmark and it has happened again with my latest project Knockout-SinglePage.

Knockout-SinglePage is a javascript library for writing single page applications using Knockout. Unlike many other libraries which use Knockout to achieve similar functionality they do so by building a framework on top of Knockout whereas Knockout-SinglePage takes the approach of being an extension to the core Knockout library. Knockout-SinglePage uses knockout components and the HTML 5 browser history API to provide most of its functionality so that we can keep the layer of code on top relatively thin.

In the rest of this article I am going to give you a basic introduction for setting up your first application with Knockout-SinglePage. In what is a first for me this library is available via Bower which is also the preferred method for distribution, it can be installed with the following command:

bower install knockout-singlepage

Once you have Knockout-Singlepage installed the next thing that needs to be done is to setup a page to host our application.

<html>
	<head>
		<script type="text/javascript" src="<path to scripts>/knockout.js"></script>
		<script type="text/javascript" src="<path to scripts>/knockout-singlepage.js"></script>
	</head>
	<body>
		<h1>App title</h1>
		<div id="app"></div>

		<script type="text/javascript" src="<path to scripts>/app.js"></script>
	</body>
</html>

Any content that is loaded by the application will be displayed in <div id="app"></div>. Where the magic starts to happen is in app.js, as mentioned previously Knockout-SinglePage uses knockout components so the first thing that we want to do is register some components that can be loaded.

ko.components.register('default', { template: 'This is the default template<br /><a href="/another-page">Go to another page</a>' });
ko.components.register('another', { template: 'This is another page<br /><a href="/">Go back to the default page</a>' });

Now that we have some content that can be loaded we need to setup some routes so that we know when to load the content. For Knockout-SinglePage routes are an array of objects each route at its simplest is made up of a name which identifies the route and the URL which will determine when it is displayed. It is important that the name of the route matches the name of a component so that we know what component should be loaded when we match the URL.

var routes = [
	{ name: 'default', url: '/' },
	{ name: 'userprofile', url: '/another-page' }
];

The URLs for the routes also support route parameters which in the form of a portion of the URL prefixed with a colon : like so /user/:id that URL will extract the route portion marked by :id into a variable passed to the component view model. Now that we have components registered and the routes defined it is time to start the application, unlike traditional Knockout where you call ko.applyBindings() we instead initialise Knockout-SinglePage like so:

ko.singlePage.init(routes, document.getElementById('app'));

The last parameter which identifies the element that we will be loading our components into is optional, when it is not supplied the content is loaded directly into the <body> tag. For more in-depth information about how to use Knockout-SinglePage you can visit the GitHub repository and view the README, if you want to contribute check out the issues at GitHub or raise an issue on how you think the library can be improved.

A follow up to writing AMD compliant jQuery plugins

2015-05-13T00:00:00+12:00

I had previously written about writing AMD compliant jQuery plugins where I went through writing a jQuery plugin that was exposed in the traditional method as well as an AMD module. The final version that was presented in that previous article was:

(function() {
	var init = function ($) {
		$.fn.colourise = function(color) {
			this.each(function(index, item) {
				$(item).css('color', color);
			});

			return this;
		};
	}

	if (typeof define === 'function' && define.amd) {
		console.log('Creating an amd module.');

		define([ 'jquery' ], init);
	} else {
		console.log('Loading into global scope.');

		init(jQuery);
	}
})();

That example works fine as long as you are exclusively loading your javascript using AMD modules or including the files manually. However it isn’t always this way in the real world if you are transitioning a legacy code base over to AMD modules there will be a period of time where you are going to want to have things like jQuery plugins exposed both via the traditional method and as an AMD module. I wanted to take the opportunity to update the original example so that it is exposed via both methods at the same time.

(function() {
	var init = function ($) {
		$.fn.colourise = function(color) {
			this.each(function(index, item) {
				$(item).css('color', color);
			});

			return this;
		};
	}

	amdAvailable = typeof define === 'function' && define.amd;

	if (amdAvailable)
		define([ 'jquery' ], init);

	if (jQuery)
		init(jQuery);
	else if (!amdAvailable)
		throw 'Unable to detect jQuery';
})();

The changes that have been made are only quite minor and are focused around where the exposition happens. First we moved the exposition via the traditional method out of the else statement and into its own if statement, it is here we also check if jQuery is available before loading something that wasn’t being done previously. The second change we made was to move the checking if AMD modules are available out of the if condition and into a variable, this was done so that when we check if jQuery is not available we can decide if we want to throw an error based on whether the plugin is exposed via another method.

Though the changes are minor it does make your code more friendly and easier to use for those developers who are currently in a transitionary stage where only some things are being exposed via AMD modules. The more friendly your code is to these different scenarios the more likely developers will choose your plugin to use.