Tim Bray

Real knowledge is to know the extent of one’s ignorance

2019-07-28T00:00:00+00:00

Confucius said in 500BC “Real knowledge is to know the extent of one’s ignorance”.

Photo by Glen Noble on Unsplash

This concept has come up in a few conversations recently.

As your experience and knowledge increases across a range of topics, you may not feel any more knowledgeable. Why might this be? It turns out there’s a simple explanation.

As your experience and knowledge expands, you encounter more topics and understand existing topics more deeply. Your circle of knowledge increases.

On the edge of this knowledge circle are the topics you may have encountered, but don’t know. As your knowledge and experience increases, you know more topics exist.

As your ‘knowns’ increase so do your ‘unknowns’. You know that you don’t know more :WINK:

Circle of knowledge

Ok, so this model might explain it. How does this help, what can we do with this knowledge? And our newfound knowledge of our lack of knowledge!?

Imposter syndrome

Imposter syndrome is a psychological pattern where you may doubt your accomplishments and fear being exposed as a fraud. You may attribute your success to luck and discount both your achievements and ability.

With the above model, knowing what you don’t know plugs in here. Being more aware of this balance between knowledge & unknown can help you evaluate your true abilities and value your achievements.

Dunning-Kruger effect

In the opposite direction is the Dunning–Kruger effect. This is a cognitive bias where people mistakenly assess their cognitive ability as greater than it is. Related to the illusory superiority bias, a cognitive bias where people fail to recognise their lack of ability.

I’ve encountered this issue regularly. The junior who over confidently proposes a risky or inappropriate solution. The manager who’s very strong opinion on a topic is misguided. Or the loud person at a party explaining something incorrectly.

Unknown Unknowns

Rumsfeld said it nicely with this quote

Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know.

When we understand the limits of our knowledge, we can make better decisions. Just identifying and recognising what we don’t know helps. We can then develop a plan to improve our knowledge, or at a minimum flag them as a risk.

Four Stages of Knowledge

There’s another model which explains progression of knowledge. The four stages of knowledge:

Four stages of knowledge

While there might be some ignorant bliss that goes with unconscious incompetence, it’s ideal to be aware of what stage your knowledge or skill is at. Then progress as many as possible through the stages, especially from the first stage!

There’s still issues with model.

For anything non-trivial, conscious competence is not a singular final state. Our knowledge is always incomplete. It either contains errors or the world has moved on. What was once true, now might not be.

But we can use this knowledge to help us problem solve and make better decisions. Understanding what is most likely fact, what is probable, what is unlikely and then what is false.

Build our self awareness and confidence in what we know. But we should also be humble. Be aware that we might be wrong, and be aware there is always more to learn.

Tracking software delivery and product velocity

2019-06-02T00:00:00+00:00

I’ve recently had a few discussions about velocity.

Photo by Mark Tuzman on Unsplash

The discussions were around how a company’s product velocity has been decreasing and how they aren’t delivering as fast as they need to. This is always a hot topic internally, as founders, leaders and teams are always looking to increase product velocity.

While the discussions specifically talked about product velocity, they were talking more broadly about the ability to deliver value to customers.

Agile made the term velocity popular. It’s defined as unit of work over an interval (usually sprints).

The tricky part with velocity is defining unit of work - as it is usually a relative estimation, eg. story points. If estimations remain consistent over time, then velocity can be compared over time (they don’t need to be accurate, just consistent). Over a few sprints or months this can be accurate, but over longer periods the consistency declines. Estimates change as the team, projects, product and type of work change.

The need for additional metrics

Without quality metrics, it’s all opinion. Usually it’s the loudest voice or most senior opinion that reigns, not the truth. Something I’ve experienced many times when I wasn’t tracking the right metrics.

If velocity is flawed as a metric, what data can we use to track software delivery?

All metrics can be flawed, so we want to combine multiple metrics across the full SDLC, hopefully creating a holistic picture. This should reduce bias and errors within individual metrics and give more reliable data to track software delivery over longer periods.

Customer Satisfaction: NPS or similar to measure customer satisfaction. No point efficiently delivering features that customers don’t like.
Engineering: Lead Time - The elapsed time between identifying a feature and delivery of the feature.
Quality: Issues found in production are a great indication of quality and a sustainable engineering process.
Production: Mean time to resolve or repair (MTTR) - how long it takes to address an issue.

Customer Satisfaction - NPS

Most important is to ensure we’re headed in the right direction. There’s not much value in efficiently delivering features that customers don’t use or need!

There are plenty of metrics that measure customer satisfaction. While I believe NPS is flawed, it’s no more of less flawed that other metrics and is very easy to track.

Engineering - Lead Time

Taken from Lean manufacturing principles. In the software world it’s the elapsed time between identifying a feature and delivery of the feature. The great thing about measuring lead time is that is encourages many good engineering practices:

Working in smaller batches - reduces risk & work in progress (WIP), increases throughput & iteration.
Encourages frequent deployment - as lead time measures when value is delivered to customers.
Backlog management - Evaluating adding items to a backlog & removing items that are not likely to ever be delivered.

Lead time can be optionally divided up into Cycle Time - the time between development starting and delivery. Measuring both Lead and Cycle allow tracking the time from the 3 core points of views - the customer/user (total time), product team (health of backlog) & engineering (time taken to build & deploy).

While tracking lead time encourages deployments, it’s also valuable to track deployment frequency as another individual metric.

Quality - Issues found in production

Tracking quality can be a simple as recording how many bugs or issues have been found in production. If this goes up, the warning bells should start to ring loudly.

Quality can also be tracked by deployment issues. Deployments need to be routine, tracking issues caused by release can give further insight into quality.

Production - MTTR

Once an issue is found in production, how long does it take to resolve? This is mean time to resolve (or recovery) - MTTR.

What’s great about MTTR - like the above metrics - gives insight into the wider software engineering processes as well as production and customer impact.

Data driven decisions

I like how these metrics all overlap & encourage a DevOps culture.

They each give insight into the full product engineering process as well as each critical component (customer satisfaction, engineering efficiency, quality & production). The overlap increases the quality of each metric and increases the ability to track over time. Which is the whole aim — tracking software delivery or velocity over time.

When tracked over time, they give an early indication of software delivery issues or changes in product velocity. All based on real data, not opinion or hearsay!

Starting an Engineering culture

2019-04-10T00:00:00+00:00

Over the past year I’ve been lucky enough to co-found a startup and build a range of products from scratch. While I’ve built many products from scratch, my previous most recent role was at the other end of the spectrum - with a mature company and products.

Photo by Sushobhan Badhai on Unsplash

It’s given me a great opportunity to think about what’s important to do when starting out. What patterns to watch out for. What decisions you can do early that will pay off later, what I’ve seen worked well, what didn’t work as well or wished we’d done differently.

There are a range of obvious decisions which I won’t touch on - eg. lean/agile processes, modern architectures, platforms, CI/CD, monitoring, analytics etc. Instead, will try to cover some less obvious decisions and patterns to watch out for.

Conscious trade-offs
Constantly attack technical debt
Architect for change
Define Done
Culture of shipping
Balance change and stability
Diversity
Summary

Conscious trade-offs

Should you write comprehensive unit tests for all code? Even if it is experimentation or prototyping and might not be released?

When should you worry about scaling? Does performance matter when you don’t know what load to expect?

Does this UI need to be a pixel perfect match to the design? Is it likely the design will change once customers start to use it?

Juggling trade-offs are hard. But it’s our job to make these decisions. There’s no perfect answer, and the answer is likely to change over time as knowledge improves and the product matures. But, it’s easy for the answers to become cultural and questions stop being asked. Everything is done in the same way - regardless of whether it’s appropriate. Misalignment between teams starts to creep in, as expectations and understanding is different.

What worked well for us was to ask these questions up front. Bake them into each story or feature and make sure everyone knows the tradeoffs and why.

Constantly attack technical debt

Tech debt is the cost of future rework by taking shortcuts - not to be confused with plain bad code!

If you haven’t worked on a green fields project recently, you may think it takes some time before tech debt accumulates. But, unless you have the luxury of time and know everything, exactly - there will be rework. We learn along the way. About the technology, platforms, services, market and customers.

If we know tech debt is always accumulating, we should regularly pay it down. Not holding off and scheduling dedicated time in the future. In my experience, that future time will never arrive. Like paying off your mortgage, it’s best if you chip away paying weekly or monthly. Prioritise debt as part of each sprint.

All tech debt isn’t created equally, it has different interest rates.

Some of it has a high interest rate - like credit card debt - and is expensive to maintain. Finance #101 says pay off your high interest debt first. These are the items slowing the team down or causing issues.

Other debt may have a lower interest rate and is cheaper to maintain. This debt may not be slowing the teams down or causing as many issues. Address this second.

Scheduling a percentage of every sprint to addressing tech debt keeps it under control. The other approach I like is the “boy scout rule” - leave the campground cleaner than you found it. Made popular by Uncle Bob, whenever anyone touches something, make it better. Clean up the code, rename variables, add tests, do whatever improves it. Even if it’s only picking up one empty beer can or a bottle cap next to the campfire.

Architect for change

Software architecture can be defined as the parts that are hard to change. They are a combination of factors - performance, security, scalability, requirements, data, compliance, and all “-ilities”.

It’s sometimes easy to know which areas that are likely to change. They’re the areas you know the least about. The areas that are most unknown - from either a product feature or a technology point of view.

I’ve also found the need to balance architectural design with a bias for action. When there are unknowns, it’s easy to over think and spend a lot of time guessing without improving outcomes. Make the best decision you can with the information at hand, within a reasonable time frame and that is reasonably easy to change in the future.

I’m also a big believer in writing only the code you need to write. Leverage as many services & standard libraries as possible.

Define Done

Set a baseline early of what is done. Does it include tests, logging, monitoring, analytics? How about documentation and new feature announcements? Is releasing a feature enough, or does it need to be used - and what type of usage?

It’s easy for steps to be forgotten, and teams to get out of sync. Writing down these expectations helps everyone stay on the same page.

Culture of shipping

There appears to be a natural tendency in software for everything to grow.

Executives, project sponsors & product teams tend to increase scope. Engineering teams can struggle to break up tasks into smaller pieces and start to deliver less frequently in larger chunks. Pull request size increases, making it harder to review the changes. Risk increases.

We know from Lean & Agile: batch size impacts delivery and increases risk. The larger the batch the more risk. I’ve noticed without external forces, batch sizes tend to increase and shipping slows down.

Create a culture that breaks up tasks, values iteration and rewards shipping.

Balance change and stability

A lot has been written about the time it takes teams to perform and deliver results. For example, Tuckman’s forming, storming, norming & performing phases made popular by Lean & Agile.

We want to protect teams to ensure they reach the performing phase. But, there are challenges with too much stability.

Stability is also the default state.

Beneficial team change requires work. Managers don’t like loosing their good people; people tend to prefer stability.

With stability, teams can start to optimise locally, serving their own interests over the wider group. Or begin to lack diversity in thought - and not search for improvements or challenge preconceived ideas.

We start to see tribalism creep in, where teams protect their patch. I’ve also seen conflict between teams, as teams understand their own situation but have little empathy or understanding of other teams challenges.

Team stability also creates narrow experts or domain champions. Early on having experts is great, makes routing issues easy and they are usually highly productive. But it creates risk - relying on individuals, is hard to scale and can cause attrition or stagnation. A narrow expert will usually be an awesome contributor and since they are experts, there’s very little the wider team can contribute or provide feedback on.

The ideal solution is to create a culture that expects some change, but not too much. Where teams are able to handle change and still perform. It’s difficult to balance and I’ve been caught out at both ends.

Diversity

We all know diversity helps companies and teams.

A few years ago I caught myself complaining about lack of gender diversity in my teams. While working towards addressing this, I lost track of other diversity planes.

During a high growth phase the business preferred people that can hit the ground running. We were also hiring for individuals with a strong cultural fit. We ended up bringing on a number of very similar people and forgot about all the other diversity planes: background, age, education, seniority, experience, skills.

A similar issue occurred with our referrals. Generally referrals are great by reducing some unknowns, but have limits, as people tend to refer alike people.

Set your diversity goals early and monitor. It’s far easier to create diversity when teams are small, than to retrofit into larger teams.

Summary

When you’re small and nimble it’s easier to change direction, adjust culture and fix issues. It’s the perfect time to start an engineering culture that delivers quality and produces real value.

While not a comprehensive list of patterns, these items have definitely helped me.

MacOS - Your disk space is critically low

2019-03-26T00:00:00+00:00

I’ve been receiving Your disk space is critically low messages for the past couple months.

It always occurs at a bad time, so I’d do the minimum. Clear downloads, empty the bin & sometimes reboot. This usually gave me about 10Gb free, which would last another few weeks, before I needed to repeat the process.

Annoyingly, most of the disk was used by the System.

Time for a proper disk clean up!

There are plenty of commercial tools that will help. But, wearing my engineer’s hat, I prefer to know how things work and wanted to do it manually.

Running through the process below, I was able to free up more than 50Gb of space.

Table of contents

The basics

Click on the Apple icon in the menu, select About this mac & then click on Storage.

This gives a quick overview of what’s using your disk space.

Click the Manage… button and follow the recommendations.

Disk Usage command

It’s easy to see what folders are using up disk space - either by the command line or Finder GUI.

In a terminal, use the du command to print out the disk usage of a folder or all folders.

List disk space and in human readable form -h and summarize -s

du -sh .

Finder also has a feature which displays folder sizes.

Open Finder, press (⌘) + J or select View > Show View Options from the menu. Then select the Calculate all sizes options. The size column will now show folder sizes.

System Files - Cache

My user cache in ~\Library\Caches was very large.

du -sh ~/Library/Caches/

Quit all apps before clearing anything in the cache. It’s apparently better to delete the contents of folders that the folders themselves - but I haven’t properly researched whether this is true. If you’re feeling lucky - clear it all, otherwise research the implications. I felt lucky & just deleted everything.

Node / yarn / npm

If you’re using Nodejs, npm or yarn you may find a large cache has built up over time. I found my yarn cache was over 8Gb and full of older packages. My npm cache also had a couple Gb which could be cleaned.

Time for a clean:

yarn cache clean

Verify & garbage collect npm as well:

npm cache verify

Xcode Junk

Support for older devices

I had 15Gb of iOS Device Support. Most of these were for older iOS versions that we can safely delete.

Browse to ~/Library/Developer/Xcode/iOS DeviceSupport and remove older iOS version folders.

Downloads

Simulator images are also saved to this location ~/Library/Caches/com.apple.dt.Xcode/Downloads, delete any dmg files you don’t need

Old simulator runtimes

Remove any old iOS versions you no use from here /Library/Developer/CoreSimulator/Profiles/Runtimes

CocoaPods

The CocoaPods cache can sometimes become large and dirty, you can delete it then re-download everything:

pod cache -all clean
pod update

If you’re feeling lucky, can delete the whole pod cache folder here ~/Library/Caches/CocoaPods

6 tips using Apollo GraphQL

2019-03-15T00:00:00+00:00

We’re using the awesome Apollo client and server libraries in Node.js & React / React Native apps. They’ve been reliable, full featured and mostly easy to use - albeit with reasonable learning curve. Which is totally understandable since they are doing a lot for you.

Here are a few items that we didn’t immediately see in the documentation, didn’t understand the importance of, or found we needed during development and roll out to production.

GraphQL vs the libraries
1. GraphQL Playground
2. Raw GraphQL
Debugging with Postman & Chrome dev tools
Logging
Simple Authentication and Authorization
Caching
1. Our caching approach
Querying only selected fields from the database
Summary

GraphQL vs the libraries

The Apollo libraries do a lot of heavy lifting for you. This makes it easier to get started, but with this help and abstraction you can easily get lost as to what the libraries are doing and what is just plain GraphQL. We found it important to understand what functionality was provided by the libraries vs what is plain GraphQL. It’s worth reading (or at least skimming) the official GraphQL spec.

GraphQL Playground

The playground is invaluable in playing around with queries and mutations.

Add your Authorization HTTP header to query anything that is not public in your graph/api.

{
  "Authorization": "Bearer eyJhbGciOiJIUzI....."
}

Raw GraphQL

Tools like the playground make it easy to get comfortable with the language by viewing, executing and testing GraphQL. It’s also worth noting that you don’t need to use the client libraries at all.

Once you understand the language syntax, you can also write queries or mutations without using the client libraries at all. For example, here we’re calling a mutation using plain old JavaScript.

const refreshToken = apiUri => () => {
  return fetch(apiUri, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      operationName: 'authRefresh',
      query: `mutation authRefresh {
                authRefresh {
                  ok
                  data
                }
              }`
    })
  })
    .then(res => res.json())
    .catch(error => console.log(error.message));
};

Debugging with Postman & Chrome dev tools

For further debugging head to Chrome’s network tools. Like any network call, you can see the full GraphQL request and response under the network tab.

Copy the request as cURL and directly import into the great Postman API editor (or API IDE?) for further debugging or manipulation.

Logging

It’s essential to log errors and requests. We found with the default setup, syntax errors and other issues became hard to debug unless we riddled our code with exception handling. We then found this feature of the Apollo server constructor, which allows logging (& modifying) errors and responses. This greatly helps with diagnosing syntax errors or issues that occur outside your resolvers.

const server = new ApolloServer({
  typeDefs,
  resolvers,
  context: graphQLContext,
  formatError: (error: any) => {
    // filter whatever errors your don't want to log
    console.error(`[GraphQL.error] ${error.message}`, error);
    return error;
  },
  formatResponse: (response: any) => {
    // don't log auth mutations or schema requests
    const name = Object.keys(get(response, 'data') || { unknown: 0 })[0];
    if (!['authSignIn', 'authRefreshToken', '__schema', ... ].includes(name)) {
      console.log(`[GraphQL.response] ${name}()`, response);
    }
    return response;
  }
});

There are more advanced logging techniques, but this is super simple to use from day 1 and can save a few debugging headaches.

Simple Authentication and Authorization

The GraphQL specification doesn’t cover authentication or authorization. It’s up to you how to implement it, which is great (you can choose whatever method best suits your situations), and could be annoying (as there isn’t a single best practice solution).

When starting our project, we initially had all GraphQL operations (API calls) public. We then added simple authentication via Express middleware, and started to pass an isAuthenticated flag down to all query and mutation resolvers.

  async user(_, args, context) {
    if (!context.isAuthenticated) {
      throw new AuthenticationError('Not authenticated.');
    }
    ...

While this works fine, it starts to get very messy as soon as the project started to grow. We then centralised these checks into the context using gql to parse the incoming operation. The gql command takes in the GraphQL string operation and returns a GraphQL object - with operation type, operation name and selected fields (via selectionSet).

const server = new ApolloServer({
  typeDefs,
  resolvers,
  context: graphQLContext
});

const graphQLContext = (context) => {
  const { req, res } = context;

  // use a express middleware to auth user
  const isAuthenticated = Boolean(req.user);

  const { name, operation } = getOperationName(req);

  // Check here if this a public operation
  if (!isAuthenticated && !isPublicOperation(name, operation)) {
    throw new AuthenticationError('Not authenticated.');
  }

  // Could also check role based authorization

  return {
    currentUser: req.user,
    isAuthenticated
  };
};

const getOperationName = (req) => {
  const query = req.body.query;
  
  // use gql to parse the given query/mutation
  const op = gql`
    ${query}
  `;

  // this is the name the user gave for the operation
  const operationNameDefault = get(req, 'body.operationName');

  // this is the name of the actual operation
  const name = get(op, 'definitions[0].selectionSet.selections[0].name.value') || operationNameDefault;
  const operation = get(op, 'definitions[0].operation') || '';

  return { name, operation };
};

There are more advanced authentication and authorization techniques, like using GraphQL schema directives.

Caching

Early on, we ran into issues when our logic became complicated and we started dealing with eternal systems/API’s. We needed to send out requests to multiple systems, then wait and combine the responses.

The natural tendency - especially coming from REST based API’s - was to just query more often. We would re-factor components to force this and changed the fetchPolicy for queries to be network-only.

We were working against GraphQL & Apollo. The solution to this growing complexity was to leverage the Apollo Cache (and subscriptions or polling)

The default fetchPolicy is cache-first (there could be an argument that cache-and-network should be the default). Cache-first checks if the requested data is in the cache, and if so, just returns it. It’s up to you to figure out if that data might be stale.

There are a few options to dealing with potentially stale data:

Periodically clear the whole cache
Bypass the cache using network-only (or request a network call after checking the cache cache-and-network)
Prevent the cache becoming stale (see below for techniques)
Set the cache directives using the cache control extension

It’s important to set up the cache when creating the Apollo Client. If you are using guid’s then it’s as easy as this, otherwise check out the InMemoryCache options.

  return new ApolloClient({
    link: ...,
    cache: new InMemoryCache({ dataIdFromObject: o => o.id })
  });

Once we understood how the cache works, everything became easier.

Our caching approach

Understand exactly how each of the fetchPolicy values work
Install the Chrome Apollo Client Developer extension. This allows inspecting the cache as well as queries and mutations.
Consider the cache on every single call.
- Does it matter if this data comes from the cache?
- Or does it need to only come from the server?
Keep track of what variables are used for queries. Each query is cached along with it’s variables
Use fetchMore and the @connection directive to handle paginated data.
Consider the cache after every mutation.
- If we return the object that was updated/created, Apollo will automatically cache it for us.
- Manually handle when we insert new records.
Determine if it’s better to update the cache manually after a mutation using writeQuery, or refetch the data from the server.

Querying only selected fields from the database

One item that annoyed us early on with our implementation was the number of SELECT * FROM ... occurring in our solution. We’re using a TypeORM (which is a solid ORM, still with all the tradeoffs and benefits of ORM’s..).

Here’s a super simple example.

Server

// define the graphql query
const UserQueryDefs = gql`
  extend type Query {
    user(id: ID): User
  }
`;

// graphql resolver
async user(_, args, context): Promise<any> {
  // return the requested user (returns the full User entity)
  return getRepository(User).findOne(args.id);
}

Client

// # graphql query operation requesting name field
query user { 
  user(id: 123456) {
    name
  }
}

Running the query above will select all fields from the users table (via the TypeORM findOne method), then GraphQL Apollo server will throw away all the fields other than name and return just it to the client.

There is a solution to this, leveraging the info parameter on resolvers. The info parameter isn’t documented and Apollo recommend it should be used for advanced cases (we’ll see why shortly).

// graphql resolver which only selects 
async user(_, args, context): Promise<any> {
  // get an array of selected fields 
  const selectedFields = info.fieldNodes[0].selectionSet.selections.map((field: any) => field.name.value);
  
  // return the requested user - with only the selected fields
  return getRepository(User).findOne({ 
    select: selectedFields, 
    where: { id: args.id } 
  });
}

There are a few issues with the solution.

Only works on the first level of the query. For example, this won’t handle situations where related entities are also requested. This can be solved through recursion.
Doesn’t handle fragments.
Using an undocumented feature of Apollo server, which may change.

This could be a great solution make some key operations as efficient as possible, without creating separate operations. But should only be used carefully with an understanding of the trade-offs.

Summary

These are a few tips and tricks that we picked up over time. Hopefully you’ll find some of them useful.

Please note, these examples are all simplifications, and are not production ready!

GraphQL and React

2019-02-20T00:00:00+00:00

GraphQL primer.

2018-12-10T00:00:00+00:00

When we started evaluating GraphQL for a new project, it seemed to have as many detractors as fans.

The GraphQL tag line explains it as a query language for your api. But I’m not sure that really helps explain what it does, how it can help and whether it’s a good option. There’s also misunderstanding (and some misinformation) out there.

We ended up selecting it as our API platform, and have built up reasonable experience with the technology. We’ve learnt a lot of hard lessons along the way, which I’ll explain in this and a few upcoming posts.

What is it

It’s a query language or way to describe data.

Around this language are a range of libraries and tools - eg. servers, clients, services etc. These all combine to enable an API (or query system) which decouples services & allow us to ship value. Boom! Or so it says on the box.

We’ve been working heavily in the JS world (React, React Native, Node), so selected the defacto standard implementation of GraphQL by Apollo. Apollo builds the popular Apollo Client and Server libraries - which have over 300,000 downloads per week.

Apollo also offer paid services on top the platform. These services are totally optional though. You can use the open source libraries for free :-)

Why not just use REST

Strongly typed schema for free. That’s nearly enough of a reason right there. Having a contract is essential to building robust applications. GraphQL comes with a built in schema which contains all the operations (queries and mutations/updates), fields and data types.

Specify just the data you want in one operation. With REST, you typically have to fire off a number of calls, collate the data and then throw away the results you don’t need. With GraphQL you can specify just the data you need in one single call.

This does depend on your Schema design. With GraphQL you need to think about your API totally differently, not in single end points, but in graphs.

Where to start

We would’ve like these Principles to be around when we started with GraphQL. It could’ve prevented some wasted time, debate and mistakes. They’re a great starting point:

Integrity: one graph, federated & tracked in a registry.
Agility: build the graph up based on real needs, when needed. Just like evolutionary architecture!
Operations: control access, layer and advanced logging.

Probably should read the official spec, or at least parts of it. It helps to understand what is GraphQL vs what is provided by libraries and third party add-ons.

Common objections

While GraphQL has now been around for some time (available publicly since 2015) and it’s usage is widespread, there are still plenty of detractors, misinformation and varied approaches.

One criticism we heard frequently is around losing control of your back-end. Front end engineers writing random inefficient queries. Will you loose control of data and back-end services?

To fuel that fire, there are tools like this which layer GraphQL directly on top of the database. They may be good for prototyping and some use cases, but the recommended approach is toward clear separation.

The power of GraphQL is highlighted when connecting a range of disparate services - which can be GraphQL, REST or really anything.

Image taken from this great article.

This separation allows GraphQL to be an API gateway. This solves a range of issues - and works in very well with micro-services. A single endpoint for clients but full control over data owners / teams / products on the back-end. This approach also enables greater control over data loading, caching, load balancing and error handling.

Like every technology, GraphQL is not the perfect choice for every situation. If you have a very simple API that is unlikely to change, then REST is probably the best solution. If you have a larger, more complex API that will grow in size and complexity over time, then GraphQL might be the right choice.

In the next few posts, I’ll write up some of our lessons learnt.

Why we switched to GitLab

2018-09-01T00:00:00+00:00

One of the first services you sign up to when creating software is an online Source Control Management (SCM) service. GitHub is the largest and the most well-known, and was the first service we signed up to.

One of my colleagues suggested we try GitLab. I was initially apprehensive - I like to use standards and whatever the masses are using. I use BitBucket for personal projects (and liked their free unlimited private repositories - up to 5 users), but hadn’t ever used GitLab.

What made the decision easier was …pricing! GitLab has an awesome free plan: Unlimited private projects and collaborators for free. We initially would have at least a dozen contributors on our project, so this sealed the deal!

Initial Impressions

Git is Git. At least from the command line, or through a GUI (I tend to use SourceTree for visualizing changes). So initially it makes no difference what service you use. It’s the add-ons and the extra functionality that really differentiate the choices.

Photo by Quinten de Graaf on Unsplash

GitLab, BitBucket & GitHub offer Issue tracking & documentation (via wiki), but GitLab has taken a different approach by including CI/CD pipelines.

Continuous Integration & Delivery Pipelines

GitLab CI/CD Pipelines enable building, testing, deploying & monitoring code. There’s a lot going here. Back in my previous post File > New Project I said:

“It’s amazing how much CI/CD improves outcomes by encouraging regular delivery of small pieces of functionality. The DevOps mindset.” - Tim Bray, 2018

It’s essential for successful projects. Using this DevOps mindset from the start of a project is super helpful. Having this built into the platform makes a lot of sense. As long as it works…

It’s super easy to get started just put a .gitlab-ci.yml file into the root of your project and configure a Runner to execute the commands. The yml file will contain options (eg. branches to monitor) and build, test or deployment commands. Whenever GitLab sees a new commit or push in a monitored branch, it will start a new Runner and execute the appropriate commands from the .gitlab-ci.yml file.

I’m not a fan of yml, but it is pretty easy to understand. What is a painful, is testing your configuration. This is usually some trial and error, make a change, commit/push, watch how it worked & repeat. Although, GitLab does have a CI Lint tool that checks if your yml is valid.

Summary

I expected to use another tool for our CI/CD piplines. Having this inbuilt into GitLab enabled us to get our pipelines up and running quickly. Can’t argue with the price either.

I expect in future we’ll outgrow GitLab’s CI/CD and want to use something more sophisticated, but until then we’ll concentrate on coding and building our software.

Technology choices

2018-08-01T00:00:00+00:00

In my last post I talked about some of my thinking around choosing technology and how it fits alongside solid processes and tooling.

This post outlines some technology I’ve been using recently.

Tools and platforms

gitlab - git, issues, CI/CD
Google cloud platform (GCP)

Core technology

postgres (hosted on GCP)
nodejs
graphql
React
React Native for building iOS & Android apps

Additional services / tools

Senty
Stackdriver
Firebase
Fastlane for publishing mobile apps
Prettier.io - formatting code

Over time, I’ll try to build out the reasoning behind each selection.

File > New Project

2018-06-01T00:00:00+00:00

I’ve been starting quite a few new software projects lately - yea!

It’s given me a chance to reflect on how to do things right from the start. But what is right? And how do we navigate all the great technology choices we have today?

What is right?

I like to have a solid methodology and rules. This helps to generate results that are predictable and repeatable. For example, I quite like the 12 Factor App methodologies, and mostly follow the factors.

While I don’t really believe in absolutes, but here are some of my some of my non-negotiables* in software projects:

Modern architectures - break that project up, don’t build another monolith. I’m also a fan of evolutionary architecture, design for change.
Modern languages - I like to choose modern languages, but not bleeding edge. Don’t want to start a new project on legacy. But don’t want to build something where it may be impossible to get help or build a team around - when the project inevitably succeeds :-)
Code formatters, linters & static analysis - use these to reduce style issues and improve code quality
Modern Platforms - choose platforms that are going to accelerate development. I don’t want to be building anything that can be pulled ‘off-the-shelf’.
API languages - how are all the services going to talk to each other? There are some cool options now. SOAP is not one of them ;-)
Source control service - In my first coding job, source control was managed by using different disks and folders: backup #1, latest backup etc. It was a nightmare, luckily was able to convince the boss there was a better way. Now it’s not whether to use source control, or what type of source control (ie use Git), it’s just about what service to use and what extras it provides.
Issue tracking software - some great options here these days, it’s also included in most source control services. I like to have the git comments link back to issues and stories.
Documentation - somewhere to write doco - internal & external. Nuff said.
CI/CD - It should be super easy to get changes tested and deployed to production through pipelines. It’s amazing how much this improves outcomes by encouraging regular delivery of small pieces of functionality. The DevOps mindset.
Monitoring - Need to know when there are issues and how the app is performing. There are now so many great tools and services to choose from here.

It’s amazing how much this improves outcomes by encouraging regular delivery of small pieces of functionality. The DevOps mindset.

(*) Rules are there to be broken, I just try to consciously balance the trade-offs. I’ve recently chosen a range of technologies that I didn’t have much experience with and somewhere very new. Mostly because I wanted to learn, improve my skills and I expect the technology to gain popularity.

Bias for action

There are dozens of good options for every one of the above. It’s easy to get distracted on research. Especially since I’ve suffered from my previous choices. This time I won’t make that mistake (I’ll just make another one instead…)

I value researching - reading, talking to people, playing around & prototyping. Plus research is easy and fun! But it’s also easy to blow days or weeks figuring out the perfect option and be no closer to selecting the right tech that will be pivotal for a new project.

After I have a certain level of confidence in a technology or direction, I prefer to make a start and learn along the journey. I also like to asses the choices regularly - especially when it early enough to change direction.

My bias for action has bitten me a few times recently. Where I’ve jumped in and started designing, architecting and building before I fully understood the technology. Most of the issues that arose are easy fixes, I just could’ve saved a little time and hassle with a better understanding.

Be happy with the choices

It’s easier to change technology choices now than ever before.

The Cloud, micro services, React/Angular/Vue components, shared packages (npm/gems etc) and interoperability between systems makes it easier to swap out parts of a platform than ever before.

It’s all secondary

…they are secondary to having the right processes, culture, automation & measurement.

While choosing great technologies and tools will help a project succeed, I’ve found they are secondary to having the right processes, culture, automation & measurement. These enable that consistent quality and delivery. Building the right product in the right way.

Tim Bray

Real knowledge is to know the extent of one’s ignorance

Imposter syndrome

Dunning-Kruger effect

Unknown Unknowns

Four Stages of Knowledge

Tracking software delivery and product velocity

The need for additional metrics

Customer Satisfaction - NPS

Engineering - Lead Time

Quality - Issues found in production

Production - MTTR

Data driven decisions

Starting an Engineering culture

Conscious trade-offs

Constantly attack technical debt

Architect for change

Define Done

Culture of shipping

Balance change and stability

Diversity

Summary

MacOS - Your disk space is critically low

The basics

Disk Usage command

System Files - Cache

Node / yarn / npm

Xcode Junk

Archives

Support for older devices

Downloads

Old simulator runtimes

CocoaPods

6 tips using Apollo GraphQL

GraphQL vs the libraries

GraphQL Playground

Raw GraphQL

Debugging with Postman & Chrome dev tools

Logging

Simple Authentication and Authorization

Caching

Our caching approach

Querying only selected fields from the database

Summary

GraphQL and React

GraphQL primer.

What is it

Why not just use REST

Where to start

Common objections

Next

Why we switched to GitLab

Initial Impressions

Continuous Integration & Delivery Pipelines

Summary

Technology choices

Tools and platforms

Core technology

Additional services / tools

File > New Project

What is right?

Bias for action

Be happy with the choices

It’s all secondary