ASCII Art in Code
19 February, 2019 ↬
This is totally my bag.
Designing Data-Intensive Applications 📚
29 January, 2019
This book surveys data storage and distributed systems and is a fantastic primer for all software developers.
It starts with naive approaches to storing data, quickly builds up to how transactions work, and works up to the complexities of building distributed systems.
I particularly enjoyed the chapter on stream processing and event sourcing. It contrasts stream processing to batch processing and highlights many of the challenges of these approaches and explores options for addressing them.
Alan Kay's Quora Answers
24 January, 2019 ↬
Still trying to learn how to think better.
Mental Models for Code and Systems
19 January, 2019 ↬
Cindy Sridharan quoting Joe Armstrong:
We should identify the error kernel. The error kernel of a system is that part which must be correct. That’s what the error kernel is. All the other code can be incorrect, it doesn’t matter. The error kernel is the part of the system that must be correct. If it’s incorrect, then all bets are off. The error kernel must be correct.
Statistical Rule of Three
19 January, 2019 ↬
John D. Cook:
The rule of three gives a quick and dirty way to estimate these kinds of probabilities. It says that if you’ve tested N cases and haven’t found what you’re looking for, a reasonable estimate is that the probability is less than 3/N. So in our proofreading example, if you haven’t found any typos in 20 pages, you could estimate that the probability of a page having a typo is less than 15%.
Using Netlify for Hosting
18 January, 2019
I recently moved the hosting of my various blogs and websites off my own server to Netlify.
I was originally going to set up an S3 bucket and Cloudfront distribution for each of my sites but Netlify provides me the CDN and hosting features I need all bundled up already. You can upload files directly for serving or hook your site up to run a static site generator when you push to a branch of a Github repository.
In short, I’m not longer paying hosting costs and they handle all of the SSL certificate renewal from Let’s Encrypt for me.
Next up I plan to clean up the tooling I use for some of my sites and tweak things on here so I have more variety in my posts.
Introduction to Unison Programming Language
15 January, 2019 ↬
Hashing syntax trees and storing them directly in a database is very interesting. I’ve long wondered what will come after the grab bag of text files approach we’ve been using to date.
A List of Beneficial Compounding Behaviours
15 January, 2019 ↬
Embrace the power of compounding.
52 Things Tom Whitwell Learned in 2018
15 January, 2019 ↬
Now you can learn them too.
The Best Television of 2018 With Sam Esmail
15 January, 2019 ↬
I ❤️ every time Sam goes on The Watch.
The Man Who Launched a Thousand Podcasts
15 January, 2019 ↬
Alex Blumberg interviews Ira Glass.
Jaron Lanier on VR and More
15 January, 2019 ↬
Come for the Feynman anecdote, stay for the exploration of technology and culture.
The Red Hand Files
15 January, 2019 ↬
A look behind the curtain with Nick Cave.
How to Prepare a Talk
27 November, 2018 ↬
An improvisational approach.
Strategy, Design, and Outcomes
6 November, 2018 ↬
Ryan Singer on the Product Love podcast talking about strategy, design, and outcomes.
Web Architecture 101
18 October, 2018 ↬
A nice summary of the various parts in a typical web application.
Unintuitive Things I’ve Learned About Management
18 October, 2018 ↬
Julie Zhuo:
Having all the answers is not the goal. Motivating the team to find the answers is the goal.
…
To evaluate the strength of a manager, look at the strength of their team.
…
The first time any of the above happens on your watch, it’s always new and hard, no matter how many books you’ve read on the topic. But the fifth or tenth time or 20th time it happens, you’re no longer freaked out. You realize that you’ll be fine.
Lots more interesting points in this and part 2.
The Hard Parts of Open Source
16 October, 2018 ↬
A look at conflict in online communities and a suggestion on how to handle it.
Engineering Productivity
11 October, 2018 ↬
Camille Fournier:
So what are the management skills that are needed to achieve [Engineering productivity]? At the first level of management, they look like:
- Breaking down the scope of projects to help your team ship frequently. An eye for the MVP, for sequencing work and for predicting likely risks and bottlenecks for project completion are the skills here. This is why I think project management is such an important part of engineering leadership development, and why I hate to hand it off to professional project managers for work that doesn’t cross teams or organizations.
Bingo.
Layering Microservices
11 October, 2018 ↬
Phil Calçado:
In product development, the closer to the customer a piece of software is, the more often it changes.
12 Factor CLI Apps
11 October, 2018 ↬
Jeff Dickey:
In short: stdout is for output, stderr is for messaging.
Some good tips sprinkled throughout.
Testing in Production, the Safe Way
11 October, 2018 ↬
Nice coverage of different approaches for testing in Production.
Rick Stein's Obituary
11 October, 2018 ↬
Nephew James Stein of Los Angeles claimed his uncle was an A&R consultant for Bad Boy records and ran a chain of legal recreational marijuana dispensaries in Colorado called Casablunta.
Via Kottke.
Constant Time Systems
18 September, 2018 ↬
Colm MacCárthaigh:
Now, let’s try a much better, O(1) control plane. DUMB AS ROCKS. Imagine if instead that the customer API pretty much edits a document directly, like a file on S3 or whatever. And imagine the servers just pull that file every few seconds, WHETHER IT CHANGED OR NOT.
This system is O(1). The whole config can change, or none of it, and the servers don’t even care. They are dumb. They just carry on. This is MUCH MUCH more robust. It never lags, it self-heals very quickly, and it’s always ready for a set of events like everyone changing everything at once. The SYSTEM DOES NOT CARE.
Forgetting Data in Event Sourced Systems
13 September, 2018
GDPR’s right to be forgotten means we have to be able to erase a person’s data from our systems. Event sourced systems work from an immutable log of events which makes erasure difficult. You probably want to think hard about storing data you need to delete in an immutable event log but sometimes that choice is already made and you need to make it work, so let’s dig in.
Erasing user data from current state projections
This is relatively straightforward. A RightToBeForgottenInvoked event is added to the event store for the person. All projectors that depend on personal data listen for this event and prune or scrub the appropriate data for the person from their projections.
Erasing data from the event stream itself
This case is trickier. We need to rewrite history in a way that doesn’t break things. Let’s look at an option for erasing data without rebuilding the event stream. This approach is also applicable for projections that are immutable change logs.
We can store personal data outside of events themselves in a separate storage layer. Each event instead stores a key for retrieving the data from this layer and any event consumers request the data when they need it. Given this data is personal the storage layer should probably encrypt the data at rest.
Once a RightToBeForgottenInvoked event is added to the event store all data for that person can be erased from the storage layer. All subsequent requests for data from the secure storage layer for that person’s data will return null objects rather than the actual data. This should make life easier for all consumers and avoid you null checking yourself to death all over the place.
Let’s see what this secure storage layer might look like.
Sketch of a secure storage layer
Our secure storage layer stores data that is scoped to a person and has a type (so we can return null objects). The store allows all data for a specific person to be erased.
Let’s start with two main models: a Person1 and a Data model.
Data Person
┌──────────┐ ┌───────────────┐
│ id │ ┌───>│ id │
├──────────┤ │ ├───────────────┤
│person_id │───┘ │encryption_key │
├──────────┤ ├───────────────┤
│ type │ │ is_erased │
├──────────┤ └───────────────┘
│ciphertext│
└──────────┘
The interface to the secure storage layer is outlined below.
class SecureStorage
def add(person_id, data_id, type, data)
# Find the Person model for person_id (lazily create one if needed).
#
# Encrypt the data using the person's encryption_key and store the
# ciphertext in the data table using the client supplied data_id and type.
#
# Clients will store this data_id in an event body and use it to retrieve
# the data later.
end
def erase_data_for_person(person_id)
# Mark the corresponding record in the person table as erased
# and delete the encryption key.
end
def get(data_id)
person = Person.find_non_erased(person_id)
if person
# Look up the row from the data table, decrypt ciphertext using the
# key on the person model, and return the data.
else
# Look up the row from the data table and return a null object for
# that data type.
end
end
end
Where does that leave us?
After a person has invoked their right to be forgotten all current state projections will be updated to erase that person’s data. The event store will return null objects for any events that contain data for the person which means that any event processors won’t see that data as they build their projections. It will also contain the RightToBeForgottenInvoked event for the person so consumers can handle that explicitly if required.
-
This could be expanded to be more general but we’ll stick with person for the purpose of this post. ↩
Replacing Google Analytics with GoAccess
3 September, 2018
I removed Google Analytics from my sites1 but still wanted access to some simple request statistics on them. Turns out GoAccess gives me most of what I need by analysing my Apache access logs.
The main challenges I ran into were, working out the correct flags for GoAccess, feeding compressed and uncompressed access logs at the same time, and ignoring junk requests from internet pests.
I pulled together a bash script that handles this, the essence of which is below.
# Analyse all log files
{ cat /var/www/mysite/logs/access.log; zcat /var/www/mysite/logs/access.log.*.gz; } | \
# Strip out junk requests
grep -v -E '\.php|jmx-console|\.cgi|phpmyadmin|dbadmin' | \
# Fire up goaccess using the correct log file format for consolidated apache logs
goaccess --log-format='%h %^[%d:%t %^] \"%r\" %s %b \"%R\" \"%u\"' --date-format='%d/%b/%Y' --time-format='%H:%M:%S' --ignore-crawlers
-
A) It does way more than I need. B) I still don’t understand how to use it properly. C) It needlessly collects and sends your information off to the Big G. ↩
Fibonacci Indentation for Vim
29 August, 2018 ↬
Clever.
Vitalik Buterin on Cryptoeconomics
27 August, 2018 ↬
This contains many things I don’t understand.
Creating God
22 August, 2018 ↬
Applying evolutionary theory to the origins and functions of religion.
A Definition of Success
16 August, 2018 ↬
Tyler Cowen:
Learning something new all the time, and staying healthy. Getting paid. Interacting with smart people. Having the chance to pass something along to others.
Solid.