December 3, 2024

38 and Grateful

It’s been a while since I’ve written an update – 2.5y! – and a long time since I wrote very frequently. On top of all the crazy ways the world has changed in the last five years, a bunch has changed for me personally in the last five years as well, along a bunch of different axes.

I think part of the reason it’s been hard to post is wanting to avoid coming across as bragging about the high points of this journey and also not really wanting to admit some of the low points, but I feel I’ve had a lot on my mind that doesn’t really make sense in blog form without the greater context; I’ve decided that I’d rather just share a lot of what’s been going on in the hopes that some part of it is relevant to someone out there or gives them some fortitude for whatever low spot they’re going through or can learn something from it.

June 15, 2022

Transacter and Intent/Result

At SQ, we had a family of Transacter interfaces:

public interface Transacter {
  void transaction(Function<Session, Void> session);
  
  public interface Session {
    DSLContext dsl();
  }
}

where DSLContext is a jOOQ concept & is the handle for doing work against the database. Then this would be used in some class like

public class KvDao {
  @Inject Transacter transacter;
  
  void put(String ns, String k, String v) {
    transacter.transaction(session -> {
        KvRecord record = new KvRecord(ns, k, v);
        session.dsl().insertInto(KV).set(record).execute();
    });
  }
}

The parameters ns, k, and v are shorthand for namespace, key, and value, respectively. We’ll show how these might be used together shortly!

August 23, 2021

hype channels

When I was managing a small team, one of the coolest protips I came across was the idea of a “hype channel”. This builds on the idea of a “hype doc” by changing the medium: instead of writing into a document, you write blurbs on your own private Slack channel.

I’m not sure whether this would help anyone but me, but maybe it’ll help someone else drowning in Chrome tabs?

May 18, 2020

interceptors

INTERCEPTORS ARE SO COOL!

Sometimes you need some “generic rails” that are still highly adaptable to other uses. This is the basic problem solved by the Interceptor pattern.

I really love the way OkHttp does interceptors for the generic rails of making HTTP calls, so I wanted to walk through a case study of why an interceptor might be useful and then try to synthesize some lessons & a minimal example of the pattern.

April 1, 2020

Hugo

I’ve decided to migrate the blog to Hugo, as you might have noticed from the new look.

This is gonna be a pretty big project and I’m probably going to be at it a while. I did keep a copy of the old blog at blog.jekyll but it seems the formatting is already broken there :-(

June 11, 2019

feeds justification

I realized I’ve left out a major part from my sequence of previous feed-related posts: a justification for why we should bother with a separate feed_sync_id. So let’s give it a shot! The fundamental problem is:

AUTO_INCREMENT ids are assigned in insertion order, but become visible to other threads in commit order.

To see how this causes a problem, consider the interactions and visibilities between three transactions to the same database:

t0: TRX0: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k0", "v0"); COMMIT;
t1: TRX1: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k1", "v1");
t2: TRX2: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k2", "v2");
t3: TRX0: SELECT MAX(id) FROM kv;
t4: TRX2: COMMIT;
t5: TRX0: SELECT MAX(id) FROM kv;
t6: TRX1: COMMIT;
t7: TRX0: SELECT MAX(id) FROM kv;

Here, we have two transactions that both insert a new kv record. The database has to assign an id value to each of those records, because we might be creating other associations to those records in our application code. But other threads – TRX0 in this case – shouldn’t be able to see those records until we COMMIT, and so indeed the SELECT at t=t3 might return 1.

May 23, 2019

macos log command

My work laptop has been randomly shutting itself off. I came across this stackoverflow post, it said to run

log show \
  --predicate 'eventMessage contains "Previous shutdown cause"' \
  --last 24h

which, for me, returns

risksys  ➜ log show --predicate 'eventMessage contains "Previous shutdown cause"' --last 24h
Filtering the log data using "composedMessage CONTAINS "Previous shutdown cause""
Skipping info and debug messages, pass --info and/or --debug to include.
Timestamp                       Thread     Type        Activity             PID    TTL
2019-05-23 11:29:01.874486-0700 0xaf       Default     0x0                  0      0    kernel: (AppleSMC) Previous shutdown cause: -128
2019-05-23 11:44:49.786722-0700 0xaf       Default     0x0                  0      0    kernel: (AppleSMC) Previous shutdown cause: -128
--------------------------------------------------------------------------------------------------------------------
Log      - Default:          2, Info:                0, Debug:             0, Error:          0, Fault:          0
Activity - Create:           0, Transition:          0, Actions:           0

This, combined with these MacOS Shutdown Causes paints a pretty bleak picture: I’ve got RAM issues :-(

April 25, 2019

absolutely minimal OLTP to OLAP pipeline

Suppose we have some data in a production OLTP database, and we need to send it to some OLAP database. This post describes one of the simplest approaches, and how to make it productional enough to rely on.

For every table t, we need to:

introduce a new field, updated_at
introduce a new index on that field, so we can get the records that changed after a certain updated_at.

For example,

February 21, 2019

smart arguments

It’s easy to love the terseness one can get out of a command-line tool with a “pass by order” convention. For example, maybe I run something like

deploy myapp sea2 production

to deploy an application called myapp to the second production datacenter near Seattle. This works great, except now I need to remember the order of arguments is: deploy [app] [datacenter] [environment].

The typical solve for this problem is introducing named arguments, so you’d end up with

January 8, 2019

feed sequences

In the mysql feeds post, I mentioned that the publisher could do

SELECT MAX(feed_sync_id)+1 
FROM kv

to find the next feed_sync_id during the publishing process, but this is actually a really bad idea. (And I knew it at the time, so forgive me for selling lies…)

Republishing

Before we jump into the problematic scenario, I’d like to motivate it with a tiny bit of background.

The republish operation is extremely useful when consumers need to receive updates. It is also extremely simple! A query like