Databases on traviscj/blog

Evolution of a Get Endpoint

Tue, 24 Jun 2025 19:47:52 -0500

My work project started pretty simple: there was a simple GetXyz endpoint that just looked up the xyz record in the database by a unique key & returned it. How complicated could it be?

It was a straightforward generalization of an old GetAbc functionality, it was really only used by oncall engineers through an admin console, it shouldn’t have been too big of a deal.

Ok, but then it outgrew its original datastore, so a separate service had to be created. We thought about migrating clients, but at the time it seemed faster to just implement once on our side & have clients continue calling us; essentially encapsulating the separate service as an implementation detail. But as part of the traffic swing, we figured it should support either the local or the remote read path.

Transacter and Intent/Result

Wed, 15 Jun 2022 08:07:47 -0500

At SQ, we had a family of Transacter interfaces:

public interface Transacter {
  void transaction(Function<Session, Void> session);
  
  public interface Session {
    DSLContext dsl();
  }
}

where DSLContext is a jOOQ concept & is the handle for doing work against the database. Then this would be used in some class like

public class KvDao {
  @Inject Transacter transacter;
  
  void put(String ns, String k, String v) {
    transacter.transaction(session -> {
        KvRecord record = new KvRecord(ns, k, v);
        session.dsl().insertInto(KV).set(record).execute();
    });
  }
}

The parameters ns, k, and v are shorthand for namespace, key, and value, respectively. We’ll show how these might be used together shortly!

feeds justification

Tue, 11 Jun 2019 00:00:00 +0000

I realized I’ve left out a major part from my sequence of previous feed-related posts: a justification for why we should bother with a separate feed_sync_id. So let’s give it a shot! The fundamental problem is:

AUTO_INCREMENT ids are assigned in insertion order, but become visible to other threads in commit order.

To see how this causes a problem, consider the interactions and visibilities between three transactions to the same database:

t0: TRX0: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k0", "v0"); COMMIT;
t1: TRX1: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k1", "v1");
t2: TRX2: BEGIN; INSERT INTO kv (ns, k, v) VALUES ("-", "k2", "v2");
t3: TRX0: SELECT MAX(id) FROM kv;
t4: TRX2: COMMIT;
t5: TRX0: SELECT MAX(id) FROM kv;
t6: TRX1: COMMIT;
t7: TRX0: SELECT MAX(id) FROM kv;

Here, we have two transactions that both insert a new kv record. The database has to assign an id value to each of those records, because we might be creating other associations to those records in our application code. But other threads – TRX0 in this case – shouldn’t be able to see those records until we COMMIT, and so indeed the SELECT at t=t3 might return 1.

absolutely minimal OLTP to OLAP pipeline

Thu, 25 Apr 2019 00:00:00 +0000

Suppose we have some data in a production OLTP database, and we need to send it to some OLAP database. This post describes one of the simplest approaches, and how to make it productional enough to rely on.

For every table t, we need to:

introduce a new field, updated_at
introduce a new index on that field, so we can get the records that changed after a certain updated_at.

For example,

feed sequences

Tue, 08 Jan 2019 00:00:00 +0000

In the mysql feeds post, I mentioned that the publisher could do

SELECT MAX(feed_sync_id)+1 
FROM kv

to find the next feed_sync_id during the publishing process, but this is actually a really bad idea. (And I knew it at the time, so forgive me for selling lies…)

Republishing

Before we jump into the problematic scenario, I’d like to motivate it with a tiny bit of background.

The republish operation is extremely useful when consumers need to receive updates. It is also extremely simple! A query like

watching progress in mysql

Fri, 04 Jan 2019 00:00:00 +0000

Pretty frequently at work, I end up polling a database with some command like

SELECT MAX(id) FROM my_tbl;
SELECT MAX(id) FROM my_tbl;
SELECT MAX(id) FROM my_tbl;
SELECT MAX(id) FROM my_tbl;
-- .... ad nauseam ....

I’ve eventually noticed a few patterns I use pretty consistently:

estimate ETA by including `NOW()`/`UNIX_TIMESTAMP()`

Generally, the point of hovering over the table is to get an estimate of when it will finish/catch up/whatever. For that, you generally want to include a timestamp in the query output, so when you come back a while later, you can know exactly how much time has elapsed. I might transform the above query into

sharded feeds

Mon, 29 Oct 2018 18:30:00 +0000

Suppose that:

our humble KV feed sees a lot of traffic.
someone needs to consume our KV feed with multiple threads.

data

The first step is to introduce a notion of “shards” into our data model:

ALTER TABLE `kv`
  ADD COLUMN `shard` INT(11) DEFAULT '0',
  ADD INDEX `k_fsi_s` (`feed_sync_id`, `shard`);

publishing

We don’t need to alter the publishing until the publishing itself is too slow to work with a single thread, but this introduces a lot of complications, so let’s just hold off for now.

Feeds as cache invalidation mechanism

Wed, 03 Oct 2018 02:31:43 +0000

One really cool use of feeds we’ve realized is that it gives a very efficient mechanism for application code to load the most recent versions of a table into memory. The basic idea is:

Set it up as a usual feed published table with an appropriate index on feed_sync_id.
Either alongside or within the cache, represent the latest loaded feed_sync_id.
Set up a cronjob/etc that reads the latest feed_sync_id and compares it to the cache’s feed_sync_id.
If they differ, reload the cache.
Ensure that all changes set feed_sync_id to null!

This works really well because the feed_sync_id in the database only gets updated on changes, so the reload cronjob mostly is a no-op. This means we can reload very frequently!

cross-dc sync with feed published KV

Tue, 10 Jul 2018 11:34:39 +0000

It’s been fun describing the feeds framework we use at Square. Today we’ll dive into a concrete problem:

We’ll stick with the feed-published kv table again.
We want two instances of some application code to bidirectionally synchronize the writes that happened on their instance to the other.
Eventually consistent is ok.

First, a bit of justification, though: I use this KV table to remember things I might have to look up without usual context, like my motorcycle’s license plate number, or that one weird python snippet I can never remember. I also have a whole slew of them at work – a bunch of random representative IDs for a bunch of things in our systems that I use from time to time. I also use a bunch of these as todo items at work, but that happens to work differently and is a topic for a future blog post :-)

history preserving data models

Mon, 02 Jul 2018 11:34:39 +0000

Start with a super simple data model:

CREATE TABLE kv (
  id BIGINT(22) NOT NULL AUTO_INCREMENT,
  k VARCHAR(255) NOT NULL,
  v LONGBLOB NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY u_k (`k`)
) Engine=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

Suppose we want to audit “changes” to this data model.

Approach 1: `kv_log`

add data model like

CREATE TABLE `kv_log` (
  id BIGINT(22) NOT NULL AUTO_INCREMENT,
  changed_at TIMESTAMP NOT NULL,
  k VARCHAR(255) NOT NULL,
  old_v LONGBLOB NOT NULL,
  new_v LONGBLOB NOT NULL,
  
)

current value query: unchanged

mysql feeds

Fri, 29 Jun 2018 00:00:00 +0000

At work, we use a pattern called feeds that gets an incredible amount of work done. I’ve been wanting to describe it here for quite a while, and now seems as good of time as any.

The basic premise is: You have a service A with some data that other “consuming” services B, C, and D want to find out about. Maybe the data is payments, maybe it’s support cases, maybe it’s password changes… whatever. The other services might include your data warehouse, some event listeners, whatever.

constraint violations in tests

Thu, 31 May 2018 00:00:00 +0000

I had some test code that looked something like

@Test public void a() {
  db.insert(0, "a");
  assertThat(db.query(1).getKey()).isEqualTo("a");
}

@Test public void b() {
  db.insert(0, "a");
  db.insert(1, "b");
  assertThat(db.query(2).getKey()).isEqualTo("b");
}

@Test public void c() {
  assertThat(db.query(3)).isNull();
}

There’s some stuff to like about this:

pretty straightforward which test is touching which records
pretty clear what you expect to see in the database.

There’s a bunch of coincidences here that make this work, though:

osquery

Fri, 12 Jan 2018 08:43:17 +0000

I’ve known about [osquery][] for a while, but recently spent some time digging around in it. [osquery]: https://osquery.io The basic idea is to provide a consistent SQL interface to a bunch of system data, instead of learning idiosyncrasies of individual commands (which themselves vary across operating systems). My hacker buddy Sharvil has worked on osquery a fair bit and explained to me a couple years ago that it actually uses [sqlite][]’s [virtual table functionality][vtab] to provide the interface – it’s a fascinating and brilliant project! [sqlite]: https://sqlite.org/ [vtab]: https://sqlite.org/vtab.html

python attrs

Thu, 31 Aug 2017 08:42:43 +0000

I came across a very interesting library in a HN thread: the python attrs library.

In particular, this seems like a great way to do the “dumb data objects” they talk about in the end of object inheritance, and also related to (but maybe lighter weight than) zope.interface.

This also seems very similar to what I use autovalue for at work.

One particularly interesting application is a “code database” – using static, checked-in-to-version-control definitions of some data model as a sort of very-fast-to-read, very-slow-to-update “Data Model”. I find this fascinating: Code shares a lot of properties with great data stores: ability to rollback (git revert) and accountability/auditability (git blame). It also makes a lot of fairly hard problems much simpler: you don’t need to poll the database for changes. You don’t need to invalidate any caches. You don’t need to consider a “split brain” environment where half of the in-memory caches have updated but the other half haven’t. You don’t need to consider failure cases of how long the in-memory cache is allowed to be invalid: you just fail to boot up on deploy. (Admittedly, there’s still an opportunity window for split brain behavior for the duration of the deploy, but this is a lot easier to reason about than an essentially arbitrary.)

relational java explorations with sqlite3 vtab

Fri, 05 May 2017 20:10:28 +0000

One idea i have been mulling over lately is exposing a java codebase relationally, with queries like:

SELECT *
FROM classes c
JOIN annotations a
  ON a.class_id = c.id
WHERE a.name = 'MyAnnotationClass';

The idea here is that you could search a codebase by pretending that you had tables for a whole bunch of things like:

classes
annotations
variables
literals
methods

This would be a useful thing because instead of relying on an IDE like IntelliJ to do “find usages” operations, you could actually script those interactions. Admittedly, of course, Java has some pretty sophisticated reflection APIs to find this sort of stuff too, but it seems like exploring the structure could be much easier writing Java code to traverse those trees.

toy/life data models

Wed, 19 Apr 2017 00:00:00 +0000

I’ve been experimenting a lot with some kinda “toy” data models based on random things that I wish there was a database to query, but isn’t. For example:

What was my average arrival time this week?
How much of my equity has vested before a certain date?
When was the last time we had spaghetti for dinner?

I’ve been doing this with flat JSON files. This is a bit of an odd choice for me; I actually love schematizing data models in protobuf and MySQL and designing proper indices for the data models I work on during work hours.

filter vs spec (draft)

Wed, 18 Jan 2017 00:00:00 +0000

Consider a silly data model to store data about cities like

message City {
  optional string city_name = 1;
  optional string state = 2;
  optional int32 population = 3;
  optional int32 year_founded = 4;
  // ... presumably others :-)
}

and some sample data like:

[
  {"city_name": "Portland", "state": "OR", "population": ...},
  {"city_name": "Portland", "state": "ME", "population": ...},
  {"city_name": "Springfield", "state": "FL", "population": ...},
  {"city_name": "Springfield", "state": "IL", "population": ...},
  {"city_name": "Springfield", "state": "CO", "population": ...}
]

There are some useful entities we can define: (DRAFT NB: don’t read too much into the matcher vs filter lingo.)

MySQLdb module in Python on Ubuntu

Sat, 30 May 2009 00:00:00 +0000

To install the mysqldb module in python on Ubuntu:

sudo apt-get install python-mysqldb

A Quick MySQL Reference

Wed, 27 May 2009 00:00:00 +0000

I got frustrated with not being able to write MySQL because I don’t do it often enough to be seeing it in my nightmares like MATLAB. But recently my datasets got annoyingly huge and it seemed like SQL might be a boon to me. So I set out to write a quick little program, and ended up writing this little reference along the way.

Also, Sharvil Shah contributed some comments to it, so thanks to him for that!