Nikolay: Hello, hello.

This is Postgres.FM.

As usual, my name is Nik, Postgres.AI, and my co-host is Michael,

pgMustard.

Hi, Michael.

Michael: Hi, Nik.

Nikolay: And we have an unexpected guest today, Simon Eskildsen, CEO and

co-founder of turbopuffer.

Hi, Simon.

Simon: Thank you for having me.

Nikolay: Yeah.

Thank you for coming and it was very unexpected because we mentioned

turbopuffer last time and you messaged me on Twitter and I think

it's a great idea sometimes to look outside of traditional Postgres

ecosystem.

I think it's beneficial for everyone, should be.

So yeah, thank you for coming for sure.

It's a great idea.

I think.

Simon: Yeah, the origin story is kind of funny and it was only

a couple of days ago.

I have a, there's a script where if turbopuffer is mentioned

anywhere, then I'll get an email or a summary.

And you guys were discussing last time, different ANN, like both

in Postgres and outside and turbopuffer was mentioned.

And so I just DM'd you and asked, hey, we get, I'd come on, chat

about Postgres, chat about MySQL, chat about like databases and

when you choose one over the other and when Postgres breaks for

some of these workloads that we've seen and when it's great.

And now it's what?

Yeah, 2 or 3 days later and we're on.

Nikolay: Including the weekend actually.

So podcast was out on Friday and we record this on Monday.

That's, that's, that's the loss that you, everyone should try

to achieve.

I like it.

Simon: Yeah.

And you've had, you've had chicken hatch in the meantime.

I got,

Nikolay: that's why my camera is overexposed because a whole,

whole night this camera, I used it to broadcast.

I didn't have time to tune it properly.

But again, like, thank you for coming.

This is about databases.

This time probably not so much about Postgres, but definitely

we should talk about vector, vectors, right, and ANN.

And maybe we should start from distance and talk about your background.

And I've heard some MySQL is involved, right?

Can you discuss it a little bit?

Simon: For sure.

Yeah, my background is I spent
almost a decade at Shopify scaling

mainly the Database layer there,
but pretty much anything that

would break as the scale increased
and increased through the

2010s.

Shopify, like most companies started
in the early 2000s was on

MySQL.

So a lot of the work that I did
was, was with MySQL, but also

every other Database that Shopify
employed, like Redis, Memcached,

Elasticsearch, and a ton of others,
Kafka, so on.

So it's been a long time there.

When I joined, it was a couple
of hundred requests per second.

And when I left, it was in the
millions and very very intimate

with the data layer there.

I was on the last resort pager
for many many years and that has

informed a lot about how I write
software today and a couple

years ago I started a company called
turbopuffer because I thought...

Nikolay: Before that, sorry for
interrupting.

I remember Shopify, actually, I
always, we always recommend the

article.

Shopify had a couple of blog posts
about UUID and how UUID version

4 is not good and version 7 is
much better.

And it doesn't matter MySQL or
Postgres, the mechanics behind

the scene is the same.

It's how B-tree behaves.

So I remember, and I think, Michael,
we mentioned that article

on podcast a few times as well.

Michael: We have mentioned it, Nik,
for sure.

And I think Shopify has come up
before.

I thought, is it a Vitess shop?

I thought it might have been 1
of the early Vitess adopters.

Simon: Shopify is using a little
bit of Vitess, but not very

much.

Vitess was not around when we
did all the sharding back in the

early 2010s.

And so we did it all at the, at
the, at the application layer.

I wasn't part of the actual sharding
decision, but I was part

of a lot of the sharding work over
the years.

And it's funny because at the time
I know that they looked at

a bunch of proxies and all of the
businesses they later looked

at had gone out of business.

It's not a great business, unfortunately.

But everything was done in Ruby
land through a, just a module

called sharding, and it did a lot
of things and a lot of monkey

patches into Rails.

But let's talk about this, like
UUID v4 thing, because I think

if we wanted to do a pros and cons,
MySQL versus Postgres, I

spent quite a bit of time with
both and this 1 actually, to my

knowledge, only really matters
for MySQL.

Well, it actually matters for Postgres
as well, but in a different

way.

So on MySQL, the primary key dictates
how the B-tree is laid

out for the primary key, right?

So for the entire row.

So if you have UUID v4, it's completely
randomly scattered along

the B-tree.

So whenever you're doing an insert,
right?

It will just kind of fall somewhere
random in the B-tree, which

makes the updates very expensive
because if you're updating 10,

if you're adding 10 rows at a time,
so you're not updating to

ads, doing 10 insertions that are
in 10 different leaves, well,

you're just doing a lot more disk
I/O and your write information

is high.

In Postgres, of course, you're
just appending to the heap with

all of its drawbacks and benefits.

And so it doesn't matter as much
other than on the indexes, right?

It's my knowledge, but on the indexes,
it will matter a lot because

on the indexes, of course, it is
sorted by that.

And if you have some temporal locality
then it's not gonna matter

as much.

So that's my understanding.

So this matters a lot in MySQL.

Now, that article I think was after
I left and Shopify doesn't

use UUIDs as primary keys for anything.

So I don't really know where this
mattered.

It must be something tangential
because MySQL really just does

auto increment.

And every shard does an auto increment
with basically like with

a 32K auto increment number.

And then every shard has a plus
offset into that to allow it

to grow to 32K shards.

Given how much of a pain in the
ass it would be to change that,

that's probably still the case.

But I always really liked that
scheme.

So some tables over time at Shopify
ended up having a primary

key on the shop ID comma the ID
because that would give locality

for a shop.

Because otherwise you have a lot
of re-damification if you're

trying to dump out a bunch of products
for a shop, because the

chance that there's gonna be multiple
products for a shop, like

in a leaf, unless you do that,
it's just a lot lower.

So that ended up working really
well.

And this is a pain to do in Postgres,
because if you want to

rewrite the primary key or the
heap by an ID, you have to rewrite

the entire thing.

That was 1 of my surprises, having
worked a bit more with Postgres

in later years.

Nikolay: Yeah, yeah, yeah.

And I agree with you and I agree
that in Postgres it also matters,

but only for B-tree itself, primary
key B-tree, if it's after increment

or in Postgres it's called big
serial, serial for example, or

auto-generated.

Right now there's another method,
but behind the scenes it's

also like sequence and inserts.

Oh no, in this case it's not sequence.

There should be a function which
generates UUID version 4 and

if it's random, like version 4
is random, Version 7 is closer

to regular numbers, basically,
because it's monotonically growing,

right?

Lexicographically ordered, right?

So in this case, you insert only
on the right side of B-tree, and

dirty pages, if you think about
how checkpointer is working.

Also, in Postgres there is also
overhead after each checkpoint

is full page write which involves
indexes as well.

So if you touch random pages all
the time, disk I/O overhead and

replication and backups, actually
everything receives additional

overhead.

While in version 7 we write on
the right side all the time.

It's much better.

But heap, yes, heap is different.

So I agree with this and anyway,
like I just wanted to say we

use MySQL article because it's
written very well.

And in Postgres we didn't have
version 7 for quite some time.

Last Thursday version 18 release
candidate was out, which will

include full implementation of
UUID v7, which was live-coded

on Postgres TV with a couple of
friends, just in Cursor, I think.

Oh no, it was not in Cursor, it
was before that, but anyway,

it was just created right online.

We did it, and right now it took
a couple of years to reach maturity

because Postgres always waits until
RFCs are finalized and so

on.

Anyway, soon version 18 will be
out and UUID version 7 is inside.

But I think everyone is already
using it on client.

Okay, great.

So you had this great career and
then decided to create another,

is it, can we call it a database
system, database management

system?

Simon: It's certainly a full-blown
database.

You know, underneath turbopuffer
is an LSM.

An LSM works really well for object
storage.

And, you know, every successful
database ends up implementing

every, every query eventually right
in the limit and turbopuffer

will end up doing the same thing,
but every good database also

starts with some specialization,
right?

And our specialization has been
on, on search workloads.

I would say that it's by no means
a replacement for Postgres.

There always becomes a time where
it starts to make sense to

move parts of data into more specialized
algorithms, more specialized

data structures, and more specialized
storage.

In general, my, my hypothesis on when it's time to create a database

is that you need 2 things in the air in order to create a generational

database company.

The first thing that you need is that you need a new storage

architecture, because if you don't have a new storage architecture,

some new way to store the data, ideally both data structure wise,

and also the actual medium that you're persisting on.

There's no reason why in the limit the other databases won't

do the workload better.

They already have the existing momentum.

They already have the workloads.

In the Postgres case, of course, you know, it's the classic relational

architecture where you replicate every byte onto 3 disks.

And this works phenomenally well, right?

We've had this in production for probably more than 40 years.

And it works great.

It has high performance, It has a very predictable performance

profile and it works really, really well with the page cache

or the buffer pool, whatever database you're using.

The problem with this model is that if the data is not very valuable,

this model is expensive.

Every gigabyte of disks, a network bound disk is about 10 cents

per gigabyte.

And unless you are really risky DBA, you're gonna run those disks

at 50% utilization on all the replicas and on the primary.

So you're paying for this disk kind of 3 times, which ends up

with a whole all-in cost of 60 cents per gigabyte.

And that's not even accounting for all the CPUs that you also

need on the replicas because you need the replicas to process

the rights as fast as the primary.

So you're kind of paying for the same thing 3 times.

So the all in sort of per terabyte cost, when you also take into

consideration that the system can be a little bit more memory

hungry is probably around 60, 60 cents to $2 per gigabyte

Nikolay: per month.

Simon: Per month.

Yeah.

Per month USD on object storage.

The base cost is 2 cents per gigabyte.

Right.

And When we need that data in RAM or on disk, we only have to

pay for 1.

We only have to pay for some percentage of the data to be in

cache at all times.

We mentioned Cursor earlier.

Cursor is a turbopuffer customer.

They don't need every single code base on SSDs or in memory at

all times.

They need some percentage in memory, the ones that are queried

a lot right now, and some percentage on disk, the ones that are

gonna be queried again in a few minutes or maybe in a few hours.

And so you end up paying a lot less because we only have to keep

1 copy of a subset of the data rather than 3 copies of all of

the data.

Now that comes with a fundamental set of trade-offs, right?

We want to be as public as that.

You can't use turbopuffer for everything.

If you want to do a write to turbopuffer, we commit that to S3,

right?

So by the time it's committed to turbopuffer, the guarantee

is actually stronger than most relational systems because we've

committed it into S3, which takes a couple hundred milliseconds,

but the durability guarantee is very strong.

But if you're building a system like Shopify, well, you can't

live with a commit time in hundreds of milliseconds.

It's just not acceptable.

So that's a trade-off that means that this is not a system that

will ever replace a relational database store.

The other downside is that because not all the data is on a disk

or in memory at all times, it means that you can have tail latency.

And that can be really catastrophic in very large systems that

are doing millions of queries per second.

If you can't rely on a very predictable query profile, you can

have massive outages to hydrate the caches.

I've seen these outages on disks, I've seen them.

And just even the workload changing slightly can mess with the

buffer pool in a way where you have a massive outage.

So these 2 things may sound like small trade-offs, but they're

massive trade-offs for very large production systems.

But it might mean that if you have let's say a billion vectors

and you're trying to store them into Postgres.

The economics just don't make sense.

You're paying thousands of thousands, not tens of thousands of

dollars in hardware costs for workload that might cost tens or

hundreds of dollars on turbopuffer.

Nikolay: How much is it really?

1 billion vectors.

If 1 vector is what's like 700

Simon: and a 768 dimensions.

Yeah.

It's, It's 3 terabytes.

Nikolay: 3 terabytes to store 1 billion vectors.

Yeah.

And also we don't have a good index for 1 billion scale.

Yeah, I mean, in Postgres, pgvector, HNSW won't work with 1 billion.

Simon: I mean, we could just run the math a couple of different

ways, right?

I'm not saying this is how it works in pgvector, I'm less familiar

with it now.

But even if you have 3 terabytes of data raw, you're probably

going to need to store more than that.

You might be able to do some tricks to make the vector smaller,

so you only have to store maybe, I don't know, a terabyte or

something along those lines, right?

But remember that a gigabyte of DRAM is $5 per month.

And you need that 3 times on all your replicas.

So you're paying $15 per gigabyte per month.

So if you're doing that, if you have to store that 3 times, you

put everything in memory and you're somehow able to get it down

to a terabyte, then you're talking about $15,000 per month,

right, across the 3 replicas, just for the RAM alone.

Nikolay: Yeah, I agree with you.

HNSW, it's like memory.

Memory is the key, and creation of index takes a lot of time.

And for billion, I already have issues with a few million vectors

scale.

I know Timescale, which I now renamed to Tiger Data, they developed

another index based on DiskANN from Microsoft, I think, which

is more like for disk, right?

But I agree with you, like for this scale, it's not convenient.

But Also, I think it's not only about vectors.

This argument that we need to save on storage costs, and it's

insane to pay for storage and memory as well when we have replicas

and we scale.

In Postgres, if it's a physical replica, it's everything.

So you replicate everything, all indexes, everything.

You cannot replicate even...

There is no ability to replicate only 1 logical database.

You need to get the whole cluster.

And it means that you multiply costs for storage and for memory.

And it would be so great to have some tiered storage maybe with

partitioning as much as automated as possible and offload all

data to S3 as basically you consider.

S3 is great for this and I remember I was actually I explored

turbopuffer through Cursor.

It's great like to see the documentation.

I knew Cursor is using Postgres.

It was a few months ago, but then I heard they considered moving

to PlanetScale.

That was before PlanetScale announced support of Postgres, so

I was thinking, are they switching to MySQL?

And then I saw vectors are stored in turbopuffer.

Great.

Then I learned that several of our clients who get consulting

from us and use Postgres, they also store vectors in turbopuffer.

It was, I think, Photoroom, a couple of more companies.

It was a surprise for me, and I started to think, oh, that's

interesting.

And then I checked your talks.

I think you also mentioned there that if we...

So there is this approach with
SPFresh algorithm, right?

Different it's not HNSW, different
types of index, but also some

additional interesting ideas about
economics you mentioned, right?

Not only these sense.

Can you elaborate a little bit?

Simon: I think it might be helpful
to just talk at a very high

level about the different algorithms
to do vector indexing into.

I'll try to simplify it as much
as possible, and we can dig into

it further if you want.

Fundamentally, the simplest way
to do vector search is that you

just store all of the vectors in
a flat array on disk, right?

And then on every search, you just
compare the query vector to

all of those.

The problem with that is that you
very quickly run up against

bandwidth limits, right?

If you have a gigabyte of vectors
and you're searching that at

maybe 10 gigabytes per second,
if you can exhaust the memory

bandwidth, which is unlikely in
a big production System, you're

only doing, you know, maybe 5 queries
per second and the Query

latency in the hundreds of milliseconds.

So if you have very, very few queries
and you don't care about

latency, this can be feasible on
small scale.

And lots of people are doing that
in production.

But if you want to search a million
vectors in less than a couple

hundred milliseconds and maybe
10 milliseconds and as part of

a bigger pipeline, you need some
kind of index.

The problem with indexing vectors
is that there is no known way

to do it in a perfect way, right?

If I search for a Query vector
about a fruit or whatever, I know

that if I'm searching for banana,
I get the closest fruit.

Maybe, I don't know, maybe there's
a plantain.

I don't know.

Right, in the cluster.

But you have to build an approximate
index in order to make this

faster.

Nikolay: Because of too many dimensions.

For small number of dimensions,
there are approaches and approaches

have them for years, but for hundreds
of thousands of dimensions

yeah.

Simon: That's right yeah there's
KD trees and so on for the

for the smaller dimensional space
which we can use for geo coordinates

and things like that and simpler
geometry.

So for very high dimensional spaces
these things fall apart.

Curse of dimensionality it's called.

So they're very large is also important
about the vectors.

If you have a kilobyte of text
it can easily turn into tens and

tens of kilobytes of vectors which
is why separating into cheaper

storage makes a lot of sense.

So there's 2 fundamental ways that
you can index the data.

There's the graph-based approaches,
HNSW and DiskANN, were the

2 you mentioned before.

And there's the clustering-based
approach.

The graph-based approach is phenomenal
if you can store all of

the data in memory and you have
very high QPS and very low latency

requirements.

So if you have, let's say a hundred
million vectors and you're

searching that at, you know, a
hundred thousand queries per second,

and you need very low latency,
you're not going to beat HNSW.

It's going to create a very, very
good graph to, to navigate

it with.

The problem is that it's very expensive
and the other problem

is that almost no workloads in
the real world actually look like

this.

So HNSW got very popular because
it's fairly simple to implement

correctly and it's very simple
to maintain.

So when you create the graph which
is essentially just you know

points that are close in vector
space or close in the graph,

it's very simple to incrementally
maintain.

You put 1 thing in, you search
the graph, and then you add it.

There's very simple rules.

You can implement something like
HNSW in tens of lines of code

if you did a very, very simple
implementation of it.

The problem with HNSW is that every
time you do a write, you

have to update a lot of data, right?

In database land, we call this
write amplification, where every

byte or every page you update,
you have to update a lot of others,

and the reason for that is that
You add something, you add a

node in the graph, and then you
have to update all these other

things to do connections to that
node in the graph.

So this works great in memory,
because memory is very fast at

updating and very fast at random
writes.

But the problem is also on the
read path.

So memory is very fast at doing
random reads.

You can do a random read at 100
nanoseconds, but a random read

on S3 or on a disk is much slower,
into hundreds of microseconds

to the hundreds of milliseconds
on S3.

And on a graph, right, you don't
really, there's no speculation

that helps, right?

If you start at the middle of the
graph and then greedily navigate

the graph from the query vector
to the closest matching vectors,

well, every single time you do
that, there's a round trip.

And on S3, that's a round trip
that takes hundreds of milliseconds.

So you're sort of navigating from
the root, it's like 200 milliseconds.

You go out 1, 200 milliseconds.

You go out another 1, 200 milliseconds.

And in general, for HNSW on a million,
this might be in the tens

of round trips.

So this gets very slow, right?

This is in the seconds to do this
on S3.

And even on a disk, this very quickly
adds up to tens of milliseconds.

That's the fundamental problem with graphs.

Now, DiskANN is essentially using a lot of the same ideas

as other graph-based indexes, but instead of trying to have the

tens of round trips that HNSW has, that's very good for memory,

DiskANN basically tries to shrink the graph so that there

are fewer jumps, right?

So instead of 200 milliseconds, 30 times, it tries to get it

to maybe 6 or 7 times, or 10 times, by shrinking the graph as

much as possible.

That's essentially the insight in DiskANN.

The problem with DiskANN is that after you have added more than

10 or so, 10 to 20% of the size of the data, you have to rebuild

the entire graph, which is an incredibly expensive operation.

That is absolutely terrifying to me, to someone that's been on

call for large databases to just have, like you could max out

like 128 cores, rebuilding this graph in prod, and it could take

you down to 3 a.m.

Because you don't know when you hit that threshold.

And if you don't do it, then the approximations start getting

bad and you start getting bad search results.

The nice thing about the graphs is just they have very low latency

but they're just very expensive to maintain.

Now the first way, then there's the other type of index which

are the clustered indexes.

Clustered indexes are very simple to picture in your head, right?

If you have a cluster of let's say you took out every song in

Spotify and you clustered them in a coordinate system and you

can visualize this in 2 dimensions.

If you, then you plotted all the songs and the songs that are

adjacent, of course, also adjacent in vector space and genres

will emerge, right?

There'll be a rap cluster, there'll be a rock cluster.

You zoom in and you get, you know, like little sub clusters,

I don't know, death metal, black metal, I don't know what all

the different rock genres are, right?

Somewhere in these clusters.

And you can generate great recommendations based on that because

you can look at, okay, what did Michael listen to and what are

some songs that are close by that he hasn't listened to and same

for Nik.

So this is, it's very simple, you create a clustering algorithm

that basically just tries to divide the data set into clusters

and when you do a query, instead of looking at all the vectors,

you look at, well, clearly the user is asking about a rock song,

so we're only going to look in the rock cluster.

And that way you divide down the number of vectors that you have

to seek.

Now the problem with this is that if you have everything in memory,

it's not necessarily as optimal because you might have to look

at more data than you do in a graph-based approach.

And because RAM has such good random
read latency, the penalty

is not necessarily worth it if
everything is in memory at all

times.

But this is great for disks, and
it's great for S3, because I

can go to S3 and in 200 milliseconds
get gigabytes of data back,

right?

It doesn't matter if I'm getting
like, you know, a megabyte or

a gigabyte, I can often get that
in the same round trip time

if I exhaust the network.

So if you then go into S3, basically,
you have to download all

the clusters.

So like, let's say clusters of
JSON to really simplify this.

And then you just look at the closest
n clusters to your query

vector.

And then you download, you know,
cluster1.json, cluster2.json,

whichever ones are closest in 2
round trips.

And now instead of on the graph-based
ones where you're doing

200 milliseconds, 200 milliseconds,
200 milliseconds to navigate

the graph, you just have to do
200 milliseconds to get the clusters

and then 200 milliseconds to get
all the clusters that were adjacent.

The nice thing about these clustered
indexes is that with algorithms

like SPFresh and lots of modifications
to them, we can incrementally

maintain these clusters because
you can imagine that when you

add a vector you just have to add
it to the cluster and it's

just 1 write.

The write amplification is very
low.

Once in a while that cluster will
grow beyond a size.

Let's say it's a thousand elements
long and you have to split

the cluster and then you have to
do some modifications.

That's essentially what SPFresh
is.

And then there's a little bit higher
write amplification.

But it's stable in the way that
you never reach this threshold

where, okay, I've added 20% of
the dataset, I have to rebuild

the entire thing as you do in DiskANN.

HNSW don't have to do that, which
is why it's very, very, like,

it's very nice, but it's just slow
still.

So SPFresh, we think, writes a
really, really nice set of trade-offs

where it's gonna be a little bit
slower, but slower in terms

of, okay, instead of a search returning
in a millisecond, it

might take 5 milliseconds, and
just no 1 cares in production

for search.

This matters for a point lookup
into a relational database, but

for search it's a perfect set of
tradeoffs.

Michael: Question on the cluster
splitting.

Does that mean we don't ever need
to rebuild the whole index?

Because I think that was a limitation
of the first cluster-based...

IVF flat, I think, we started with
in pgvector, and that didn't

have the splitting as far as I'm
aware, and therefore we had

to rebuild the whole index from
time to time if the data changed

significantly.

Simon: That's right.

And there's also MERGE,

right?

Nikolay: There's also not only split, there
is also MERGE in SPFresh,

as I remember.

Simon: Yes, there's MERGE as
well.

Michael: Makes sense.

Simon: And because, because you
might do deletes, which are

also a pain.

To my knowledge, pgvector does
not implement SPFresh.

It's a, it is very difficult to
implement correctly.

Nikolay: But also funny, I did
a little bit research in May,

I think, when I discovered turbopuffer
and started reading about

this.

I saw original implementation was
actually on forked Postgres

SPFresh.

I saw some, you need to dig into
it, like in some Microsoft repositories

also I think some Chinese engineers
were involved, something

like there, Like I saw some repository
which was basically forked

Postgres and original SPFresh
implementation was on top of that.

Maybe I'm hugely mistaken, but
I saw something like this.

But it's hard.

I agree.

And I also recalled what I wanted
to ask.

I was lost in my previous question
because it was too many things.

I recalled in your talks, you discuss
that S3 is ready to store

data.

S3 is ready because over the last
few years, they added important

features.

Can you recall what you talk about
at the talks?

Simon: Yeah, I mentioned that
there were 2 things that you

needed to build a new database.

And there's a reason why a database
like turbopuffer hasn't already

been built.

It's a new storage architecture.

It's only really possible now.

The 3 things that have enabled
a database like turbopuffer to

exist with this sort of pufferfish
architecture, right, where

it's, you know, when the pufferfish
is deflated, it's in S3,

and when it's, you know, somewhere
in between, it's in SSD, and

then it's in memory when it's all
the way inflated.

The reason that's possible is because
of 3 things.

The first 1 is our NVMe SSDs.

NVMe SSDs have a new set of trade-offs.

They act completely differently
than other disks, right?

SSDs are just sort of like...

The old SSDs were very fast, but
NVMe SSDs have just phenomenal

performance potential where basically
on an NVMe SSD, the cost

per gigabyte is a hundred times
lower than memory, but the performance,

if you use it correctly, is only
about 5 times worse.

And you have to, by using NVMe
SSD correctly is really that you

have to put a lot of concurrency
on the disk, but again, similar

to S3, every single round trip
takes into hundreds of microseconds,

but you can drive a lot of bandwidth.

Old storage engines have not been
designed for that.

You have to design from that, from
the day that you write the

first line of code.

Otherwise, it takes a very long
time to retrofit.

It happens to be that that exact
usage pattern is also what's

very good for S3.

NVMe SSDs were not available in
the cloud until 2017, 2018.

So this is, in database languages,
this is relatively new.

Second thing that needs to happen
is that S3 was not consistent

until December of 2020.

I think this is the most counterintuitive
to most because most

of us just think that it always
has been, but it hasn't.

What that means is that when, if
you put an object on S3 and

then you read it immediately after
you were, as of December of

2020, guaranteed that it was going
to be read after write consistency.

The third thing that needs to happen,
and this is very informed

by the fact that I was on call
for Shopify for so long.

When you're on call for a long
time, you gravitate towards very

simple systems and ideally systems
that are constantly tested

on their resiliency.

So you don't get paged when something
abnormal happens.

Nikolay: Yeah.

Simon: And for us, what was very
important for us to be on

call for a database for Justin
and I was that it only had 1 dependency.

And that dependency could only
be 1 of the most reliable systems

on earth, which is S3, Google Cloud
Storage, right?

And the other derivatives like
Azure blob storage and so on.

They're very, very, very reliable
systems.

But you could not build the metadata
layer on top, right?

So Snowflake and Databricks and
others that have built on top

of this in the last generation
of new databases needed another

metadata layer, some consensus
layer like FoundationDB or their

own Paxos or RAF protocol to essentially
enforce the read off

the right consistency, but also
to do various metadata operations

atomically.

But in late 2024, S3 finally announced
that re:Invent

compare-and-swap.

And what compare-and-swap allows
you to do is to put a file,

let's say metadata.json, you download
the file, you do some modifications

to it, and then you upload it again
with a version number, and

you only upload it if the version
number is the same.

Basically guaranteeing that you
did an atomic and nothing has

changed in the interim.

Very important when you're building
distributed systems, right,

you can really implement anything
on top of that, as long as

you're willing to take the performance
hit of going back and

forth to S3.

And of course, they have a whole metadata, Paxos, whatever thing

to implement that.

In GCS, it's Spanner, but I don't have to worry about that.

That's for them to formally verify and whatever they need to

do to uphold those constraints.

Those were the 3 things that needed to happen.

And that is requirement number 1, to build a new database.

And so That's what was in the air for turbopuffer to grab.

The second thing that you need for a new database is that you

need a new workload that's begging for that particular storage

architecture.

So for Snowflake and Databricks, we saw that in, well, we want

big scale analytics and it doesn't make sense also for the dollar

per gigabyte that we have to pay in operational databases and

adding indexes on everything.

So there was a new OLAP workload, and there was at least an acceleration

of the OLAP workload with all the successful web applications,

and then the new storage architecture on top of S3, but they

had to do some more work because these APIs that I just mentioned

weren't available.

And a new workload now is connecting lots of data to AI.

And this storage architecture is a very good fit for it.

Nikolay: Yeah, So that's great.

Thank you for elaborating about this 3 changes at S3.

But they also made another change recently and they announced

S3 vectors, right?

What do you think about this comparing to your system?

Simon: Yeah, I think that S3 Vectors is, if you, if you are

writing the vectors once and you don't query them that much,

and you don't care that much about the query latency, it can

be a useful, useful product in the same way that you might be

using S3 today.

But S3 Vectors is not, doesn't do full text search, right?

It doesn't do lots of these features that you need for a serious

system.

And even S3 Vectors recommends that you load into OpenSearch.

And so for archival vectors, this can make sense.

So they're still operational, but there's lots of limitations

that would make it very difficult for it to go into production

systems, right?

If you do a query to S3 vectors, it takes hundreds and hundreds

of milliseconds to get the result back.

Whereas the turbopuffer, you can get the result back in less

than 10 milliseconds.

Nikolay: Thanks to cache, right?

Simon: Thanks to cache.

Nikolay: Yeah.

Yeah, that's totally makes sense.

I still I have skeptical questions more of them, if you don't

mind.

Simon: Let's go.

Nikolay: Yeah, 1 of them is your
your cache layer is on local

NVMEs, right?

Yeah.

But why?

Why?

Like, we could store it there.

PlanetScale recently came to Postgres
ecosystem and they said

let's stop having fears about using
local NVMEs and yes, like

ephemeral storage and so on and
there is a reliable failover

and so on.

Let's do it super fast.

Yes, I agree, super fast.

And 4TB for a billion or 3TB for
a billion vectors probably won't

cost too much because this price
is usually in AWS, for example,

in the instances which have local
NVMe, it's basically included.

And the limit is many, many dozens
of terabytes of local NVMe

storage these days on larger instances.

So we are fine to store not only
Vectors, but everything else.

So my question is like back from
S3 to MySQL, for example, My

SQL supports storage engines for
many years.

Have you considered building storage
engine for, for MySQL,

for example, and, and using local
NVMe?

Simon: You can't outrun the economics,
right?

The economics are still the same.

You, you have to replicate whether
it's local NVMe or not, which

is not necessarily cheaper, maybe
only marginally than, than

a, than a network volume.

You still have to replicate it
3 times, right?

You still have to put it on 3 machines.

You still need all the cores and
memory and so on that you would

need on the primary to keep up,
unless you start to have a heterogeneous

replica stack, which for a variety
of reasons would be a really

bad idea.

So you're still paying for paying
for all of that.

Now up to a certain point that
makes a lot of sense right.

If I have a customer who gets on
a call with our sales team and

they have a couple million vectors
in pgvector there is no reason

to move off of it.

That is perfect.

You should not be taking on the
complexity of ETLs and so on.

But if you have like tens of terabytes
of vector data, it is

not economical for a lot of businesses.

Now for some businesses it are,
right?

But the art of a business is to
earn a return on the underlying

costs.

And for some businesses, it's very,
very challenging to earn

a return on storing this on 3 replicas,
this vector data, it's

generally not valuable enough to
the business.

So turbopuffer doesn't make sense
to as a storage engine into

into a MySQL or into a Postgres.

It's just fundamentally not compatible
with the way that we do

things outside of replica chains,
right?

You could maybe come up with a
storage engine where you page

everything into S3 and all of that,
but you're now trying to

build a new database inside of
an extremely old database, right?

The storage layer is, is too, especially,
more so in Postgres,

I think that in MySQL is very,
very weaved into, into how the,

how the query planner works and
all of that.

So at that point, you're rebuilding
the query planner to be very

round trip sensitive, you're rebuilding
a storage layer.

It doesn't make sense anymore.

It's a completely different database.

Nikolay: Okay.

Another skeptical question.

I go to turbopuffer website and,
by the way, great design, very

popular these days, with mono,
mono space font and, like this

types of graphic and your the ties
for 1 billion scale by 1 billion

vectors or 1 billion documents.

But there, if I check 1 billion,
the console of namespaces pops

up.

And namespaces, it's, if I choose
1 billion, if I choose a hundred

million, it's okay.

I can have 1 namespace, but if
it's 1 billion, there is a warning

that probably you should split
it to 10 namespaces.

And this means 10 different indexes,
right?

Simon: That's right.

Nikolay: Yeah.

So it's not actual 1 billion scale.

Or like something is off here in
my mind.

Like 1 billion is a single index,
should be index, single index.

If we talk about 1 billion but
divided by 10 collections and

indexes also, like it's already
a different story.

And so can you explain this to
me?

I saw this in the beginning and
I'm happy in this position I

can ask the founder himself about
this.

Simon: Yeah, we try to be extremely
transparent in our limits,

right?

And I think your mental model is
correct.

Before this, we just had a limit
that said that we could do in

a single namespace around 250 million
vectors, right?

But I mean, even that is a simplification
because how big are

the vectors?

If they're 128 dimensions, they
are a lot less space, which is

ultimately what matters here, which
is why we also put the gigabytes.

So when we in the past, we had
just 250 million vectors on there

as a limit, people came to us and
said, or we knew that people

weren't testing with us, because
they wanted to sort of search

a billion at once.

And they didn't realize that you
could just do id % N and

then you could do a billion and
it would be very economical for

them.

So we sort of had to put in the
docs like, yes, you can do a

billion at once, but you have to
shard it.

Now I would love to handle that
sharding for you, right?

I mean that's what Elasticsearch
does and it's what a lot of

databases do because the only way
to scale any database is sharding,

right?

You don't get around it.

The question is where the complexity
lives, right?

Does the complexity live inside
of the database to handle it

for you.

Some of the most sophisticated
sharding you will find lives inside

of Cockroach and Spanner and these
kinds of systems.

And the simplest type of sharding
is what we're exposing, where

every single shard is just a directory
on S3, and you can put

as many of them as you want and
you can query as many of them

as you want.

Now over time, of course, we need
to create an orchestration

layer on top of that so that a
logical namespace to the user

is actually multiple namespaces
underneath.

But we're challenging ourselves
to make every individual namespace

as large as it possibly can.

When I ran an Elasticsearch cluster,
or was involved in scaling

Elasticsearch clusters, every shard
was around 50 gigabytes.

That was roughly what was recommended
for Elasticsearch.

Nikolay: Quite small, right?

Simon: It's quite small, right?

Like on Postgres it's a lot larger
than that.

It's basically the size of the
machine.

But the problem with a small shard
size is that for something

like a B-tree, right, it's obviously
like it's log n, right?

The number of searches you have
to do.

And if you have log n and n is
very high, well, that's a great

number of operations that you have
to do.

But for every shard, there's sort
of like an m log n.

And now m is very high if the shard
size is small and n is small.

So you're doing a lot more operations.

So we want the shards to be as
large as possible, because of

course you can get to a billion
by just doing, you know, a thousand,

1 million shards, but like that's
incredibly competitionally

ineffective.

So we have shards now for some
users that we're testing that

do almost a billion documents at
once, right?

But it requires some careful tuning
on our end.

So we wanna push that number as
high as possible.

The other thing with namespaces
is that turbopuffer is a multi-tenant

system and we have to bin pack
these namespaces.

So if a namespace is 5 terabytes
large, it's much harder for

us to bin pack on node sizes that make sense.

So we have to walk some threshold there where we try to make

the shards as small as possible from the logical data.

We have to bin pack and then we have to index by without slowing

the indexing down because the larger an ANN index gets, the harder

it is to maintain the index.

So those are the constraints we walk.

So over time, we update these limits continuously.

You will see that shard size increase.

But you're not going to find anyone who's doing 100 billion vectors

in a single index on a single machine, AKA M is 1 and N is a

hundred billion, but we want to get as high as possible because

that is the most competitionally ineffective.

That's how we get our users the best prices.

And we see the most ambitious use cases.

Nikolay: And, and, shards are these actual shards or partitions?

So it's like shards, meaning that the different compute nodes

behind the scenes are used if it's 10 namespaces is it 10 shards

10 different compute nodes?

Simon: So a namespace for us and this is also why we've chosen

a different name for it right A namespace to us is a directory

on S3, and which compute node that goes to is essentially just

a consistent hash of the name of the prefix and the User ID,

right?

So compute node 1 could hash to maybe, you know, a hundred thousand

different namespaces.

And they share the compute and then we scale the compute right

with, with that, and that's, that's why the bin packing is

Nikolay: flexible.

I see.

Simon: Yes.

And that's also part of why we can do get really good economics.

Nikolay: Yeah.

I wanted to shout out again, like front page is beautiful.

It explains like basics, architecture and pricing right here,

latencies right here and case studies.

This is like how it should be, right?

For engineers, you know, so you quickly see all the numbers and

understand.

That's great.

Thank you for transparency.

That's great.

Simon: I think it's, it just comes from the fact that I was

on that, you know, your side of the table, my whole career, right?

I've been buying databases, evaluating databases, and sometimes

I swear I end up on a Database website and I don't know if they're

marketing a sneaker or a Database.

Nikolay: Yeah, go figure out storage or compute price in AWS

or GCP.

It's like always a task.

Simon: So we really wanted to
just put the foot forward of

like the diagram is right there.

Okay.

This is my mental model.

This is what it does.

This is what it costs.

This is how fast it is.

These are the limits.

These are the customers and how
they use it.

You go to the documentation.

We talk about guarantees.

We talk about trade-offs.

The kinds of things that I always
look for immediately to slot

it into my model.

Because I don't want you to use
our database at all costs.

I want you to use it if it makes
sense for you.

I want it to be a competitive advantage
to you.

I want to save you like millions
of dollars a year and in order

for that to make sense You have
to just put all the bullshit

aside and make it very clear what
you're good at and what you're

not good at

Nikolay: What about for example
if I I know there is a vector

search and full text search support
to try it But what about

additional filtering when we have
some dimensions when we want

to filter on them.

Is it possible?

Yes.

For example, some categories or
like time ranges or something,

it's all possible, right?

In both types of search in turbopuffer.

Simon: I'll go back to my line
before of every successful database

eventually supports every query,
right?

And, we're, I don't know, I like
maybe 5% of the way there.

So we support filters and filtering
on vector search is actually

very difficult, right?

Because if you do something like,
let's say I'm searching for

banana and it's an e-commerce site
and I have to filter for ships

to Canada.

Well that might cut off a fruit
cluster which is actually where

I want it to be and then like you
know I get to the banana on

a t-shirt cluster but it's really
far away.

So you have to do very sophisticated
filtering to get good results.

I don't know what pgvector does,
but I know that our query planner

is recall aware.

It does a lot of statistics and
distribution of where the vectors

are to ensure that we have high
recall.

And for a percentage of queries
in production, we check against

an exhausted search what the recall
is, the accuracy of the index.

I don't know if pgvector does this,
but I know it's been a monumental

effort for us to

Nikolay: do it.

It's a big headache there.

It's a big headache.

Actually, it's a good point that
additional filtering can change

semantics of the search.

If it's searching for bananas,
size equals nano, it's very different.

Michael: 1 thing they've implemented
relatively recently is iterative

index scans so I think they had
a problem where for example you'd

put a limit you do a similarity
search with a limit and a filter

and you'd ask for a hundred results
and you'd get back fewer

even though they were so it's a
they had a problem so they go

back to the index and request more
so yes it's like a work around

I'd say more than a solution but
it's pretty effective for a

lot of these cases.

Simon: I think the problem there
is that I would be very skeptical

of the recall in a case like that,
right?

Because, okay, so you just like
you got 100, but you probably

should have looked at a lot more
data to know what the true top

K was.

So, you know, that solution was
not acceptable to us.

Like we had to go much, much further
to ensure high recall.

And I think the problem with a
lot of these solutions is that

people are not evaluating their
recall because it's actually

very difficult to do.

And it's very competitionally expensive.

You're not gonna have your production
Postgres run an exhaustive

search on millions of vectors.

It's too dangerous to do that while
you're doing your transactional

workloads.

But again, if you're hitting these
kinds of problems, then it

may be time to consider maybe for
a subset of your workloads,

whether something more specialized
makes sense, whether the trade-offs

make sense to you.

But I think with the pre-filtering
and post-filtering, it can

be, it's very challenging to create
a query planner that can

do that well.

So we support all the queries,
right?

You can do, and you can do range
queries.

You can do exact queries.

You can operate with arrays.

You can do all types of queries,
set intersections that people

use for permissions.

We can do full-text search queries,
we can do GROUP BYs, we

can do simple aggregates.

So we can do more and more queries
and we're constantly expanding

that with what our users demand
from the system that we've built.

Nikolay: 1 more unknowing question.

I know it's not open source.

There is no free version at all.

Even I'm pretty sure it was decision
not to go with open core

or open source model at all, and
can you explain why?

Simon: Yeah, I think there's
never been any, any, any particular

resistance to open source.

I mean, I love open source.

The reason we're not open source
is because open source is, if

you want to do it well, it's a
lot of effort.

And it's also a lot of effort to
build a company.

It's a lot of effort to find product
market fit.

And we decided to pour all of our
energy into that.

And it's similar argument for the
minimum.

It's really the, the only thing
that a startup has over, you

know, the big incumbents is focus.

And so our focus has been on the
customers that are willing to

pay.

And 1 day I would love to give
a free tier.

I would love for people's blogs
to run on turbopuffer.

But for now, I'm afraid that we
have to prioritize the customers

that are willing to put some money
behind it.

It's not because we have provisioned
hardware or anything like

that.

It's really just that we need to
know that you're willing to

put some money behind it so we
can give you amazing support,

right?

A lot of the times there'll be
engineers working on the database

who are supporting your tickets
and we can't do that with a free

tier.

Nikolay: Yeah.

Makes sense.

Yeah.

And we, Mike and I have different
points of view on this area.

Michael: I think that was a good
answer.

I'm on your side, Simon.

I had a question on the full text
search and semantic search.

We had a discussion a while back
about hybrid, I think it's generally

called hybrid search.

Are you seeing much of that where
people are wanting to kind

of mix the results and order between
them?

Simon: Yeah, we do.

And what we see is that the embedding
models are pretty phenomenal

at finding good results for subjects
that they know about, right?

And terms that they know about.

You know, you, you, you run a project
called pgMustard, right?

And maybe the embedding model doesn't
know.

I think it's popular enough that
it will know what it is, but

let's say it didn't know.

And then it puts it in the cluster
close to ketchup.

And the results are just horrible.

And the text, the full text search
tends to be really good at

this.

Like recall for things that the
embedding model can't know about.

If you're searching for an algae,
you know, these like TV SKUs,

it's like, you know, it's indecipherable.

Actually, the embedding models
are actually quite good at these

because they happen enough on the
web.

But imagine that it wasn't these
kinds of searches it's good

for.

The other thing is search as your
type.

So if I'm searching for SI, the
embedding model might find that,

well, he's searching for something
agreeable in Spanish, right?

But really what I wanted was the
document that started with Simon.

So sometimes you just need to compliment
these twos.

And I think that that's why we've
doubled down on the full-text

search information in turbopuffer
to do this hybrid.

But people can get a lot of mileage
of embedding models alone.

Michael: Wonderful.

Nikolay: Yeah, thank you so much.

Yeah, it was super interesting.

Good luck with your system.

And yeah, I think it's an interesting
idea, I guess, for Postgres

users who need, as you said, to
store more and have all characteristics

turbopuffer have.

Maybe it's worth considering, right,
to keep all OLTP workloads

and data in regular Postgres while
moving vectors to turbopuffer,

right?

Simon: I mean, it's just similar
to, people have taken workloads

out of Postgres.

It's like updating a posting list.

It's a very expensive operational
and a transactional store.

And it's similar in an index updating
that with the same kinds

of ACID semantics that Postgres
upholds is very expensive.

And I've ripped full-text search
out of Postgres many times because

it's very, very expensive to do.

So we do that because we don't
want to shard Postgres, because

it's like a lot of work.

And so we start by moving some
of these workloads out first and

search is 1 of the early ones to
go.

Nikolay: So your point is that
it probably postpones the moment

when you need to shard.

Simon: Yeah, same reason from
memcache and redis, right?

You separate out these workloads
as soon as possible to avoid

sharding.

Kick the can.

Nikolay: Interesting idea.

Okay, again, thank you so much
for coming.

It was a super interesting discussion.

I enjoyed a lot.

Simon: Thank you so much.

Michael: Yeah, really nice to meet
you.

Thanks for joining us.