Michael: Hello and welcome to PostgresFM,
a weekly show about all things PostgreSQL.

I am Michael, founder of pgMustard,
and I'm joined as always by

Nik, founder of Postgres.ai.

Hey, Nik

Nik: Well, Michael

Michael: Hello, and we have
a special guest today, Tudor

Golobenco from Xata, CTO there.

Really nice to have you, Tudor.

Thanks for joining us

Tudor: hi Michael, hi Nick, hi everyone.

Thank you so much for having me

Michael: It is our pleasure.

So to kick us off, I wonder if
you could give us a bit of the

background story of Xata and take
us through to where we are today

Tudor: Yeah, so I will actually start
with what it is today because we did a

change once, like a pivot or like a--
I call it more like a technical pivot.

So it's a Postgres platform which has
as differentiating features a very fast

copy-on-write branching, so it can create
a branching in a matter of seconds, even

for databases that, that are terabytes in
sizes, and scale to zero which means that

if a database is inactive, the compute
part shuts down, and then on the next

connection, it automatically wakes up.

And this wake up is very fast.

We're targeting one second.

So that makes it very convenient,
especially for developer workflows.

So these non-production use cases, things
like if you want a Postgres branch for

each pull request or if you want to create
an, sort of an ephemeral database for each

agent run or for your agent and so on.

Now, these are our more common use cases.

But on the Xata platform we can and
do run production workloads as well.

So the differentiating features are
on the developer experience if you

want, but you can use it as a normal
Postgres hosting provider as well.

So that's what Xata is today.

It didn't really start like that.

The initial idea five years ago my
partner Monica Sarbu, she wanted to do

a database that's really easy to use.

It was a lot of inspiration from Airbase
or spreadsheets and because they were

like like developer using Airbase as a
database or Google Sheets as a database.

So the initial idea was to make
one that's as easy as to use as a

spreadsheet but that scales up so you
don't have to switch at some point.

So having features from both.

And we did that for a while
with some moderate success.

But at some point, we decided
that it's pretty hard to build a

long-term business on that audience.

And it n-never really took
off as a primary database

for, like for more production

Nik: This…

Tudor: use as teams and things like this.

Nik: Apologies for interrupting,
because you're packing a lot of

stuff in one a very short time.

I just wanted to pause a little bit here
and it- this reminds me a bit IO, right?

Th- they also like drag and drop CSV file.

You don't think about schema or
anything, but it's Postgres behind it.

So you had similar thing, right?

Tudor: Yeah, but p.io
were actual Postgres.

Ours, we were actually hiding
Postgres behind our own API.

So it was a REST API that you
interacted with data a bit like

PostgREST or something like this.

Nicer, I would say, but but
that, that type of interfaces.

Yeah

Nik: So it's I like this idea
a lot but Bit.io was closed.

I remember like a very short notice,
like two weeks notice or something.

People were like outraged.

That was yet another not fully Postgres
platform, but I have something,

like I store my data somewhere and
but then it's closed and I have

very short notice to grab my data.

And you are telling us right now that
also this model didn't work well, right?

In terms of-

Tudor: yeah.

Yeah, like there's definitely
companies that do-- that address

this sort of audience successfully.

Like I'm thinking of
Supabase, for example, right?

But they have like millions of developers
and such, and out of those millions,

you're going to have a small minority
that will become large databases, so

they will pay a lot and so on, right?

And we had a few tens of thousand.

I think at the peak we had 50,000 users.

Turns out of those, there's
not enough paying customers

to sustain such a business.

It didn't work for

Nik: Thank you for sharing.

This is super interesting.

50,000 it's actually a lot.

But

Tudor: yeah.

It was quite a lot.

We had the growth in the number
of users, but that didn't

reflect in the growth in revenue.

Nik: Many of them are not paying.

Most

Tudor: Yeah.

Nik: Yeah

I feel you well.

I understand this problem also, like
from different kinds of businesses.

Yeah.

It's hard to find business model on this.

Unless you're Supabase.

This is a good comparison.

Thank you so much.

It's super.

Like before th-this recording I
said we were, we're going to focus

on technical stuff, but the start
is very different than what…

and I like it.

Yeah.

Thank you so much.

Michael: I think it's really useful
context to then dive into the technical

side because I remember one of your
posts that I thought was really good

about the architecture which I guess
might have been the architecture all

along, but it explained how you could
offer a free plan and how the free

plan didn't actually cost you very much
because of your architectural choices.

And that, that was a really nice post
from the business perspective but also

from a technical perspective, like
understanding how those trade-offs

actually help you as a business.

Tudor: Yeah, we-- So on the new
platform, we don't have a free tier.

this is again, the business side, and
our experience with the old platform.

So we call the old platform Zeta Lite.

It's now closed.

We migrated the users that we
could and so on, but it's now

closed, so now it's only Zeta

but the old platform had a had a free
tier and when doing the new platform, we

were not very eager to add another one.

Although with our current architecture,
we can do it again because we have

this scale to zero, which is also very
efficient from that point of view.

But we're not very eager because it adds a
lot of noise and like I was saying earlier

you have some growth, but as long as…

If that doesn't really result
in in the type of business that

you want, you get a bunch of
feedback that's not really useful.

And it confuses things a little
bit more on the business side.

We might come back to that
decision and maybe eventually

we'll add again a free tier plan,
but currently we don't have one.

Nik: Yeah, let's shift to more
technical stuff because I know you

use CloudNativePG, you have branching,
you have very interesting open

source tools like pgStream developed
and others like pgRole, right?

I'm very curious to talk
with you about any of them.

Tudor: Yeah, maybe we
start at the Xata Store.

It's our storage engine that
makes a lot of this possible.

And then we can go about how
we use CloudNativePG to get

the very fast wake-up times.

That's also, I think, interesting.

Nik: So copy on write and branching,

right?

Tudor: yeah.

so we do copy-on-write branching, but
we do it at the underlying volume level.

So because our feature set is
pretty similar with Neon, right?

Also like copy-on-write branching
and then scale to zero with

fast wake-up times, these

Nik: decouple storage and compute, right?

Tudor: Yeah, the

Nik: And you re- and you recently
made everything open source,

Apache 2.0 or no, or not everything

Tudor: it not quite everything, but
enough to run the whole platform yourself.

But it won't scale to the same levels that
we scale our own platform, and you won't

get as fast times as with our platform.

I can go into more details about
exactly why, but we have op-open

source enough to be, to be running
it yourself and to be useful.

But it's not quite everything
open source in there

Nik: Cool.

So if we talk about storage,
like what's interesting there

Tudor: Yeah, so Zeta Store, we call
it like the Zeta Store and it's-- this

is like one of the proprietary pieces.

And we have quite a
long history into that.

We tried a bunch of products and
Kubernetes tools for managing

storage, the storage layer.

As in the end, we decided to build
our own, but also use a bunch of ZFS

functionality, so it's based on that.

The way it works is, and I'm not sure
how familiar you are with ZFS, it's

Nik: let me interrupt you here.

my company, Postgres.ai, has Database
Lab Engine, which implemented

database branching in 2019 before
Neon and before Xata, right?

And I have a lot of questions related
to your experience with database

branching and how you convince
users that it's really needed.

Because we do it seven years and
some of them are solved very easily,

but some of them still struggle
and are like, "Oh, production data,

we need to change it," and so on.

Let's talk about this separately,
but I just to give you context

that you probably don't have.

Tudor: yeah.

Yeah

Nik: And it's on ZFS.

Yeah.

Tudor: Oh, are you using ZFS?

I thought you are using BTRFS.

Uh, no

Nik: it was not stable when
we started, and I don't know

Tudor: Gotcha.

Okay.

I was actually remembering

wrong then

Nik: It was actually interesting.

When I created it first, it was out
came out of ex-experience with GitLab.

We just needed to give backend
engineers capabilities to experiment

and see ex-explain plans not touching
a few database experts, right?

And it's, it was great.

It worked great.

And then we scaled it and the problem
always was that first of all, production

data how to protect this, right?

And also when Neon started working
with branching, it's very different.

So we do it very locally, and we achieve
static cost for all clones and branches.

So you have, you pay once and you
have everything, like mu-multi-tenancy

in one virtual machine, basically.

You have very different model, so I'm very
curious how it works and what's your…

what do you observe how people use it?

And because my honest take, I
spent a lot of time in this area,

market is still not understanding
that branching is super powerful.

And I hope AI will

Tudor: I think--

Nik: but yeah.

Tudor: I think it's it's
it's growing pretty nicely.

But yeah the, like the use cases that
we see most often are branch per PR.

So they do PR and have some
sort of preview environments.

This could be, like Vercel or
their own stuff or different

other platforms for ephemeral
environments or preview environments.

But then they want also the database to
be part of that with the data, right?

So then a platform like
ours makes that possible.

You have the database with the data.

It doesn't have to be the
exact production data.

And the way we typically recommend
people to do is we tell them,

"Look, keep production where it is."

Let's say it's RDS.

And then we have, this is
why we develop pgStream.

We create a replica of it.

And pgStream can copy the data once.

So for small databases, we recommend
people to set up a nightly job

or a weekly job or something
like this because it's simpler.

For large databases, pgStream can
also do the initial snapshot and then

keep it up via a logical replication.

Nik: And DDL?

And

Tudor: and DDL,

Nik: you solve DDL.

I looked at it briefly, and I'm
very curious, like, how… if

you do it and what's your scale?

Is it working really well?

If it's working really well,
why it doesn't go to core

of Postgres in that form?

Tudor: Yeah.

I think there's some work in
progress to make it in in, In co-

Nik: it-- there are several, there,
there are several ways, and if you

check recent activities it's not
super progress-- it's not progressing

Tudor: And not progressing, yeah.

actually a little bit of
backstory on, on pgStream.

We were doing in the old product,
so Zetalight, we were doing, we were

offering this REST API, and then
most of it was powered by Postgres.

But we also had a free text search
with, BM25 and relevance and so on.

And that was actually
searched from Elasticsearch.

So we wanted to have this replication
between Postgres and Elasticsearch.

And we could do, because we had like our
own API for modifying the schema and such,

you could do it in both at the same time.

But that's also tricky because if
you replicate data but the schema

changes come out of order, then
again you have a problem because

it's not going to fit the schema, the
classical CDC type of problems, right?

So it's actually better if the
schema changes are in the same

replication stream as the actual data.

It's better that way.

So that's why we did this event
hook approach, listening for schema

changes and then apply them that.

There, with Zetalight we had this
advantage that we, you couldn't

do arbitrary schemas in Postgres.

It it was a particular set
of data types and so on.

It was a little bit more into
our control, so we knew we could

test all possible schema changes
and make sure they replicate

correctly and these sort of things.

But then we generalized this
approach a little bit more.

Nik: Yeah.

So that's interesting.

So you have limited set of
what's happening, right?

And then using event triggers, you, Yeah.

Tudor: Yeah, that's our approach.

And of course, it needs-- we need
to make sure it's in the right

place in the replication stream.

And the way we do it is we write
in a kind of, in a shadow table.

We write what has changed and which
w- with what altered statement that

happened, and that gets put into
the logical replication stream.

And then pgStream reads it
from there and applies it on

the target in the right order.

That's the general approach.

Nik: And I'm very curious at what
scale it is working and are you

fully satisfied or there are some
like plans to improve or something?

Tudor: Yeah.

We have people using pgStream at
like four or five terabytes of data.

And the initial snapshot takes like
24 hours or something like this.

And then And it b- that depends on a
number of factors or something like

this, and then it's kept up to date.

But then it, like it, it's from
company to company, and sometimes

we have to work on pgStream a little
bit , to make it work for them.

But in the meantime, we had we it
had enough battles, let's say that

now we're feeling like, it covers
already the majority of, uh, of,

cases pretty well.

Nik: Yeah.

Battle proven already, right?

Tudor: Yeah, to some degree.

Like there, there are still problems.

The CDC logical replication

Nik: oh yes

Tudor: is still quite a pla- I w- I would
love if DDL would be supported in core

Nik: everyone waits for it s- but
it's super not trivial as I saw

last time when I checked this topic.

That's cool.

I like s-small non-non-technical
side note side question.

These pieces are… Like you
mentioned some proprietary stuff

about p- pgroll and pgstream

Tudor: Yeah, these are fully open source,

Nik: yeah, that's and Apache 2.0, right?

Tudor: Yes.

yeah.

Yeah, and especially pgStream,
it's important for us to be open

source because we recommend people
to run it in their environment

because it also does anonymization.

This is the, you know, like
kind of what we started from.

It's important to do anonymization so
that when you do this copy-on-write

branching, the data is already anonymized
and you have branches, very fast branches

that don't contain the sensitive data.

Like it's up to you what what
transformation rules you set and so on.

But in many cases, it's like 80% of
the data is fine as it is, and you have

some data that it's sensitive and you
need to transform it and make sure it's

not in any developer copy and so on.

And this I think is the, is the
perfect balance of things if you want.

But it does depend on
this replication story.

And we want to run it in their environment
so they have full control over it.

That's why it needs to
be open source really.

Nik: Yeah.

this answers my question

Michael: on the anonymization
front, I saw you acquired a company.

I saw some blog posts I think you wrote.

Was it beginning of this year?

Is that what you've then
ro- rolled into this

Tudor: Yeah, partially partially.

The it's called Privacy Dynamics, this
company, and they were, like, really

deep into the anonymization front.

it's like there's levels to this stuff.

In pgStream we already had
the anonymization, which was

simple column rule-based.

Like you can say, okay, this
column, you can hash it like this

deterministically and, it's hashed.

But then like for example, for HIPAA
compliance and so on, you need to do

the type of anonymization where if you,
le-let's say there's patient's data and

you have a woman that's 46 years old
and so on, and has this email address

or something like this, and if you just
anonymize the email address, but it's

the only woman in the data set with
those characteristics then you haven't

really anonymized anything because
you can, like even if the name is not

there, there's enough information,
unique information there that you

can still work backwards what it is.

So they doing these sort of things
where they see exactly combining the

data set into groups and figuring
out, okay, we need to anonymize this

data so that no group is really small.

Every group has more than 10
data points and such, so you

cannot find the individual.

And this is like more like, like HIPAA
compliant sort of things and such.

So that's not, that's also something
that's not in pgStream at the moment.

Maybe we will roll out more of those
features in there that we set up

more like manually for customers that
actually need this complex stuff.

Nik: Yeah, that's great.

So these two pieces are,
they're interesting themselves.

They could they could be
separate episodes, honestly.

This is what I… Yeah, because if we
dive deeper, there are so many levels.

As I said I'm in that space as well.

We usually use some other
tools, like we didn't build.

And we-- I remember also
there was a company, Tonika.

Yeah I'm curious what happened to it.

Als-also

Tudor: They still exist, I think, right?

Nik: Yeah.

So a lot of… if you look to
what they build, you understand

that so many layers go there.

It's like

Tudor: yeah.

Absolutely.

Absolutely.

Yeah.

And Tonic AI I think they still exist,
but it's just proprietary and they

open source like a simple version
of it, but it doesn't scale at all.

So it's,

Nik: and important to also
remember, like use cases are

very different for branching.

If you just want to… Like you said you
don't recommend to take production and if

it's like simple development it's better
to have some like preview environments.

It can be some synthetic
data, for example, right?

You don't need to synchronize it all.

But If you bring data for a various
like closer to production testing,

and especially if it's re- related to
performance testing, I'm-- I, I was

always concerned about what plans we
will be getting because if you change

data, I stop trusting those plans.

Tudor: Yeah, of course.

Nik: So I usually try to convince people
to keep like red data a- as is production

data and just limit access and consider
it as part of production and still have

branching and test it closer to production
because in… this is the only way you can

have really realistic performance testing.

What you think about this?

Tudor: Yeah.

So there, there is this deterministic
hashing which basically means if you

hash a value, it will always hash
to the same thing, which is also

useful for referential integrity.

Let's say, like if you have a foreign
key on a string column or something

like this, then you want it to
anonymize the same in both places.

So that helps a little bit with this.

But yeah, with the Postgres planner,
there can be a lot of nuances around

Nik: we, as I said, this
could be whole episode.

If we dive deeper, like when you
say hashing, okay pgStream takes

data from RDS, understand this.

You have ability, like
you copy anyway, right?

But if you talk about your own platform,
you have physical data directory,

Tudor: You're right.

Yes.

Yeah

Nik: W- when you touch it with any
update, with anything, it's better,

like you have, like we have think
cloning branching on physical level.

And if you start touching it, you
already will Mi- Michael will tell

better what, what will happen with plans
because he builds pgMustard, right?

Which is focusing only on, on the plans.

So if you physically - change
something, you like, different

number of blocks and so on, it's
already different plans, right?

Tudor: Yeah

Nik: Relpages can… reltuples
numbers will become different.

Statistics might start different.

Even if you don't, if you say update
some row set column equals the same

column ID something, logically nothing
changed, but physically it changed.

And eventually if you massive
scale, it can drift plans

Tudor: Yeah, and logical
replication already like kind

of throws that away because

Nik: Right.

But if it's, if we go back to ZFS
and thin cloning, the copy-on-write,

I, I-- my point was always like,
let's not touch the data and we will

have ability to have this, the exact

Tudor: and you can you can still do that.

You can have… like if you run
production on Xata what we typically

recommend is have a prod project and
then something like a staging project

or something like this, and pgStream
between them to do the anonymization.

But then the prod project still
supports copy-on-write branching,

and that would be like ZFS

Clone type of thing.

And you can use that but you would
only use this, use it when you

actually need it for like performance,

Nik: And like I, I see Michael
wants to ask something.

But I will ask very last question because
I sit on this topic seven years already,

so I like, I'm super curious to find
guys who have like also spent time there.

Last question from me.

Unit economics.

If we want to build like pipelines which
will test performance plans, right?

If we keep data in Zetta production
and build these pipelines will

each every CI pipeline lead to
provisioning on the whole node?

Will I be paying separately
for whole… For if I run

10 pipe- or 100 pipelines, you…
I need to pay for every one, right?

Or it's

Tudor: Because we do the separation
of storage and compute, right?

And on the storage side,
there's this thin copy, right?

So you only pay for the difference.

That's on the

Nik: so it basically doesn't change.

If I don't change data,
it's the same price.

I

Tudor: Right.

With, of course, some gotchas
there as well, but at the

high level, let's say that.

And on the compute side they would we use
Kubernetes so that would be-- they will

be starting on different compute nodes
potentially, but not each new branch

will be a new node necessarily, right?

They will fill existing nodes and such.

And there you only pay if you use
our Xata Cloud platform because

you also have bring your own cloud,
which is a little bit different.

But if you use our cloud platform you only
pay for the time the branch is active.

And with the scale to zero that that,
it's like for development it makes a

lot of sense because it wakes up in a
second when you connect to it, right?

And you can set like a five
minutes expiration time.

This is like a, a lot of our customers do.

It's like you pay almost nothing.

I actually have a blog post.

You can run thousand… It's
just how the math works.

You can run a thousand branches for
one dollar basically a tenth of a cent

each if they run for five minutes.

Nik: yeah, that's great.

So this is much better than what
I observe with thing cloning

at, in Aurora and Neon, right?

Because there is absolutely
it, it's linear scaling.

If you wa- if you want many
pipelines in CI to test, you

amplify your spending budgets.

And this limits, This is super important
because this makes engineers to

stop testing enough, and they start

Tudor: Yeah.

Yeah.

And it discouraged to have a
branch for each PR, which is like

Nik: so that's the kudos for that.

This is cool.

I, as I understand, it's not like a
constant price, but definitely you've

thought about this and because this
is super important to make testing

more like intensive and complete.

Tudor: Yeah.

Yeah.

Yeah.

Yeah.

So definitely Neon ch-- Neon,
like you get 20 branches for free.

You still pay the compute.

They're not completely free.

You would pay for the compute but above
20 branches you pay a fixed amount for

each of them, even if they are inactive.

So we don't do that.

We don't charge that.

That's why you can have a thousand
branches with only $1, which I don't

know, maybe in a year we'll have
another call and I will say that

does- that doesn't work economically.

Nik: I you start to add nuances.

I know which exactly which nuance.

If you have some branch or clone
or like it holds snapshot, and

if production runs away into the
future, you keep… You start paying

for all data to be stored, right?

Tudor: Yeah, Exactly.

Nik: That it is.

I know this problem very well

Tudor: Yeah like sometimes hard to
explain because you create a branch,

its cost is zero on the storage, and
then you delete all the data into it,

and then the-- you would expect-- you
wouldn't expect the cost to go up, but

that's actually what happens because now
it's different from the parent, right?

So it's like, "Hey, why am I paying
more by deleting data," right?

That, that can be confusing

Nik: this is exactly what we
also ha-had and still having.

Yeah, that's cool.

Cool.

Yeah.

Yeah, Michael, I, I
apologize you had questions.

I, I'm just super excited to talk about

Michael: I was gonna ask more,
but actually now I'm thinking,

I think this is great in terms
of using Xata for your developer,

environments, let's say like
pre-production those early maybe per

agent, per branch, that kind of thing.

But I think you've also got some
interesting stuff on the developer

experience of running Postgres in
production, like pgRole for example.

But I suspect you'll have opinions
on a lot of the other things that

are currently quite hard to do as a
developer running Postgres in production.

So I'd, I wondered about getting
your opinion on some of those.

So the big one for me, I think
still that's difficult for people

to manage is major version upgrades.

Like that, like things like that, I
wonder if you have any takes or any,

any interesting features that most
managed services don't offer or things

that you want to do in the future
to make things like that easier?

Tudor: Yeah, we did put a lot of effort
into the schema changes part of it.

It felt to us like one of the
biggest pain points in, in Postgres.

There's a lot of gotchas there as well,
like you do a schema change and it works

on your small database that you test with,
but then you run it in production and it

locks the whole table for a long time.

So then you have to do this
process and orchestrate multiple

PRs, change the application, do
that, do the backfill, and so on.

So that w- that's how,
like how we started pgroll.

It's also related to the old platform,
which again, it was all about making

the database as easy as possible to use.

so then we had an API
to do schema changes.

And that was not as, you couldn't
do as much as as Postgres can do.

But everything that you did
there was guaranteed to be safe.

You could never lock
your database and such.

But then again, you hit the problem
that people want to do those things.

So it's like that Postgres
functionality was not added for nothing.

And then you end up having to
support everything, which was,

whi- which was our problem.

So yeah, pgroll is is like it's an attempt
to make that as as simple as possible.

It's working.

It also has some, some gotchas that
currently we're not super focused on

it because we're focused primarily
on the Xata platform itself with copy

and write branching and supporting
many agents and these sort of things.

That's where our focus is.

So we didn't have that much
time for pgroll lately.

Yeah, that's wh- that's where we are.

Nik: So now focus is to support
to give database to agents, right?

To-- This is

Tudor: Right.

And that can be in two ways.

One is during development,
like you give like I was saying

earlier, branches per PR.

Now with agents, you create a lot more
PRs, and we're seeing, like we're seeing

companies that have two engineers, and
they have 250 open PRs, and they're

actually all making progress and such.

We see people that that don't do reviews.

It's only AI reviews, and
they get merged automatically.

The tests are running.

A lot of these things are happening now
and will probably happen more, right?

So that's one.

And the second way is like AI
platforms that need a Postgres database

as part of their product right?

And then they want to spin up like an FML
database for that application, which could

be an experiment that they throw away.

So then you want these cheap
databases that spin up and down

very dynamically and scale a lot,

scaling to

Nik: Yeah.

Yeah, these guys even don't
need HA at all, right?

It's just single node no failover

Tudor: Yeah, exactly.

Exactly.

U-usually you don't need
replicas for those use cases, yes

Nik: Yeah, so no high availability needed.

Maybe just backups and that's it, right?

Like you also feel this like new demand

Tudor: yeah.

Yeah, and we do take advantage
of that because the way we solve

these very fast wake-up times is
by doing what we call warm pools.

We keep CNPG clusters ready to go.

And then when there's a connection for
a hibernated branch or database we take

one already provision clustered from
from the pool, and we connect it to the

right ZFS volume on the ZetaStore storage
node, and then serve it to the user.

And that's how, that's
how it happens so fast.

But this means the clusters in the
pool have to all look the same because

if they don't look the same and
and they have, they have different

RAM and so on, it's obvious, but
also different Postgres settings.

If that causes a restart, then
you did nothing because you

need to restart it then too.

So we take advantage of that.

The cluster pools have
no, , no read replicas.

It's-- They're very fast,
but it is what it is,

Nik: how fast it is?

Tudor: So like we're targeting a second.

pro-- Yeah, provisioning a, a new
cluster is the simplest operation

because we just take one of
the, from the pool and give it.

It's it's, yeah, also second.

But it's that most of the cost
there is really the control plane.

We have a single global control plane
region, so we have to go to it and so on.

But that's fast.

The more-- The wake-up is actually more
interesting because there you cannot

just take one from the pool and serve it.

You have to take one of the, from
the pool, connect it to the right

volume, and then serve it, right?

And this is what the, this is what
the Xata store enables be- and this

separation of storage and compute to
some degree And have a bit of a trick

that Cloud Native PG doesn't support any
of this, so we are maintaining kind of

our fork of it to enable some of that.

But there's the instance manager in in
Cloud Native PG, and we modified it to

essentially wait for the PGDATA folder
to show up, and until then it's waiting.

So the pod is completely provisioned.

It's just waiting for that folder to
show up before letting Postgres start.

And behind the scenes, we mount it
in the right place, and as soon as

that shows up there, the instance
manager will let Postgres start.

And Postgres starts in
300 milliseconds or so.

If the, it depends as well if
the, it, if the checkpoint was

done and then things like this.

But if it doesn't have to do the
recovery, it's actually really fast.

But what's expensive is provisioning
the whole stuff in Kubernetes

around it, the pod and the init
containers and these sort of things.

that's where the time was going.

So that's why we do all that
thing, and then the connection

the, the work that we do at the
connection time is actually minimal.

That's why it's so fast.

Nik: That's impressive.

That's cool.

I have a

tricky question.

Oh.

okay, Michael first

Michael: I maybe have
the same tricky question.

I wondered about your-- i-in a
development environment, you probably

want the same extensions as production,
or you might you might need them

in order to test various things.

And also, you might have changed
a lot of configuration parameters.

I'm wondering how…

maybe it's not quite relevant to the same
pooling system, but does that, that system

you just described limit your use of
extensions and changing of configuration

Nik: I just want superuser.

That's

Tudor: Oh, okay.

two different questions.

Let me start with the one from Michael.

Yeah it does.

Because the, the problem is
you cannot keep a cluster pool.

This is how we call them.

You cannot keep a cluster pool for each
combination of, like you have instance

size, and then you s- have a Postgres
setting or list of - extensions and so on.

Preload libraries is more relevant.

Because the extensions, they
can be on the image, you just

don't enable them or enable them.

That's fine.

That's not a problem.

But if you need different set of
preload libraries, that is a problem.

So we just chose to enable
more more common stuff in the

preload libraries by default.

And then you have to, like you have
to keep in mind that if you are not

on the happy path and we don't have a
cluster pool for you, it still works.

It's just slower.

It will take a few seconds then s-
six to 10 seconds instead of one

if you're not on the happy path.

And then the other thing is
that we're-- we, we haven't done

yet, but we want to do, which is
auto-scaling for the cluster pools.

Because some some configurations,
if you want, are going to be a lot

more common than others, right?

The default settings that we
have, you do a, a cluster pool

of 100 waiting to go, right?

So that you can do 100 at once and
you don't-- y- it's not overloaded.

If you have a more complex one, it
could be just one is enough, right?

Because you start that one, and
then you immediately provision

the replacement for it.

That will take 10 seconds, . As long as
someone else doesn't do the same operation

in those 10 seconds, you're okay.

So it's enough to keep one around.

But making this -- via static
configuration is definitely annoying.

So that's why we want to do some
sort of auto-scaling based on that.

So depending on the demand that we've
seen historically, we would make these

pools smaller or bigger, and that would
automatically optimize costs for us.

That, that's something that we haven't
done yet but want to look into it.

Yeah, superuser is like I happen
to know your opinion on that.

We don't give superuser to to our user.

We give the, give up s-

Nik: But will you think about it at least?

Tudor: Yeah, the, the issue is
because we run on Kubernetes

it's pretty hard to secure it at

that level.

If you can run copy program on

Nik: Let's wrap up this question.

I have much more interesting
and we don't have a lot of time.

So

Tudor: Mm-hmm.

Nik: question to you, tricky one.

If you started this platform to build
this once again right now from scratch,

would you choose Kubernetes still?

Tudor: We do use a lot of stuff
from Kubernetes and it's good.

Yeah, I'm happy with-- Overall,

I'm

Nik: most important.

Okay, cool.

Are you

Tudor: I'm happy

Nik: cloud native PG?

Tudor: I'm-- Yeah, you know, it's like
that's more on the fence, I would say

for me.

Nik: opinion about it

Tudor: I know your opinion about it.

Nik: Yeah and more is coming actually.

Literally this weekend I didn't
work only on this PG SimCity

Tudor: Oh, that's awesome by the way.

Nik: Yeah, thank you.

It's, it was a simple one prompt.

It was unexpected.

but this thing like what's happening,
we have a le- a, a winning leader among

Kubernetes operators, which is completely
like ignoring the facts like about

split brains and data loss and so on.

Do you use synchronous replication
if you have multi-node setup?

Tudor: It's configurable.

It's

Nik: It's configurable, but
it's not on def- by default,

Right?

It's not d- it's not on by default.

And it w- which means I have demonstration
of split brain is still possible

in multiple f- in multiple ways.

Michael: Tudor, do you mean you
let your users configure it?

Tudor: yeah.

We can enable it for them

Nik: And in Cloud Native it's not
enabled by default, it's configurable.

It's the same.

Yeah, that's the problem.

And so haven't you already… Like maybe
I'm over like maybe we don't need proper

HA, split brain protection and so on.

Like maybe this is like it's okay
to run this thing and if your users,

if you don't know about your users
suffering may- at your scale, that

this is good enough already, right?

Like it's okay.

What's your take on this?

Tudor: No, I think it's definitely
a problem that like I'm, yeah I'm

also a little bit disappointed
by the response to that issue.

I was expecting, like to be closed sooner.

I understand it's tricky because in
CNPG there's no there's no proxy layer.

So they rely on Kubernetes services to
expose things and to move them around.

That makes it, I think, a
little bit more complex for them

compared to Patroni, for example.

The funny thing is we do have a gateway

Nik: it's not complex, it's impossible.

It's impossible.

You can… There is no
consensus algorithm used.

There is only quorum, and there
is mixing of concepts like quorum

is needed to handle transactions.

It's not n- it's not for
deciding who is the primary.

So I have demonstrations.

You can have multiple--
You have two primaries.

You can have split brain, you can have
data loss, and more to be published soon.

I'm pissed off because m- because of
their popularity clients keep asking us.

I literally have right now two asks
from two different companies to

explain, should we use CloudNativePG
because it's super popular.

And now, like I meet you
and you use it at scale.

So you don't have problems with
no-nobody lost data, no-nobody

dealt with split brains.

Do we have… M-maybe we have already good
cloud resources which don't experience

like outages, like maybe th-this is it.

Maybe we don't need the HA
replicas may-maybe even, right?

Like it's already good enough
to leave a single node.

So there is no such a problem as HA,
high availability, no more such problems.

Or you had such issues like split
brains or data losses because

there's no consensus algorithm

Tudor: Yeah, it's like I, like I would
have to go back and look how often.

Like I do remember maybe one case
where to be honest, we couldn't

figure out exactly what happened.

in these things there's always
a combination of factors.

But I do remember once we thought,
like this might actually be the

split brain scenario hitting us.

So like I, I think it's a bug really.

Nik: Unfortunately, in their DNA, they
like, it's like this is a basic decision.

If you read recent post.

I have an answer unpublished.

I'm going to publish it, so more stuff is

Tudor: It's coming

Nik: And I sent you a LinkedIn request.

Please accept it.

Let's talk, let's keep in touch
on this topic because I want the

industry understand that consensus
algorithm is needed, and either

CloudNativePG needs to be, to change
or people should stop using it.

Or maybe we like, again,
like one node is enough.

We don't need the HA, everything is good.

AWS and Google Cloud they stopped being
unreliable as it was 10 to 15 years ago.

Tudor: Yeah, on, on our side, we're
already I told you we are on a fork of it.

It's not that I would, I would not
recommend people to use our fork

because we would, we, we-- It's open
source as well, but we didn't make

it with the goal of becoming, like a
CNPG successor or anything like that.

But we did we did go a bit in a different
direction when it comes to backups.

For example, we're using pgBackRest,
and we made it in the operator itself

rather than the plug-in solution,
which has a number of disadvantages.

So we have the, like we have the precedent
of us going a little bit in different

directions regarding these things.

Also, we do have a gateway anyway.

We need a gateway, like a SQL gateway.

So then, like the main benefit
that you are proxy-less actually

doesn't really apply to us.

So it might, that might be also
something that we would choose to

do a little bit different there.

Nik: Yeah.

And by default CloudNative PG has
a Barman or, for backups or what?

Tudor: I- it's a plugin that's-- It
used to be, Barman in the core, but

then now it's Barman as a plugin.

I think as of right now it's the
only, though, plugin that exists.

I think there's one for pgBackRest,

but in…

Nik: So if you replace and if you
pl- if you have pgBackRest, this

next step is just to add Patroni

Tudor: Yeah, you're right

Nik: And then my concerns
are fully resolved.

I'm okay.

Sensor algorithm works well and so on.

But I understand it can be super
challenging at your scale because if

you have a lot of smaller clusters
and so on ETCD or something, like you

sh- you need to properly engineer it.

I think it's done by some guys,
but it's not straightforward to

not-- like, because we don't…

If we have a lot of small clusters, we
cannot afford ETCD for everyone, right?

We need to have shared
ETCD, and there are some

limitations

remember.

Tudor: exactly.

Nik: But it's like it
would be natural step.

Like next time, like you bump into a
couple of other weird cases and maybe you

should add Patroni, something like this.

Tudor: Yeah.

Yeah.

but that's, like there's the advantage
that we don't really do this yet, but it

does make sense to have a bit of different
configuration for your development

branches versus your production branches.

Because the the thousands of branches
and so on, the really large numbers don't

even have replicas like you're saying,
because that like the point is that

you shut them down, they come back up.

The storage is separated and
it's secure no matter what.

It doesn't depend on Postgres at all.

So then, you just don't use read
replicas at all for those use cases.

And the production ones are
just much fewer and then you can

Nik: And computer's smaller, right?

So like less bu- shared buffers is
small, everything like this right?

But then you can fool the planner
and make it work as to choose

the same plans as in production.

This is magic I realized some time
ago, and it's super cool that you

can s- have a tiny me- like memory
small, shared buffers is tiny, but

you say effective cache size is huge.

Other p- planning parameters
are huge, some other things, and

you get exactly same plans as in

production.

Yeah, and then you need to just
adjust you, you need s- small mind

shift and engineer minds to make them
focus on buffers numbers mo- more

because timing might be different.

This is what we talk about buffers
three years already or four years?

Yeah

Michael: I'm super conscious of time,

but Tudor, is there any, any
last things or shout-outs you

wanted to give on the Xata side?

Like I'm very aware we haven't
even talked about DeltaX,

which looks really promising.

Tudor: Maybe we do another one in a few,

Nik: Yeah, let's have part two

quite soon.

Yeah.

It's, it was cool.

And thank you for coming.

I enjoyed to- a lot.

I think we do similar things, like
in terms of where we ha- headed.

I think everyone should
use database branching.

It's super cool, we will have all
the links in show notes, right?

And so I encourage everyone
to try out if you haven't.

And I guess you just need to tell your
agent to try to go and use it, right?

That's it.

Tudor: Yeah, pretty much, yes.

Nik: That's cool.

Thank you for coming.

Let's have part, part two soon.

Michael: Yeah.

Thanks so much, Judah

Tudor: Thank you.

It was really nice.

Thanks both